You are not logged in.
The laptop hangs after extended use - about 7 hours or so, or a few minutes into graphics intensive applications (kerbal space program - native linux game btw). Its a hard crash, can't switch to another tty or use the magickey sequence.. or well anything. The laptop has to be rebooted by holding down power button.
The journalctl logs are unremarkable
log of the crashed session https://pastebin.com/t188gy36
log of the session after crash https://pastebin.com/wMVbDQzx
I am using xfce desktop environment, and with minimal software installed. The rest of the hardware works fine - multi touch touch/trackpad is working, no over heating or exorbitant memory/cpu loads - even leading up to the crash.
This wasn't isolated, the crashes started soon after installation when either of the above conditions are met. I don't think its unique to arch even, i was using sparky (debian derivative distro) before and it too was intermittently crashing - switched to Arch hoping the newer kernels and drivers would help.
I wasn't running the stable arch kernel during the above crash, however I was trying it out to see if it would help as it had patched drivers https://aur.archlinux.org/packages/linu … -next-git/. If required I can switch to stable arch kernel and reproduce the crash - if so what other logs should I check? Should I enable any other type of logging?
The windows 10 installation runs just fine (without any consistent crashes that is) - its not a dual boot setup as arch and windows are on separate drives.
What should I try next? All the relevant drivers and software are up to date. I don't really mind the games crashing(I can switch to windows if i badly wanted to play something), however the random crashing after extended usage is troublesome as whatever I am working on could be lost.
Last edited by aravindpallipara (2020-02-15 11:49:48)
Offline
Try the LTS kernel, I'm running v4.19 with that CPU & graphics combination and I've just had a quick blast at CS:GO after 7½ hours uptime with no crashes. I don't think I've ever had a hard freeze under that kernel.
EDIT: https://wiki.archlinux.org/index.php/Microcode?
Last edited by Head_on_a_Stick (2020-01-22 19:15:45)
Para todos todo, para nosotros nada
Offline
Ok I will try 4.19, I hope there is an LTS version then. Any idea why the newer kernels aren't supported?
I have the latest microcode updates from Acer - dont they package that with the bios updates? If so I am running the latest version.
Last edited by aravindpallipara (2020-01-23 09:54:26)
Offline
Any idea why the newer kernels aren't supported?
*shrugs* Regressions, perhaps?
I have the latest microcode updates from Acer
But have you installed and applied the µcode packages? They are recommended but I'm not sure if the second generation Ryzens have had any fixes yet. Anyway read the linked ArchWiki page.
Para todos todo, para nosotros nada
Offline
I did read through the microcode section, however I haven't managed to get my hands on any microcode update packages - the amd site link in the page is broken and a google search for 2500u chipset microcode updates was fruitless - Have you manged to obtain any updates for the 2500u? If so where should I search. The acer bios updates were released as binaries for windows, I installed the latest bios update before installing arch
And I tried the 4.19.97-1-lts kernel. Kerbel space program crashes in the same manner. I suspect it could be the game's fault then probably running out of memory or something. I will try to play CS:GO and see if the same problem persists. I will keep the current session live to see if the random crashes happen.
Any other suggestions?
Last edited by aravindpallipara (2020-01-23 10:25:12)
Offline
where should I search
Para todos todo, para nosotros nada
Offline
It's flagged out of date, but I will setup microcode updation ASAP.
EDIT : apparently AMD microcode updates happend automatically without any need for setup - I have installed the amd-ucode package. And here's the result of
dmesg | grep microcode
[ 1.127913] microcode: CPU0: patch_level=0x08101007
[ 1.127924] microcode: CPU1: patch_level=0x08101007
[ 1.127944] microcode: CPU2: patch_level=0x08101007
[ 1.127951] microcode: CPU3: patch_level=0x08101007
[ 1.127970] microcode: CPU4: patch_level=0x08101007
[ 1.127978] microcode: CPU5: patch_level=0x08101007
[ 1.127997] microcode: CPU6: patch_level=0x08101007
[ 1.128003] microcode: CPU7: patch_level=0x08101007
[ 1.128068] microcode: Microcode Update Driver: v2.2.
I checked the journalctl logs and these messages are unchanged from before amd-ucode installation
Last edited by aravindpallipara (2020-01-23 10:38:41)
Offline
I have installed the amd-ucode package
Have you configured your bootloader to install the µcode?
Para todos todo, para nosotros nada
Offline
Yes I have configured bootloader (using GRUB, ran
grub-mkconfig -o /boot/grub/grub.cfg
and I can see the amd-ucode.img file in the boot folder). Though the result remains same as above for
dmesg | grem microcode
. And searching the web i found a page which seem to indicate the latest patch level is
microcode: CPU0: patch_level=0x0810100b
instead of
microcode: CPU0: patch_level=0x08101007
here in this site https://www.phoronix.com/forums/forum/h … tree/page3
Can you check if the patch levels for microcode with your laptop are same as what I have or is it different? If so it seems early loading is not working - else its not logged - the wiki page did mention early loading might not be logged. Should I try to manually reorder the img file order to makes sure it's loaded?
This is such a wild goose chase. Much thanks for taking the time to help me here.
Last edited by aravindpallipara (2020-01-24 06:41:49)
Offline
Can you check if the patch levels for microcode with your laptop are same as what I have or is it different?
My patch levels are the same, I downloaded the latest linux-firmware tarball from kernel.org and applied it manually but the patch level didn't change so I don't think there are any fixes for the 2500u. Yet.
So I have no idea why you're having these problems, sorry.
Para todos todo, para nosotros nada
Offline
The linux-ck kernel also froze after sometime, now trying the lts one for stability. Do you use any particular kernel parameters? I am using
iommu=pt
idle=nomwait
Should I try disabling them? Also is your microcode patch_level=0x08101007 (just checking as you didn't differentiate between patch_level=0x08101007 and patch_level=0x0810100b)
Offline
Do you use any particular kernel parameters?
empty@E485:~ $ cat /proc/cmdline
BOOT_IMAGE=/debian/vmlinuz root=/dev/nvme0n1p3 rootflags=subvol=debian rw quiet acpi_backlight=thinkpad_acpi kaslr pti=on slab_nomerge page_poison=1 slub_debug=FPZ ivrs_ioapic[32]=00:14.0 ivrs_ioapic[33]=00:0.1 amd_iommu=on iommu=pt
empty@E485:~ $
The ACPI & IRVS options are to correct faults in my (ThinkPad E485) firmware and the non-IOMMU options are for hardening. I'm using Debian stable rather than Arch but that's also on the 4.19 LTS branch. I'm pretty sure Ben Hutchings is passing any patches back upstream so they should also be present in Arch's version.
Should I try disabling them?
Why not, eh? I think we're scraping the bottom of the barrel here though
Also is your microcode patch_level=0x08101007
Oh dear, I think I need a reading skills refresher course because I've just checked again and my patch level is actually newer than yours:
|empty@E485:~ $ sudo dmesg | grep microcode
[ 3.561141] microcode: CPU0: patch_level=0x0810100b
[ 3.561165] microcode: CPU1: patch_level=0x0810100b
[ 3.561180] microcode: CPU2: patch_level=0x0810100b
[ 3.561192] microcode: CPU3: patch_level=0x0810100b
[ 3.561211] microcode: CPU4: patch_level=0x0810100b
[ 3.561245] microcode: CPU5: patch_level=0x0810100b
[ 3.561265] microcode: CPU6: patch_level=0x0810100b
[ 3.561276] microcode: CPU7: patch_level=0x0810100b
[ 3.561379] microcode: Microcode Update Driver: v2.2.
empty@E485:~ $
I would like to claim that it was different last time but no packages have been updated since then so I would be talking out of my ass (again).
But my µcode package is from 2018-11-28, which is older than Arch's version. I am confused. Nevertheless, installing a later version and loading the µcode manually doesn't change the patch level.
Try downloading the firmware tarball from kernel.org and applying it manually, see if the patch level changes afterwards.
Para todos todo, para nosotros nada
Offline
And I have tried reinstalling the arch amd-ucode package, grub, and different kernels. No change in patch levels. I think the outdated patch level is the problem here.
I am not sure how to directly apply patches from linux-firmware tarball obtained from kernel.org, I will have to figure that out (sorry about the delay in reply).
https://www.reddit.com/r/archlinux/comm … n_5_3500u/ this discussion points to some microcode binary files - however I am not able to ascertain which is the right one for 2500u or if there is anything about 2500u. Can you make any sense of this git hub repo? https://github.com/platomav/CPUMicrocod … master/AMD
Should I try a different boot loader? Because I strongly suspect there is no patching happening from linux kernel side for some reason, and the current patch level is from the bios update I did through windows awhile ago.
Offline
I am not sure how to directly apply patches from linux-firmware tarball obtained from kernel.org
Download the tarball, unpack it and copy microcode_amd*.bin to /usr/lib/firmware/amd-ucode/ then run
# echo 1 > /sys/devices/system/cpu/microcode/reload
https://wiki.archlinux.org/index.php/Mi … te_loading
Should I try a different boot loader?
No, let's keep it simple. GRUB is perfectly capable of loading amd-ucode.img
But we should probably check if it's found the image and added the configuration:
grep amd-ucode.img /boot/grub/grub.cfg
If it has the correct initrd line and the new firmware from kernel.org changes your patch level then you can make your own µcode image with the new firmware by following the "Early loading of microcode" section in the relevant Linux From Scratch chapter:
http://www.linuxfromscratch.org/blfs/vi … mware.html
Place the custom image at /boot/amd-ucode.img so that GRUB can find it and pacman can over-write it when the package from the repositories is updated.
Para todos todo, para nosotros nada
Offline
Can't run
echo 1 > /sys/devices/system/cpu/microcode/reload
permission denied- even if attempted to run as the root user. Isn't there safeguards to prevent microcode being live loaded in linux distros now? Do I have to tinker with kernel settings to enable live loading?
Offline
sudo echo 1 > /sys/devices/system/cpu/microcode/reload
The redirection is performed before sudo. Instead try
sudo sh -c 'echo 1 > /sys/devices/system/cpu/microcode/reload'
Offline
If loqs' suggestion does not work then check /usr/lib/tmpfiles.d/linux-firmware.conf, which should look like this:
w /sys/devices/system/cpu/microcode/reload - - - - 1
Arch enables µcode late loading by default.
Para todos todo, para nosotros nada
Offline
Download the tarball, unpack it and copy microcode_amd*.bin to /usr/lib/firmware/amd-ucode/ then run
Did that and
sudo sh -c 'echo 1 > /sys/devices/system/cpu/microcode/reload'
this - you were right about
sh -c
modifier, I really need to learn more terminal commands.
Checked
/usr/lib/tmpfiles.d/linux-firmware.conf
and it does show
w /sys/devices/system/cpu/microcode/reload - - - - 1
, so that's setup as intended.
and
grep amd-ucode.img /boot/grub/grub.cfg
returns
initrd /boot/amd-ucode.img /boot/initramfs-linux.img
initrd /boot/amd-ucode.img /boot/initramfs-linux.img
still the microcode patch levels remain the same.... Any other ideas?
Offline
After much reading, and finally getting a little insight into how microcode is loaded early, the .img file ( amd-ucode.img here) is formed from concatenation of bin files in /lib/firmware/amd-ucode/ and subsequent conversion to cpio archive with bsdcpio utility. The necessary updates are in the microcode_amd_fam17h.bin for ryzen 2500u apu - if there are any .
The arch amd-ucode package gives the img file obtained with
bsdcpio -o -H newc -R 0:0 > /boot/amd-ucode.img
And I have run grub-mkconfig after placing the amd-ucode.img file. And it is detected by grub and set as initrd image.
From what I can tell, I have done all the proper setup possible
I am suspecting there aren't any kernel patches for 2500u, as from your earlier reply
|empty@E485:~ $ sudo dmesg | grep microcode
[ 3.561141] microcode: CPU0: patch_level=0x0810100b
[ 3.561165] microcode: CPU1: patch_level=0x0810100b
[ 3.561180] microcode: CPU2: patch_level=0x0810100b
[ 3.561192] microcode: CPU3: patch_level=0x0810100b
[ 3.561211] microcode: CPU4: patch_level=0x0810100b
[ 3.561245] microcode: CPU5: patch_level=0x0810100b
[ 3.561265] microcode: CPU6: patch_level=0x0810100b
[ 3.561276] microcode: CPU7: patch_level=0x0810100b
[ 3.561379] microcode: Microcode Update Driver: v2.2.
empty@E485:~ $
This is different from what wiki example shows as a proper early loading -
[ 2.119089] microcode: microcode updated early to new patch_level=0x0700010f
[ 2.119157] microcode: CPU0: patch_level=0x0700010f
[ 2.119171] microcode: CPU1: patch_level=0x0700010f
[ 2.119183] microcode: CPU2: patch_level=0x0700010f
[ 2.119189] microcode: CPU3: patch_level=0x0700010f
[ 2.119269] microcode: Microcode Update Driver: v2.2.
So the fault seems to be from the manufacture side (acer) as the necessary cpu patch level is obtained from the bios update and not from linux kernel early loading - in both mine and your cases it seems.
So unless there are some other bin files which contains said patches form some other sources than kernel.org.... it probably wont work.
Offline
Yes, there are 2 ways to update microcode : from uefi firmware on startup of system and from microcode updating by kernel .
Both have the same endresult : updated microcde .
Keep in mind that the patch level shown is different for each processor model and there doesn't seem to be a list where you can find the latest microde version for your processor .
Disliking systemd intensely, but not satisfied with alternatives so focusing on taming systemd.
clean chroot building not flexible enough ?
Try clean chroot manager by graysky
Offline
I am aware that patch levels for individual processor models will differ, what I am specifically referring to is this line:
[ 2.119089] microcode: microcode updated early to new patch_level=0x0700010f
Since neither mine or @Head_on_a_Stick's dmesg output has that particular line. I assume there aren't any kernel ucode patches available for 2500u apu,
Have you updated the bios on the thinkpad @Head_on_a_Stick?
Any ideas for a source on those binaries?
Offline
Have you updated the bios on the thinkpad @Head_on_a_Stick?
Yes, I think I did. It wasn't crashing beforehand though and it didn't fix the various IVRS table problems under Linux that Lenovo seem determined to ignore.
Para todos todo, para nosotros nada
Offline
After giving up on that issue for awhile, today I managed to obtain the microcode binary file and load it early after a surprisingly small amount of leg work.
Here are the steps involved:-
Many microcode binaries for different processors(both intel and amd) are available at this github repo (this link was posted in an earlier reply of mine). The hard part as mentioned above is figuring out which is the right one for your processor. For amd2500u apu the cpuid is cpu00810F10
I haven't found a resource that matches cpu/apu models to the cpuid of the files given in that github repo. The way I figured out mine was to go to lenova's website and download the bios update .iso file for Thinkpad E485(Head_on_a_Stick's laptop) then extract the microcode binaries using MCExtractor tool then figure out the right cpuid from the extracted binary file name.
The binary file in whichever way obtained needs to be converted to a linux suitable container(files with .h extension) using this amd-ucodegen tool In my searches I did see an analogues iucode-tool for intel.
Then by following the directions at Gentoo Wiki the .h file(s) can be converted to a .img file suitable for early loading with initrd.
Copy the .img file to /boot directory, uninstall amd-ucode package if you have it installed. Then update grub.cfg.
Here's my current
dmesg | grep micro
output:-
[ 1.273708] microcode: microcode updated early to new patch_level=0x08101016
[ 1.273766] microcode: CPU0: patch_level=0x08101016
[ 1.273804] microcode: CPU1: patch_level=0x08101016
[ 1.273828] microcode: CPU2: patch_level=0x08101016
[ 1.273836] microcode: CPU3: patch_level=0x08101016
[ 1.273855] microcode: CPU4: patch_level=0x08101016
[ 1.273864] microcode: CPU5: patch_level=0x08101016
[ 1.273866] microcode: CPU6: patch_level=0x08101016
[ 1.273873] microcode: CPU7: patch_level=0x08101016
[ 1.273973] microcode: Microcode Update Driver: v2.2.
I played some hollow knight for an hour and it didn't crash, so it seems my problem was associated with microcode.
Thanks for the help everyone, I think above the steps can be used to obtain microcode updates for other models of processors too. Be warned though the binary files are obtained from a random website, if possible it might be better to obtain from a more trustworthy source like the manufacturer's site (in my case I had little choice).
Offline