Crash during systemd and boot loop

pisauraxtx · 2023-09-22 22:45:29

Hello everyone.
I've been running Arch on my computer without any major issues for more than two years. Up until september 10, when I booted my computer and installed updates like I usually do. The next time I turned on my PC, it got to the systemd screen, showed the first couple of lines, and then crashed. And entered a boot loop.

After trying the fallback initramfs, checking the PSU and the fans, I decided to save my data and reinstall Arch - to no avail. The install medium booted without any issues, but the finished install crashed during the first split second of the systemd screen again.

Now, I can't do without my computer, so I tried installing Linux Mint instead, and it worked. I've been running Mint on the same hardware without any problems for almost two weeks. But I'm not happy with Mint, so today I tried installing Endeavour. The live disk booted and the install went the way it should, but when I tried to boot from the installed OS - crash and boot loop. I just tried installing it on an external USB drive to make sure that it's not the main drive that's causing the issue, but the result is the same - crash and boot loop.

I'm running Memtest86 right now to check for corrupted RAM, but other than that, I'm at my wits' end.

Long story short - Arch and Arch-based distros have suddenly stopped working on my computer; at least one other Linux distro works fine; the SSD is not the issue; I'm trying to get to the root of the problem.

Does anyone have an idea what could be causing this? Can anyone point me in the right direction in order to find out?

I'm using the following hardware:

CPU: AMD Ryzen 7 5800X
Motherboard: MSI MPG B550 Gaming Edge WIFI AMD B550 So.AM4 Dual Channel
RAM: 2x 32GB Corsair Vengeance LPX DDR4-3200 DIMM CL16-20-20-38
Main SSD: 1TB Corsair Force Series MP600 M.2 2280 PCIe 4.0 x4 NVMe 1.3
GPU: PowerColor Radeon RX 6600 XT Red Devil 8G OC (Retail)

loqs · 2023-09-22 23:06:57

Please post the journal for a boot with the issue from the USB drive install you created.

seth · 2023-09-23 06:31:04

Spontanous reboot is CPU, PSU or RAM and it's a ryzen 5800X … https://wiki.archlinux.org/title/Ryzen#Random_reboots
What kernel version does mint boot?

pisauraxtx · 2023-09-23 09:33:52

loqs wrote:

Please post the journal for a boot with the issue from the USB drive install you created.

I booted from a live disk again, mounted the drives in question and ran arch-root on them. Running "journalctl" yields the output "No journal files were found. -- No entires --".
This is true for both the install on my M.2 drive and on the USB drive. Makes sense since neither of them has ever booted successfully.

Are there any other logs or meaningful info I can get on the failed boots? If so, how?

pisauraxtx · 2023-09-23 09:46:52

seth wrote:

Spontanous reboot is CPU, PSU or RAM and it's a ryzen 5800X … https://wiki.archlinux.org/title/Ryzen#Random_reboots
What kernel version does mint boot?

Thanks. I tried disabling DRAM power down, increasing the voltage on my CPU, and adding the kernel parameters idle=nomwait and processor.max_cstate=5, but none of these helped.
My UEFI doesn't seem to have any relevant options for the PSU, and I'm not entirely sure if I found the right options for increasing voltage. I was only able to add a positive offset of one.

The current release of Mint is shipping with kernel version 5.15.

seth · 2023-09-23 14:01:25

The install medium booted without any issues … the install on my M.2 drive

processor.max_cstate=1 iommu=soft noacpi pci=nomsi nvme_core.default_ps_max_latency_us=0 idle=nomwait nomodeset

https://wiki.archlinux.org/title/Solid_ … leshooting

And try to install the LTS kernel

pisauraxtx · 2023-09-23 18:24:15

seth wrote:

And try to install the LTS kernel

I installed linux-lts via chroot and ran grub-mkconfig, but the LTS kernel isn't showing up in Grub. Am I missing a step here?

seth wrote:

processor.max_cstate=1 iommu=soft noacpi pci=nomsi nvme_core.default_ps_max_latency_us=0 idle=nomwait nomodeset
https://wiki.archlinux.org/title/Solid_ … leshooting

I tried running the default kernel with all these parameters, but it still crashed.

I don't think the M.2 drive is the culprit here. Like I stated above, I also tried installing Endeavour OS on a usb drive and it resulted in the same crash.

Speaking of Endeavour, I compared kernel and driver packages between the live medium and the installed version, and the only difference seems to be slightly newer AMD microcode and MESA packages.
Live: amd-ucode 20230625.ee91452d-5
Installed: amd-ucode 20230804.7be2766d-2
Live: mesa 1:23.1.5-1
Installed: mesa 1:23.1.8-1

(Btw, sorry for going on and on about an Arch derivative here - I'm only using it to figure out where the problem lies. The GUI live disk makes that slightly easier for me. And the problem is exactly the same between Endeavour and Arch)

seth · 2023-09-23 19:58:02

Did you maybe forget to mount the /boot partition?

seth wrote:

noacpi pci=nomsi

but meant

pci=nomsi,noacpi

but you can also just try "acpi=off" instead/in addition

hiddengerbil · 2023-10-23 14:45:59

I am having the same issue. There were some MCE errors on my boot log, but I've checked these out and it appears my CPU Ryzen 9 5950x is not supported by MCE, however, that should not prevent the machine from booting.

I can load the Arch ISO, and install the system using the Archinstall command (just to be quick). I did a manual install (which I've done at least 30 times on this machine) and I'd never had any issues until this June 2023. I have the same boot loop, as you talk about, and after reboot I never get beyond the GRUB screen.

I tried Garuda Linux (live ISO works fine), Manjaro, Endevor and Arcolinux. All of them fail after the install and reboot.

I can, however, install a number of Ubuntu/Debian based distros without issues.

I have Arch running on an older Threadripper 1 machine, without issue, so it is certainly something about my Ryzen 9 5950x machine it doesn't like. Windows works fine on this newer machine, and I've done everything to try and expose any CPU issues, but I'm not seeing anything.

Oh, the other distro I can't install and boot into is OpenSUSE,

pisauraxtx wrote:

Hello everyone.
I've been running Arch on my computer without any major issues for more than two years. Up until september 10, when I booted my computer and installed updates like I usually do. The next time I turned on my PC, it got to the systemd screen, showed the first couple of lines, and then crashed. And entered a boot loop.
After trying the fallback initramfs, checking the PSU and the fans, I decided to save my data and reinstall Arch - to no avail. The install medium booted without any issues, but the finished install crashed during the first split second of the systemd screen again.
Now, I can't do without my computer, so I tried installing Linux Mint instead, and it worked. I've been running Mint on the same hardware without any problems for almost two weeks. But I'm not happy with Mint, so today I tried installing Endeavour. The live disk booted and the install went the way it should, but when I tried to boot from the installed OS - crash and boot loop. I just tried installing it on an external USB drive to make sure that it's not the main drive that's causing the issue, but the result is the same - crash and boot loop.
I'm running Memtest86 right now to check for corrupted RAM, but other than that, I'm at my wits' end.
Long story short - Arch and Arch-based distros have suddenly stopped working on my computer; at least one other Linux distro works fine; the SSD is not the issue; I'm trying to get to the root of the problem.
Does anyone have an idea what could be causing this? Can anyone point me in the right direction in order to find out?
I'm using the following hardware:
CPU: AMD Ryzen 7 5800X
Motherboard: MSI MPG B550 Gaming Edge WIFI AMD B550 So.AM4 Dual Channel
RAM: 2x 32GB Corsair Vengeance LPX DDR4-3200 DIMM CL16-20-20-38
Main SSD: 1TB Corsair Force Series MP600 M.2 2280 PCIe 4.0 x4 NVMe 1.3
GPU: PowerColor Radeon RX 6600 XT Red Devil 8G OC (Retail)

pisauraxtx · 2023-10-23 16:08:46

hiddengerbil wrote:

I tried Garuda Linux (live ISO works fine), Manjaro, Endevor and Arcolinux. All of them fail after the install and reboot.
I can, however, install a number of Ubuntu/Debian based distros without issues.

By this point, it looks to me like it's an issue in the latest versions of the linux kernel. I am now running Fedora on my machine and I can boot perfectly fine with kernel 6.4, but all newer versions of the kernel result in the instant crash. This also goes to kernels I compiled myself.

I submitted a bug report here and they suggested bisecting the kernel to get to the root of this problem. This far exceeds my skills and knowledge though. If you feel comfortable with this type of thing, feel free to give it a shot and let me know if you find anything.

You could also try installing Arch again with an older kernel or LTS kernel if you're as unhappy with Mint as I was.

ewaller · 2023-10-23 16:48:36

pisauraxtx wrote:

You could also try installing Arch again with an older kernel or LTS kernel if you're as unhappy with Mint as I was.

Or, one could install the LTS kernel along side the mainline kernel in the existing install and choose which one to use in your bootloader.

hiddengerbil · 2023-10-23 17:06:57

I think you're correct. I know that OpenSUSE was working for me until an update ruined it.

I'll look into the older kernel option, as it would be nice to get my old Arch setup running again.

I did consider learning more about compiling my own kernel, and I can do the compiling part, but I would not know how to make alterations to the kernel, so returning to an older kernel is a much safer option.

pisauraxtx wrote:

hiddengerbil wrote:
I tried Garuda Linux (live ISO works fine), Manjaro, Endevor and Arcolinux. All of them fail after the installation and reboot.
I can, however, install a number of Ubuntu/Debian based distros without issues.
By this point, it looks to me like it's an issue in the latest versions of the linux kernel. I am now running Fedora on my machine and I can boot perfectly fine with kernel 6.4, but all newer versions of the kernel result in the instant crash. This also goes to kernels I compiled myself.
I submitted a bug report here and they suggested bisecting the kernel to get to the root of this problem. This far exceeds my skills and knowledge though. If you feel comfortable with this type of thing, feel free to give it a shot and let me know if you find anything.
You could also try installing Arch again with an older kernel or LTS kernel if you're as unhappy with Mint as I was.

hiddengerbil · 2023-10-23 17:08:53

I did think about the LTS, and I've not tried that yet. Good idea...

ewaller wrote:

pisauraxtx wrote:
You could also try installing Arch again with an older kernel or LTS kernel if you're as unhappy with Mint as I was.
Or, one could install the LTS kernel along side the mainline kernel in the existing install and choose which one to use in your bootloader.

pisauraxtx · 2023-10-23 20:50:42

hiddengerbil wrote:

I did think about the LTS, and I've not tried that yet. Good idea...

Let me know how it goes!

pisauraxtx · 2023-10-30 11:41:17

Update: it looks like the crash was caused by a Firewire extension card. I removed it from my computer and now I can boot with the 6.5 and newer kernels without issues.
See also:
https://bugzilla.kernel.org/show_bug.cgi?id=217993#c11

Arch Linux

#1 2023-09-22 22:45:29

Crash during systemd and boot loop

#2 2023-09-22 23:06:57

Re: Crash during systemd and boot loop

#3 2023-09-23 06:31:04

Re: Crash during systemd and boot loop

#4 2023-09-23 09:33:52

Re: Crash during systemd and boot loop

#5 2023-09-23 09:46:52

Re: Crash during systemd and boot loop

#6 2023-09-23 14:01:25

Re: Crash during systemd and boot loop

#7 2023-09-23 18:24:15

Re: Crash during systemd and boot loop

#8 2023-09-23 19:58:02

Re: Crash during systemd and boot loop

#9 2023-10-23 14:45:59

Re: Crash during systemd and boot loop

#10 2023-10-23 16:08:46

Re: Crash during systemd and boot loop

#11 2023-10-23 16:48:36

Re: Crash during systemd and boot loop

#12 2023-10-23 17:06:57

Re: Crash during systemd and boot loop

#13 2023-10-23 17:08:53

Re: Crash during systemd and boot loop

#14 2023-10-23 20:50:42

Re: Crash during systemd and boot loop

#15 2023-10-30 11:41:17

Re: Crash during systemd and boot loop

Board footer