[SOLVED] Catastrophic system failure after kernel 6.6.10 update

thedanbob · 2024-01-10 15:09:57

I run a large number of docker containers on my home server. After updating the system this morning and restarting, the system broke when the docker containers came up. Trying to run any command returned an error message along the lines of "<path to executable> exists but is not executable". I tried deleting /var/lib/docker and starting from scratch but that didn't help. I'm starting to suspect it's a resource exhaustion issue; I could start a few containers (didn't seem to matter which ones) but at some point it brought the system down.

I downgraded linux-lts from 6.6.10 to 6.1.71 since that was one of the updates this morning and that fixed the problem. All the containers come up properly now. However, that is of course only a temporary solution. How can I diagnose this issue? Once the system hangs no commands work and nothing gets logged.

Last edited by thedanbob (2024-01-10 23:45:49)

seth · 2024-01-10 15:15:45

error message along the lines

Please don't paraphrase, https://bbs.archlinux.org/viewtopic.php?id=57855

Inspecting that path might be helpful and did you maybe forget to build some OOT modules?

thedanbob · 2024-01-10 16:04:07

Unfortunately I didn't think to copy the messages, I was in a hurry to just get it working again. I'll get the exact error messages tonight when I have time to troubleshoot.

However, one detail I remember is that at least some of the executables referenced (maybe all) were located in /usr/local/bin and /usr/local/sbin. I just checked and those are both empty directories on my system. So maybe I can keep everything working and diagnose the problem by just removing those directories from $PATH.

To answer your second question, I use ZFS but it built successfully and worked properly until running docker.

2ManyDogs · 2024-01-10 18:40:22

Moving to Newbie Corner.

thedanbob · 2024-01-10 23:45:34

No error messages because I just upgraded to 6.6.10 again and the problem didn't come back. I guess my server just didn't like me this morning.

thedanbob · 2024-01-21 17:26:37

The problem came back with the latest kernel upgrade. This time I was able to figure out that it was a problem with my NVME drive:

Jan 21 10:07:28 potados kernel: nvme nvme0: controller is down; will reset: CSTS=0x3, PCI_STATUS=0x10
Jan 21 10:07:28 potados kernel: nvme0n1: Read(0x2) @ LBA 249200592, 8 blocks, Host Aborted Command (sct 0x3 / sc 0x71)
Jan 21 10:07:28 potados kernel: I/O error, dev nvme0n1, sector 249200592 op 0x0:(READ) flags 0x80700 phys_seg 1 prio class 2
# Last two lines repeated with different sector numbers

The root filesystem was getting remounted read-only which was causing the errors I mentioned before ("file exists but is not executable"). I worked around the issue by moving the directories I was using for docker volume mounts to another drive.

Last edited by thedanbob (2024-01-21 17:47:30)

seth · 2024-01-21 20:23:31

https://wiki.archlinux.org/title/Solid_ … leshooting

thedanbob · 2024-01-22 12:18:03

Unfortunately disabling APST and/or ASPM didn't work. (I tried both nvme_core.default_ps_max_latency_us=0 and pcie_aspm=off kernel parameters.) I think it must be a different bug with similar symptoms. There was a firmware update available for my drive which I've applied, between that and my workaround everything seems to be running fine now.

seth · 2024-01-22 14:11:18

Did you test "iommu=soft"?

thedanbob · 2024-01-22 23:50:05

I added that to my kernel cmdline and moved my docker files back to the NVME drive. Rebooted and everything came up properly, so I'll tentatively say problem solved. Thanks!

Edit: I tried removing "iommu=soft" and the server still came back up, so maybe the new NVME firmware did the trick. In any case the problem seems to be fixed.

Last edited by thedanbob (2024-01-22 23:59:11)

Arch Linux

#1 2024-01-10 15:09:57

[SOLVED] Catastrophic system failure after kernel 6.6.10 update

#2 2024-01-10 15:15:45

Re: [SOLVED] Catastrophic system failure after kernel 6.6.10 update

#3 2024-01-10 16:04:07

Re: [SOLVED] Catastrophic system failure after kernel 6.6.10 update

#4 2024-01-10 18:40:22

Re: [SOLVED] Catastrophic system failure after kernel 6.6.10 update

#5 2024-01-10 23:45:34

Re: [SOLVED] Catastrophic system failure after kernel 6.6.10 update

#6 2024-01-21 17:26:37

Re: [SOLVED] Catastrophic system failure after kernel 6.6.10 update

#7 2024-01-21 20:23:31

Re: [SOLVED] Catastrophic system failure after kernel 6.6.10 update

#8 2024-01-22 12:18:03

Re: [SOLVED] Catastrophic system failure after kernel 6.6.10 update

#9 2024-01-22 14:11:18

Re: [SOLVED] Catastrophic system failure after kernel 6.6.10 update

#10 2024-01-22 23:50:05

Re: [SOLVED] Catastrophic system failure after kernel 6.6.10 update

Board footer