Failing to suspend - udev-worker refusing to freeze

qherring · 2023-10-03 19:12:11

After my system is running for a few days I cannot put it to sleep nor reboot properly (I have to hard reset with power button). Appears some issue with udev-worker being unresponsive. My system activity tell me the unresponsive process (udev-worker) is in "disk sleep" and I cannot kill it no matter what I try.

If I hard reset I am then afterwards able to reboot and put the system to sleep normally, however after some time has passed (2-3 days) while system is running that I see this issue arise again.
This started happening in the last 2 weeks and may correspond with an upgrade to the linux kernel.

Here is last 10000 lines from journalctl:
https://pastebin.com/1RgsGQap

seth · 2023-10-03 21:03:37

It's full of nvme errors, https://wiki.archlinux.org/title/Solid_ … leshooting

nvme_core.default_ps_max_latency_us=0 iommu=soft

Possibly "pcie_aspm=off" (can't tell because the journal tail is too short)

qherring · 2023-10-03 21:55:27

The nvme errors have been persistent since I built the system a few months ago. I haven't been able to resolve them.

However these errors haven't caused any suspend/reboot issue up until now, so are you certain that is the culprit?

how long of a log do you need for journalctl? I keep reaching the limits for pastebin

seth · 2023-10-04 05:58:11

so are you certain that is the culprit?

No, but it's the errors that /are/ in the journal you posted and the only obvious indicator for trouble so far.
Plus udev-worker waits for some IOCTL what always makes IO devices a prime suspect.
And the wiki literally says (errheeemmm)

The Wiki wrote:

Controller failure due to broken suspend support
Some users (for example, see Laptop/HP) have reported suspend failures with certain NVMe drives. As above, the failure renders the device inoperable until system reset, with kernel messages

So I'm taking an educated guess.

See whether you can get rid of them by disabling APST and/or the HW IOMMU

I keep reaching the limits for pastebin

Please post your complete system journal for the boot:

sudo journalctl -b | curl -F 'file=@-' 0x0.st

The limit there is 512MB

Last edited by seth (2023-10-04 05:58:30)

qherring · 2023-10-04 19:00:24

here is current my journalctl -b

(its rather large so I had to break into 4 segments to be under the 512mb limit)

1of 4
https://pastebin.com/LDcdwuQP

2of4
https://pastebin.com/VAVzPAAC

3of4
https://pastebin.com/d04Buv70

4of4
https://pastebin.com/QRnxBC3F

I had to move the PC, so no choice but a hard shutdown. This log will probably not give what you are looking for, I will now have to wait another few days until the system refuses to suspend/reboot again.

seth · 2023-10-04 20:07:11

The combination of all files is 1.5MB - you could have easily posted that to 0x0.st
Your system supports ASPM, so pcie_aspm might help.

The journal neither includes an attempt to S3 nor dormant udev-worker - try the kernel parameters in #2

qherring · 2023-10-04 20:16:19

My bad, misread that as 512kb. oh well next time

I did just trying adding iommu=soft as a kernel parameter, nvme errors are still present

qherring · 2023-10-04 20:35:12

seth wrote:

Your system supports ASPM, so pcie_aspm might help.

I set pcie_aspm=off and it seems to have gotten rid of the nvme errors. Im gonna let the system run for a 3-4 days and see if the issue resurfaces.

I also noticed a faster system boot after adding iommu=soft. I am not sure if I should keep this parameter but for now I see no downside to having it. From what I was reading it should draw a small amount of processor power, but im not worried.

I will leave this unsolved for a few days until I am certain the suspend/reboot issue is gone.

Is this ACPI BIOS Error anything critical?

Thanks again for all your insight!

seth · 2023-10-04 20:58:31

What about "nvme_core.default_ps_max_latency_us=0"?
pcie_aspm will hit everything, not just the nvme.

ACPI errors are unfortunately common, the reason is that board vendors implement the ACPI by poking around until windows stops crashing (reading specs is almost like real work, you know…)
The usual strategy is to ignore them as long as there're no further issues since the only thing you can do to actually, maybe, fix that is a UEFI update.

qherring · 2023-10-05 16:47:58

added nvme_core.default_ps_max_latency_us=0

below is latest journalctl -b

https://pastebin.com/ARrgeJmL

seth wrote:

ACPI errors are unfortunately common

noted, I will ignore

seth · 2023-10-05 18:54:13

The journal is in conjuction w/ iommu=soft and pcie_aspm=off, you certainly want to explore the behavior w/o "pcie_aspm=off"

qherring · 2023-10-07 17:14:47

I removed pcie_aspm=off

Just did a reboot, I'm now getting a nvme error again. Not nearly as pervasive as before but still present.

journalctl -b
https://pastebin.com/79AUuhAh

qherring · 2023-10-07 17:23:04

I also noticed this in both of the two recent journals I posted:

nvme 0000:02:00.0: platform quirk: setting simple suspend

seth · 2023-10-07 17:39:58

They're also in https://bbs.archlinux.org/viewtopic.php … 1#p2124611

More important than the

PCIe Bus Error: severity=Corrected, type=Physical Layer, (Receiver ID)

is whether you can suspend the system w/o APST and the SW IOMMU now.
If yes, try to remove "iommu=soft" next - you want to use as little of these things as necessary.

qherring · 2023-10-07 18:41:08

thanks,

Just to clarify, your suggestion is to only use nvme_core.default_ps_max_latency_us=0
this means omitting both iommu=soft and pcie_aspm=off

seth · 2023-10-07 18:43:10

If you've no issues currently w/o pcie_aspm=off: Yes.
And if that doesn't work, try only iommu=soft - you want to isolate the parameter combo that's absolutely necessary to allow the S3, not more.

qherring · 2023-10-24 09:18:54

Well the it looks like the disk has now become corrupted.

Just now I had a chromium process that was preventing me from suspending or rebooting. I got stuck rebooting and had to hard reset. Now it appears the disk cannot be found. This was a fully encrypted drive with btrfs. I am not even sure what to do at this point to try and recover the drive.

seth · 2023-10-24 13:31:47

ftr, the OP has opened https://bbs.archlinux.org/viewtopic.php?id=289800 about this

qherring · 2023-10-26 18:34:52

Im getting pretty frustrated here. The inability to suspend and restart has not been resolved still. Even with all three parameters set (pcie_aspm=off, iommu=soft, and nvme_core.default_ps_max_latency_us=0)
I also tried to set only each one of these on their own and still the issue persists.

Now the more pressing issue is this:
I just ran an update again this morning (linux kernel had an update once again) and now I cannot boot up again. This time I dont get the "device not found" screen but rather it just hangs on the screen showing Asus (bios logo)

Last time I was able to boot back into the fallback image and reinstall the linux package however this time that option no longer exists. the suggestion it seems would be to arch-chroot into the system and try to run the pacman -S linux update to regenerate the initramfs?
when doing so do I need to mount all btrfs subvolumes or should I only mount the root subvol or should I mount them all?

seth · 2023-10-26 19:44:10

You should first check whether you can mount the disk at all - if the system refuses to boot at all (What do you use as bootloader? Can you enter the UEFI? Does the disk show up there?) that's not a kernel-problem.
You'll have to mount the root and an optional boot partition.

But b/c of the previous bus errors, the nvme might have become completely inaccessible (badly seated?)

qherring · 2023-10-26 19:57:16

seth wrote:

the nvme might have become completely inaccessible (badly seated?)

Doesnt seem to be the case since I chrooted into the system through my other arch system with any issue. I then ran pacman -S linux, and its giving me an error, here is the output of the terminal:
https://pastebin.com/6j4hmRpp

After trying to reinstall the linux package its giving me the same frozen screen at boot.
what am I doing wrong here?

Last edited by qherring (2023-10-26 19:59:19)

seth · 2023-10-26 20:02:20

(4/5) Refreshing PackageKit...
Error connecting: Could not connect: No such file or directory
error: command failed to execute correctly

The error is from packagekit.

Frozen how, where and after what and displaying what?
What is your bootloader? Grub? I'd re-install that.

qherring · 2023-10-26 20:13:36

seth wrote:

Frozen how, where and after what

frozen after I select arch linux as first option in the grub menu. It shows two lines of code as usuall and then goes to the logo from Asus (motherboard/bios manufacturer) and then freezes on this screen.

This is the normal procedure when booting this system (up until the freeze of course)

typically after this logo screen appears it asks me for my password to decrypt the disk, it never gets there now

seth wrote:

What is your bootloader? Grub? I'd re-install that.

grub. Its appears to work fine though right? If it was a grub issue I shouldnt even get to the point am I getting to correct?

qherring · 2023-10-26 20:15:09

seth wrote:

The error is from packagekit.

so this is not related? or irrelevant for the time being?

seth · 2023-10-26 20:24:44

If it was a grub issue I shouldnt even get to the point am I getting to correct?

Correct - the asus logo sounded like you're not even getting to the bootloader.

Try to install and boot the lts kernel.
You might also be running into https://bbs.archlinux.org/viewtopic.php?id=286293 ?

Arch Linux

#1 2023-10-03 19:12:11

Failing to suspend - udev-worker refusing to freeze

#2 2023-10-03 21:03:37

Re: Failing to suspend - udev-worker refusing to freeze

#3 2023-10-03 21:55:27

Re: Failing to suspend - udev-worker refusing to freeze

#4 2023-10-04 05:58:11

Re: Failing to suspend - udev-worker refusing to freeze

Controller failure due to broken suspend support

#5 2023-10-04 19:00:24

Re: Failing to suspend - udev-worker refusing to freeze

#6 2023-10-04 20:07:11

Re: Failing to suspend - udev-worker refusing to freeze

#7 2023-10-04 20:16:19

Re: Failing to suspend - udev-worker refusing to freeze

#8 2023-10-04 20:35:12

Re: Failing to suspend - udev-worker refusing to freeze

#9 2023-10-04 20:58:31

Re: Failing to suspend - udev-worker refusing to freeze

#10 2023-10-05 16:47:58

Re: Failing to suspend - udev-worker refusing to freeze

#11 2023-10-05 18:54:13

Re: Failing to suspend - udev-worker refusing to freeze

#12 2023-10-07 17:14:47

Re: Failing to suspend - udev-worker refusing to freeze

#13 2023-10-07 17:23:04

Re: Failing to suspend - udev-worker refusing to freeze

#14 2023-10-07 17:39:58

Re: Failing to suspend - udev-worker refusing to freeze

#15 2023-10-07 18:41:08

Re: Failing to suspend - udev-worker refusing to freeze

#16 2023-10-07 18:43:10

Re: Failing to suspend - udev-worker refusing to freeze

#17 2023-10-24 09:18:54

Re: Failing to suspend - udev-worker refusing to freeze

#18 2023-10-24 13:31:47

Re: Failing to suspend - udev-worker refusing to freeze

#19 2023-10-26 18:34:52

Re: Failing to suspend - udev-worker refusing to freeze

#20 2023-10-26 19:44:10

Re: Failing to suspend - udev-worker refusing to freeze

#21 2023-10-26 19:57:16

Re: Failing to suspend - udev-worker refusing to freeze

#22 2023-10-26 20:02:20

Re: Failing to suspend - udev-worker refusing to freeze

#23 2023-10-26 20:13:36

Re: Failing to suspend - udev-worker refusing to freeze

#24 2023-10-26 20:15:09

Re: Failing to suspend - udev-worker refusing to freeze

#25 2023-10-26 20:24:44

Re: Failing to suspend - udev-worker refusing to freeze

Board footer