kernel panic during pacman update

gugah · 2024-03-25 15:18:15

I have experienced this issue at least 3 times in the past 2 months but it is difficult to reproduce:

When updating my system using yay (pacman -Syu) I get a kernel panic when running the post-transaction hooks, after all packages are supposedly downloaded and installed:

( 2/16) Reloading system manager configuration...

Then the CapsLock key starts blinking, SysReq keys don't help and I need to force reboot with the power button. After rebooting kernel panics again and I have to fix my system following this section https://wiki.archlinux.org/title/pacman … an_upgrade, by re-installing the packages that were being updated during the crash (I find those using this command:

find /var/cache/pacman/pkg -size 0

). It could be that the kernel panicked because the update was nuking my system with empty files, but I'm not 100% sure because of the hard reboot. yay could be faulty too (I switched to paru just in case), but the logs point to pacman.

These are the relevant logs I could find:

/var/log/pacman.log

[2024-03-24T13:11:08-0400] [PACMAN] Running 'pacman -S -y --config /etc/pacman.conf --'
[2024-03-24T13:11:08-0400] [PACMAN] synchronizing package lists
[2024-03-24T13:11:32-0400] [PACMAN] Running 'pacman -S -y -u --config /etc/pacman.conf -- extra/go'
[2024-03-24T13:11:32-0400] [PACMAN] synchronizing package lists
^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@
^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@
^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@
^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@
^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@
^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@
^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@
^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@
^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@[2024-03-24T13:23:07-0400] [PACMAN] Running 'pacman -Syu'
[2024-03-24T13:23:07-0400] [PACMAN] synchronizing package lists
[2024-03-24T13:37:55-0400] [PACMAN] Running 'pacman -Syu'
[2024-03-24T13:37:55-0400] [PACMAN] synchronizing package lists

(go was being installed as a make dependency for yay)

This is the journalctl from the time of the crash:

$ journalctl --since "2024-03-24 13:11:00" --until "2024-03-24 13:20:00"
Mar 24 13:11:00 thinkpad ntpd[810]: Listen normally on 6 wlp4s0 192.168.0.231:123
Mar 24 13:11:00 thinkpad ntpd[810]: Listen normally on 7 wlp4s0 [fe80::2067:3443:ecff:6546%3]:123
Mar 24 13:11:00 thinkpad ntpd[810]: new interface(s) found: waking up resolver
Mar 24 13:11:01 thinkpad rtkit-daemon[1080]: Supervising 7 threads of 5 processes of 1 users.
Mar 24 13:11:01 thinkpad rtkit-daemon[1080]: Supervising 7 threads of 5 processes of 1 users.
Mar 24 13:11:03 thinkpad root[85978]: ACPI group/action undefined: button/up / UP
Mar 24 13:11:03 thinkpad root[85980]: ACPI group/action undefined: button/up / UP
Mar 24 13:11:03 thinkpad root[85982]: ACPI group/action undefined: button/up / UP
Mar 24 13:11:03 thinkpad root[85984]: ACPI group/action undefined: button/up / UP
Mar 24 13:11:03 thinkpad root[85986]: ACPI group/action undefined: button/up / UP
Mar 24 13:11:04 thinkpad root[85988]: ACPI group/action undefined: button/up / UP
Mar 24 13:11:04 thinkpad root[85990]: ACPI group/action undefined: button/up / UP
Mar 24 13:11:04 thinkpad root[85992]: ACPI group/action undefined: button/up / UP
Mar 24 13:11:04 thinkpad root[85994]: ACPI group/action undefined: button/up / UP
Mar 24 13:11:04 thinkpad root[85996]: ACPI group/action undefined: button/up / UP
Mar 24 13:11:05 thinkpad root[85998]: ACPI group/action undefined: button/up / UP
Mar 24 13:11:06 thinkpad root[86000]: ACPI group/action undefined: button/down / DOWN
Mar 24 13:11:06 thinkpad kernel: usb 1-9: reset full-speed USB device number 3 using xhci_hcd
Mar 24 13:11:07 thinkpad systemd[1]: NetworkManager-dispatcher.service: Deactivated successfully.
Mar 24 13:11:08 thinkpad fprintd[85906]: LED blinking failed with error Operation was cancelled
Mar 24 13:11:08 thinkpad sudo[86017]:  gugah : TTY=pts/1 ; PWD=/home/gugah ; USER=root ; COMMAND=/usr/bin/pacman -S -y --config /etc/pacman.conf --
Mar 24 13:11:08 thinkpad sudo[86017]: pam_unix(sudo:session): session opened for user root(uid=0) by gugah(uid=1000)
Mar 24 13:11:08 thinkpad sudo[86017]: pam_unix(sudo:session): session closed for user root
Mar 24 13:11:11 thinkpad fprintd[85906]: USB write transfer error: transfer timed out
Mar 24 13:11:11 thinkpad fprintd[85906]: Data exchange failed at state 1, usb error: transfer timed out
Mar 24 13:11:11 thinkpad fprintd[85906]: Deactivation failed at state 1, unexpected device reply during deactivation
Mar 24 13:11:32 thinkpad sudo[90008]:  gugah : TTY=pts/1 ; PWD=/home/gugah ; USER=root ; COMMAND=/usr/bin/pacman -S -y -u --config /etc/pacman.conf -- extra/go
Mar 24 13:11:32 thinkpad sudo[90008]: pam_unix(sudo:session): session opened for user root(uid=0) by gugah(uid=1000)
Mar 24 13:11:46 thinkpad dbus-broker-launch[1130]: Noticed file-system modification, trigger reload.
Mar 24 13:11:46 thinkpad dbus-broker-launch[1773]: Noticed file-system modification, trigger reload.
Mar 24 13:11:46 thinkpad at-spi-bus-launcher[1773]: Missing configuration file in /usr/share/defaults/at-spi2/accessibility.conf +1: /usr/share/defaults/at-spi2/accessibility.conf
Mar 24 13:11:46 thinkpad dbus-broker-launch[1773]: Invalid configuration, ignored.
Mar 24 13:11:46 thinkpad dbus-broker-launch[1773]: Noticed file-system modification, trigger reload.
Mar 24 13:11:46 thinkpad at-spi-bus-launcher[1773]: Policy to allow eavesdropping in /usr/share/defaults/at-spi2/accessibility.conf +15: Eavesdropping is deprecated and ignored
Mar 24 13:11:46 thinkpad at-spi-bus-launcher[1773]: Policy to allow eavesdropping in /usr/share/defaults/at-spi2/accessibility.conf +17: Eavesdropping is deprecated and ignored
Mar 24 13:11:46 thinkpad dbus-broker-launch[1130]: Service file '/usr/share/dbus-1/services/org.kde.dolphin.FileManager1.service' is not named after the D-Bus name 'org.freedesktop.FileManager1'.
Mar 24 13:11:46 thinkpad dbus-broker-launch[1130]: Service file '/usr/share/dbus-1/services/org.kde.kscreen.service' is not named after the D-Bus name 'org.kde.KScreen'.
Mar 24 13:11:46 thinkpad dbus-broker-launch[1130]: Service file '/usr/share/dbus-1/services/org.kde.plasma.Notifications.service' is not named after the D-Bus name 'org.freedesktop.Notifications'.
Mar 24 13:11:46 thinkpad dbus-broker-launch[1130]: Policy to allow eavesdropping in /usr/share/dbus-1/session.conf +31: Eavesdropping is deprecated and ignored
Mar 24 13:11:46 thinkpad dbus-broker-launch[1130]: Policy to allow eavesdropping in /usr/share/dbus-1/session.conf +33: Eavesdropping is deprecated and ignored
Mar 24 13:11:46 thinkpad dbus-broker-launch[1130]: Noticed file-system modification, trigger reload.

edit: some extra info df -h shows the root partition has plenty of free space: 32G free out of 64G but maybe I should check if my SSD are failing?

Last edited by gugah (2024-03-25 15:26:53)

V1del · 2024-03-26 14:39:11

I'd tell systemd and their "udev rule updating guidance" to go eff itself

sudo touch /etc/systemd/do-not-udevadm-trigger-on-update

https://bugs.archlinux.org/task/77789

But if you really want to figure the underlying trigger... just from this log I'd point my finger(print) at fprintd and the driver you have for that, maybe check what happens if you disable fprintd/unload the fingerprint module (whichever that is on your system). Other common suspects though usually limited to xorg crashes rather than kernel panics, having old and broken evdev xorg config files lying around

grep -R evdev /{etc,usr/share}/X11/xorg.conf*

should only have a "10-quirks.conf" hit in /usr/share/xorg.conf.d

Last edited by V1del (2024-03-26 14:48:03)

gugah · 2024-03-27 01:27:27

$ grep -R evdev /{etc,usr/share}/X11/xorg.conf*
/usr/share/X11/xorg.conf.d/70-synaptics.conf:# This option is recommend on all Linux systems using evdev, but cannot be
/usr/share/X11/xorg.conf.d/10-quirks.conf:# Explicitly tell evdev to not ignore the absolute axes.
/usr/share/X11/xorg.conf.d/10-quirks.conf:        MatchDriver "evdev"
/usr/share/X11/xorg.conf.d/10-quirks.conf:        MatchDriver "evdev"

looks good to me, the other hit is a comment.

Just did a pacman -S systemd to check the hook, but didn't trigger any issue.

Maybe unrelated but just before these issues I updated to plasma6 which defaults to wayland instead of X11.

Last edited by gugah (2024-03-27 01:28:42)

gugah · 2024-04-02 15:26:18

I ran

sudo touch /etc/systemd/do-not-udevadm-trigger-on-update

Haven't experienced these updates crashes since.

V1del · 2024-04-02 15:42:00

Do you run the nvidia driver? We're currently discussing a revert of this in https://gitlab.archlinux.org/archlinux/ … /issues/26 and one of the systemd maintainers is adamant this is a bug in nvidia and triggered on daemon-reload which wouldn't match with this file being the fix.

seth · 2024-04-02 16:01:25

From https://bbs.archlinux.org/viewtopic.php?id=291773 - yes.

If there's reason to suspect it's nvidia's fault (I checked some of the reports, I could indeed not find one w/o the nvidia driver), /etc/systemd/do-not-udevadm-trigger-on-update should be added to the nvidia-utils package(s, also in AUR) to hopefully silence this.

And people haven't "properly investigated" this because the kernel halts, they press the power button, all information is lost and reboot to ground zero that they then have to adress.
Proper investigation would mean to ask them to risk to deliberately please repeat that.

so it's really hard for me to believe that this is causes by udevd reload.

Nobody's saying that it's the cause, but the (uneccsary?) trigger - you also don't jam a bullet personally into someones head, but you're not getting scot free on the theory that you just pulled a little lever and how bad could that possibly be.

Last edited by seth (2024-04-02 16:10:35)

gugah · 2024-04-02 21:05:49

V1del wrote:

Do you run the nvidia driver? We're currently discussing a revert of this in https://gitlab.archlinux.org/archlinux/ … /issues/26 and one of the systemd maintainers is adamant this is a bug in nvidia and triggered on daemon-reload which wouldn't match with this file being the fix.

yes, latest version with latest kernel:

$ pacman -Qi nvidia 
Name            : nvidia
Version         : 550.67-3
Description     : NVIDIA drivers for linux
Architecture    : x86_64
...

I could remove the /etc/systemd/do-not-udevadm-trigger-on-update file and report I have a system crash during an update...

Last edited by gugah (2024-04-02 21:09:26)

seth · 2024-04-02 21:08:08

@V1del, I'll hopefully get the gitlab account reset by tomorrow and comment on the bug asap.

seth · 2024-04-05 21:39:25

Hi, do you still recall the nature of the update crash?
Did the system suddenly reboot or did it stall and you then rebooted it with the power button?

---
This is a mass-inquiry, so please excuse if your thread actually detailed that.
We're trying to get some data on the situation, so it would be very helpful if you can just briefly respond.
Thanks a lot.

gugah · 2024-04-08 17:07:16

hi seth: I recall the last crash only (had 3 this year):

pacman -Syu (on yakuake, ffox running too) -> got to ( 2/16) Reloading system manager configuration... -> CapsLock blinking (kernel crash)

Then I force-rebooted with the power button and restored my partition with an arch iso

Grandfather-Paradox · 2024-04-14 21:19:33

seth wrote:

Hi, do you still recall the nature of the update crash?
Did the system suddenly reboot or did it stall and you then rebooted it with the power button?
---
This is a mass-inquiry, so please excuse if your thread actually detailed that.
We're trying to get some data on the situation, so it would be very helpful if you can just briefly respond.
Thanks a lot.

This has been happening to me constantly for the last 1-2 months. Like the OP here, I have the same issue where the kernel panics during an update after running the post-transaction hooks. The screen stays frozen indefinitely, and SysRq keys don't work. I hold down the power button to force the computer to shut down. The system is often corrupted to the point of being unable to boot (depending on which packages were updated), requiring booting from a live system from repair.

I've experienced this with both the linux and linux-lts kernels, and I'm also running nvidia and nvidia-lts.

seth · 2024-04-14 21:23:49

https://bbs.archlinux.org/viewtopic.php … 6#p2165046

gugah · 2024-04-18 13:58:17

back here to report that my system crashed again on a pacman update even with /etc/systemd/do-not-udevadm-trigger-on-update present.

Some remarks:
- laptop was sleeping before I updated and yt video playing on ffox while I started the update
- pacman updated a large bunch of packages including nvidia and linux, crashed on post process again
- kernel panic (CapsLock blinking) as mentioned by other users: no sysreq keys working, only thing I could do was to use the power button
- crash nukes system, all the packages updated were broken: needed to https://wiki.archlinux.org/title/pacman … an_upgrade and also recover my boot partition https://unix.stackexchange.com/question … mount-boot

Last edited by gugah (2024-04-18 15:18:49)

seth · 2024-04-18 20:32:34

Do you currently have nvidia_uvm blacklisted?

gugah · 2024-04-18 20:44:54

nvidia_uvm is not blacklisted

$ lsmod | grep uvm       
nvidia_uvm           6631424  0
nvidia              60370944  27 nvidia_uvm,nvidia_modeset

I'll try disabling it since I don't use CUDA all the time.

brandon.arnold · 2024-04-20 17:15:57

Hi all. This happened to me last night, rendering my system un-bootable. I do indeed use the nvidia driver. I've just booted to live installation media and am trying to figure out how to fix the system, according to the Wiki op linked to. I get a bunch of errors on gdk-pixbuf2 saying it already exists in the filesystem. But anyway, +1.

jumperfly · 2024-04-21 21:44:16

Hi all. These lockups have happened to me several times. I'm an Intel+nvidia user. The crash often occurs during pacman upgrades at "reloading system configuration" stage, resulting in the need to repair using live media. I was also able to trigger a freeze performing a systemctl daemon-reload. Often also locks up on shutdown/restart attempt.

Not sure of the trigger, it never happens if just booted, but after some period of use it's a crash waiting to happen! I resorted to running systemctl daemon-reload before attempting any pacman operations, as this causes less damage on a crash!

Anyway... I have tried
- Using both xorg and wayland
- Switching to nouveau driver - this gave me other xorg/wayland crashing issues
- Blacklisting nvidia_uvm

None of the above helped.

I'm currently running on nvidia-open for the last few days. No crashes yet.

seth · 2024-04-21 22:24:06

For all we know nvidia-open isn't affected (unfortunately, because then one could look up the source code)
An alternative mitigation would be to return to the 545xx nvidia-dkms and nvidia-utils packages along the LTS kernel (they won't compile against 6.8 anymore)

gugah · 2024-04-26 14:36:58

I confirm, as many nvidia users commented in other posts, that I can trigger the crash with:

$ sudo systemctl daemon-reload

I run that before trying to update my system to avoid filesystem corruptions.
... and the kernel panic only happens if the laptop was in sleep mode previously.

reztho · 2024-05-01 23:04:14

It happened to me again yesterday... this time I could see stuff in the kernel messages. The desktop survived it somehow but with issues (doing a ls inside a terminal never gives an output and never finishes with no cpu usage, for example). It said kernel NULL pointer dereference bug and then a lot of traces about the cpu soft-locking and stalling: I couldn't keep all the logs, but at least I know next time I can try again to save the logs to a file or at least, take a photo of the screen.

And yeah, it was definitely due to the "systemctl --system daemon-reload" hook and I had the do-not-udevadm-trigger-on-update file in place.

Last edited by reztho (2024-05-01 23:05:04)

Arch Linux

#1 2024-03-25 15:18:15

kernel panic during pacman update

#2 2024-03-26 14:39:11

Re: kernel panic during pacman update

#3 2024-03-27 01:27:27

Re: kernel panic during pacman update

#4 2024-04-02 15:26:18

Re: kernel panic during pacman update

#5 2024-04-02 15:42:00

Re: kernel panic during pacman update

#6 2024-04-02 16:01:25

Re: kernel panic during pacman update

#7 2024-04-02 21:05:49

Re: kernel panic during pacman update

#8 2024-04-02 21:08:08

Re: kernel panic during pacman update

#9 2024-04-05 21:39:25

Re: kernel panic during pacman update

#10 2024-04-08 17:07:16

Re: kernel panic during pacman update

#11 2024-04-14 21:19:33

Re: kernel panic during pacman update

#12 2024-04-14 21:23:49

Re: kernel panic during pacman update

#13 2024-04-18 13:58:17

Re: kernel panic during pacman update

#14 2024-04-18 20:32:34

Re: kernel panic during pacman update

#15 2024-04-18 20:44:54

Re: kernel panic during pacman update

#16 2024-04-20 17:15:57

Re: kernel panic during pacman update

#17 2024-04-21 21:44:16

Re: kernel panic during pacman update

#18 2024-04-21 22:24:06

Re: kernel panic during pacman update

#19 2024-04-26 14:36:58

Re: kernel panic during pacman update

#20 2024-05-01 23:04:14

Re: kernel panic during pacman update

Board footer