[SOLVED] Kernel panic after update crash

TotoShampoin · 2024-03-03 09:26:18

Hello, me again!

2 days ago, I had this issue, which got solved pretty quickly.

Today, I did a yay update again, which crashed AGAIN... But this time, it won't even boot, because of this:

error while loading shared libraries: /usr/lib/libmount.so.1: file too short

So, I tried following the same steps as previously. The problem is that pacman is unusable in arch-chroot, so instead I tried using pacman --root, or even pacstrap.

But no matter what I try, I can't seem to reinstall anything, because of that corrupted libmount file.

What can I do to fix this time?

Last edited by TotoShampoin (2024-03-03 15:31:27)

TotoShampoin · 2024-03-03 11:49:53

This is the result of

sudo LC_ALL=C pacman -Qkk --root=/mnt 2>&1 | grep -v ', 0 altered files'

https://0x0.st/H7ZV.txt

Trying to do "pacman --root=/mnt -S --dbonly ..." outputs the same "libmount file too short" error

Trying to do "pacstrap -K ..." also gives the error

Last edited by TotoShampoin (2024-03-03 11:50:18)

seth · 2024-03-03 12:15:46

/usr/lib/libmount.so.1 is provided by util-linux-libs
Do NOT chroot into the installed system and use "pacman --root /mnt --cachedir /mnt/var/cache/pacman/pkg -S --dbonly util-linux-libs util-linux" from the install iso
Then "pacman --root /mnt --cachedir /mnt/var/cache/pacman/pkg -S util-linux-libs util-linux"
Then the rest of the packages.

pacman --root /mnt --cachedir /mnt/var/cache/pacman/pkg -S --dbonly gcr-4 krita lib32-expat lib32-libnghttp2 libadwaita libass libdbusmenu-qt5 libfontenc libjxl libluv libnghttp2 libunibreak libvirt libvirt-python linux mdadm msgpack-c noto-fonts openshadinglanguage pahole python-aiohttp python-ytmusicapi
pacman --root /mnt --cachedir /mnt/var/cache/pacman/pkg -S gcr-4 krita lib32-expat lib32-libnghttp2 libadwaita libass libdbusmenu-qt5 libfontenc libjxl libluv libnghttp2 libunibreak libvirt libvirt-python linux mdadm msgpack-c noto-fonts openshadinglanguage pahole python-aiohttp python-ytmusicapi

If there're remaining issues, post the entire context.

pacman … 2>&1 | tee /dev/stderr > /tmp/pacman.session

should™ hopefully log everything (idk whetehr the iso ships https://man.archlinux.org/man/core/util … cript.1.en )

You maybe want to look into why and how the system frequently "crashes" on updates.

TotoShampoin · 2024-03-03 13:59:15

Sorry for the delay

The first command didn't work...

https://0x0.st/H7NP.txt

TotoShampoin · 2024-03-03 14:05:02

By the way, I have installed Timeshift last time, 2 days ago. Just saying, in case it might help

Last edited by TotoShampoin (2024-03-03 14:39:09)

seth · 2024-03-03 15:19:31

The errors are from some alpm hooks (incl. packageshit…) just continue w/ the installation of the packages (ie. same command w/o dbonly)

TotoShampoin · 2024-03-03 15:30:58

Well, the errors were very scary, but I took your word for it, and... They worked so far.

Now, I'm going one last package upgrade in chroot, and hopefully everything work again.

Ah, the update is done now. Let's try and reboot...

Everything is working again!

Thank you Seth, you are a life saver! ;–;

But now, I have to look into what's going on with those update crashes...
In both cases, it crashed at

( 2/16) Reloading system manager configuration...

Last edited by TotoShampoin (2024-03-03 15:35:14)

seth · 2024-03-03 15:55:38

https://bugs.archlinux.org/task/77789 ?

sudo /etc/systemd/do-not-udevadm-trigger-on-update

But probably post your pacman.log, or at least a segment covering a succesfull and a bad update to illustrate what's going on at the time.

Thank you Seth, you are a life saver! ;–;

https://en.wikipedia.org/wiki/Set_(deity) has a way less favorable take…

Last edited by seth (2024-03-31 23:35:30)

reztho · 2024-03-04 23:12:51

/Hi seth,

this kind of crash happened to me last week and today, just now. The end result is that the pacman.log doesn't get updated at all, the pacman cache files are with 0 bytes in /var/cache/pacman and the installed files during the upgrades ends with 0 bytes too. This is quite nasty.

I confirm too that today's crash happened when the "( 2/16) Reloading system manager configuration..." hook message appeared on the screen.

The problem is I don't have a way to reproduce it in a deterministic manner (and I don't have journal logs for this either; I'm using XFS & nvidia drivers if that helps). I'm considering switching to the LTS kernel in the meantime.

I've noticed that this is happening a lot to many people, example: https://www.reddit.com/r/archlinux/comm … perienced/

In fact, I was watching videos on firefox while doing the upgrade so maybe it's related to the nvidia drivers and the 6.7 kernel

seth · 2024-03-05 07:33:54

@TotoShampoin, wrt your DM

error: error while reading file /var/lib/pacman/local/python-aiohttp-3.9.3-1/mtree: Unrecognized archive format
 altered files
python-aiohttp: no mtree file

python-aiohttp is still broken - the rest looks pretty harmless - but you may have copypasted it out of a terminal? The format is weird at places and there seem stray tokens.
I guess you moved around /boot/intel-ucode.img ?

@reztho

reddit wrote:

REMOVED: Rule 4. This qualifies as a shitpost.

What was/is the nature of the crash? Insta-reboot or did the system just freeze and you rebooted w/ the power button?
The latter is prone to leave behind corrupted files. Esp. w/ caching SSDs - so that's not overly suspicious then.
If the system spontanously reboots, that's a CPU, temperature or RAM issue.

"Reloading system manager configuration" reloads the systemd daemon:
You two could correlate the output of

find /etc/systemd -type l -exec test -f {} \; -print | awk -F'/' '{ printf ("%-40s | %s\n", $(NF-0), $(NF-1)) }' | sort -f

The next hook would (most likely, check your pacman log - should be same on successful updates) be packagekit. Do you have/use that, too?

TotoShampoin · 2024-03-05 07:40:46

seth wrote:

but you may have copypasted it out of a terminal? The format is weird at places and there seem stray tokens.

Ah damnit, that was visible? Sorry for that

seth wrote:

error: error while reading file /var/lib/pacman/local/python-aiohttp-3.9.3-1/mtree: Unrecognized archive format
 altered files
python-aiohttp: no mtree file

python-aiohttp is still broken

I saw that, I fixed it

seth wrote:

I guess you moved around /boot/intel-ucode.img ?

... Not that I remember? I think I kinda remember doing something along the lines of manually doing "pacman -S intel-ucode", but why would that do something that pacman doesn't expect to see?

seth wrote:

the rest looks pretty harmless

Well... Okay then. But what do they indicate?

TotoShampoin · 2024-03-05 07:45:01

seth wrote:

You two could correlate the output of

find /etc/systemd -type l -exec test -f {} \; -print | awk -F'/' '{ printf ("%-40s | %s\n", $(NF-0), $(NF-1)) }' | sort -f

Here you go : https://0x0.st/H7j8.status

seth wrote:

What was/is the nature of the crash? Insta-reboot or did the system just freeze and you rebooted w/ the power button?

In my case, it's the latter. Both times. So, I guess that means it is a kernel issue, if I'm not the only one experiencing those crashes?

Last edited by TotoShampoin (2024-03-05 07:46:57)

seth · 2024-03-05 08:05:47

Not necessarily the kernel, see eg. https://bbs.archlinux.org/viewtopic.php … 5#p2073025
For visibility, you're using

bluetooth.service                        | bluetooth.target.wants
dbus-org.bluez.service                   | system
dbus-org.freedesktop.nm-dispatcher.service | system
dbus-org.freedesktop.timesync1.service   | system
display-manager.service                  | system
docker.service                           | multi-user.target.wants
fstrim.timer                             | timers.target.wants
gcr-ssh-agent.socket                     | sockets.target.wants
getty@tty1.service                       | getty.target.wants
gnome-keyring-daemon.socket              | sockets.target.wants
libvirtd-ro.socket                       | sockets.target.wants
libvirtd.service                         | multi-user.target.wants
libvirtd.socket                          | sockets.target.wants
NetworkManager.service                   | multi-user.target.wants
NetworkManager-wait-online.service       | network-online.target.wants
p11-kit-server.socket                    | sockets.target.wants
pipewire-pulse.socket                    | sockets.target.wants
pipewire-session-manager.service         | user
pipewire.socket                          | sockets.target.wants
power-profiles-daemon.service            | graphical.target.wants
remote-fs.target                         | multi-user.target.wants
speech-dispatcherd.service               | multi-user.target.wants
speech-dispatcher.service                | system
supergfxd.service                        | getty.target.wants
systemd-timesyncd.service                | sysinit.target.wants
timeshift.service                        | multi-user.target.wants
virtlockd.socket                         | sockets.target.wants
virtlogd.socket                          | sockets.target.wants
wireplumber.service                      | pipewire.service.wants
xdg-user-dirs-update.service             | default.target.wants

You could also try to explpicitly reload the daemon and see whether that shows any roadbumps.

reztho · 2024-03-05 16:43:38

seth wrote:

Not necessarily the kernel, see eg. https://bbs.archlinux.org/viewtopic.php … 5#p2073025
For visibility, you're using

bluetooth.service                        | bluetooth.target.wants
dbus-org.bluez.service                   | system
dbus-org.freedesktop.nm-dispatcher.service | system
dbus-org.freedesktop.timesync1.service   | system
display-manager.service                  | system
docker.service                           | multi-user.target.wants
fstrim.timer                             | timers.target.wants
gcr-ssh-agent.socket                     | sockets.target.wants
getty@tty1.service                       | getty.target.wants
gnome-keyring-daemon.socket              | sockets.target.wants
libvirtd-ro.socket                       | sockets.target.wants
libvirtd.service                         | multi-user.target.wants
libvirtd.socket                          | sockets.target.wants
NetworkManager.service                   | multi-user.target.wants
NetworkManager-wait-online.service       | network-online.target.wants
p11-kit-server.socket                    | sockets.target.wants
pipewire-pulse.socket                    | sockets.target.wants
pipewire-session-manager.service         | user
pipewire.socket                          | sockets.target.wants
power-profiles-daemon.service            | graphical.target.wants
remote-fs.target                         | multi-user.target.wants
speech-dispatcherd.service               | multi-user.target.wants
speech-dispatcher.service                | system
supergfxd.service                        | getty.target.wants
systemd-timesyncd.service                | sysinit.target.wants
timeshift.service                        | multi-user.target.wants
virtlockd.socket                         | sockets.target.wants
virtlogd.socket                          | sockets.target.wants
wireplumber.service                      | pipewire.service.wants
xdg-user-dirs-update.service             | default.target.wants

You could also try to explpicitly reload the daemon and see whether that shows any roadbumps.

Thank you for answering, Seth. Yeah, I had to poweroff with the power button so that explains the 0 byte files.

Here's what we have in common:

bluetooth.service                        | bluetooth.target.wants
dbus-org.bluez.service                   | system
dbus-org.freedesktop.nm-dispatcher.service | system
display-manager.service                  | system
fstrim.timer                             | timers.target.wants
gcr-ssh-agent.socket                     | sockets.target.wants
getty@tty1.service                       | getty.target.wants
gnome-keyring-daemon.socket              | sockets.target.wants
libvirtd-ro.socket                       | sockets.target.wants
libvirtd.service                         | multi-user.target.wants
libvirtd.socket                          | sockets.target.wants
NetworkManager.service                   | multi-user.target.wants
NetworkManager-wait-online.service       | network-online.target.wants
p11-kit-server.socket                    | sockets.target.wants
pipewire-pulse.socket                    | sockets.target.wants
pipewire-session-manager.service         | user
pipewire.socket                          | sockets.target.wants
remote-fs.target                         | multi-user.target.wants
virtlockd.socket                         | sockets.target.wants
virtlogd.socket                          | sockets.target.wants
wireplumber.service                      | pipewire.service.wants
xdg-user-dirs-update.service             | default.target.wants

(I did your command plus comm -1 -2 with the output in my computer and their output)

(About the reddit link, that poor folk had the exact same issue whilst updating with pacman)

I'm, for now, on LTS kernel to see if I avoid the issue again and I don't use PackageKit.

Last edited by reztho (2024-03-05 16:50:03)

seth · 2024-03-05 18:17:21

From that list, libvirt looks at me.
Otherwise I'd almost have said supergfxd.service looks like a natural contender.
Can either of you cause the freeze by simply reloading the daemon? ("sudo systemctl daemon-reload")

I'd suggest to record your complete service list, it might be the same impact from different causes (eg. if you're using sth. like bumblebee which is damn close to supergfxd)

reztho · 2024-03-05 18:27:05

Hi Seth, thank you. I don't use bumblebee, just pure nvidia prime (not reverse, but direct) configuration (using XFCE but apart from that, the only nvidia things I have active with systemd are nvidia-resume/nvidia-suspend for suspend / hibernation). As a matter of fact, before your last comment, I uninstalled libvirt too just in case since it looks I had it installed without any use here.

Anyway, here's my full service list:

acpid.service                            | multi-user.target.wants
auditd.service                           | multi-user.target.wants
bluetooth.service                        | bluetooth.target.wants
chronyd.service                          | multi-user.target.wants
clamav-freshclam-once.timer              | timers.target.wants
cups.path                                | multi-user.target.wants
cups.service                             | multi-user.target.wants
cups.service                             | printer.target.wants
cups.socket                              | sockets.target.wants
dbus-org.bluez.service                   | system
dbus-org.fedoraproject.FirewallD1.service | system
dbus-org.freedesktop.home1.service       | system
dbus-org.freedesktop.NetworkManager.service | system
dbus-org.freedesktop.nm-dispatcher.service | system
dbus-org.freedesktop.resolve1.service    | system
dbus-org.freedesktop.thermald.service    | system
dbus.service                             | system
dbus.service                             | user
display-manager.service                  | system
earlyoom.service                         | multi-user.target.wants
firewalld.service                        | multi-user.target.wants
fstrim.timer                             | timers.target.wants
gcr-ssh-agent.socket                     | sockets.target.wants
getty@tty1.service                       | getty.target.wants
gnome-keyring-daemon.socket              | sockets.target.wants
linux-modules-cleanup.service            | basic.target.wants
NetworkManager.service                   | multi-user.target.wants
NetworkManager-wait-online.service       | network-online.target.wants
nvidia-resume.service                    | systemd-hibernate.service.wants
nvidia-resume.service                    | systemd-suspend.service.wants
nvidia-suspend.service                   | systemd-suspend.service.wants
p11-kit-server.socket                    | sockets.target.wants
pipewire-pulse.socket                    | sockets.target.wants
pipewire-session-manager.service         | user
pipewire.socket                          | sockets.target.wants
reflector.timer                          | timers.target.wants
remote-fs.target                         | multi-user.target.wants
smartd.service                           | multi-user.target.wants
systemd-homed.service                    | multi-user.target.wants
systemd-resolved.service                 | multi-user.target.wants
systemd-userdbd.socket                   | sockets.target.wants
thermald.service                         | multi-user.target.wants
tlp.service                              | multi-user.target.wants
wireplumber.service                      | pipewire.service.wants
xdg-user-dirs-update.service             | default.target.wants

And no, I tried to do the daemon-reload a lot yesterday after the latest crash to see if I trigger the failure, but nothing.

Last edited by reztho (2024-03-05 18:36:58)

seth · 2024-03-05 18:42:45

I tried to do the daemon-reload a lot yesterday after the latest crash

Might be coincidental. Is this a very recent problem?
Try to add "numa=off" to the kernel parameters, https://lore.kernel.org/all/20240216111 … ul@sk.com/ can pot. throw a lot of things off

reztho · 2024-03-05 19:00:52

seth wrote:

I tried to do the daemon-reload a lot yesterday after the latest crash
Might be coincidental. Is this a very recent problem?
Try to add "numa=off" to the kernel parameters, https://lore.kernel.org/all/20240216111 … ul@sk.com/ can pot. throw a lot of things off

Alright, I'll do a little sacrifice and I'll be back on kernel 6.7.8 with numa=off for a while. Let's see how it goes now without libvirt too.

Edit: Regarding how recent these events are, at least for me, first crash was last Wednesday and second one yesterday.

Last edited by reztho (2024-03-05 23:47:01)

reztho · 2024-03-16 08:18:11

It went well for a week or so, and since I didn't want more of these random hangs everytime when I update with pacman, I decided to go with the LTS kernel... now guess what, just now, it happened to me too again (kernel LTS 6.6.21)... the only packages being installed were ghostscript and ostree. I had firefox and a video on the background too... not sure if that's the pattern for me.

Last edited by reztho (2024-03-16 11:29:50)

seth · 2024-03-16 08:28:38

LTS kernel + nvidia 550xx?
Tried LTS + nvidia 535xx/545xx?

reztho · 2024-03-16 11:29:09

nvidia dkms 550.54.14-5 & kernel 6.6.21-1-lts, yes. Haven't tried any other combination.

Right now, I tried to setup the systemd-pstore service to see if in next hang, this gives me any info. Let's see.

Last edited by reztho (2024-03-16 11:29:24)

seth · 2024-03-16 14:40:02

You most likely want to downgrade to 545 or 535 dkms

gugah · 2024-03-26 14:18:08

Hi seth, I've been experiencing the same issue in my system -3 times this year already- (in my case my kernel panics, confirmed since the capslock key was blinking https://bbs.archlinux.org/viewtopic.php?id=294216). The 2 coincidences I see with this post is 1) nvidia drivers (using nvidia package with latest 550 release) and 2) yay update calls pacman.
I do want to look into the why my system crashes on updates.

reztho · 2024-03-28 22:13:08

Hi gugah, could you run this command line and post the output here? Thanks. I'm suspecting of the pacman hook for running systemd tmpfiles "cleaning" currently.

for i in /usr/lib/tmpfiles.d/*.conf; do pacman -Qo $i ; done | cut -d ' ' -f5 | sort -u

Last edited by reztho (2024-03-28 22:13:58)

gugah · 2024-03-29 20:17:20

reztho:

$ for i in /usr/lib/tmpfiles.d/*.conf; do pacman -Qo $i ; done | cut -d ' ' -f5 | sort -u
alsa-utils
audit
dbus
dhcpcd
filesystem
geoclue
kio-fuse
libutempter
man-db
mkinitcpio
openssh
pam
sddm
shadow
sudo
systemd
tpm2-tss
udisks2
util-linux

(couldn't reproduce the bug yet, running paru now instead of yay)

Arch Linux

#1 2024-03-03 09:26:18

[SOLVED] Kernel panic after update crash

#2 2024-03-03 11:49:53

Re: [SOLVED] Kernel panic after update crash

#3 2024-03-03 12:15:46

Re: [SOLVED] Kernel panic after update crash

#4 2024-03-03 13:59:15

Re: [SOLVED] Kernel panic after update crash

#5 2024-03-03 14:05:02

Re: [SOLVED] Kernel panic after update crash

#6 2024-03-03 15:19:31

Re: [SOLVED] Kernel panic after update crash

#7 2024-03-03 15:30:58

Re: [SOLVED] Kernel panic after update crash

#8 2024-03-03 15:55:38

Re: [SOLVED] Kernel panic after update crash

#9 2024-03-04 23:12:51

Re: [SOLVED] Kernel panic after update crash

#10 2024-03-05 07:33:54

Re: [SOLVED] Kernel panic after update crash

#11 2024-03-05 07:40:46

Re: [SOLVED] Kernel panic after update crash

#12 2024-03-05 07:45:01

Re: [SOLVED] Kernel panic after update crash

#13 2024-03-05 08:05:47

Re: [SOLVED] Kernel panic after update crash

#14 2024-03-05 16:43:38

Re: [SOLVED] Kernel panic after update crash

#15 2024-03-05 18:17:21

Re: [SOLVED] Kernel panic after update crash

#16 2024-03-05 18:27:05

Re: [SOLVED] Kernel panic after update crash

#17 2024-03-05 18:42:45

Re: [SOLVED] Kernel panic after update crash

#18 2024-03-05 19:00:52

Re: [SOLVED] Kernel panic after update crash

#19 2024-03-16 08:18:11

Re: [SOLVED] Kernel panic after update crash

#20 2024-03-16 08:28:38

Re: [SOLVED] Kernel panic after update crash

#21 2024-03-16 11:29:09

Re: [SOLVED] Kernel panic after update crash

#22 2024-03-16 14:40:02

Re: [SOLVED] Kernel panic after update crash

#23 2024-03-26 14:18:08

Re: [SOLVED] Kernel panic after update crash

#24 2024-03-28 22:13:08

Re: [SOLVED] Kernel panic after update crash

#25 2024-03-29 20:17:20

Re: [SOLVED] Kernel panic after update crash

Board footer