[SOLVED] Kernel panic during boot after crash during upgrade.

LinearView88 · 2024-04-29 19:54:32

Hi,

I recently upgraded my packages with pacman -Syu. After the packages had been downloaded and while they was upgrading, the system crashed hard, to the point of audio stuttering and keyboard lights blinking. Attempting to reboot into my Linux kernel resulted in a kernel panic, and linux-lts got stuck and refused to load at all.

I was able to mount the system and my home partition from a live USB and arch-chroot into them, but I'm not really sure where to go from here. My best guess is that I should downgrade my system to where it was before I ran the last pacman -Syu, but I'm not quite sure how to do that. This is the first time I've dealt with a kernel panic like this. Would anyone be able to advise me as to what I can do to try to fix my system?

I read about an issue with Nvidia recently that could cause a panic like this, but that's just a shot in the dark. At this point I'm more interested in the proper steps to get my system up and running again.

Last edited by LinearView88 (2024-05-01 01:03:56)

seth · 2024-04-29 21:44:31

Boot the install iso, mount the root filesystem of the installed system into /mnt, if you have one, the boot partition into /mnt/boot
Do NOT chroot and run

LC_ALL=C pacman -Qkk --root /mnt | grep -v ', 0 altered files'

to gage how bad the situation is.

Every missing or broken "mtree" package will have to be re-installed w/

pacman -Sy --dbonly --root /mnt --cachedir /mnt/var/cache/pacman/pkg <list of broken packages>

first
Then again along all packages w/ suspicious file deviations (everything in /usr/lib and /usr/bin is bad, some deviations in esp. /etc are perfectly normal - your kernel *is* broken one way or another and needs to be re-installed)

pacman -Sy --root /mnt --cachedir /mnt/var/cache/pacman/pkg <list of broken packages> # this time w/o --dbonly

If the list of broken packages is *really* long, it's probably easier to just re-install all packages - nb. the "--root /mnt --cachedir /mnt/var/cache/pacman/pkg" and you'll still have to do this twice if (first with "--dbonly", second without) if you didn't selectively sanitize the database before.

Last edited by seth (2024-04-29 21:45:18)

LinearView88 · 2024-04-29 23:54:35

Thanks for the response.

seth wrote:

Boot the install iso, mount the root filesystem of the installed system into /mnt, if you have one, the boot partition into /mnt/boot
Do NOT chroot and run
LC_ALL=C pacman -Qkk --root /mnt | grep -v ', 0 altered files'
to gage how bad the situation is.

I tried this. I have such an enormous list of files with md5 hash mismatches that I think its probably going to be faster (and less fraught with errors) to reinstall all packages.

Seth wrote:

If the list of broken packages is *really* long, it's probably easier to just re-install all packages - nb. the "--root /mnt --cachedir /mnt/var/cache/pacman/pkg" and you'll still have to do this twice if (first with "--dbonly", second without) if you didn't selectively sanitize the database before.

I'm a bit confused by what you mean, especially by "-nb." Are you saying that I should run the command detailed in the arch wiki page? Where do I need to add what arguments? I tried looking at the pacman manual page and I'm still not sure exactly where to go.

seth · 2024-04-30 07:17:47

https://en.wikipedia.org/wiki/Nota_bene
DON'T FORGET TO MOUNT THE BOOT PARTITION into /mnt/boot - provided you have one.

pacman --root /mnt -Sy # sync the database, just in case
pacman --root /mnt -Qnq | pacman --root /mnt --cachedir /mnt/var/cache/pacman/pkg -S --dbonly - # sanitize the metadata of all packages
pacman --root /mnt -Qnq | pacman --root /mnt --cachedir /mnt/var/cache/pacman/pkg -S  - # re-install all packages

Instead of --root and --cachedir you could also just use --sysroot or chroot into the installed system and use its pacman, but depending on how fucked up the installed system is, that may not be possible (eg. if pacman, glibc, libcurl or probbly libcurses are affected)

LinearView88 · 2024-04-30 21:03:04

seth wrote:

https://en.wikipedia.org/wiki/Nota_bene
DON'T FORGET TO MOUNT THE BOOT PARTITION into /mnt/boot - provided you have one.
pacman --root /mnt -Sy # sync the database, just in case
pacman --root /mnt -Qnq | pacman --root /mnt --cachedir /mnt/var/cache/pacman/pkg -S --dbonly - # sanitize the metadata of all packages
pacman --root /mnt -Qnq | pacman --root /mnt --cachedir /mnt/var/cache/pacman/pkg -S  - # re-install all packages
Instead of --root and --cachedir you could also just use --sysroot or chroot into the installed system and use its pacman, but depending on how fucked up the installed system is, that may not be possible (eg. if pacman, glibc, libcurl or probbly libcurses are affected)

Gotcha. Unfortunately, attempting to sync the database results in an error: failed to synchronize all databases (unable to lock database)

Is it safe to remove this lock and continue, and if so should I?

seth · 2024-04-30 21:10:48

stat /mnt/var/lib/pacman/db.lck

If there's no running pacman process and the timestamps are dated (from when the update crashed) it's ok to remove the file - you can't make things worse here

LinearView88 · 2024-04-30 21:43:24

seth wrote:

stat /mnt/var/lib/pacman/db.lck
If there's no running pacman process and the timestamps are dated (from when the update crashed) it's ok to remove the file - you can't make things worse here

Nice! Genuinely comforting to know that I can't make things worse

Confirmed that there was no pacman process and the lock timestamps were dated to the original crash, then went ahead and deleted the lock and synced successfully. No problems there.

The second command yielded a no targets specified error. Confirmed that the command was typed correctly and that the cachedir was not empty.

seth · 2024-04-30 21:45:08

Does this output anything?

pacman --root /mnt -Qnq

Edit: If not, see https://wiki.archlinux.org/title/Pacman … l_database

Last edited by seth (2024-04-30 21:46:14)

LinearView88 · 2024-04-30 21:50:22

seth wrote:

Does this output anything?
pacman --root /mnt -Qnq
Edit: If not, see https://wiki.archlinux.org/title/Pacman … l_database

Yep! Outputs native packages just like it should.

seth · 2024-04-30 21:55:01

Try to reverse this

… -S --dbonly -

… --dbonly -S  -

or use process substitution

pacman --root /mnt --cachedir /mnt/var/cache/pacman/pkg -S --dbonly $(pacman --root /mnt -Qnq)

LinearView88 · 2024-04-30 22:10:15

Making progress! The process substitution did the trick.

A number of the packages have signatures of unknown trust, leading pacman to indicate they are invalid or corrupted. If deleted, errors are said to have occurred and the upgrade does not proceed.

seth · 2024-04-30 22:34:05

Aare you operating from a rather dated install iso?

pacman-key --refresh-keys # no --root here

Then try to explcitly install archlinux-keyring first though you should™ live off the iso's pacman keyring… you're not chrooting into the system, are you?

LinearView88 · 2024-04-30 22:38:46

seth wrote:

Aare you operating from a rather dated install iso?
pacman-key --refresh-keys # no --root here
Then try to explcitly install archlinux-keyring first though you should™ live off the iso's pacman keyring… you're not chrooting into the system, are you?

I am operating from a very dated install iso. No worries, not chrooting in.

Rather than doing it the way I shouldn't, I'll take the opportunity to install a newer iso, then report back.

LinearView88 · 2024-05-01 01:03:04

That did the trick! Many, many thanks.

I'll be marking this topic as solved. Still have no idea what caused the initial crash, sadly, but I'm just happy to have a useable system again.

seth · 2024-05-01 06:27:08

There's a decent chance that you hit the nvidia bug, https://bbs.archlinux.org/viewtopic.php?id=293400
https://aur.archlinux.org/packages/nvidia-535xx-dkms

Arch Linux

#1 2024-04-29 19:54:32

[SOLVED] Kernel panic during boot after crash during upgrade.

#2 2024-04-29 21:44:31

Re: [SOLVED] Kernel panic during boot after crash during upgrade.

#3 2024-04-29 23:54:35

Re: [SOLVED] Kernel panic during boot after crash during upgrade.

#4 2024-04-30 07:17:47

Re: [SOLVED] Kernel panic during boot after crash during upgrade.

#5 2024-04-30 21:03:04

Re: [SOLVED] Kernel panic during boot after crash during upgrade.

#6 2024-04-30 21:10:48

Re: [SOLVED] Kernel panic during boot after crash during upgrade.

#7 2024-04-30 21:43:24

Re: [SOLVED] Kernel panic during boot after crash during upgrade.

#8 2024-04-30 21:45:08

Re: [SOLVED] Kernel panic during boot after crash during upgrade.

#9 2024-04-30 21:50:22

Re: [SOLVED] Kernel panic during boot after crash during upgrade.

#10 2024-04-30 21:55:01

Re: [SOLVED] Kernel panic during boot after crash during upgrade.

#11 2024-04-30 22:10:15

Re: [SOLVED] Kernel panic during boot after crash during upgrade.

#12 2024-04-30 22:34:05

Re: [SOLVED] Kernel panic during boot after crash during upgrade.

#13 2024-04-30 22:38:46

Re: [SOLVED] Kernel panic during boot after crash during upgrade.

#14 2024-05-01 01:03:04

Re: [SOLVED] Kernel panic during boot after crash during upgrade.

#15 2024-05-01 06:27:08

Re: [SOLVED] Kernel panic during boot after crash during upgrade.

Board footer