[SOLVED] System cannot boot after pacman upgrade crash

troymcfont · 2024-03-18 20:41:35

My system cannot boot anymore after a pacman update crash. It fails to boot after I decrypt the hard drive; a black screen with the cursor appears instead of the GDM login. I have tried to recover it with the live USB medium without luck so far. The first thing I have tried in the live medium is to `arch-chroot` into my system and run `pacman -Syu`, but this fails with

pacman: error when loading shared libraries: /usr/lib/libbz2.so.1.0 file too short

Then, from the USB medium instead (not chrooted) I have tried

pacman --sysroot /mnt -Syu

and this exists without errors ("there is nothing to do").
But of course the state of many libraries is corrupted since when I run

find /mnt/usr/lib -size 0

A very long list of libraries show up.

I have also tried to reinstall base and bzip2 with

pacstrap -K /mnt --overwrite base bzip2

but even if this runs successfully, I still cannot run pacman when chrooted (same error of "/usr/lib/libbz2.so.1.0 file too short file too" appears). At this point I do not know what else to try. Any help would be appreciated. Thanks!

Last edited by troymcfont (2024-03-20 08:16:16)

onemyndseye · 2024-03-18 20:50:12

** Message Deleted **

Last edited by onemyndseye (2024-03-20 17:54:31)

troymcfont · 2024-03-18 21:40:59

Here it is: http://sprunge.us/BZlW4O

Last edited by troymcfont (2024-03-18 21:58:11)

troymcfont · 2024-03-18 22:11:52

So now when I try to run

pacman --sysroot /mnt -S bzip2

I get the error

error: could not open file: /etc/mtab: No such file or directory
error: could not determine filesystem mount points

I am not sure what changed to get this error

seth · 2024-03-18 23:04:54

LC_ALL=C pacman --root /mnt -Qkk 2>&1 | grep -v ', 0 altered files' | tee /tmp/howbadisit.txt
cat /tmp/howbadisit.txt | curl -F 'file=@-' 0x0.st

You'll have to re-install all damaged packages and everything w/ a broken mtree w/ "--dbonly" first.
You can probably also not chroot into the installed system?
You can then not use "--sysroot" but "--root /mnt --cachdir /mnt/var/cache/pacman/pkg" instead.

The journal is from the iso, that's useless.

troymcfont · 2024-03-18 23:15:29

Thanks for the help. I can chroot into the installed system. I posted that journalctl -b because the installed system one was empty (literally) and that confused me.
Here is the output of how bad it is (I used --sysroot /mnt since it worked okay): https://0x0.st/XraX.txt

Last edited by troymcfont (2024-03-18 23:15:54)

seth · 2024-03-18 23:23:47

"Very bad. Very, very bad."

You'll have to install those --db-only first

awk -F ': ' '/mtree/{print $1}' /tmp/howbadisit.txt | sort -u | grep -v error

But there're so many errors that you might as well re-install all packages
First --dbonly, then proper.
Don't forget to mount any root partition into /mnt/boot, if you have one.

The installed pacman is broken, you'll have to use the one from the iso (don't chroot)

troymcfont · 2024-03-18 23:32:16

Using

pacman --sysroot /mnt -Qnq | pacman --sysroot /mnt -S --dbonly -

results in the error

error: could not open file: /etc/mtab: No such file or directory
error: could not determine filesystem mount points

I did mount my efi and boot partitions.

Last edited by troymcfont (2024-03-18 23:32:39)

seth · 2024-03-18 23:35:51

Yeah, *not* sysroot, use --root and --cachedir

troymcfont · 2024-03-19 00:07:02

I managed to fix it finally. Just for reference, the steps were:

mount -t proc proc /mnt/proc; mount --rbind /sys /mnt/sys; mount --rbind /dev /mnt/dev
pacman --root /mnt -Qnq | pacman --cachedir=/mnt/var/cache/pacman/pkg --root /mnt -S --dbonly -
pacman --root /mnt -Qnq | pacman --cachedir=/mnt/var/cache/pacman/pkg --root /mnt -S -

While this issue has been fixed, it is the second time in 3 weeks that I get a crash during a pacman upgrade. I am not sure why though, so it would nice if someone could point me on how to do a system check or anything that can help prevent this. Finally, for reference too, the error happened when upgrading the following packages:

aom bzip2 e2fsprogs libverto ca-certificates-mozilla ctemplate nss harfbuzz libxml2 librsvg discord harfbuzz-icu libbsd gnome-control-center gnnome-keyring grub gtkmm3 lib32-libxml2 lib-nss libinih libopenmpt libpipeline pkgconf procps-ng python-markdown vapoursynth xorg-server-common xorg-server xorg-server-devel xorg-server-xephyr xorg-server-xnest xorg-server-xvfb

Again, if anyone has an idea about the culprit please let me know. And thanks again Seth, you are a life saver!

Last edited by troymcfont (2024-03-19 00:10:41)

seth · 2024-03-19 00:10:44

https://bugs.archlinux.org/task/77789 is OTR to crash stuff wildly.
Try to add

sudo touch /etc/do-not-udevadm-trigger-on-update

Your life was probably not in danger, but please always remember to mark resolved threads by editing your initial posts subject - so others will know that there's no task left, but maybe a solution to find.
Thanks.

troymcfont · 2024-03-19 00:17:38

Done and noted, thanks!

troymcfont · 2024-03-19 21:55:24

I am sorry to reopen this but it is not yet fully solved, as it appears. During a `pacman -Syu` today the following errors showed up

ldconfig: File /usr/lib32/libxml2.so.2.12.6 is empty, not checked.
ldconfig: File /usr/lib32/libsoftokn3.so is empty, not checked.
ldconfig: File /usr/lib32/libxml2.so is empty, not checked.
ldconfig: File /usr/lib32/libssl3.so is empty, not checked.
ldconfig: File /usr/lib32/libnss3.so is empty, not checked.
ldconfig: File /usr/lib32/libfreeblpriv3.so is empty, not checked.
ldconfig: File /usr/lib32/libnsssysinit.so is empty, not checked.
ldconfig: File /usr/lib32/libfreebl3.so is empty, not checked.
ldconfig: File /usr/lib32/libnssutil3.so is empty, not checked.
ldconfig: File /usr/lib32/libxml2.so.2 is empty, not checked.
ldconfig: File /usr/lib32/libsmime3.so is empty, not checked.

Here is the output of LC_ALL=C pacman -Qkk | grep -v ', 0 altered files'
http://0x0.st/Xren.txt

Trying to reinstall this using eg:

pacman -S libxml2 --overwrite '*'

does not fix it. Any help is appreciated, thanks!

Last edited by troymcfont (2024-03-19 21:59:02)

seth · 2024-03-19 22:02:10

lib32-libxml2: no mtree file
lib32-nss: no mtree file

Did you remove/disable the multilib repo from your config?

pacman-conf

troymcfont · 2024-03-19 22:10:19

I think I did not.. Here is output pacman-conf http://0x0.st/XreF.txt

seth · 2024-03-19 22:14:22

Re-install them, first --dbonly, then completely.

Do you use a weird™ FS (zfs, xfs, btrfs) or an aggressive trimming strategy w/ an SSD/nvme?
It would seem that there remained some corruption or new was caused after your previous fix.
You can check your pacman log for whether they were updated in your last run

troymcfont · 2024-03-19 22:25:11

Running for example

pacman -S libxml2 --dbonly
pacman -S libxml2

yields the same error (in both cases). My FS is ext4, indeed on NVME. The only weird™ part is that I have LVM encypted with LUKS. I have 2 logical partitons, root and home.
Looking into pacman log shows indeed that at least libxml2 was not updated when I reinstalled all packages

cat /var/log/pacman.log | grep libxml2
[2024-01-23T21:02:58+0000] [ALPM] installed libxml2 (2.12.4-1)
[2024-01-28T13:18:03+0100] [ALPM] installed lib32-libxml2 (2.12.4-1)
[2024-02-06T00:19:39+0100] [ALPM] upgraded libxml2 (2.12.4-1 -> 2.12.5-1)
[2024-02-06T00:19:39+0100] [ALPM] upgraded lib32-libxml2 (2.12.4-1 -> 2.12.5-1)
[2024-02-07T23:04:08+0100] [ALPM] installed perl-alien-libxml2 (0.19-2)

seth · 2024-03-20 07:08:33

The error isn't from pacman but ldconfig when rebuilding the cache and the broken pacakges are "lib32-libxml2" and "lib32-nss" (nb. the leading "lib32")

troymcfont · 2024-03-20 08:08:46

I see, but how can I fix these? I have some applications (admittedly steam) which cannot start because of problems with libxml2

/usr/share/themes/Adwaita-dark/gtk-2.0/main.rc:733: error: unexpected identifier `direction', expected character `}'
/usr/share/themes/Adwaita-dark/gtk-2.0/hacks.rc:28: error: invalid string constant "normal_entry", expected valid string constant
MESA-LOADER: failed to open iris: /usr/lib32/libxml2.so.2: file too short (search paths /usr/lib32/dri, suffix _dri)
failed to load driver: iris

seth · 2024-03-20 08:09:54

pacman -S --dbonly lib32-libxml2 lib32-nss
pacman -S lib32-libxml2 lib32-nss

Edit: the lib32 thing is important, this is an (old) 32bit client, the regular 64bit version seems fine.

Last edited by seth (2024-03-20 08:11:13)

troymcfont · 2024-03-20 08:16:00

This worked, thanks. I tried the same with libxml2 before as it was also showing up as a ldconfig error but it was not fixing it. This did the trick!

seth · 2024-04-05 21:44:02

Hi, do you still recall the nature of the update crash?
Did the system suddenly reboot or did it stall and you then rebooted it with the power button?

---
This is a mass-inquiry, so please excuse if your thread actually detailed that.
We're trying to get some data on the situation, so it would be very helpful if you can just briefly respond.
Thanks a lot.

troymcfont · 2024-04-06 20:14:07

Hi, yes, the system stalled and I had to reboot it with the power button. I took a picture of the stalled screen.

I see in this thread that they used yay, and I used it too when this happened.

Last edited by troymcfont (2024-04-06 20:18:24)

0xwojak · 2024-04-17 21:59:20

seth wrote:

Hi, do you still recall the nature of the update crash?
Did the system suddenly reboot or did it stall and you then rebooted it with the power button?
---
This is a mass-inquiry, so please excuse if your thread actually detailed that.
We're trying to get some data on the situation, so it would be very helpful if you can just briefly respond.
Thanks a lot.

Hi Seth, don't know if it is still useful. Just in case it is.

It reliably happens on `(2/9)reloading system manager configuration` phase. Laptop freezes. Can't switch tty. Unsurprisingly almost always corrupts pacman database.

Im on Blade 15 with 3080. Only started to happen with 550 driver. Does not happen with Nouveau.

Very likely related to https://forums.developer.nvidia.com/t/s … /284772/64

troymcfont · 2024-04-17 22:08:00

Yeap, and I am using a NVIDIA GeForce RTX 4060 Max-Q / Mobile with current 550 driver. From what I read in the linked post, it looks very much related indeed.

Arch Linux

#1 2024-03-18 20:41:35

[SOLVED] System cannot boot after pacman upgrade crash

#2 2024-03-18 20:50:12

Re: [SOLVED] System cannot boot after pacman upgrade crash

#3 2024-03-18 21:40:59

Re: [SOLVED] System cannot boot after pacman upgrade crash

#4 2024-03-18 22:11:52

Re: [SOLVED] System cannot boot after pacman upgrade crash

#5 2024-03-18 23:04:54

Re: [SOLVED] System cannot boot after pacman upgrade crash

#6 2024-03-18 23:15:29

Re: [SOLVED] System cannot boot after pacman upgrade crash

#7 2024-03-18 23:23:47

Re: [SOLVED] System cannot boot after pacman upgrade crash

#8 2024-03-18 23:32:16

Re: [SOLVED] System cannot boot after pacman upgrade crash

#9 2024-03-18 23:35:51

Re: [SOLVED] System cannot boot after pacman upgrade crash

#10 2024-03-19 00:07:02

Re: [SOLVED] System cannot boot after pacman upgrade crash

#11 2024-03-19 00:10:44

Re: [SOLVED] System cannot boot after pacman upgrade crash

#12 2024-03-19 00:17:38

Re: [SOLVED] System cannot boot after pacman upgrade crash

#13 2024-03-19 21:55:24

Re: [SOLVED] System cannot boot after pacman upgrade crash

#14 2024-03-19 22:02:10

Re: [SOLVED] System cannot boot after pacman upgrade crash

#15 2024-03-19 22:10:19

Re: [SOLVED] System cannot boot after pacman upgrade crash

#16 2024-03-19 22:14:22

Re: [SOLVED] System cannot boot after pacman upgrade crash

#17 2024-03-19 22:25:11

Re: [SOLVED] System cannot boot after pacman upgrade crash

#18 2024-03-20 07:08:33

Re: [SOLVED] System cannot boot after pacman upgrade crash

#19 2024-03-20 08:08:46

Re: [SOLVED] System cannot boot after pacman upgrade crash

#20 2024-03-20 08:09:54

Re: [SOLVED] System cannot boot after pacman upgrade crash

#21 2024-03-20 08:16:00

Re: [SOLVED] System cannot boot after pacman upgrade crash

#22 2024-04-05 21:44:02

Re: [SOLVED] System cannot boot after pacman upgrade crash

#23 2024-04-06 20:14:07

Re: [SOLVED] System cannot boot after pacman upgrade crash

#24 2024-04-17 21:59:20

Re: [SOLVED] System cannot boot after pacman upgrade crash

#25 2024-04-17 22:08:00

Re: [SOLVED] System cannot boot after pacman upgrade crash

Board footer