[SOLVED] System not bootable after borked upgrade

mbork · 2024-03-31 07:17:01

So I wanted to update my system (using `yay -Syu`, which AFAIK calls `pacman -Syu` under the hood). The computer hanged during the update, so I rebooted it. It didn't boot, so I booted it from the Arch boot drive. I mounted my filesystem (which succeeded), but `arch-chroot /mnt` entered a weird state where I cannot "see" any files. When I say `ls` in any directory, I get an empty output (but the status code is 0). Weirdly, I can cd around, and `tab`-autocompletion shows that the files apparently are there, but I cannot access them. Unsurprisingly, `pacman -Syu` yields the same result (no output, exit code 0).

When I look around outside chroot, everything seems fine - I can even see that the files look correct (I tried `cat`ting one of my files to see that the contents look good.)

I tried `pacstrap /mnt base` and got lots of errors:

ldconfig: File /usr/lib/<whatever> is empty, not checked

I also tried `pacstrap /mnt linux`, and got a similar result, with one more thing:

(1/1) Updating linux initcpios
call to execv failed (Exec format error)
error: command failed to execute correctly

I could probably reinstall the whole system, but I'd really prefer to fix this "in place". While I have backup of (most of) my files, reinstalling and configuring everything is quite a bit of work, so I'd prefer to avoid it.

Any ideas?

Last edited by mbork (2024-04-02 10:40:44)

schard · 2024-03-31 09:13:49

Don't use yay - especially when it's obviously not working for you.
pacman -Syu not generating any output at all is certainly strange.
Don't pacstrap over an existing system. You're asking for trouble here.
Please submit your pacman log (use a pastebin service) and the out put of # pacman-conf here in code-tags - both from your installed (chrooted) system.

seth · 2024-03-31 09:16:25

The filesystem got corrupted.
Probably https://bugs.archlinux.org/task/77789, Try to add

sudo touch /etc/systemd/do-not-udevadm-trigger-on-update

To address the immediate situation you'll first have to figure how bad things are:

sudo LC_ALL=C pacman -Qkk | grep -v ', 0 altered files'

Since you cannot use the installed pacman, do NOT chroot and use the pacman from the install iso along

--root /mnt --cachedir /mnt/var/cache/pacman/pkg

for all of this.
Don't forget to mount the boot partition into /mnt/boot if you have one.

Everything that shows up w/ a broken mtree needs to be re-installed "--dbonly" first and then again properly.
Also every package w/ suspiciously deviant files (any deviation in /usr/lib and /usr/bin is bad, some deviations in /etc are normal, the rest is gray area) has to be re-installed explicitly.
Finally close w/ a

pacman --root /mnt --cachedir /mnt/var/cache/pacman/pkg -Syu

and make sure there're no pending errors.

Last edited by seth (2024-04-01 06:50:52)

mbork · 2024-03-31 14:00:56

schard wrote:

Don't use yay - especially when it's obviously not working for you.

why? It used to work until today...

schard wrote:

pacman -Syu not generating any output at all is certainly strange.
Don't pacstrap over an existing system. You're asking for trouble here.

why?

schard wrote:

Please submit your pacman log (use a pastebin service) and the out put of # pacman-conf here in code-tags - both from your installed (chrooted) system.

this is not very easy - I'm writing from a different machine, and I certainly don't want to retype my pacman log ;-). But if this turns out to be necessary, I think I can figure out how to post it from CLI via curl. Thing is, it's Easter, so I don't want to sit at my laptop all day instead of going for a walk with my family ;-).

As for `pacman-conf`, the output is empty.

seth · 2024-03-31 14:04:36

The biggest issue w/ all the pacman wrappers is that they blend the AUR w/ the repos - and some do nasty stuff like basically running DOS attacks against the API and then they get blocked and don't work and yay specifically has some suprises if you link the "wrong" commandline switches that are not supposed to be used together and then will do random stuff, presenting you weird errors instead of "you can't combine those"

None of that has however caused your situation, it's just the pacman wrapper SNAFU.

Edit:

I think I can figure out how to post it from CLI via curl

By clicking the first link below…

As for `pacman-conf`, the output is empty.

In what context? The install iso? Did you pass "--root"?

Last edited by seth (2024-03-31 14:05:57)

mbork · 2024-03-31 14:14:28

seth wrote:

Try to add

sudo touch /etc/do-not-udevadm-trigger-on-update

Where do I add this?

seth wrote:

To address the immediate situation you'll first have to figure how bad things are:
sudo LC_ALL=C pacman -Qkk | grep -v ', 0 altered files'
Since you cannot use the installed pacman, do NOT chroot and use the pacman from the install iso along
--root /mnt --cachedir /mnt/var/cache/pacman/pkg
for all of this.
Don't forget to mount the boot partition into /mnt/boot if you have one.

Did this and got lots of output (136 lines). Most said "no mtree file" or "modification time/size/md5 checksum/sha256 checksum mismatch"

seth wrote:

Everything that shows up w/ a broken mtree needs to be re-installed "--dbonly" first and then again properly.
Also every package w/ suspiciously deviant files (any deviation in /usr/lib and /usr/bin is bad, some deviations in /etc are normal, the rest is gray area) has to be re-installed explicitly.
Finally close w/ a
pacman --root /mnt --cachedir /mnt/var/cache/pacman/pkg -Syu
and make sure there're no pending errors.

I assume that you meant saying

pacman --root /mnt --cachedir /mnt/var/cache/pacman/pkg -S <package> --dbonly

. This didn't work - I got similar output as with pacstrap before:

ldconfig: File /usr/lib/... is empty, not checked

a lot of times.

Is there a way to bypass pacman asking "proceed with installation"? With it, I can't redirect its output to a file – if it's necessary that I put its output on pastebin, it would make things easier... (If this can't be done, I'll search for how to save the output in tmux - I guess it's possible, but a simple redirection would be easier.)

Thanks a lot for your help, but it seems it's not enough, unfortunately...

mbork · 2024-03-31 14:19:04

seth wrote:

The biggest issue w/ all the pacman wrappers is that they blend the AUR w/ the repos - and some do nasty stuff like basically running DOS attacks against the API and then they get blocked and don't work and yay specifically has some suprises if you link the "wrong" commandline switches that are not supposed to be used together and then will do random stuff, presenting you weird errors instead of "you can't combine those"
None of that has however caused your situation, it's just the pacman wrapper SNAFU.

I see. What is better then?

seth wrote:

Edit:
I think I can figure out how to post it from CLI via curl
By clicking the first link below…

Silly me. Thanks.

seth wrote:

As for `pacman-conf`, the output is empty.
In what context? The install iso? Did you pass "--root"?

In chrooted shell in the install iso. When I use

--root /mnt

, I get this: http://0x0.st/XzMw.txt

Edit: and here is the result of

grep -a 2024-03-31 /mnt/var/log/pacman.log | curl -F 'file=@-' 0x0.st

: http://0x0.st/XzMx.bin

Last edited by mbork (2024-03-31 14:23:18)

seth · 2024-03-31 14:20:52

Where do I add this?

Nowhere, it's a command. "man touch" to understand what it does. Later.

I assume that you meant saying

The command in that box isn't the first step, but

pacman --root /mnt --cachedir /mnt/var/cache/pacman/pkg -S <package> --dbonly

is what you#re supposed to do fo the packages w/ broken mtrees, yes. (You can install them all at once and you might have to "-Sy" at this point already)

This didn't work

It probably did, at this point we haven't fixd any file on your disk, so there's still gonna be a lot of ldconfig complaints.
That's fine and to be addressed by the next steps.

With it, I can't redirect its output to a file

Yes of course you can

pacman … 2>&1 | tee /tmp/mypacman.log

mbork · 2024-03-31 14:35:37

seth wrote:

Where do I add this?
Nowhere, it's a command. "man touch" to understand what it does. Later.

Yes, I know what touch does, you used the verb "add" and that's why I didn't understand you. My bad.

seth wrote:

I assume that you meant saying
The command in that box isn't the first step, but
pacman --root /mnt --cachedir /mnt/var/cache/pacman/pkg -S <package> --dbonly
is what you#re supposed to do fo the packages w/ broken mtrees, yes. (You can install them all at once and you might have to "-Sy" at this point already)

Yes, I thought so, and that's what I did.

seth wrote:

This didn't work
It probably did, at this point we haven't fixd any file on your disk, so there's still gonna be a lot of ldconfig complaints.
That's fine and to be addressed by the next steps.

I see. So, do I need to reinstall every package where pacman complains about mtrees? (There's quite a few of them, but I can do it if that's what's needed.)

seth wrote:

With it, I can't redirect its output to a file
Yes of course you can
pacman … 2>&1 | tee /tmp/mypacman.log

Thanks, it's rather obvious in hindsight!

Again, thanks a lot for your help! I have at least some hope now...

seth · 2024-03-31 14:50:18

I see. So, do I need to reinstall every package where pacman complains about mtrees?

You'll have to re-install each of them --dbonly to avoid file collisions.
Then you'll have to re-install all packages pacman complains about (minus a couple of false positives, but they won't harm either)
Then you'll have to run a full system upgrade to create a coherent state.

mbork · 2024-03-31 15:03:58

Thanks.

I did the pacman dance you showed me for most of them, and there is definite progress - I can now chroot into my system, and there are less errors when I issue next pacman spells.

However, I have another problem: http://0x0.st/XzM7.txt (this is what happens if I answer "y"). Sounds a bit scary...

Edit: I turned the computer off for now and I'm going to take a walk in the woods. I am now full of hope that my computer will work again soon thanks to you!

Last edited by mbork (2024-03-31 15:11:33)

seth · 2024-03-31 15:12:08

https://archlinux.org/packages/core/x86_64/cryptsetup/ depends on a specific mkinitcpio version, update mkinitcpio first or both together.

mbork · 2024-03-31 18:07:33

Thanks, that helped! Still no complete success, though: after I chrooted into /mnt and tried pacman -S linux, this happened: http://0x0.st/XzSJ.txt

Thinking that I need to update archlinux-keyring, I did this: pacman -S archlinux-keyring, and a similar thing happened: http://0x0.st/XzSy.txt

What do I do now?

Edit: the reason I wanted to do pacman -S linux was that I still could not boot my laptop normally (without the Arch iso drive) - it told me that /vmlinuz-linux file could not be found.

Edit 2: I also tried pacman --root /mnt --cachedir /mnt/var/cache/pacman/pkg -S linux from outside chroot, with the following result: http://0x0.st/XzSF.txt

Edit 3: I found this: https://wiki.archlinux.org/title/Pacman … l_the_keys, which solved this issue. It still won't boot, though.

This is what I get from pacman -Qkk | grep -v ', 0 altered files' : http://0x0.st/XzQT.txt

When I try to boot w/o the arch iso, I get (more or less) this:

/init: line 16: can't open /proc.cmdline: no such file
:: running early hook [udev]
:: running hook [udev]
::  Triggering uevents...
:: running hook [keymap]
:: Loading keymap...done.
ERROR: device '' not found. Skipping fsck.
:: mounting '' on real root
:: running cleanup hook [udev]
ERROR: Failed to mount the real root device.
Bailing out, you are on your own. Good luck.

sh: can't access tty; job control turned off
[rootfs ~]#

and the computer is unresponsive - my keypresses aren't even echoed.

Last edited by mbork (2024-03-31 19:19:15)

seth · 2024-03-31 21:34:32

This is what I get from pacman -Qkk | grep -v ', 0 altered files' : http://0x0.st/XzQT.txt

util-linux-libs: 79 total files, 48 altered files
glib2: 785 total files, 4 altered files

You'll have to at least re-install those two packages.

mbork · 2024-04-01 05:59:37

Thanks!!! I reinstalled every package for which "altered files" was non-zero, and it seems my system is operational now!!! What a relief!

Now, two questions remain.

1. What could have been the reason for this weird behavior? Should I worry about it happening again? I do my backups every day, but still...

2. What also worries me is that when I was rebooting the system from the arch iso image (many times), a few times I got a hang, and once a kernel panic. I'm worried that it might mean some hardware problem with my machine. I understand that it's nearly impossible to diagnose such things over an internet forum, but how would you assess the probability of such a problem? I once had a weird, intermittent issue with my memory apparently not compatible with my processor (or motherboard), and the symptoms were random changes of some bytes in memory or something like that. That was in the nineties, though... Any suggestions/precautions (other than the daily backups I make) I should consider?

(I will wait a bit with marking this as solved, since I'm afraid hardly anyone will look at the above questions when I do that, but I'll definitely do it today.)

Again, thanks a lot for your help and patience!

seth · 2024-04-01 06:50:27

What could have been the reason for this weird behavior?

https://bugs.archlinux.org/task/77789, Try to add

sudo touch /etc/systemd/do-not-udevadm-trigger-on-update

when I was rebooting the system from the arch iso image (many times), a few times I got a hang, and once a kernel panic

Can you reproduce this and has this ever occurred w/ the installed system?
https://bbs.archlinux.org/viewtopic.php?id=292840
Otherwise https://wiki.archlinux.org/title/Stress_testing

Maples31 · 2024-04-02 06:06:26

seth wrote:

I see. So, do I need to reinstall every package where pacman complains about mtrees?
You'll have to re-install each of them --dbonly to avoid file collisions.
Then you'll have to re-install all packages pacman complains about (minus a couple of false positives, but they won't harm either)
Then you'll have to run a full system upgrade to create a coherent state.

I meet exactly the same problem!

So now I should first reinstall the packages which have no mtree with dbonly option, then install all packages which have nonzero alter files without dbonly option. Finally upgrade the whole system.

Is that right?

seth · 2024-04-02 07:33:39

You'll also have to reinstall the packages for which you restored the mtree (ie. "no/broken mtree == nonzero alter files")
nb. that some file deviations, esp. in /etc are perfectly normal
Once you're done you should close everything w/ -Syu to make sure the entire system is in a coherent state and you didn't perform partial updates.

seth · 2024-04-05 21:40:46

Hi, do you still recall the nature of the update crash?
Did the system suddenly reboot or did it stall and you then rebooted it with the power button?

---
This is a mass-inquiry, so please excuse if your thread actually detailed that.
We're trying to get some data on the situation, so it would be very helpful if you can just briefly respond.
Thanks a lot.

Vamp898 · 2024-04-06 02:55:53

I see. What is better then?

I personally would not recommend using any wrapper tools at all and just access AUR with an independent tool.

I personally use aurutils where you setup a custom repo for your own packages and aurutils takes care on downloading/building them and putting them in that repo.

For example if you want anki from aur you'd run

aur sync anki

And this will pull the PKGBUILD file from git, build anki using makepkg, put it in the custom repo using repo-add and afterwards you run pacman -S anki which will install it from your own custom repo. And in the future, aur sync -u will take care of keeping your AUR packages up to date

That way, pacman stays pacman and if anything changes in pacman, you're safe

Last edited by Vamp898 (2024-04-06 02:56:43)

Arch Linux

#1 2024-03-31 07:17:01

[SOLVED] System not bootable after borked upgrade

#2 2024-03-31 09:13:49

Re: [SOLVED] System not bootable after borked upgrade

#3 2024-03-31 09:16:25

Re: [SOLVED] System not bootable after borked upgrade

#4 2024-03-31 14:00:56

Re: [SOLVED] System not bootable after borked upgrade

#5 2024-03-31 14:04:36

Re: [SOLVED] System not bootable after borked upgrade

#6 2024-03-31 14:14:28

Re: [SOLVED] System not bootable after borked upgrade

#7 2024-03-31 14:19:04

Re: [SOLVED] System not bootable after borked upgrade

#8 2024-03-31 14:20:52

Re: [SOLVED] System not bootable after borked upgrade

#9 2024-03-31 14:35:37

Re: [SOLVED] System not bootable after borked upgrade

#10 2024-03-31 14:50:18

Re: [SOLVED] System not bootable after borked upgrade

#11 2024-03-31 15:03:58

Re: [SOLVED] System not bootable after borked upgrade

#12 2024-03-31 15:12:08

Re: [SOLVED] System not bootable after borked upgrade

#13 2024-03-31 18:07:33

Re: [SOLVED] System not bootable after borked upgrade

#14 2024-03-31 21:34:32

Re: [SOLVED] System not bootable after borked upgrade

#15 2024-04-01 05:59:37

Re: [SOLVED] System not bootable after borked upgrade

#16 2024-04-01 06:50:27

Re: [SOLVED] System not bootable after borked upgrade

#17 2024-04-02 06:06:26

Re: [SOLVED] System not bootable after borked upgrade

#18 2024-04-02 07:33:39

Re: [SOLVED] System not bootable after borked upgrade

#19 2024-04-05 21:40:46

Re: [SOLVED] System not bootable after borked upgrade

#20 2024-04-06 02:55:53

Re: [SOLVED] System not bootable after borked upgrade

Board footer