How to get protected from pacman killing /boot img?

patate · 2020-05-12 00:07:23

I faced today a full crash (blackscreen) while pacman - Syu was running.
At reboot, grub did not find any of my kernel (4 img: "normal" kernell and its fallback, LTS kernel and its fallback), all was gone.
From what I see by Googling, it seems pretty common and there is a section in the wiki about it:

https://wiki.archlinux.org/index.php/Pa … _rebooting

All is fine and I recovered (usb boot, chroot etc..)

But now I wonder: is there something that could be done to avoid this?
Maybe Is there a way to install a "protected" kernell that pacman won't touch? (so I can boot, even with no usb stick around)

seth · 2020-05-12 06:19:53

You can either install an additional kernel from the repo and ignore it in your pacman.conf or install an AUR kernel (since pacman doesn't manage AUR packages), but that does not mean the system will boot because pretty much anything could have gotten screwed by the broken update.
You could use https://wiki.syslinux.org/wiki/index.php?title=MEMDISK to boot eg. the archiso image from the disk (but the boot partition must be large enough to hold the image)
Edit: this also implies that the grub installation must not get broken during an update, you'll still require grub to load and be able to load memdisk.

seems pretty common

About as common as not having a USB key around ;-)

Last edited by seth (2020-05-12 06:21:18)

patate · 2020-05-12 07:39:03

seth wrote:

You can either install an additional kernel from the repo and ignore it in your pacman.conf or install an AUR kernel (since pacman doesn't manage AUR packages), but that does not mean the system will boot because pretty much anything could have gotten screwed by the broken update.

it worth to give a try! :-)
I'm pretty sure pacman killed the .img file "only":
- the img files were here in /boot, but only 21 bytes
- other files in /boot seems ok (not 21 byte)
- preset files in /etc/mkinitcpio.d was renamed in .preset.pacsave instead of .preset (the guilty has left some clue!!)

seth wrote:

You could use https://wiki.syslinux.org/wiki/index.php?title=MEMDISK to boot eg. the archiso image from the disk (but the boot partition must be large enough to hold the image)
Edit: this also implies that the grub installation must not get broken during an update, you'll still require grub to load and be able to load memdisk.

I prefer previous idea, all this is quite new to me (I only start to understand how this /boot works) and I'm not fully comfortable in booting an iso in the middle of my regular /boot :-)

seth wrote:

seems pretty common
About as common as not having a USB key around ;-)

:-) you're right!
well, I didn't meant it happen very often, but for sure it's really easy to find something with searching "no boot after pacman" or similar.
and yes, it's a good habit to keep an arch-iso usb stick well labelled in your drawer.
I would even suggest to have a live distro as well, so you can browse the web if needed (if no other computer around).
Nevertheless, it can happen if you launch a pacman -Syu while away from home (no risk no fun)...

Trilby · 2020-05-12 13:09:19

EDIT: apparently, given the next few posts, recent changes make my assessment here mostly wrong. Sorry for the noise. My Original post is below.

---

patate wrote:

I'm pretty sure pacman killed the .img file "only"

Sure, this time. But there should be no pattern to that. If pacman is killed abruptly, it could be in the middle of operations on any package or file. That (I believe) was Seth's point. Coming up with a strategy that protects you only from what failed last time this happened is of very little value in protecting you next time.

As far as it being common, in 10 years of using arch as my primary OS (for many of those years on multiple systems including servers) I don't recall every having such a problem. I've had many problems, but not the one you describe. It has certainly happened to others, but the fact that there is a strategy to cope with it on the wiki doesn't mean it's common - it just means it's significant enough that on the very odd occasiona that it might happen to someone, they will likely need guidance to work their way out of that situation.

In other words, while I certainly hope I never get hit by lightning, I'm not particuarly worried about it. The fact that it has happened to others and hospitals have strategies to treat struck patients doesn't mean that it's a common occurrence. And if it were to happen to me (and I survived), I'd not then be inspired to go buy lightning insurance after that. Freak accidents can happen, and it's unfortunate that one may have happened on your system, but devising strategies to prevent the exact same freak accident from happening again doesn't seem like a wise use of effort. If what should be a very rare freak accident is happening to you repeatedly, you would be better served by getting to the bottom of what's making your system a susceptible target.

patate wrote:

it's really easy to find something with searching "no boot after pacman" or similar.

Have you read those threads? They are not going to be so similar to your situation. In fact, a vast majority of those threads are not even a problem booting: someone's display manager or X session didn't start and they concluded that their system "can't boot". Most of the time it is due to failing to read pacman output, the front page news, or perhaps to partial upgrades or other bad practices. They are not often due to pacman being killed mid process. Some are, but even among those, not many would be due to pacman being in the middle of generating kernel images at that moment but rather something else, like the boot loader, or systemd, etc, etc.

Last edited by Trilby (2020-05-12 19:06:06)

eschwartz · 2020-05-12 18:16:14

pacman didn't kill your .img file, mkinitcpio did. Recent versions of mkinitcpio contain a hook that deletes your .img, deletes your preset, and also deletes your kernel, every time the /usr/lib/modules/$(uname -r) directory changes. This occurs before any packages are installed, removed, or upgraded. After the transaction is *fully* complete, it reinstalls these files from scratch (in the case of the kernel, it will be copied over from /usr/lib/modules). Note that the removal process destroys any modifications you have made to the preset.

This change was introduced as part of a general rework of the core/linux PKGBUILD to make it no longer install mkinitcpio configs because they're not used if you install dracut instead, and also to avoid installing anything at all to /boot because Fedora expects your kernel to be installed in /usr/lib/modules/$(uname -r) and be additionally copied to your boot directory/partition by dracut.

Unfortunately, moving a file from one directory to another is, as far as pacman hooks know, "uninstalling" the file...

(Given the precise nature of the change, I am unable to think of a way to handle this correctly without making the kernel package itself provide the preset or a similar tracking file, and that's been rejected as unpalatable...)

As a mkinitcpio user I've taken to building a custom kernel (WIP first draft that simply reverts to the old way of doing things, here: https://pkgbuild.com/~eschwartz/linux-v … src.tar.gz)

Last edited by eschwartz (2020-05-12 18:22:35)

Trilby · 2020-05-12 18:55:02

Hmm. Eschwartz, if I'm reading that right, you're saying that the removal of the kernel and initramfs happens at the very start of the whole pacman transaction, and replacing them doesn't happen until the end? If so, my analysis would be wrong as a power cut at any time in a relevant pacman transaction would result in this problem not just if it were during an update of mkinitcpio or the kernel. That seems like a very bad design.

I don't see why the old kernel and images (under /boot) would need to be deleted at the start of such a transaction. Couldn't the post-transaction hook simply overwrite the old ones with a new one? (or remove them right before creating the new ones).

Last edited by Trilby (2020-05-12 18:57:42)

eschwartz · 2020-05-12 18:59:47

That's correct. There's a mkinitcpio-provided pacman hook which is PreTransaction and triggered by *removing* a usr/lib/modules/*/vmlinuz file, which wipes everything, and a PostTransaction hook which is triggered by installing or upgrading such files, and installs everything. During the actual transaction process, no files exist except in /usr/lib/modules/.

Trilby wrote:

I don't see why the old kernel and images (under /boot) would need to be deleted at the start of such a transaction. Couldn't the post-transaction hook simply overwrite the old ones with a new one? (or remove them right before creating the new ones).

As I said earlier:
Unfortunately, moving a file from one directory to another is, as far as pacman hooks know, "uninstalling" the file...

The hook is *supposed* to be for removing the kernel/images when you pacman -R somekernel.
What the hook actually does is remove the kernel/images whenever you perform any sort of transaction that causes you to no longer have a *versioned* file installed. The versioned file in question is versioned by using the $pkgver-$pkgrel in the filename.

One solution to this entire issue is "don't try to remove this, that's not the job of a hook, pacman already tracks files for you if you tell it to".

Last edited by eschwartz (2020-05-12 19:05:43)

Awebb · 2020-05-12 19:01:21

Not 100% on topic, but there has been a number of "vmlinuz-linux not found after an update" threads recently. I had this too, but fixed it without understanding the cause. Maybe something is broken that doesn't show up every time.

EDIT: Okay, that number was one and there were multiple people with different problems.

Last edited by Awebb (2020-05-12 19:02:50)

Arch Linux

#1 2020-05-12 00:07:23

How to get protected from pacman killing /boot img?

#2 2020-05-12 06:19:53

Re: How to get protected from pacman killing /boot img?

#3 2020-05-12 07:39:03

Re: How to get protected from pacman killing /boot img?

#4 2020-05-12 13:09:19

Re: How to get protected from pacman killing /boot img?

#5 2020-05-12 18:16:14

Re: How to get protected from pacman killing /boot img?

#6 2020-05-12 18:55:02

Re: How to get protected from pacman killing /boot img?

#7 2020-05-12 18:59:47

Re: How to get protected from pacman killing /boot img?

#8 2020-05-12 19:01:21

Re: How to get protected from pacman killing /boot img?

Board footer