Arch Linux

lolposting

Member

Registered: 2023-10-16

Posts: 1

Freeze During Upgrade result: vmlinuz-linux not found"

I've had archlinux running on my media server (plex, sonarr, radarr, etc.) for ~2 years now and update all the packages every other month or so. I do not have physical access to this server so any maintenance or intervention not doable via ssh needs to be done via remote hands.

I updated all outdated packages 2 days ago but sadly was disconnected near the end of the process after the server seemed to freeze on these lines:

:: Running post-transaction hooks...
( 1/27) Creating system user accounts...
Creating group 'rtkit' with GID 133.
Creating user 'rtkit' (RealtimeKit) with UID 133 and GID 133.
( 2/27) Registering binary formats...
( 3/27) Updating journal message catalog...
( 4/27) Reloading system manager configuration...
client_loop: send disconnect: Broken pipe

After rebooting the server with the help of a remote admin on-location I was met with the following error (grub only shows 2 boot options: one archlinux kernel and another failsafe kernel and both fail with this error):

Loading Linux linux  ...
error: file '/boot/vmlinuz-linux' not found.
Loading initial ramdisk ...
error: you need to load the kernel first.

To me it looks like the post-install scripts either failed to create the new kernel image file due to the premature freezing+reboot or that the grub config somehow was replaced and now points to an erroneous location?

Normally I would mount the HD of my arch install, chroot into it and with that shell finish rebuilding the linux kernel and such but with my remote server requiring remote hands for anything this is a bit trickier and I'll have to plan any fix well in advance. I'd like to confirm a few things prior to arranging this:

1. Is the older pre-update version of my kernel somewhere I can boot by updating the grub cfg?
2. Is there any way without a live USB to fix this?
3. Could I verify exactly what happened only via the grub console?

V1del

Forum Moderator

Registered: 2012-10-16

Posts: 21,870

Re: Freeze During Upgrade result: vmlinuz-linux not found"

1. Generally no, on kernel updates a pre-transaction hook runs that will kill all prior kernel images

2. Probably not trivially so -- unless you have some way to chroot into the system that doesn't involve a live usb, after which you could just reinstall the linux package

3. Check your ESP/boot directory, as the message mentions you'll likely lack both vmlinuz-linux and initramfs-linux.img

Mitigations strats for the future, run an update in a terminal client that doesn't kill the remote process when your client disconnects, e.g. a screen or a tmux session so that even if your client disconnects, the server can continue the operation on it's own. Seeing where you got kicked you're another victim of "totally safe, clients are just buggy lol" systemd suggestion to retrigger all udev rules which got included in the package. You can prevent said retrigger from happening by creating the file /etc/systemd/do-not-udevadm-trigger-on-update

See https://bugs.archlinux.org/task/77789 for background information

Last edited by V1del (2023-10-17 13:23:25)