[solved] Safety of rolling back a kernel update before rebooting

eternal_flame-AD · 2024-11-27 23:09:11

(Not sure if this is 100% the correct board to post this because it is the result of an upgrade but I think all solutions are mostly system administration related so I decided to post here)

Hi, I have a cloud server running on Arch that has a website that I would like to prevent downtime as much as possible, 2 days ago I performed a scheduled update which involved bumping linux from 6.11 to 6.12 and migrating to iptables-nft, However I after the update is complete I did not expect that the VNC console from my cloud provider broke down and I know if I reboot there is a high likelihood I won't be able to enter the LUKS password from the console. (Yes I did restart agetty and did everything I can, the system recognized my screen changed after restarted my getty but still won't give me a VNC)

And since it was black friday they still have not replied back on whether this issue is likely fixable by a reboot or I will completely lose access until an intervention on their side if I reboot.

The OS is still functioning but I can no longer change any firewall or docker containers because of the kernel inconsistency. I was wondering since I have not rebooted maybe it is safe to just roll this kernel update back and get the docker access back, and once the VNC is recovered redo the update and reboot?

Another alternative I have considered is to temporarily add an empty password to my LUKS volume and configure it to try the empty password on boot, but I never had a proven workflow on how to do this and still do not want to risk it. However if someone has done something like this successfully (adding an empty password and setting the try-empty-password parameter to make this next boot not require intervention) I am open to giving it a try! My security model is mostly preventing people stealing my drive from the data center not backdoor by the provider so I think this is acceptable to me.

This is the pacman.log for this update ( I removed some python and haskell libraries for brevity):

[2024-11-25T21:24:20-0600] [PACMAN] Running 'pacman -S -y --config /etc/pacman.conf --'

[2024-11-25T21:24:20-0600] [PACMAN] synchronizing package lists

[2024-11-25T21:25:08-0600] [PACMAN] Running 'pacman -S -y -u --config /etc/pacman.conf --'

[2024-11-25T21:25:08-0600] [PACMAN] synchronizing package lists

[2024-11-25T21:25:08-0600] [PACMAN] starting full system upgrade

[2024-11-25T21:25:13-0600] [ALPM] running '60-mkinitcpio-remove.hook'...

[2024-11-25T21:25:14-0600] [ALPM] running 'ghc-unregister.hook'...

[2024-11-25T21:25:15-0600] [ALPM] transaction started

[2024-11-25T21:25:15-0600] [ALPM] upgraded filesystem (2024.04.07-1 -> 2024.11.21-1)

[2024-11-25T21:25:15-0600] [ALPM] upgraded pam (1.6.1-3 -> 1.7.0-1)

[2024-11-25T21:25:15-0600] [ALPM] upgraded ca-certificates-mozilla (3.106-1 -> 3.107-1)

[2024-11-25T21:25:15-0600] [ALPM] upgraded git (2.47.0-1 -> 2.47.1-1)

[2024-11-25T21:25:15-0600] [ALPM] upgraded iproute2 (6.11.0-1 -> 6.12.0-1)

[2024-11-25T21:25:16-0600] [ALPM] upgraded linux (6.11.8.arch1-2 -> 6.12.1.arch1-1)

[2024-11-25T21:25:16-0600] [ALPM] upgraded lsof (4.99.3-2 -> 4.99.4-1)

[2024-11-25T21:25:16-0600] [ALPM] upgraded luajit (2.1.1727870382-1 -> 2.1.1731601260-1)

[2024-11-25T21:25:16-0600] [ALPM] upgraded mariadb-libs (11.6.2-1 -> 11.6.2-2)

[2024-11-25T21:25:16-0600] [ALPM] upgraded qemu-guest-agent (9.1.1-2 -> 9.1.2-1)

[2024-11-25T21:25:16-0600] [ALPM] upgraded runc (1.2.1-1 -> 1.2.2-1)

[2024-11-25T21:25:16-0600] [ALPM] warning: /etc/sudoers installed as /etc/sudoers.pacnew

[2024-11-25T21:25:16-0600] [ALPM] upgraded sudo (1.9.16-1 -> 1.9.16.p1-1)

[2024-11-25T21:25:17-0600] [ALPM] upgraded vault (1.18.1-1 -> 1.18.2-1)

[2024-11-25T21:25:17-0600] [ALPM] transaction completed

[2024-11-25T21:25:17-0600] [ALPM] running '20-systemd-sysusers.hook'...

[2024-11-25T21:25:17-0600] [ALPM] running '30-systemd-daemon-reload-system.hook'...

[2024-11-25T21:25:18-0600] [ALPM] running '30-systemd-sysctl.hook'...

[2024-11-25T21:25:18-0600] [ALPM] running '30-systemd-tmpfiles.hook'...

[2024-11-25T21:25:18-0600] [ALPM] running '30-systemd-udev-reload.hook'...

[2024-11-25T21:25:19-0600] [ALPM] running '30-systemd-update.hook'...

[2024-11-25T21:25:19-0600] [ALPM] running '40-update-ca-trust.hook'...

[2024-11-25T21:25:20-0600] [ALPM] running '60-depmod.hook'...

[2024-11-25T21:25:21-0600] [ALPM] running '90-mkinitcpio-install.hook'...

[2024-11-25T21:25:21-0600] [ALPM-SCRIPTLET] ==> Building image from preset: /etc/mkinitcpio.d/linux.preset: 'default'

[2024-11-25T21:25:21-0600] [ALPM-SCRIPTLET] ==> Using default configuration file: '/etc/mkinitcpio.conf'

[2024-11-25T21:25:21-0600] [ALPM-SCRIPTLET] -> -k /boot/vmlinuz-linux -g /boot/initramfs-linux.img

[2024-11-25T21:25:21-0600] [ALPM-SCRIPTLET] ==> Starting build: '6.12.1-arch1-1'

[2024-11-25T21:25:21-0600] [ALPM-SCRIPTLET] -> Running build hook: [base]

[2024-11-25T21:25:22-0600] [ALPM-SCRIPTLET] -> Running build hook: [systemd]

[2024-11-25T21:25:23-0600] [ALPM-SCRIPTLET] -> Running build hook: [udev]

[2024-11-25T21:25:23-0600] [ALPM-SCRIPTLET] -> Running build hook: [autodetect]

[2024-11-25T21:25:23-0600] [ALPM-SCRIPTLET] -> Running build hook: [microcode]

[2024-11-25T21:25:23-0600] [ALPM-SCRIPTLET] -> Running build hook: [modconf]

[2024-11-25T21:25:23-0600] [ALPM-SCRIPTLET] -> Running build hook: [kms]

[2024-11-25T21:25:24-0600] [ALPM-SCRIPTLET] -> Running build hook: [keyboard]

[2024-11-25T21:25:24-0600] [ALPM-SCRIPTLET] -> Running build hook: [keymap]

[2024-11-25T21:25:24-0600] [ALPM-SCRIPTLET] -> Running build hook: [consolefont]

[2024-11-25T21:25:24-0600] [ALPM-SCRIPTLET] ==> WARNING: consolefont: no font found in configuration

[2024-11-25T21:25:24-0600] [ALPM-SCRIPTLET] -> Running build hook: [sd-encrypt]

[2024-11-25T21:25:25-0600] [ALPM-SCRIPTLET] ==> WARNING: Possibly missing firmware for module: 'qat_420xx'

[2024-11-25T21:25:25-0600] [ALPM-SCRIPTLET] -> Running build hook: [block]

[2024-11-25T21:25:26-0600] [ALPM-SCRIPTLET] -> Running build hook: [filesystems]

[2024-11-25T21:25:26-0600] [ALPM-SCRIPTLET] -> Running build hook: [fsck]

[2024-11-25T21:25:26-0600] [ALPM-SCRIPTLET] ==> Generating module dependencies

[2024-11-25T21:25:26-0600] [ALPM-SCRIPTLET] ==> Creating zstd-compressed initcpio image: '/boot/initramfs-linux.img'

[2024-11-25T21:25:27-0600] [ALPM-SCRIPTLET] -> Early uncompressed CPIO image generation successful

[2024-11-25T21:25:27-0600] [ALPM-SCRIPTLET] ==> Initcpio image generation successful

[2024-11-25T21:25:27-0600] [ALPM-SCRIPTLET] ==> Building image from preset: /etc/mkinitcpio.d/linux.preset: 'fallback'

[2024-11-25T21:25:27-0600] [ALPM-SCRIPTLET] ==> Using default configuration file: '/etc/mkinitcpio.conf'

[2024-11-25T21:25:27-0600] [ALPM-SCRIPTLET] -> -k /boot/vmlinuz-linux -g /boot/initramfs-linux-fallback.img -S autodetect

[2024-11-25T21:25:27-0600] [ALPM-SCRIPTLET] ==> Starting build: '6.12.1-arch1-1'

[2024-11-25T21:25:27-0600] [ALPM-SCRIPTLET] -> Running build hook: [base]

[2024-11-25T21:25:27-0600] [ALPM-SCRIPTLET] -> Running build hook: [systemd]

[2024-11-25T21:25:28-0600] [ALPM-SCRIPTLET] -> Running build hook: [udev]

[2024-11-25T21:25:29-0600] [ALPM-SCRIPTLET] -> Running build hook: [microcode]

[2024-11-25T21:25:29-0600] [ALPM-SCRIPTLET] -> Running build hook: [modconf]

[2024-11-25T21:25:29-0600] [ALPM-SCRIPTLET] -> Running build hook: [kms]

[2024-11-25T21:25:31-0600] [ALPM-SCRIPTLET] ==> WARNING: Possibly missing firmware for module: 'ast'

[2024-11-25T21:25:35-0600] [ALPM-SCRIPTLET] -> Running build hook: [keyboard]

[2024-11-25T21:25:35-0600] [ALPM-SCRIPTLET] ==> WARNING: Possibly missing firmware for module: 'xhci_pci_renesas'

[2024-11-25T21:25:36-0600] [ALPM-SCRIPTLET] -> Running build hook: [keymap]

[2024-11-25T21:25:36-0600] [ALPM-SCRIPTLET] -> Running build hook: [consolefont]

[2024-11-25T21:25:36-0600] [ALPM-SCRIPTLET] ==> WARNING: consolefont: no font found in configuration

[2024-11-25T21:25:36-0600] [ALPM-SCRIPTLET] -> Running build hook: [sd-encrypt]

[2024-11-25T21:25:37-0600] [ALPM-SCRIPTLET] ==> WARNING: Possibly missing firmware for module: 'qat_420xx'

[2024-11-25T21:25:38-0600] [ALPM-SCRIPTLET] -> Running build hook: [block]

[2024-11-25T21:25:38-0600] [ALPM-SCRIPTLET] ==> WARNING: Possibly missing firmware for module: 'aic94xx'

[2024-11-25T21:25:38-0600] [ALPM-SCRIPTLET] ==> WARNING: Possibly missing firmware for module: 'bfa'

[2024-11-25T21:25:39-0600] [ALPM-SCRIPTLET] ==> WARNING: Possibly missing firmware for module: 'qed'

[2024-11-25T21:25:39-0600] [ALPM-SCRIPTLET] ==> WARNING: Possibly missing firmware for module: 'qla1280'

[2024-11-25T21:25:39-0600] [ALPM-SCRIPTLET] ==> WARNING: Possibly missing firmware for module: 'qla2xxx'

[2024-11-25T21:25:39-0600] [ALPM-SCRIPTLET] ==> WARNING: Possibly missing firmware for module: 'wd719x'

[2024-11-25T21:25:41-0600] [ALPM-SCRIPTLET] -> Running build hook: [filesystems]

[2024-11-25T21:25:41-0600] [ALPM-SCRIPTLET] -> Running build hook: [fsck]

[2024-11-25T21:25:44-0600] [ALPM-SCRIPTLET] ==> Generating module dependencies

[2024-11-25T21:25:44-0600] [ALPM-SCRIPTLET] ==> Creating zstd-compressed initcpio image: '/boot/initramfs-linux-fallback.img'

[2024-11-25T21:25:46-0600] [ALPM-SCRIPTLET] -> Early uncompressed CPIO image generation successful

[2024-11-25T21:25:46-0600] [ALPM-SCRIPTLET] ==> Initcpio image generation successful

[2024-11-25T21:25:46-0600] [ALPM] running 'ghc-register.hook'...

[2024-11-25T21:25:53-0600] [PACMAN] Running 'pacman -S --config /etc/pacman.conf -- core/iptables-nft'

[2024-11-25T21:26:00-0600] [ALPM] transaction started

[2024-11-25T21:26:00-0600] [ALPM] removed iptables (1:1.8.10-2)

[2024-11-25T21:26:00-0600] [ALPM] installed iptables-nft (1:1.8.10-2)

[2024-11-25T21:26:00-0600] [ALPM] transaction completed

[2024-11-25T21:26:00-0600] [ALPM] running '30-systemd-daemon-reload-system.hook'...

[2024-11-25T21:26:00-0600] [ALPM] running '30-systemd-update.hook'...

[2024-11-25T21:26:00-0600] [PACMAN] Running 'pacman -D -q --asexplicit --config /etc/pacman.conf -- iptables-nft'

Last edited by eternal_flame-AD (2024-11-28 06:58:01)

V1del · 2024-11-27 23:38:14

You can definitely do that, to generally avoid such a situation consider installing kernel-module-hooks or so.

eternal_flame-AD · 2024-11-28 06:57:50

Thanks for the reassurance, I have reinstalled the old kernel and was able to restore functionality!

Arch Linux

#1 2024-11-27 23:09:11

[solved] Safety of rolling back a kernel update before rebooting

#2 2024-11-27 23:38:14

Re: [solved] Safety of rolling back a kernel update before rebooting

#3 2024-11-28 06:57:50

Re: [solved] Safety of rolling back a kernel update before rebooting

Board footer