You are not logged in.
ZFS native encrypted root, systemd-boot, and zfs-dkms. Any update that requires a rebuild of the ramdisk images / call to mkinitcpio causes a initrd failure on next boot.
I can boot with arch iso, mount partitions/datasets, arch-chroot in and run mkinitcpio -P, unmount and export pool then reboot and all is well again.
This can be reporduced by manually running 'mkinitcpio -P' from the running system which will then require a recovery on next boot.
I have this configuration on two systems (desktop & laptop) and am seeing the exact same issue on both. All pacman and mkinitcpio logs look good without any warnings or errors and are identical between arch-chroot and the normal running system. It must have somthing to do with the zfs hook but with no apparent difference in logs after running inside and out of chroot I'm at a loss as to where to go from here.
I can post any logs or configs that might be helpful, including the last pacman update that caused a failed boot below.
[2023-08-17T20:56:17-0500] [PACMAN] Running 'pacman -Syu'
[2023-08-17T20:56:17-0500] [PACMAN] synchronizing package lists
[2023-08-17T20:56:18-0500] [PACMAN] starting full system upgrade
[2023-08-17T20:56:21-0500] [ALPM] transaction started
[2023-08-17T20:56:22-0500] [ALPM] upgraded util-linux-libs (2.39.1-1 -> 2.39.2-1)
[2023-08-17T20:56:22-0500] [ALPM] upgraded device-mapper (2.03.22-1 -> 2.03.22-2)
[2023-08-17T20:56:22-0500] [ALPM] upgraded kbd (2.6.1-1 -> 2.6.2-1)
[2023-08-17T20:56:22-0500] [ALPM] upgraded util-linux (2.39.1-1 -> 2.39.2-1)
[2023-08-17T20:56:22-0500] [ALPM] transaction completed
[2023-08-17T20:56:22-0500] [ALPM] running '20-systemd-sysusers.hook'...
[2023-08-17T20:56:22-0500] [ALPM] running '30-systemd-daemon-reload.hook'...
[2023-08-17T20:56:23-0500] [ALPM] running '30-systemd-udev-reload.hook'...
[2023-08-17T20:56:26-0500] [ALPM] running '30-systemd-update.hook'...
[2023-08-17T20:56:26-0500] [ALPM] running '90-mkinitcpio-install.hook'...
[2023-08-17T20:56:27-0500] [ALPM-SCRIPTLET] ==> Building image from preset: /etc/mkinitcpio.d/linux.preset: 'default'
[2023-08-17T20:56:27-0500] [ALPM-SCRIPTLET] ==> Using default configuration file: '/etc/mkinitcpio.conf'
[2023-08-17T20:56:27-0500] [ALPM-SCRIPTLET] -> -k /boot/vmlinuz-linux -g /boot/initramfs-linux.img --microcode /boot/intel-ucode.img
[2023-08-17T20:56:27-0500] [ALPM-SCRIPTLET] ==> Starting build: '6.4.10-arch1-1'
[2023-08-17T20:56:27-0500] [ALPM-SCRIPTLET] -> Running build hook: [base]
[2023-08-17T20:56:28-0500] [ALPM-SCRIPTLET] -> Running build hook: [udev]
[2023-08-17T20:56:29-0500] [ALPM-SCRIPTLET] -> Running build hook: [autodetect]
[2023-08-17T20:56:29-0500] [ALPM-SCRIPTLET] -> Running build hook: [modconf]
[2023-08-17T20:56:29-0500] [ALPM-SCRIPTLET] -> Running build hook: [kms]
[2023-08-17T20:56:34-0500] [ALPM-SCRIPTLET] -> Running build hook: [keyboard]
[2023-08-17T20:56:34-0500] [ALPM-SCRIPTLET] ==> WARNING: Possibly missing firmware for module: 'xhci_pci'
[2023-08-17T20:56:34-0500] [ALPM-SCRIPTLET] -> Running build hook: [keymap]
[2023-08-17T20:56:34-0500] [ALPM-SCRIPTLET] -> Running build hook: [consolefont]
[2023-08-17T20:56:34-0500] [ALPM-SCRIPTLET] ==> WARNING: consolefont: no font found in configuration
[2023-08-17T20:56:34-0500] [ALPM-SCRIPTLET] -> Running build hook: [block]
[2023-08-17T20:56:37-0500] [ALPM-SCRIPTLET] -> Running build hook: [zfs]
[2023-08-17T20:56:38-0500] [ALPM-SCRIPTLET] -> Running build hook: [filesystems]
[2023-08-17T20:56:38-0500] [ALPM-SCRIPTLET] ==> Generating module dependencies
[2023-08-17T20:56:38-0500] [ALPM-SCRIPTLET] ==> Creating zstd-compressed initcpio image: '/boot/initramfs-linux.img'
[2023-08-17T20:56:39-0500] [ALPM-SCRIPTLET] ==> Image generation successful
[2023-08-17T20:56:39-0500] [ALPM-SCRIPTLET] ==> Building image from preset: /etc/mkinitcpio.d/linux.preset: 'fallback'
[2023-08-17T20:56:39-0500] [ALPM-SCRIPTLET] ==> Using default configuration file: '/etc/mkinitcpio.conf'
[2023-08-17T20:56:39-0500] [ALPM-SCRIPTLET] -> -k /boot/vmlinuz-linux -g /boot/initramfs-linux-fallback.img -S autodetect --microcode /boot/intel-ucode.img
[2023-08-17T20:56:39-0500] [ALPM-SCRIPTLET] ==> Starting build: '6.4.10-arch1-1'
[2023-08-17T20:56:40-0500] [ALPM-SCRIPTLET] -> Running build hook: [base]
[2023-08-17T20:56:40-0500] [ALPM-SCRIPTLET] -> Running build hook: [udev]
[2023-08-17T20:56:41-0500] [ALPM-SCRIPTLET] -> Running build hook: [modconf]
[2023-08-17T20:56:41-0500] [ALPM-SCRIPTLET] -> Running build hook: [kms]
[2023-08-17T20:56:46-0500] [ALPM-SCRIPTLET] ==> WARNING: Possibly missing firmware for module: 'ast'
[2023-08-17T20:57:02-0500] [ALPM-SCRIPTLET] -> Running build hook: [keyboard]
[2023-08-17T20:57:03-0500] [ALPM-SCRIPTLET] ==> WARNING: Possibly missing firmware for module: 'xhci_pci'
[2023-08-17T20:57:05-0500] [ALPM-SCRIPTLET] -> Running build hook: [keymap]
[2023-08-17T20:57:06-0500] [ALPM-SCRIPTLET] -> Running build hook: [consolefont]
[2023-08-17T20:57:06-0500] [ALPM-SCRIPTLET] ==> WARNING: consolefont: no font found in configuration
[2023-08-17T20:57:06-0500] [ALPM-SCRIPTLET] -> Running build hook: [block]
[2023-08-17T20:57:06-0500] [ALPM-SCRIPTLET] ==> WARNING: Possibly missing firmware for module: 'aic94xx'
[2023-08-17T20:57:07-0500] [ALPM-SCRIPTLET] ==> WARNING: Possibly missing firmware for module: 'qla2xxx'
[2023-08-17T20:57:07-0500] [ALPM-SCRIPTLET] ==> WARNING: Possibly missing firmware for module: 'qed'
[2023-08-17T20:57:08-0500] [ALPM-SCRIPTLET] ==> WARNING: Possibly missing firmware for module: 'bfa'
[2023-08-17T20:57:08-0500] [ALPM-SCRIPTLET] ==> WARNING: Possibly missing firmware for module: 'qla1280'
[2023-08-17T20:57:09-0500] [ALPM-SCRIPTLET] ==> WARNING: Possibly missing firmware for module: 'wd719x'
[2023-08-17T20:57:15-0500] [ALPM-SCRIPTLET] -> Running build hook: [zfs]
[2023-08-17T20:57:16-0500] [ALPM-SCRIPTLET] -> Running build hook: [filesystems]
[2023-08-17T20:57:22-0500] [ALPM-SCRIPTLET] ==> Generating module dependencies
[2023-08-17T20:57:22-0500] [ALPM-SCRIPTLET] ==> Creating zstd-compressed initcpio image: '/boot/initramfs-linux-fallback.img'
[2023-08-17T20:57:25-0500] [ALPM-SCRIPTLET] ==> Image generation successful
System failed to boot after this but re-running 'mkinitcpio -P' inside arch-chroot, while producing the same result as above, seems to fix whatever is broke and all is well again.
I ran across some posts about a systemd udev race bug of some kind or other that was causing some boot failures with other configurations but I haven't had time to dig into it yet. Ref: https://bugs.archlinux.org/task/77789 and https://gitlab.archlinux.org/archlinux/ … issues/177
Any help appreciated!
Update:
Went for today's kernel update and it left me unable to boot no matter what I tried so I decided to eliminate dkms and current kernel from the equation.
I'm now on the zfs-linux-lts kernel and back in the same spot I was before. Manually running 'mkinitcpio -P' or anything that triggers it while inside the running system leaves me with the same initrd error on the next boot. Again, booting into recovery and running 'mkinitcpio -P' in arch-chroot gets me back to a running system with no issues but if mkinitcpio is envoked again from the running system for any reason I'm back to a failed boot with an initrd failure.
Have a few hours to burn tonight so I'm digging in.....
Update 2:
I don't know that I fixed anything but after switching to zfs-utils-git & zfs-dkms-git and updating to latest kernel agian I have now sucessfully updated both systems without issue... so far. Leaving this up in case someone else experiences somthing similar and could maybe share some insite.
Last edited by knujeo (2023-08-19 16:45:45)
Offline