Arch Linux

Jaxon

Member

Registered: 2018-06-05

Posts: 6

[SOLVED] mkinitcpio sometimes creates unbootable initramfs on upgrade

I've found a few topics about this that might be the same issue, but as they don't mention a specific detail I'm creating a new topic.

Sometimes when an upgrade triggers an initramfs rebuild, the resulting image will fail to boot, and put me in a recovery shell without keyboard response. I am unable to reproduce this, and have already overwritten the faulty initramfs so I cannot provide the exact output I see on screen, but it complains about not finding the root filesystem on the partition with the correct UUID. The line has some corruption at the start of the UUID, which starts with a backslash followed by what might be the glyph for the character code point given by the first few digits of the UUID. I should specify that my root partition is on a NVMe drive.

The issue has occurred a couple of times, and each time I've gone back and looked at the pacman logs to find a discrepancy in the mkinitcpio output compared to what I usually see. Specifically the lines beginning with 'sort' and 'modprobe'. Here's the part of /var/log/pacman.log (timestamp and ALPM/ALPM-SCRIPTLET prefixes removed):

running '90-mkinitcpio-install.hook'...
==> Building image from preset: /etc/mkinitcpio.d/linux.preset: 'default'
  -> -k /boot/vmlinuz-linux -c /etc/mkinitcpio.conf -g /boot/initramfs-linux.img
==> Starting build: '6.2.13-arch1-1'
  -> Running build hook: [base]
  -> Running build hook: [systemd]
  -> Running build hook: [autodetect]
sort: cannot read: '/sys/devices/pci0000:00/0000:00:08.1/0000:0c:00.3/usb5/5-4/5-4.2/5-4.2:1.1/ep_01/uevent': No such file or directory
modprobe: ERROR: missing parameters. See -h.
  -> Running build hook: [modconf]
  -> Running build hook: [kms]
  -> Running build hook: [keyboard]
  -> Running build hook: [sd-vconsole]
  -> Running build hook: [block]
  -> Running build hook: [filesystems]
  -> Running build hook: [fsck]
==> Generating module dependencies
==> Creating zstd-compressed initcpio image: '/boot/initramfs-linux.img'
==> Image generation successful
==> Building image from preset: /etc/mkinitcpio.d/linux.preset: 'fallback'
  -> -k /boot/vmlinuz-linux -c /etc/mkinitcpio.conf -g /boot/initramfs-linux-fallback.img -S autodetect
==> Starting build: '6.2.13-arch1-1'
  -> Running build hook: [base]
  -> Running build hook: [systemd]
  -> Running build hook: [modconf]
  -> Running build hook: [kms]
==> WARNING: Possibly missing firmware for module: 'ast'
  -> Running build hook: [keyboard]
==> WARNING: Possibly missing firmware for module: 'xhci_pci'
  -> Running build hook: [sd-vconsole]
  -> Running build hook: [block]
==> WARNING: Possibly missing firmware for module: 'aic94xx'
==> WARNING: Possibly missing firmware for module: 'qla1280'
==> WARNING: Possibly missing firmware for module: 'qed'
==> WARNING: Possibly missing firmware for module: 'wd719x'
==> WARNING: Possibly missing firmware for module: 'qla2xxx'
==> WARNING: Possibly missing firmware for module: 'bfa'
  -> Running build hook: [filesystems]
  -> Running build hook: [fsck]
==> Generating module dependencies
==> Creating zstd-compressed initcpio image: '/boot/initramfs-linux-fallback.img'
==> Image generation successful

Re-running mkinitcpio has so far (n > 10) not resulted in a faulty initramfs, and the fallback image has worked every time the default one hasn't.

Any ideas as to what could be the issue?

Last edited by Jaxon (2023-05-03 21:07:41)

V1del

Forum Moderator

Registered: 2012-10-16

Posts: 23,252

Re: [SOLVED] mkinitcpio sometimes creates unbootable initramfs on upgrade

Yes, the udev trigger call in /usr/share/libalpm/scripts/systemd-hook which get's triggered by systemd updates. While the settle command that got introduced in an attempt to fix this in 253.3-3 should™ fix this it apparently has corner cases where it doesn't.

What logically happens here is that will trigger a logical reload of every single piece of hardware on your system. and then hopefully wait for that to finish, to then hopefully not enter a race with mkinitcpio. You could make the race more guaranteed with some hard sleep for like 5-10 secs or so, but I still consider the reasoning for doing this in the first place quite murky. it fixed a one time regression for an userspace tool from systemd and has since caused various amounts of headaches for a variety of different usecases...

See https://bugs.archlinux.org/task/77789 and https://gitlab.archlinux.org/archlinux/ … issues/177 -- my personal workaround is to simply  add the file to NoUpgrade = directive in pacman.conf and commenting the udev trigger and udev settle lines in the offending file.

Last edited by V1del (2023-05-03 17:27:06)

Jaxon

Member

Registered: 2018-06-05

Posts: 6

Re: [SOLVED] mkinitcpio sometimes creates unbootable initramfs on upgrade

Thanks for the reply! That sounds like quite the annoying issue. Hopefully the race will be resolved properly soon.