[Frustrated] mkinitcpio was changed without notice to system admins

ReaperX7 · 2026-06-15 19:54:35

I only figured this out after 4 hours of needless debugging a failed install of ArchLinux on ZFS and trying to figure out why the boot process was stopping at sysroot and leaving me without a recovery console.

Traditionally, the mkinitcpio has used BusyBox which loads beforehand and then passes the system off to systemd after loading the kernel and drivers, unlocking the file systems, and other things before the main init system begins the remainder of the boot process.

Recently, someone changed the BusyBox hooks to systemd hooks without warning users and system administrators alike of the change, which resulted in broken boot cycles, and hours of needless debugging of processes to discover this oversight.

This change needs to be made public on the news section so system administrators understand why a sysroot lockout has occurred since systemd doesn't allow for a recovery console if the mkinitcpio fails to initialize like BusyBox does.

I understand systemd is the default init system of ArchLinux, but this is the initramfs where if something goes wrong, system administrators need tools to start the recovery of a boot process, and if a tool is not allowing a system administrator to do their job, then that tool should not be the default.

SimonJ · 2026-06-15 19:58:58

Would this not have been the change you mentioned?

https://archlinux.org/news/mkinitcpio-h … microcode/

dimich · 2026-06-15 20:13:31

ReaperX7 wrote:

Recently, someone changed the BusyBox hooks to systemd hooks without warning users and system administrators alike of the change

Could you please clarify what change do you mean? I use busybox based initramfs and didn't encounter any issues yet, but want to be aware.

ReaperX7 wrote:

if something goes wrong, system administrators need tools to start the recovery of a boot process

Boot with live Arch ISO?

seth · 2026-06-15 20:27:10

An untouched mkinitcpio.conf would have been updated/migrated to the new defaults but if you've customized it you get a .pacnew you can ignore.
Makes me wonder how that could not have been the case on a ZFS system

@SimonJ the mentioned change was in late 2025 (october or so)

ReaperX7 · 2026-06-15 20:36:55

SimonJ wrote:

Would this not have been the change you mentioned?
https://archlinux.org/news/mkinitcpio-h … microcode/

The hooks were mentioned, but it was never clearly stated what the defaults were changed to. Adding hooks means only what is available to the system administrator to use.

Traditionally, due to limitations with systemd inside the initramfs, udev was used for device initializations, not systemd. While the systemd specific hooks are available as mentioned in the mkinitcpio page of the wiki, it was never implied the BusyBox hooks would not be the default any longer. If systemd failed during the initramfs initialization, it left the user without a recover console. If BusyBox failed, it launched a recovery console, and allowed the system administrator to fix the boot process inside the initramfs and complete the boot process, and then fix the initramfs post-boot without having to rely on an external boot disk.

Whoever made this change performed a huge oversight with deep reaching consequences for system administrators, especially newly installed systems.

For the TL;DR people, changing the initramfs device manager hook, broke the system and disabled the recovery console, and nobody was thoroughly warned of this default change.

ReaperX7 · 2026-06-15 20:42:43

seth wrote:

An untouched mkinitcpio.conf would have been updated/migrated to the new defaults but if you've customized it you get a .pacnew you can ignore.
Makes me wonder how that could not have been the case on a ZFS system
@SimonJ the mentioned change was in late 2025 (october or so)

It shouldn't have been changed to begin with. There was no need to switch the device manager to systemd from udev and busybox. This was a process that lasted less a second and was negligible to mess with. There is no net-positive gain for this change.

Even now it still breaks the recovery console in the initramfs. You don't break tools needed by system administrators just to shave off a millisecond of negligible speed without showing negligence to the system.

seth · 2026-06-15 21:14:56

https://gitlab.archlinux.org/archlinux/ … =heads#L48
idk whether boot-speed was any concern here but rather doubt it.
Does your ZFS system really not require any local changes to the mkinitcpio?

Speaking of what "should" be: https://terms.archlinux.org/docs/code-o … opicsrants

ReaperX7 · 2026-06-15 21:26:59

Taken from the wiki:

https://wiki.archlinux.org/title/Mkinit … leshooting

The systemd hook disables the root account. To enable the emergency shell, temporarily add SYSTEMD_SULOGIN_FORCE=1 to the kernel parameters.

The root account is needed to diagnose the boot process and forcing reliance upon an optional flag at the kernel command line, which you may not have access to outside of specific bootloaders like Grub which allow kernel command line editing prior to boot, is not good practice in any regards. Busybox, by default, allow for an emergency shell.

Shaving down the

HOOKS=(base udev usr resume autodetect microcode modconf kms keyboard keymap consolefont block filesystems fsck)

to be

HOOKS=(base systemd autodetect modconf kms keyboard sd-vconsole block filesystems fsck)

Saved what? A few bytes of data space and made up like 0.25 seconds of boot time, while breaking the console to perform root recovery. That was smart...

ReaperX7 · 2026-06-15 21:32:15

seth wrote:

https://gitlab.archlinux.org/archlinux/ … =heads#L48
idk whether boot-speed was any concern here but rather doubt it.
Does your ZFS system really not require any local changes to the mkinitcpio?
Speaking of what "should" be: https://terms.archlinux.org/docs/code-o … opicsrants

Yes, it uses the "zfs" hook which also specifically requires "udev" because due to zfs-utils. We dropped, on the wiki, using sd-zfs due to it not being part of the core project and was from an untrustworthy source that required an entire extra repository to introduce and broke filesystem encryption. While sd-zfs was available, it was not endorsed or published by archzfs in their repository specifically.

Last edited by ReaperX7 (2026-06-15 21:35:02)

seth · 2026-06-15 22:06:55

So… how did you end up in this situation? Unsupervised pacdiff merge?

WorMzy · 2026-06-15 22:32:18

Mod note: moving to AUR Issues.

ReaperX7 · 2026-06-15 23:01:07

seth wrote:

So… how did you end up in this situation? Unsupervised pacdiff merge?

I do rebuilds of ArchLinux with ZFS root every 6 months to ensure nothing is broken and the wiki page isn't neglected where someone with a pet project comes in and makes a mess just to do it their way and not the way the package is intended to be used by the developers. I don't just ride out the same system like my main system which gets updates but not on the same track.

I rebuild from scratch to ensure new users have a seamless experience going through the install process. I remembered the hooks update, but wasn't able to check around January due to my work schedule, and because I'm the only current maintainer of the ZFS as Root wiki page, things didn't go as planned till now when I had time to actually do a test build. Since there was no mentioning of changes to the defaults, everything was to go as is. There are some contributions to offshoots like zfsbootmenu and others which have been neglected for some time, but not on the same level as the core how-to-guide being maintained.

Until last night when I spent 4 hours trying to figure out what went wrong when the system locked at at sysroot mount, and then remembered last minute, "Why is the systemd hook being used and not the udev BusyBox hook? The systemd hook is broken for the recovery console."

That's how. Me spending an hour auditing my own wiki contribution only to end up spending 4 more hours to diagnose the problem which shouldn't have even been a problem to begin with.

Last edited by ReaperX7 (2026-06-15 23:05:05)

Lone_Wolf · 2026-06-16 09:07:22

Maybe section 3.6 of the installation guide could add ZFS as a setup needing modifying of mkinitcpio ?

seth · 2026-06-16 15:45:35

https://wiki.archlinux.org/index.php?ti … did=879365
Not sure how reasonable it would be to drop a random zfs comment into the general installation guide (notably since you've to read the linked wiki anyway)

The subject of the thread is rather that the OP very much would have preferred a heads-up for the changed default.
Which is understandable but given theirs is also a rather exceptional case.

cryptearth · 2026-06-16 18:09:32

Lone_Wolf wrote:

Maybe section 3.6 of the installation guide could add ZFS as a setup needing modifying of mkinitcpio ?

as someone using (and supporting) zfs myself: if, at all, such information should be placed rather on some zfs page - as root on <any raid> isn't a default setup in the first place - root on zfs is yet another step up because it already requires special handling of the boot stuff

aside from that ...

@OP
although i do appreciate the effort - my point on "root on raid" is: just DON'T! - for a simple reason: consumer-grade hardware just can'T do it anyway! consumer grade hardware always boots from ONE drive - without a modified bios/uefi to support raid within it (like amd fakeraid or intel rst) one always ends up to replicate a bootloader across all drives part of the array - and one has to either have some luck or take specific actions so the systems firmware can silently fail-over to the next drive if one fails - which in itself is already something most firmwares would just hang until a drive fails completely

plus: what seth said: any arch user is supposed to pay attention on the output of every update - if you had you had read that mkinitcpio threw a line like "blah blah blah ... mkinitcpio ... .pacsave ..." which signals the user: "hey, you need to pay extra manual actions here"

TL;DR: running root on <insert your fav type of raid here> and ending up with a system not booting is somewhat user error - if, for some reason, you think you need to run your os on some fault tolerant setup - you most likely actualy don't and by this you do something wrong
just setup a cheap 200gb sata ssd as your os drive ... and don't make the mistake to mount your array at /home - it will cause you about the same issues

seth · 2026-06-16 19:45:43

any arch user is supposed to pay attention on the output of every update

The OP ran into this w/ a fresh installation - updates should™ not cause this at all because the mkinitcpio.conf had been altered and would not be replaced and one *probably* would notice the init system change in pacdiff (or not run pacdiff at all)
The OPs is a rather special case in that regard, though it also led to the ZFS intallation wiki stressing that you cannot use the systemd hooks, so there's actually some value come of of their frustration.

Foxboron · 2026-06-16 21:12:20

This change was announce on `arch-dev-public` and there was no announcement as there is not going to be anything that would prevent people from updating their systems.

https://lists.archlinux.org/archives/li … T6F4RUPBJ/

If you run exotic setups, like zfs on Arch *and* edit the archwiki, I would completely expect someone to follow updates on a-d-p.

Arch Linux

#1 2026-06-15 19:54:35

[Frustrated] mkinitcpio was changed without notice to system admins

#2 2026-06-15 19:58:58

Re: [Frustrated] mkinitcpio was changed without notice to system admins

#3 2026-06-15 20:13:31

Re: [Frustrated] mkinitcpio was changed without notice to system admins

#4 2026-06-15 20:27:10

Re: [Frustrated] mkinitcpio was changed without notice to system admins

#5 2026-06-15 20:36:55

Re: [Frustrated] mkinitcpio was changed without notice to system admins

#6 2026-06-15 20:42:43

Re: [Frustrated] mkinitcpio was changed without notice to system admins

#7 2026-06-15 21:14:56

Re: [Frustrated] mkinitcpio was changed without notice to system admins

#8 2026-06-15 21:26:59

Re: [Frustrated] mkinitcpio was changed without notice to system admins

#9 2026-06-15 21:32:15

Re: [Frustrated] mkinitcpio was changed without notice to system admins

#10 2026-06-15 22:06:55

Re: [Frustrated] mkinitcpio was changed without notice to system admins

#11 2026-06-15 22:32:18

Re: [Frustrated] mkinitcpio was changed without notice to system admins

#12 2026-06-15 23:01:07

Re: [Frustrated] mkinitcpio was changed without notice to system admins

#13 2026-06-16 09:07:22

Re: [Frustrated] mkinitcpio was changed without notice to system admins

#14 2026-06-16 15:45:35

Re: [Frustrated] mkinitcpio was changed without notice to system admins

#15 2026-06-16 18:09:32

Re: [Frustrated] mkinitcpio was changed without notice to system admins

#16 2026-06-16 19:45:43

Re: [Frustrated] mkinitcpio was changed without notice to system admins

#17 2026-06-16 21:12:20

Re: [Frustrated] mkinitcpio was changed without notice to system admins

Board footer