Arch Linux

marcus

Member

Registered: 2012-09-26

Posts: 34

[solved] LVM on LUKS on MD-RAID: systemd-based initiramfs fails

Hi forum,

I am using LVM on LUKS on MD-RAID on one of my machines. The LUKS volume is decrypted with a key file on a USB pen drive using the sd-encrypt hook in a systemd-based initramfs. mkinitcpio.conf:

MODULES="ext4 raid10 nvidia nvidia_modeset nvidia_uvm nvidia_drm"
BINARIES=(mdmon)
FILES=""
HOOKS=(systemd keyboard autodetect microcode kms sd-vconsole sd-encrypt modconf block mdadm_udev lvm2 filesystems fsck)
COMPRESSION="lz4"
COMPRESSION_OPTIONS="-9"

after upgrading, the array does not assemble anymore:

linux-lts (6.12.75-1 -> 6.18.16-1)
mdadm (4.5-1 -> 4.5-2)

The previous versions worked fine.

In order to debug the issue, I have added the sd-debug hook from https://aur.archlinux.org/packages/mkin … emd-extras to my initramfs. Change in mkinitcpio.conf:

HOOKS=(systemd keyboard autodetect microcode kms sd-vconsole sd-encrypt modconf block mdadm_udev lvm2 filesystems fsck sd-debug)

To my surprise, the system booted normally afterwards.

The hook basically adds busybox and a root shell on tty2 during boot. Comparing the broken and working initramfs with lsinitcpio shows the following difference:

etc/pam.conf
etc/systemd/
etc/systemd/system/
etc/systemd/system/sysinit.target.wants/
etc/systemd/system/sysinit.target.wants/init-debug.service
usr/bin/agetty
usr/bin/arch
usr/bin/ascii
usr/bin/ash
usr/bin/awk
usr/bin/base32
usr/bin/base64
usr/bin/basename
usr/bin/bc
usr/bin/blkdiscard
usr/bin/busybox
usr/bin/bzip2
usr/bin/cat
usr/bin/chgrp
usr/bin/chmod
usr/bin/chown
usr/bin/chroot
usr/bin/clear
usr/bin/cp
usr/bin/cpio
usr/bin/crc32
usr/bin/cttyhack
usr/bin/cut
usr/bin/dd
usr/bin/df
usr/bin/dirname
usr/bin/dmesg
usr/bin/du
usr/bin/echo
usr/bin/env
usr/bin/expr
usr/bin/fallocate
usr/bin/false
usr/bin/fatattr
usr/bin/free
usr/bin/fsfreeze
usr/bin/fstrim
usr/bin/fsync
usr/bin/getopt
usr/bin/grep
usr/bin/gzip
usr/bin/halt
usr/bin/head
usr/bin/hexdump
usr/bin/hexedit
usr/bin/i2ctransfer
usr/bin/ifconfig
usr/bin/init
usr/bin/install
usr/bin/ip
usr/bin/ipaddr
usr/bin/iplink
usr/bin/ipneigh
usr/bin/iproute
usr/bin/iprule
usr/bin/iptunnel
usr/bin/kbd_mode
usr/bin/kill
usr/bin/killall
usr/bin/less
usr/bin/link
usr/bin/ln
usr/bin/loadfont
usr/bin/loadkmap
usr/bin/login
usr/bin/losetup
usr/bin/ls
usr/bin/lsscsi
usr/bin/lzop
usr/bin/md5sum
usr/bin/mim
usr/bin/mkdir
usr/bin/mkfifo
usr/bin/mknod
usr/bin/mkpasswd
usr/bin/mktemp
usr/bin/mountpoint
usr/bin/mv
usr/bin/nc
usr/bin/netstat
usr/bin/nologin
usr/bin/nproc
usr/bin/nsenter
usr/bin/nslookup
usr/bin/nuke
usr/bin/openvt
usr/bin/partprobe
usr/bin/paste
usr/bin/pgrep
usr/bin/pidof
usr/bin/ping
usr/bin/ping6
usr/bin/poweroff
usr/bin/printf
usr/bin/ps
usr/bin/pwd
usr/bin/readlink
usr/bin/realpath
usr/bin/reboot
usr/bin/resume
usr/bin/rm
usr/bin/rmdir
usr/bin/route
usr/bin/run-init
usr/bin/sed
usr/bin/seedrng
usr/bin/seq
usr/bin/setfattr
usr/bin/sh
usr/bin/sha1sum
usr/bin/sha256sum
usr/bin/sha512sum
usr/bin/shuf
usr/bin/sleep
usr/bin/sort
usr/bin/stat
usr/bin/strings
usr/bin/sync
usr/bin/tac
usr/bin/tail
usr/bin/tar
usr/bin/tcpsvd
usr/bin/tee
usr/bin/telnet
usr/bin/test
usr/bin/tftp
usr/bin/touch
usr/bin/tree
usr/bin/true
usr/bin/truncate
usr/bin/ts
usr/bin/tsort
usr/bin/udhcpc
usr/bin/udhcpc6
usr/bin/uname
usr/bin/uniq
usr/bin/unix_chkpwd
usr/bin/unlink
usr/bin/unshare
usr/bin/unzip
usr/bin/uptime
usr/bin/vi
usr/bin/wc
usr/bin/wget
usr/bin/which
usr/bin/xxd
usr/bin/xz
usr/bin/yes
usr/lib/libgssapi_krb5.so.2
usr/lib/libgssapi_krb5.so.2.2
usr/lib/libk5crypto.so.3
usr/lib/libk5crypto.so.3.1
usr/lib/libkeyutils.so.1
usr/lib/libkeyutils.so.1.10
usr/lib/libkrb5.so.3
usr/lib/libkrb5.so.3.3
usr/lib/libkrb5support.so.0
usr/lib/libkrb5support.so.0.1
usr/lib/libnsl.so.3
usr/lib/libnsl.so.3.0.0
usr/lib/libpam_misc.so.0
usr/lib/libpam_misc.so.0.82.1
usr/lib/libresolv.so.2
usr/lib/libtirpc.so.3
usr/lib/libtirpc.so.3.0.0
usr/lib/security/
usr/lib/security/pam_unix.so
usr/lib/systemd/system/init-debug.service

So I guess, some binary or lib called from a udev hook is missing?

$ for l in $(cat missing.txt ); do grep "$l" /usr/lib/udev/rules.d/*.rules; done
/usr/lib/udev/rules.d/63-md-raid-arrays.rules:41:ENV{MD_MON_THIS}=="?*", TEST=="/etc/initrd-release", PROGRAM="/usr/bin/basename $env{MD_MON_THIS}", ENV{SYSTEMD_WANTS}+="mdmon@initrd-%c.service"
/usr/lib/udev/rules.d/63-md-raid-arrays.rules:42:ENV{MD_MON_THIS}=="?*", TEST!="/etc/initrd-release", PROGRAM="/usr/bin/basename $env{MD_MON_THIS}", ENV{SYSTEMD_WANTS}+="mdmon@%c.service"
/usr/lib/udev/rules.d/63-md-raid-arrays.rules:43:ENV{MD_RESHAPE_ACTIVE}=="True", PROGRAM="/usr/bin/basename $env{MD_MON_THIS}", ENV{SYSTEMD_WANTS}+="mdadm-grow-continue@%c.service"
/usr/lib/udev/rules.d/71-seat.rules:78:SUBSYSTEM=="input", ATTR{name}=="Wiebetech LLC Wiebetech", RUN+="/usr/bin/loginctl lock-sessions"
/usr/lib/udev/rules.d/63-md-raid-arrays.rules:40:ENV{MD_LEVEL}=="raid[1-9]*", ENV{MD_CONTAINER}=="?*", PROGRAM="/usr/bin/readlink $env{MD_CONTAINER}", ENV{MD_MON_THIS}="%c"
/usr/lib/udev/rules.d/56-multipath.rules:5:     RUN+="/usr/bin/rm -f /run/multipath/find_multipaths/$major:$minor"

So I tried adding the matches to my mkinitcpio.conf:

BINARIES=(mdmon basename readlink rm)

Unfortunately this did not solve the issue. Any other suggestions?

THX!

Last edited by marcus (2026-03-09 12:59:15)

frostschutz

Member

Registered: 2013-11-15

Posts: 1,636

Re: [solved] LVM on LUKS on MD-RAID: systemd-based initiramfs fails

To my surprise, the system booted normally afterwards.

Perhaps it's simply because you rebuilt the initcpio. The mdadm update by itself does not trigger that unfortunately. Usually it does not matter since even an older version of mdadm will assemble arrays just fine, but with buggy mdadm it's a different story.

If you still have the broken initrd, you could extract the mdadm binary from it and check its version. Unfortunately they both print the same `mdadm --version` string. But they have different checksums, md5sum:

$ md5sum mdadm*/usr/bin/mdadm
8222d28648fe761e6dfec6e368867d06  mdadm-4.5-1-x86_64.pkg.tar.zst.dir/usr/bin/mdadm
034596c7d22637913c773020a9312325  mdadm-4.5-2-x86_64.pkg.tar.zst.dir/usr/bin/mdadm

If it still breaks after reverting your changes to mkinitcpio.conf, then you have a different problem...

You could deliberately revert mdadm to 4.4 from the archlinux package archive (or your package cache if it's around) and see if it works with old mdadm. https://archive.archlinux.org/packages/m/mdadm/

Is there anything weird in your mdadm.conf?

Last edited by frostschutz (2026-03-09 10:09:44)

marcus

Member

Registered: 2012-09-26

Posts: 34

Re: [solved] LVM on LUKS on MD-RAID: systemd-based initiramfs fails

Thanks for your reply @frostschutz. Unfortunately that was not the issue. I checked the following:

$ rm /boot/initramfs-linux-lts.img /boot/EFI/Linux/archlinux-linux-lts-*
# mkinitcpio.conf without sd-debug
$ mkinitcpio -p linux-lts
$ mv /boot/EFI/Linux/archlinux-linux-lts.efi /boot/EFI/Linux/archlinux-linux-lts-broken.efi
# mkinitcpio.conf with sd-debug
$ mkinitcpio -p linux-lts
$ mv /boot/EFI/Linux/archlinux-linux-lts.efi /boot/EFI/Linux/archlinux-linux-lts-debug.efi

The debug UKI boots, the other one does not.

$ mkdir broken debug
$ cd broken
$ lsinitcpio --extract /boot/EFI/Linux/archlinux-linux-lts-broken.efi
$ cd ../debug
$ lsinitcpio --extract /boot/EFI/Linux/archlinux-linux-lts-debug.efi
$ cd ..
$ md5sum broken/usr/bin/mdadm debug/usr/bin/mdadm
034596c7d22637913c773020a9312325  broken/usr/bin/mdadm
034596c7d22637913c773020a9312325  debug/usr/bin/mdadm

Both mdadm binaries match 4.5-2.

marcus

Member

Registered: 2012-09-26

Posts: 34

Re: [solved] LVM on LUKS on MD-RAID: systemd-based initiramfs fails

Downgrading to mdadm 4.4-2 indeed produces a bootable initramfs. Extracting and comparing all files in both UKIs shows differences in /usr/bin/mdmon, /usr/bin/mdadm and /usr/lib/udev/rules.d/63-md-raid-arrays.rules. Diff of the udev file:

$ diff -u broken/usr/lib/udev/rules.d/63-md-raid-arrays.rules downgrade/usr/lib/udev/rules.d/63-md-raid-arrays.rules
--- broken/usr/lib/udev/rules.d/63-md-raid-arrays.rules 2026-03-09 11:11:11.119651366 +0100
+++ downgrade/usr/lib/udev/rules.d/63-md-raid-arrays.rules      2026-03-09 11:29:40.343988311 +0100
@@ -15,6 +15,7 @@
 ATTR{md/metadata_version}=="external:[A-Za-z]*", ATTR{md/array_state}=="inactive", GOTO="md_ignore_state"
 TEST!="md/array_state", ENV{SYSTEMD_READY}="0", GOTO="md_end"
 ATTR{md/array_state}=="clear*|inactive", ENV{SYSTEMD_READY}="0", GOTO="md_end"
+ATTR{md/sync_action}=="reshape", ENV{RESHAPE_ACTIVE}="yes"
 LABEL="md_ignore_state"

 IMPORT{program}="/usr/bin/mdadm --detail --no-devices --export $devnode"
@@ -29,7 +30,6 @@
 IMPORT{builtin}="blkid"
 OPTIONS+="link_priority=100"
 OPTIONS+="watch"
-OPTIONS+="db_persist"
 ENV{ID_FS_USAGE}=="filesystem|other|crypto", ENV{ID_FS_UUID_ENC}=="?*", SYMLINK+="disk/by-uuid/$env{ID_FS_UUID_ENC}"
 ENV{ID_FS_USAGE}=="filesystem|other", ENV{ID_PART_ENTRY_UUID}=="?*", SYMLINK+="disk/by-partuuid/$env{ID_PART_ENTRY_UUID}"
 ENV{ID_FS_USAGE}=="filesystem|other", ENV{ID_FS_LABEL_ENC}=="?*", SYMLINK+="disk/by-label/$env{ID_FS_LABEL_ENC}"
@@ -40,6 +40,6 @@
 ENV{MD_LEVEL}=="raid[1-9]*", ENV{MD_CONTAINER}=="?*", PROGRAM="/usr/bin/readlink $env{MD_CONTAINER}", ENV{MD_MON_THIS}="%c"
 ENV{MD_MON_THIS}=="?*", TEST=="/etc/initrd-release", PROGRAM="/usr/bin/basename $env{MD_MON_THIS}", ENV{SYSTEMD_WANTS}+="mdmon@initrd-%c.service"
 ENV{MD_MON_THIS}=="?*", TEST!="/etc/initrd-release", PROGRAM="/usr/bin/basename $env{MD_MON_THIS}", ENV{SYSTEMD_WANTS}+="mdmon@%c.service"
-ENV{MD_RESHAPE_ACTIVE}=="True", PROGRAM="/usr/bin/basename $env{MD_MON_THIS}", ENV{SYSTEMD_WANTS}+="mdadm-grow-continue@%c.service"
+ENV{RESHAPE_ACTIVE}=="yes", PROGRAM="/usr/bin/basename $env{MD_MON_THIS}", ENV{SYSTEMD_WANTS}+="mdadm-grow-continue@%c.service"

 LABEL="md_end"

My mdadm.conf is fairly standard, as far as I can tell (comments removed only here):

DEVICE partitions
MAILADDR root
ARRAY /dev/md/boot metadata=1.0 UUID=ee974527:1e6264ca:2ec898b6:4b75f37d
ARRAY /dev/md/lvm metadata=1.2 UUID=4c8dddd2:0a6cc6dd:9d080e8d:1f01a4f1

marcus

Member

Registered: 2012-09-26

Posts: 34

Re: [solved] LVM on LUKS on MD-RAID: systemd-based initiramfs fails

Replacing the udev rules file with for 4.5-2 with its 4.4-2 version does not produce a bootable UKI.

frostschutz

Member

Registered: 2013-11-15

Posts: 1,636

Re: [solved] LVM on LUKS on MD-RAID: systemd-based initiramfs fails

Then it's either another bug in mdadm or something timing-related but not sure how that could be. Also it works fine for me and the assembly process is more or less the same for non-systemd based initramfs (pretty much everything triggered by udev rules).

mdadm bugtracker mentions one bug affecting dracut, but it seems to be related to DDF so it should not affect you, unless you got old raid headers on these disks. (I only skimmed the bug report, so I could be wrong here.)

It would be fixed in mdadm v4.6 https://github.com/md-raid-utilities/mdadm/issues/249 so maybe wait and see...

But it does not explain why the problem would be gone with busybox or debug hooks added. So it's probably something else that just doesn't trigger for everyone. And you'd have to find some other means to debug udev, mdadm to find out what's actually happening.

As for mdadm.conf you could test if it works if you remove it altogether.

Last edited by frostschutz (2026-03-09 11:52:14)

marcus

Member

Registered: 2012-09-26

Posts: 34

Re: [solved] LVM on LUKS on MD-RAID: systemd-based initiramfs fails

The issue was my mkinitcpio.conf. The MODULES section was still configured as a string, not an array. Wrong:

MODULES="ext4 raid10 nvidia nvidia_modeset nvidia_uvm nvidia_drm"

correct:

MODULES=(ext4 raid10 nvidia nvidia_modeset nvidia_uvm nvidia_drm)

Interestingly, when adding sh to BINARIES, both worked.

Marking the issue solved. Thx for the hints @frostschutz

frostschutz

Member

Registered: 2013-11-15

Posts: 1,636

Re: [solved] LVM on LUKS on MD-RAID: systemd-based initiramfs fails

Hmmm, initcpio should convert MODULES string to array by itself (/usr/lib/initcpio/functions :: arrayize_config) so both should work.

So it's about sh, hmmm. Not sure which change in mdadm would be responsible for that.

I wouldn't notice since I run busybox based initrd still.