[SOLVED] initramfs tries to mount root before RAID is assembled

niteskunk · 2017-03-06 09:50:08

I'm trying to run Arch on a 2 disk RAID-1 using onboard Intel Rapid Storage Technology (Fake RAID) and LVM. The problem I am encountering is during boot, where the init script tries to mount my root device (/dev/mapper/Arch-root) before mdadm has had a chance to assemble it. This causes the boot to halt and drop me to a recovery shell, where I must manually invoke mdadm to assemble the RAID. Once I manually assemble the RAID, I'm able to exit the recovery shell and boot into Arch where everything works perfectly.

This is a GPT system using GRUB, MD, and LVM. I'm confident GRUB, MD and LVM are all configured properly -- the issue seems to be that the init script doesn't know to assemble the RAID before trying to mount root. I am using the mdadm_udev hook in mkinitcpio.conf, which I figured would take care of this. Does anyone know what the issue may be?

----------------
/etc/default/grub

GRUB_DEFAULT=0
GRUB_TIMEOUT=5
GRUB_DISTRIBUTOR="Arch"
GRUB_CMDLINE_LINUX_DEFAULT="resume=/dev/mapper/Arch-swap"
GRUB_CMDLINE_LINUX=""

# Preload both GPT and MBR modules so that they are not missed
GRUB_PRELOAD_MODULES="part_gpt part_msdos"

# Uncomment to enable Hidden Menu, and optionally hide the timeout count
#GRUB_HIDDEN_TIMEOUT=5
#GRUB_HIDDEN_TIMEOUT_QUIET=true

# Uncomment to use basic console
GRUB_TERMINAL_INPUT=console

# Uncomment to disable graphical terminal
#GRUB_TERMINAL_OUTPUT=console

# The resolution used on graphical terminal
# note that you can use only modes which your graphic card supports via VBE
# you can see them in real GRUB with the command `vbeinfo'
GRUB_GFXMODE=auto

# Uncomment to allow the kernel use the same resolution used by grub
GRUB_GFXPAYLOAD_LINUX=keep

# Uncomment if you want GRUB to pass to the Linux kernel the old parameter 
# format "root=/dev/xxx" instead of "root=/dev/disk/by-uuid/xxx" 
GRUB_DISABLE_LINUX_UUID=true

# Uncomment to disable generation of recovery mode menu entries
GRUB_DISABLE_RECOVERY=true

# Uncomment and set to the desired menu colors.  Used by normal and wallpaper 
# modes only.  Entries specified as foreground/background.
#GRUB_COLOR_NORMAL="light-blue/black"
#GRUB_COLOR_HIGHLIGHT="light-cyan/blue"

# Uncomment one of them for the gfx desired, a image background or a gfxtheme
#GRUB_BACKGROUND="/path/to/wallpaper"
#GRUB_THEME="/path/to/gfxtheme"

# Uncomment to get a beep at GRUB start
#GRUB_INIT_TUNE="480 440 1"

#GRUB_SAVEDEFAULT="true"

Menu entry (from /boot/grub/grub.cfg):

menuentry 'Arch Linux' --class arch --class gnu-linux --class gnu --class os $menuentry_id_option 'gnulinux-simple-11579af4-df36-4491-b6ea-753410f8340f' {
	load_video
	set gfxpayload=keep
	insmod gzio
	insmod part_gpt
	insmod ext2
	if [ x$feature_platform_search_hint = xy ]; then
	  search --no-floppy --fs-uuid --set=root  f3977d32-2934-420f-a2b7-2a146a1785e4
	else
	  search --no-floppy --fs-uuid --set=root f3977d32-2934-420f-a2b7-2a146a1785e4
	fi
	echo	'Loading Linux linux ...'
	linux	/vmlinuz-linux root=/dev/mapper/Arch-root rw  resume=/dev/mapper/Arch-swap
	echo	'Loading initial ramdisk ...'
	initrd  /initramfs-linux.img
}

/etc/mkinitcpio.conf

# vim:set ft=sh
# MODULES
# The following modules are loaded before any boot hooks are
# run.  Advanced users may wish to specify all system modules
# in this array.  For instance:
#     MODULES="piix ide_disk reiserfs"
MODULES="dm_mod md_mod raid1 ext4"

# BINARIES
# This setting includes any additional binaries a given user may
# wish into the CPIO image.  This is run last, so it may be used to
# override the actual binaries included by a given hook
# BINARIES are dependency parsed, so you may safely ignore libraries
BINARIES="/sbin/mdmon"

# FILES
# This setting is similar to BINARIES above, however, files are added
# as-is and are not parsed in any way.  This is useful for config files.
FILES=""

# HOOKS
# This is the most important setting in this file.  The HOOKS control the
# modules and scripts added to the image, and what happens at boot time.
# Order is important, and it is recommended that you do not change the
# order in which HOOKS are added.  Run 'mkinitcpio -H <hook name>' for
# help on a given hook.
# 'base' is _required_ unless you know precisely what you are doing.
# 'udev' is _required_ in order to automatically load modules
# 'filesystems' is _required_ unless you specify your fs modules in MODULES
# Examples:
##   This setup specifies all modules in the MODULES setting above.
##   No raid, lvm2, or encrypted root is needed.
#    HOOKS="base"
#
##   This setup will autodetect all modules for your system and should
##   work as a sane default
#    HOOKS="base udev autodetect block filesystems"
#
##   This setup will generate a 'full' image which supports most systems.
##   No autodetection is done.
#    HOOKS="base udev block filesystems"
#
##   This setup assembles a pata mdadm array with an encrypted root FS.
##   Note: See 'mkinitcpio -H mdadm' for more information on raid devices.
#    HOOKS="base udev block mdadm encrypt filesystems"
#
##   This setup loads an lvm2 volume group on a usb device.
#    HOOKS="base udev block lvm2 filesystems"
#
##   NOTE: If you have /usr on a separate partition, you MUST include the
#    usr, fsck and shutdown hooks.

# temporarily removing autodetect
HOOKS="base udev modconf block mdadm_udev lvm2 filesystems keyboard fsck"

# COMPRESSION
# Use this to compress the initramfs image. By default, gzip compression
# is used. Use 'cat' to create an uncompressed image.
#COMPRESSION="gzip"
#COMPRESSION="bzip2"
#COMPRESSION="lzma"
#COMPRESSION="xz"
#COMPRESSION="lzop"
#COMPRESSION="lz4"

# COMPRESSION_OPTIONS
# Additional options for the compressor
#COMPRESSION_OPTIONS=""

/etc/mdadm.conf (generated by "mdadm --examine --scan > /etc/mdadm.conf")

ARRAY metadata=imsm UUID=0099e2ff:958e6f52:71e7e423:d45eaec2
ARRAY /dev/md/data container=0099e2ff:958e6f52:71e7e423:d45eaec2 member=0 UUID=3220c3db:1a961be5:93e4a8fd:d96575c3

lsblk -f (from booted system)

NAME            FSTYPE          LABEL       UUID                                   MOUNTPOINT
sda             isw_raid_member                                                    
└─md126                                                                            
  ├─md126p1     vfat                        FAEC-EF48                              
  ├─md126p2     ext2                        f3977d32-2934-420f-a2b7-2a146a1785e4   /boot
  └─md126p3     LVM2_member                 sdlQDN-mxrS-8hBC-4jLD-StVz-G4LB-lgevuI 
    ├─Arch-swap swap                        97b3dd97-9df3-4e51-bb5b-3c9527068ad8   [SWAP]
    ├─Arch-root ext4                        11579af4-df36-4491-b6ea-753410f8340f   /
    └─Arch-home ext4                        4e99806e-f2f1-47f4-af7c-c9e144e57658   /home
sdb             isw_raid_member                                                    
└─md126                                                                            
  ├─md126p1     vfat                        FAEC-EF48                              
  ├─md126p2     ext2                        f3977d32-2934-420f-a2b7-2a146a1785e4   /boot
  └─md126p3     LVM2_member                 sdlQDN-mxrS-8hBC-4jLD-StVz-G4LB-lgevuI 
    ├─Arch-swap swap                        97b3dd97-9df3-4e51-bb5b-3c9527068ad8   [SWAP]
    ├─Arch-root ext4                        11579af4-df36-4491-b6ea-753410f8340f   /
    └─Arch-home ext4                        4e99806e-f2f1-47f4-af7c-c9e144e57658   /home

When the system boots, I see:

:: running early hook [udev]
:: running early hook [lvm2]
:: running hook [udev]
:: Triggering uevents...
Waiting 10 seconds for device /dev/mapper/Arch-root
ERROR: device '/dev/mapper/Arch-root' not found. Skipping fsck.
ERROR: Unable to find root device '/dev/mapper/Arch-root'.
You are being dropped to a recovery shell
  Type 'exit' to try and continue booting
sh: can't access tty: job control turned off

If I run:

[rootfs ]# IMSM_NO_PLATFORM=1 mdadm -As   # note: we need IMSM_NO_PLATFORM=1 to force mdadm to assemble because EFI vars are not available at this point & mdadm uses them to try to verify the OROM exists.
[rootfs ]# exit

Arch then boots normally.

Last edited by niteskunk (2017-03-07 01:00:23)

Lone_Wolf · 2017-03-06 11:52:39

I noticed your mkinitcpio.conf doesn't use the autodetect hook , maybe you are missing a needed module.

Please post mkinitcpio --automods output and lsinictpio --analyze /path/to/your-initramfs-file

falconindy · 2017-03-06 14:14:28

Does fakeraid support incremental assembly? I'm not sure it does. The mdadm_udev hook requires that your array supports this.

In general, fakeraid is not well supported by much of anything.

And, autodetect doesn't add modules. Ever. It only removes modules.

niteskunk · 2017-03-07 00:33:21

Lone_Wolf wrote:

I noticed your mkinitcpio.conf doesn't use the autodetect hook , maybe you are missing a needed module.
Please post mkinitcpio --automods output and lsinictpio --analyze /path/to/your-initramfs-file

As falconindy said, autodetect only serves to remove unneeded modules. I removed it to eliminate that risk, but it didn't seem to do anything.

falconindy wrote:

Does fakeraid support incremental assembly? I'm not sure it does. The mdadm_udev hook requires that your array supports this.
In general, fakeraid is not well supported by much of anything.
And, autodetect doesn't add modules. Ever. It only removes modules.

Hmm... you may be on to something here. Based on some tinkering, I don't think Fakeraid supports incremental assembly.

I've been looking deeper into this and mdadm_udev is just an installation hook that adds udev rules (63-md-raid-arrays.rules and 64-md-raid-assembly.rules) to the initramfs. The former triggers actions when an md device is added to the system, the latter actually attempts incremental assembly when udev starts.

The default action is this (and I've confirmed this runs).

ACTION=="add|change", IMPORT{program}="/usr/bin/mdadm --incremental --export $devnode --offroot ${DEVLINKS}"

Now, since I know mdadm -As runs, I modified the rule to:

ACTION=="add|change", IMPORT{program}="/usr/bin/mdadm -As"

But it still doesn't seem to assemble the RAID... strange.

niteskunk · 2017-03-07 00:59:26

...Wow, I figured it out.

As I mentioned, mdadm has a feature where it "will only allow an IMSM array to be created or modified if detects that it is running on an Intel platform which supports IMSM". This can be turned off by setting the environment variable IMSM_NO_PLATFORM=1 before issuing mdadm -As.

http://www.reactivated.net/writing_udev … ternal-run states that in the context of running a program, "udev does not run these programs on any active terminal, and it does not execute them under the context of a shell", so I assume it did not have access to that environment variable. I checked out the source code of mdadm and found that it determines the existence of an IMSM platform by checking EFI variables. I adjusted my mkinitcpio udev hook to:

run_hook() {
    msg ":: Triggering uevents"
    udevadm trigger --action=add --type=subsystems
    udevadm settle
    mount -t efivarfs efivarfs /sys/firmware/efi/efivars
    udevadm trigger --action=add --type=devices
    udevadm settle
}

...and my aforementioned change to 64-md-raid-assembly.rules worked. The RAID assembled when udev started. It had been failing because EFI variables were not present, so the OROM check was failing.

I'm going to probably write a tutorial on working with Intel FakeRAID, as there is currently a vast amount of outdated/conflicting information that made getting this working very challenging. falconindy, your mention about incremental assembly definitely got me on the right path here. Thanks a lot!

Edit: I restored the udev rule file to the default (with incremental assembly) and that seems to work too. The root of this issue was that the EFI variables need to be accessible before mdadm runs with Intel FakeRAID.

Last edited by niteskunk (2017-03-07 02:03:29)

Arch Linux

#1 2017-03-06 09:50:08

[SOLVED] initramfs tries to mount root before RAID is assembled

#2 2017-03-06 11:52:39

Re: [SOLVED] initramfs tries to mount root before RAID is assembled

#3 2017-03-06 14:14:28

Re: [SOLVED] initramfs tries to mount root before RAID is assembled

#4 2017-03-07 00:33:21

Re: [SOLVED] initramfs tries to mount root before RAID is assembled

#5 2017-03-07 00:59:26

Re: [SOLVED] initramfs tries to mount root before RAID is assembled

Board footer