[SOLVED] Encrypted mdadm raid0 device fails to mount on boot 20% of...

maxweiss · 2017-10-21 16:56:29

Update: After a handful of reboots, it looks like adding "rootdelay=20" to the GRUB_CMDLINE_LINUX_DEFAULT line in /etc/default/grub, as well as moving "block" to be listed before "autodetect" in the HOOKS line of /etc/mkinitcpio.conf may have solved the problem. If I go a week or two without the issue happening, I will come back and mark this thread as solved.

Update 2: After many more restarts, the problem has yet to appear again, so I am marking as solved. If it does happen again, I will come back and remove the "solved".

X-post from reddit.

I have three hard hard drives on a 7 year old BIOS (no uefi) computer. Two drives are solid state, and one is a slow old drive. The two solid state disks are the same size and connected via mdadm software raid in raid0. The raid0 device is then encrypted with luks, and the luks block device is used for lvm2 to make my boot, swap, and filesystem faux-partitions.

You can see the setup in this picture. sda and sdc are the solid state drives, and sdb is the hard disk.

Grub2 handles this pretty well. I have grub2 installed on both sda and sdc, and my BIOS is set to try to boot from sda, then sdc, then the cdrom, then network, then give up. So when I boot, I am presented with this screen, at which point I enter my password, grub slowly derives the crypto key, and it loads /boot and goes to the grub menu and auto-selects my OS, and tries to boot.

In addition to the password, the cryptodisk is also decrypted by a keyfile, which grub is set to send to systemd. About 80% of the time, this works great, and I see this screen shortly before LightDM and Xorg take over and I am booted into my desktop.

However, about 20% of the time, the process fails, and the cryptodisk isn't (re)mounted by systemd, and it fails into busybox, as shown here.

I am hoping somebody here can help me figure out how to fix this, either with a script that busybox can run to remount the crypt disk, or with any other fix. I don't care if booting takes a little longer, I just want the process to be unattended after I enter the password initially at the grub welcome screen. I've researched this for hours upon hours, and I can't for the life of me figure out how to fix this, or why it's working some of the time and not others. My guess is that the mdadm raid device doesn't finish setting up quickly enough, but I don't know that for sure, and I also don't know what to do about that even if it is the problem.

I don't know if this will help, but here's relevant lines in my /etc/default/grub configuration file:

GRUB_CMDLINE_LINUX_DEFAULT="quiet splash luks.options=timeout=30s nvidia-drm.modeset=0 modeset=0 nomodeset cryptdevice=/dev/md0:vg0 cryptkey=rootfs:/path/to/keyfile"
GRUB_ENABLE_CRYPTODISK=y
GRUB_PRELOAD_MODULES="part_gpt part_msdos mdraid cryptodisk luks"

(I have an Nvidia card, hence all the nomodeset stuff.)

And here's my /etc/mkinitcpio.conf file:

MODULES="nvidia nvidia_modeset nvidia_uvm nvidia_drm"
BINARIES=""
FILES="/path/to/keyfile"
HOOKS="base udev autodetect modconf block mdadm encrypt lvm2 filesystems keyboard fsck"
COMPRESSION="gzip"

Any suggestions anybody has would be greatly appreciated. Thanks everybody!

MOD EDIT: please don't add [Closed] to your post title as it indicates a topic that has been closed by a moderator, which this has not.

Last edited by maxweiss (2017-10-25 19:02:21)

seth · 2017-10-21 19:01:52

wild guess (didn't actually read your post ;-)
"rootdelay=<seconds>" or "rootwait" kernel parameter?

maxweiss · 2017-10-21 22:02:35

I've tried those before and they don't seem to help. Thanks though.

mich41 · 2017-10-21 22:37:06

They are intended for the kernel and meaningless with initramfs. In this case it's a bug in init scripts or systemd service files missing some dependencies so that they fail to wait for LVM before trying to run fsck and mount.

I can't help much with that, it's been ages since I have last used stock kernels and initramfs. Either something is still wrong with your mkinitcpio.conf (not sure what could it be) or probably there is a bug in mkinitcpio, some scripts, systemd, hell knows what. Yeah, I guess I'm very helpful.

seth · 2017-10-22 07:24:21

Does it matter how long it takes you to insert the passwort? Ie. can you turn the value into 100% failure when, instead of entering a password, you first get a cup of coffee?

jasonwryan · 2017-10-22 07:30:08

Have you tried adding the raid, lvm and crypto modules to your initramfs?

maxweiss · 2017-10-22 13:35:54

seth wrote:

Does it matter how long it takes you to insert the passwort? Ie. can you turn the value into 100% failure when, instead of entering a password, you first get a cup of coffee?

I only enter the password once, right at the beginning for grub, so grub can decrypt the disk (since /boot is encrypted as well). After that, during the boot process, the setup uses the keyfile. That's when the failure is occuring, when systemd is starting up.

jasonwryan wrote:

Have you tried adding the raid, lvm and crypto modules to your initramfs?

Is that not what the HOOKS line in my mkinitcpio.conf does? What else do I need to do to add them to the initramfs?

mich41 wrote:

Either something is still wrong with your mkinitcpio.conf (not sure what could it be) or probably there is a bug in mkinitcpio, some scripts, systemd, hell knows what. Yeah, I guess I'm very helpful.

Haha, thanks anyway.

Last edited by maxweiss (2017-10-22 13:37:08)

seth · 2017-10-22 13:59:36

maxweiss wrote:

I only enter the password once, right at the beginning for grub, so grub can decrypt the disk (since /boot is encrypted as well). After that, during the boot process, the setup uses the keyfile. That's when the failure is occuring, when systemd is starting up.

What bothers me is that the behavior is non-deterministic - smells like a timing issue and the obvious non-deterministic element in the chain is you :-P
Bottom line is whether the amount delay you introduce when entering the passwort matters (in one or the other direction)

mich41 · 2017-10-22 16:32:45

The nondeterministic element is whether LVM volumes are fully initialized by the time mount is ran, which for some reason doesn't wait for LVM. Maybe it's stupid systemd spawning these jobs concurrently, maybe udev is taking too long to create some symlinks, dunno. You guys need to either unpack initramfs (google it) and see how that stuff works or bother Arch developers as it seems it's them who maintain mkinitcpio. Bugtracker maybe?

BTW, if you have this emergency shell, check if /dev/mapper/vg0-rootfs exists. Maybe LVM fails to start altogether. Bugs happen.

jasonwryan · 2017-10-22 17:54:57

maxweiss wrote:

jasonwryan wrote:
Have you tried adding the raid, lvm and crypto modules to your initramfs?
Is that not what the HOOKS line in my mkinitcpio.conf does? What else do I need to do to add them to the initramfs?

No, that would be the MODULES array.

maxweiss · 2017-10-23 00:21:22

Thank you all for the comments and the help. At least I have some idea of what's happening and why. I will try Jason's suggestion and see if that helps, and if it doesn't, try what a user in the Reddit thread suggested, and hook systemd and try using sd-encrypt and sd-lvm and seeing if I can get that working.

seth · 2017-10-23 18:00:29

Cross check here: https://bbs.archlinux.org/viewtopic.php?pid=1744748

maxweiss · 2017-10-24 00:05:07

seth wrote:

Cross check here: https://bbs.archlinux.org/viewtopic.php?pid=1744748

Holy shit, yeah that's exactly my issue. Thanks for the link, will read the whole thread. I'm glad to finally know I'm not alone!!

Arch Linux

#1 2017-10-21 16:56:29

[SOLVED] Encrypted mdadm raid0 device fails to mount on boot 20% of...

#2 2017-10-21 19:01:52

Re: [SOLVED] Encrypted mdadm raid0 device fails to mount on boot 20% of...

#3 2017-10-21 22:02:35

Re: [SOLVED] Encrypted mdadm raid0 device fails to mount on boot 20% of...

#4 2017-10-21 22:37:06

Re: [SOLVED] Encrypted mdadm raid0 device fails to mount on boot 20% of...

#5 2017-10-22 07:24:21

Re: [SOLVED] Encrypted mdadm raid0 device fails to mount on boot 20% of...

#6 2017-10-22 07:30:08

Re: [SOLVED] Encrypted mdadm raid0 device fails to mount on boot 20% of...

#7 2017-10-22 13:35:54

Re: [SOLVED] Encrypted mdadm raid0 device fails to mount on boot 20% of...

#8 2017-10-22 13:59:36

Re: [SOLVED] Encrypted mdadm raid0 device fails to mount on boot 20% of...

#9 2017-10-22 16:32:45

Re: [SOLVED] Encrypted mdadm raid0 device fails to mount on boot 20% of...

#10 2017-10-22 17:54:57

Re: [SOLVED] Encrypted mdadm raid0 device fails to mount on boot 20% of...

#11 2017-10-23 00:21:22

Re: [SOLVED] Encrypted mdadm raid0 device fails to mount on boot 20% of...

#12 2017-10-23 18:00:29

Re: [SOLVED] Encrypted mdadm raid0 device fails to mount on boot 20% of...

#13 2017-10-24 00:05:07

Re: [SOLVED] Encrypted mdadm raid0 device fails to mount on boot 20% of...

Board footer