Arch Linux

phoenixkc

Member

Registered: 2015-10-30

Posts: 5

[Solved] Can no longer boot to RAID array after update

I recently updated my system and after a reboot I get the following:

ERROR: device ‘UUID=...’ not found. Skipping fsck.
mount: /new_root: can’t find UUID=...
You are now being dropped into an emergency shell.

My root file system is on a RAID array and running blkid from the emergency shell shows that the array is not listed.

I have /dev/sda2 and /dev/sdb2 in RAID1 and typically it mounts to /boot. They have a UUID="b45... and are assembled into /dev/md1 with a UUID="436.... While /dev/sda3 and /dev/sdb3 in RAID0 typically mounted to /. They have a UUID="8dc... and are assembled into /dev/md2 with a UUID="d0d.... The error at boot is looking for d0d....

I booted to live USB and ran blkid and the array was there. So I mounted the arrays, proc, sys, dev, and boot. I checked /etc/mdadm.conf and saw that all the ARRAY lines were commented out so I ran:

# mdadm --detail --scan >> /mnt/etc/mdadm.conf

I checked that the UUIDs matched the two partitions I am using in the arrays (b45... and 8dc...). I then chrooted in.

I then checked /etc/mkinitcpio.conf and saw that mdadm_udev was missing from the HOOKS setting. So I readded that.

From there, I tried each of the following followed by a reboot:

# pacman -S linux
# mkinitcpio -p linux
# mkinitcpio -P

For each I can see it build the mdadm_udev hook and it tells me that:

Custom /etc/mdadm.conf file will be used in initramfs for assembling arrays

But when I reboot, I get the same error and using blkid in the emergency shell shows that my RAID arrays are not listed (but the partitions that make it up are there and the UUIDs match what is in mdadm.conf) .

I also added the kernel parameter raid0.default_layout=2 with the same result. It's never been there before but the wiki seems to indicate that I need it.

The Linux package is linux-5.12.6.arch1-1 and before the ERROR on boot it tells me that it is:

Starting version 248.3-2-arch

What am I missing that is causing the array not to be assembled on boot?

Last edited by phoenixkc (2021-05-25 22:37:55)

frostschutz

Member

Registered: 2013-11-15

Posts: 1,647

Re: [Solved] Can no longer boot to RAID array after update

any error messages in dmesg on that initramfs shell?

cat /proc/mdstat exists and lists "Personalities : [raid1]"?

does 'mdadm  --verbose --assemble --scan' work or do anything? does it work if you remove the mdadm.conf in initramfs shell?

also check there is only one mdadm.conf sometimes there is both /etc/mdadm.conf and /etc/mdadm/mdadm.conf and it can be confusing which one applies

--detail --scan tends to be too verbose for mdadm.conf, ARRAY lines only need UUID= and none of the other conditions.

phoenixkc

Member

Registered: 2015-10-30

Posts: 5

Re: [Solved] Can no longer boot to RAID array after update

any error messages in dmesg on that initramfs shell?

Only thing that stood out was the following:

 
GPT:Primary header thinks Alt. header is not at the end of the disk. 
GPT:1546903! = 122879999
GPT:Alternate header not at the end of the disk. 
GPT:1546903! = 122879999
GPT: Use GNU Parted to correct GPT errors. 

I'll look into this after I'm done posting. I'm sure I have an old GParted disk around here somewhere...

cat /proc/mdstat exists and lists "Personalities : [raid1]"?

When I chroot, yes. /proc/mdstat exists and lists "Personalities : [raid0] [raid1]". From the initramfs shell, /proc/mdstat does not exist.

does 'mdadm  --verbose --assemble --scan' work or do anything? does it work if you remove the mdadm.conf in initramfs shell?

From the initramfs shell, the mdadm does not exist and neither does mdadm.conf. In the initramfs shell, /etc only has:

fstab
initrd-release
ld.so.cache
ld.so.conf
modprobe.d
mtab

That seems weird to me. Especially since I can see the mdadm_udev module being built when I run mkinitcpio.

When run in chroot, it alternately tells me that the partitions used for the array are either busy or that they have the wrong uuid. Is this related to the GPT errors in dmesg? Or is it irrelevant because I'm chrooted?

also check there is only one mdadm.conf sometimes there is both /etc/mdadm.conf and /etc/mdadm/mdadm.conf and it can be confusing which one applies

As mentioned above, the initramfs shell does not contain anything related to mdadm. And when chrooted, it is there is only /etc/mdadm.conf.

--detail --scan tends to be too verbose for mdadm.conf, ARRAY lines only need UUID= and none of the other conditions.

I removed the extra parameters but ran into the same error.

I'm going to look into the GPT errors. But I'm more concerned that the initramfs doesn't seem to have any of the mdadm tools or config files. I made sure that mdadm_udev was built and included when running mkinitcpio. So where am I going wrong?

frostschutz

Member

Registered: 2013-11-15

Posts: 1,647

Re: [Solved] Can no longer boot to RAID array after update

mdadm should exist, maybe you forgot to add mdadm hook after all, or you are booting the wrong initrd file?

is mdadm itself not installed correctly in your system?

gnu parted is not gparted. you can fix GPT with any GPT-aware partitioner of your choice.

phoenixkc

Member

Registered: 2015-10-30

Posts: 5

Re: [Solved] Can no longer boot to RAID array after update

I still have no idea what caused the issue in the first place. But the solution was dead simple and I feel dumb. But thanks to frostschutz for making me go through those checks. It helped point the way.

The issue was that I had run:

# mount /dev/md1 /mnt/boot

from the live USB before chrooting and running mkinitcpio because I wanted to check the settings in grub. So the initramfs image I created with mdadm was on a RAID array. The initramfs that got loaded had no clue about mdadm and so wasn't building the array.

Solution was to run mkinitcpio in chroot WITHOUT mounting an array to /boot.