Arch Linux

renatocan

Member

Registered: 2012-10-31

Posts: 6

Problem configuring RAID: system halts when I simulate a disk failure

Hi everyone!

I'm having some problems trying to set up a RAID 1 config for a fresh install of Arch Linux on a server using UEFI and two SATA hard disks.

In fact, the system is loading properly and RAID is in sync, but when I simulate a disk failure by unplugging the second disk, it loads GRUB but fails to find the root md partition. I receive the following message:

device UUID=1f016eec-7b66-4e59-b1fa-d4998a31c2a3 not found. Skipping
mount: /new_root: can't find UUID=1f016eec-7b66-4e59-b1fa-d4998a31c2a3
ERROR: Failed to mount 'UUID=1f016eec-7b66-4e59-b1fa-d4998a31c2a3' on real root
You are now being dropped into an emergency shell

in which UUID=1f016eec-7b66-4e59-b1fa-d4998a31c2a3 refers to md device that should be mounted as /.

Following the suggestions from Arch Wiki I have set a pair of FAT32 partitions /dev/sda1 and /dev/sdb1 to be used by the EFI system and two more pairs of partitions of type RAID:
- /dev/sda2 and /dev/sdb2 used by /dev/md127, that is mounted as /
- /dev/sda3 and /dev/sdb3 used by /dev/md126, that is mounted as /home

I have tried to regenerate initramfs and grub.cfg including GRUB_PRELOAD_MODULES="... mdraid09 mdraid1x" on /etc/default/grub and reinstalled GRUB with grub-install but that did not help. The system keeps booting without problems, but halts when I unplug the second hard disk. It boots normally if I replug it.

I'd appreciate it if you can give some suggestions on how to proceed.

Here are some more info about config files:

- mdstat

# cat /proc/mdstat
Personalities : [raid1]
md126 : active raid1 sda3[0] sdb3[1]
  	1464711168 blocks super 1.2 [2/2] [UU]
  	bitmap: 2/11 pages [8KB], 65536KB chunk

md127 : active raid1 sda2[0] sdb2[1]
  	146352128 blocks super 1.2 [2/2] [UU]
  	bitmap: 0/2 pages [0KB], 65536KB chunk

unused devices: <none>

- mdadm.conf

# cat /etc/mdadm.conf
ARRAY /dev/md/root metadata=1.2 UUID=06531176:fa0a4686:7175cb63:537d550f
ARRAY /dev/md/home metadata=1.2 UUID=833bfc5b:ef7d3ef0:beb49549:f8496386

- blkid

# blkid
/dev/md127: UUID="1f016eec-7b66-4e59-b1fa-d4998a31c2a3" BLOCK_SIZE="4096" TYPE="ext4"
/dev/sdb2: UUID="06531176-fa0a-4686-7175-cb63537d550f" UUID_SUB="d4eab4ff-3d84-cf97-db72-b3c1a3b3decf" LABEL="any:root" TYPE="linux_raid_member" PARTUUID="56b33ac5-0d64-4692-9167-e86e73ff0a32"
/dev/sdb3: UUID="833bfc5b-ef7d-3ef0-beb4-9549f8496386" UUID_SUB="b2f47b28-798b-f9ec-0b41-b546d2e15505" LABEL="any:home" TYPE="linux_raid_member" PARTUUID="6e98e51f-41bb-4050-9e7c-efbd9fd59032"
/dev/sdb1: UUID="8BEA-8760" BLOCK_SIZE="512" TYPE="vfat" PARTUUID="ba0b8fe6-d080-4956-b304-f2fa7265b77b"
/dev/md126: UUID="bc8b74fd-b2f3-406f-8fde-55e040473af3" BLOCK_SIZE="4096" TYPE="ext4"
/dev/sda2: UUID="06531176-fa0a-4686-7175-cb63537d550f" UUID_SUB="1bd2c6e3-885a-170a-03b5-d16ed3863817" LABEL="any:root" TYPE="linux_raid_member" PARTUUID="56b33ac5-0d64-4692-9167-e86e73ff0a32"
/dev/sda3: UUID="833bfc5b-ef7d-3ef0-beb4-9549f8496386" UUID_SUB="43b9d256-9a0e-0815-ffd6-b21cea7678cd" LABEL="any:home" TYPE="linux_raid_member" PARTUUID="6e98e51f-41bb-4050-9e7c-efbd9fd59032"
/dev/sda1: UUID="8BB0-BFD2" BLOCK_SIZE="512" TYPE="vfat" PARTUUID="ba0b8fe6-d080-4956-b304-f2fa7265b77b"

Thank you!

frostschutz

Member

Registered: 2013-11-15

Posts: 1,476

Re: Problem configuring RAID: system halts when I simulate a disk failure

Check dmesg in the emergency shell. mkinitcpio relies on mdadm udev rules (64-md-raid-assembly.rules) to do incremental assembly. And most likely that means it waits indefinitely for that 2nd disk to appear.

In systemd environment it eventually kicks off the mdadm-last-resort timer but in initcpio without systemd this fallback does not exist (even with systemd-based initcpio I'm not sure if its included).

You'd have to write a custom hook to the same effect; or write a custom mdadm hook which assembles arrays the oldfashioned way (not using incremental assembly).

The behavior might be different if, instead of missing altogether, the drive is still around but marked as failed. Incremental assembly is described in more detail in the mdadm man page.

Manpage suggest running this:

Note that neither of these approaches is really ideal.  If it can be known that all
       device discovery has completed, then
          mdadm -IRs
       can be run which will try to start all arrays that are being incrementally assembled.

The last resort timer just mdadm --run /dev/mdX

Last edited by frostschutz (2024-08-30 06:22:05)

renatocan

Member

Registered: 2012-10-31

Posts: 6

Re: Problem configuring RAID: system halts when I simulate a disk failure

Thank you for the suggestion!

mdadm -IRs worked and I ended up with the default configuration. I took a look about the custom hook solution but I think it will not be easy to scale for multiple servers with more people maintaining them.

In order to have updated efi data on the second drive (/boot/efi2), I followed the suggestion at https://wiki.debian.org/UEFI#RAID_for_t … _Partition

And to allow the system to start when there is a fail in the first drive, I added a second entry to the UEFI boot menu with

efibootmgr --create --disk /dev/sdb --part 1 --loader '\EFI\GRUB\grubx64.efi' --label 'linux-sdb' --unicode

[]'s

Renato