[SOLVED] gummiboot + mdadm RAID 1?

pgoetz · 2014-02-26 15:01:29

I'm working on my second Arch install. The first was to a simple /dev/sdX disk, and the instructions in the Beginner's Guide worked perfectly. Now I'm trying to set up a small server which has 2 SSD disks I'd like to configure as RAID1. Of course the beginner guide doesn't cover this use case, so I improvised:

sda1 + sdb2 = /dev/md0 formatted using FAT32 and mounted as /boot
sda2 + sdb2 = /dev/md1 formatted as ext4 and mounted as /

(It's hard to provide lsblk output, as the system isn't functional on the network right now)

I read somewhere that using RAID1 for the EFI boot partition should be OK (maybe this only applies to hardware RAID?), but I can't get gummiboot to install, and the options for efibootmgr don't seem to be applicable (in any case, I can't get it to work).

I would like the system to be able to boot if one of the drives fails, and would like to use gummiboot, given that this is a new UEFI system board. I have gotten a setup like this working with grub2 in the past, but of course that didn't involve a boot partition on the disk. Before going back to grub2, I'll just un-raid the boot partition and mirror it by hand after the fact; however better would be for it to just work.

So, is there any way to trick gummiboot into working with this configuration, or is it not technically feasible?

Last edited by pgoetz (2014-03-10 14:02:52)

brain0 · 2014-02-26 15:54:40

I read somewhere that using RAID1 for the EFI boot partition should be OK (maybe this only applies to hardware RAID?), but I can't get gummiboot to install, and the options for efibootmgr don't seem to be applicable (in any case, I can't get it to work).

IF that should work, you need a RAID format that has the metadata at the end of the partition (otherwise your firmware won't find the file system). As far as I can tell, mdadm metadata formats 1.0 and 0.9 should do it. I don't think gummiboot can handle this, so after you're done, you'd need to use efibootmgr manually to create the boot entries.

All that is guessing on my part, I never tried it.

pgoetz · 2014-02-26 16:32:59

Thanks for the tip. I'll look at the RAID wiki to see if there are any tips on how to set mdadm metadata formats.

It's not clear how one would use efibootmgr, in this case, though. The Gummiboot wiki page (https://wiki.archlinux.org/index.php/Gummiboot) suggests using it this way:

If gummiboot install command failed, you can create a EFI boot entry manually using efibootmgr utility:

# efibootmgr -c -d /dev/sdX -p Y -l /EFI/gummiboot/gummibootx64.efi -L "Gummiboot"

First of all, I'm pretty sure there is a typo there, as -p requires a numerical argument (i.e. Y probably needs to be 1, but I don't know enough about this yet to edit this page myself)

I tried running this command:

# efibootmgr -c -d /dev/md0  -l /EFI/gummiboot/gummibootx64.efi -L "Gummiboot"

and got a warning: "This MBR disk does not have a unique signature. If this is not the first disk found, you might not be able to boot from it [actually, that wouldn't be a problem]. ... Run efibootmgr with the -w flag." So then I tried

# efibootmgr -w -c -d /dev/md0  -l /EFI/gummiboot/gummibootx64.efi -L "Gummiboot"

This gave another error:

** Warning ** Boot0007 has same label as Gummiboot

But the bottom line is no gummibootx64.efi showed up in /boot/EFI/gummiboot

Are all these just side effects of having the wrong mdadm metadata format?

Last edited by pgoetz (2014-03-07 19:40:02)

brain0 · 2014-02-27 09:17:06

No no no, the firmware does NOT understand the MD RAID. You need to install boot entries for both -d /dev/sda -p 1 and -d /dev/sdb -p1. This means you'll have two boot entries, one for each partition, so the firmware can fall back on the second if the first disk fails. Before you do that, you must run gummiboot install to actually copy the files.

However, this is still all just guessing.

brain0 · 2014-03-01 01:18:19

Okay, I managed to get this done in a qemu VM with OVMF and two hard drives. First, I created the usual GPT partitions, identical on both drives. Then I created two RAID arrays, the one for /boot had metadata format 1.0:

# mdadm --create /dev/md0 -e 1.0 -n 2 -l 1 -n internal /dev/sda1 /dev/sdb1

Metadata 1.0 is important here, since the superblock is at the end of the device and thus the file system looks like an ordinary FAT32. The firmware only accesses the ESP in read-only mode, so nothing should break. However, do not try to dual-boot this with Windows (Windows will perform read-write access to the ESPs separately, since it does not recognize the RAID).

Now, gummiboot will refuse to install onto /boot, so you'll need to do it manually:

# mkdir -p /boot/EFI/{Boot,gummiboot}
# cp /usr/lib/gummiboot/gummibootx64.efi /boot/EFI/Boot/bootx64.efi
# cp /usr/lib/gummiboot/gummibootx64.efi /boot/EFI/gummiboot/

The second line only installs the "default loader" and can be omitted. However, having it here is a good fallback if the firmware forgets about its boot settings. Of course, you also need to create the usual configuration files. Now, you need to install the boot entries:

# efibootmgr -c -L Gummiboot1 -l \\EFI\\gummiboot\\gummibootx64.efi -d /dev/sda -p 1
# efibootmgr -c -L Gummiboot2 -l \\EFI\\gummiboot\\gummibootx64.efi -d /dev/sdb -p 1

This will allow you to boot from both hard drives with gummiboot - again, the firmware is unaware of the RAID, so this just looks like two separate FAT32 partitions with identical contents. You can also add EFISTUB boot entries:

# efibootmgr -c -L Arch\ Linux\ 1 -l \\vmlinuz-linux -d /dev/sda -p 1 -u "$(cat /proc/cmdline)"
# efibootmgr -c -L Arch\ Linux\ 2 -l \\vmlinuz-linux -d /dev/sdb -p 1 -u "$(cat /proc/cmdline)"

pgoetz · 2014-03-07 14:47:40

This is almost working. However, on boot the system drops down into an emergency shell with this message:

:: running early hook [udev]
:: running hook [udev]
:: triggering uevents...
fsck: fsck.linux_raid_member: not found
fsck: error 2 while executing fsck.linux_raid_member for /dev/sdb1
ERROR: fsck failed on '/dev/sdb1'
:: mounting /dev/sdb1 on real root
mount: unknown filesystem type 'linux_raid_member'
You are now being dropped into an emergency shell

The odd thing about this is that in the boot order /dev/sda1 is the first boot device. I tried booting through and selecting the boot entry explicity. I also removed it using

efibootmgr -b 0000 -B

and then re-created it using

efibootmgr -c -L Arch\ Linux\ 1 -l \\EFI\\gummiboot\\gummibootx64.efi -d /dev/   sda -p 1

Here is the complete set of steps following a fresh boot from the current Arch ISO:
(previously set up: /dev/md0 = RAID 1 of /dev/sda1 and /dev/sdb1; /dev/md1 = RAID 1 of /dev/sda2 and /dev/sdb2)

mount /dev/md1 /mnt
mount /dev/md0 /mnt/boot
arch-chroot /mnt /bin/bash

mkdir -p /boot/EFI/Boot
mkdir -p /boot/EFI/gummiboot
cp /usr/lib/gummiboot/gummibootx64.efi /boot/EFI/Boot/bootx64.efi
cp /usr/lib/gummiboot/gummibootx64.efi /boot/EFI/gummiboot/

efibootmgr -c -d /dev/sda -p 1 -L Arch\ Linux\ 1 -l \\EFI\\gummiboot\\gummibootx64.efi
efibootmgr -c -d /dev/sdb -p 1 -L Arch\ Linux\ 2 -l \\EFI\\gummiboot\\gummibootx64.efi

I then used efibootmgr -o to change the boot order and set up the gummiboot boot entries:

mkdir -p /boot/loader/entries
  vi arch1.conf
     title          Arch Linux
     linux         /vmlinuz-linux
     initrd         /initramfs-linux.img
     options      root=/dev/sda1 rw
  vi arch2.conf
     title          Arch Linux
     linux         /vmlinuz-linux
     initrd         /initramfs-linux.img
     options     root=/dev/sdb1 rw

exit; umount -R /mnt; reboot

Last edited by pgoetz (2015-01-06 16:40:48)

brain0 · 2014-03-07 16:18:51

Your root= option is wrong. You need to use root=/dev/md1 (Linux is aware of your RAID, unlike the firmware). You also need the mdadm_udev hook added in mkinitcpio.conf. Of course, fstab must also list /dev/md0 as /boot.

Also, please use \[code\] tags.

pgoetz · 2014-03-10 14:16:02

Awesome -- that worked. Thanks for all your help with this. The PEBKAC in this case is that I had no idea about mkinitcpio. I've read quite a bit of documentation and was surprised to be surprised by this, so went back and checked. There's no mention of this in the Wiki Beginner's Guide (that's probably OK), but it is covered briefly in the Installation Guide, which I overlooked.

One point: it wasn't clear where mdadm_udev should go in the hooks list. The /etc/mkinitcpio.conf file itself states that order is important, but then there doesn't seem to be any suggestion on the mkinitcpio Wiki page what the appropriate order is (other than what you can figure out using common sense; e.g. obviously udev needs to come before mdadm_udev, and blocks likely does as well). In fact, the mkinitcpio Wiki page doesn't even mention that the order matters; I assumed it didn't until I opened the conf file for editing. Either an order hierarchy (or text explicitly stating that the order of hooks listed IS in the correct hierarchy) would be useful; at the very least mentioning that order matters. I would do this myself, but don't know the answer.

For the benefit of anyone using this forum post for hints on how to configure this, I'll append one more reply detailing every step necessary to get a system with such a configuration (Mirrored disks, where the EFI boot partition is mirrored as well) working.

pgoetz · 2014-03-10 14:56:17

Step by Step Summary:

System: Intel Z87 i5 with two 105GB SSD disks

Boot Arch ISO somehow (I used a USB stick)
Use gdisk to create partitions on /dev/sda:
- 512MB EF02 partition (for EFI boot)
- 102GB linux partition
- 8 GB swap partition

Copy the /dev/sda partition table to /dev/sdb:

sgdisk --backup=table /dev/sda
sgdisk --load-backup=table /dev/sdb

Create the RAIDs. The EFI boot partition must be created using an older version of the mdadm metadata:

mdadm --create /dev/md0 --metadata 1.0 --raid-devices=2 --level=1 /dev/sd[ab]1
mdadm --create /dev/md1 --raid-devices=2 --level=1 /dev/sd[ab]2

Once the RAIDs have been created, you need to create appropriate filesystems on the devices (the EFI boot partition must be Fat32).
```
mkfs.fat -F32 /dev/md0
mkfs.ext4 /dev/md1
```

Get your swap on:

mkswap /dev/sda3
mkswap /dev/sdb3
swapon /dev/sda3
swapon /dev/sdb3

Mount the disk partitions for software installation and configuration:
```
mount /dev/md1 /mnt
mkdir /mnt/boot
mount /dev/md0 /mnt/boot
```
Install the Arch base:
```
pacstrap -i /mnt base
```
Create the /etc/mdadm.conf and /etc/fstab configuration files:
```
mdadm --detail --scan >> /mnt/etc/mdadm.conf
genfstab -U -p /mnt >> /mnt/etc/fstab
```
Edit /etc/fstab and remove the pri= line from the swap entries in /etc/fstab in order to equalize the swap partitions
chroot to the disk (note that arch-root is shell script which does some other stuff for you)
```
arch-chroot /mnt /bin/bash
```
Set up locale, hardware clock, hostname, and networking as outlined in the Beginner's Guide
Install the gummiboot package:
```
pacman -S gummiboot
```

The usual automated gummiboot install (e.g. gummiboot install) will fail if you try it, as gummiboot doesn't know about RAID. Instead, install by hand:

mkdir -p /boot/EFI/gummiboot
mkdir -p /boot/EFI/Boot # [optional]
cp /usr/lib/gummiboot/gummibootx64.efi /boot/EFI/gummiboot/
cp /usr/lib/gummiboot/gummibootx64.efi /boot/EFI/Boot/bootx64.efi  # [optional]
efibootmgr -c -d /dev/sda -p 1 -L Arch\ Linux\ 1 -l \\EFI\\gummiboot\\gummibootx64.efi
efibootmgr -c -d /dev/sdb -p 1 -L Arch\ Linux\ 2 -l \\EFI\\gummiboot\\gummibootx64.efi

(The lines marked [optional] above install a "default loader" and aren't strictly necessary. However, having it here is a good fallback if the firmware forgets about its boot settings.)

Most likely the EFI boot entries won't be in the order you want, at this point. You can check using
```
efibootmgr
```
and then change the order using; for example:
```
efibootmgr -o 0003,0004,0000
```
Now manually create configuration files for the gummiboot manager:
```
mkdir -p /boot/loader/entries
```
```
vi /boot/loader/entries/arch1.conf
```
title Arch Linux
linux /vmlinuz-linux
initrd /initramfs-linux.img
options root=/dev/md1 rw
Finally, you must add mdadm_udev to the list of hooks in /etc/mkinitcpio.conf:
HOOKS="base udev autodetect modconf block mdadm_udev filesystems keyboard fsck"
```
mkinitcpio -p linux
```
Exit from chroot, unmount the partitions, and reboot, as explained in the Beginner's Guide.
The system will now boot with a mirrored EFI boot partition

Gummiboot has now been tricked into booting the system from one of the boot partitions (whichever you put first in the EFI boot hierarchy), and if one of the disks fails, the system should just proceed to boot the other one (even if the failed disk was the primary boot device). Disclaimer I haven't actually tried failing a disk to see if this works.

Edit (2014-05-03): the original instructions had filesystems before mdadm_udev in the list of HOOKS:
HOOKS="base udev autodetect modconf block filesystems mdadm_udev keyboard fsck"
but some users have reported that their system won't boot unless this order is reversed

Last edited by pgoetz (2015-01-06 16:41:51)

brain0 · 2014-03-10 15:28:34

pgoetz wrote:

mdadm --create /dev/md0 --metadata 1.0 --raid-devices=2 level=1 /dev/sd[ab]1
mdadm --create /dev/md1 --raid-devices=2 level=1 /dev/sd[ab]2

Add the '-b internal' option to create an internal bitmap - apparently, this speeds up some raid recovery cases. You can add the bitmap later as well, but I forgot how (mdadm manpage contains is somewhere).

mount -t efivarfs efivarfs /sys/firmware/efi/efivars

arch-chroot nowadays mounts efivarfs.

The usual automatic gummiboot will fail if you try it, as gummiboot doesn't know about RAID. Instead, install by hand:
efibootmgr -c -L Arch\ Linux\ 1 -l \\EFI\\gummiboot\\gummibootx64.efi -d /dev/  sda -p 1
efibootmgr -c -L Arch\ Linux\ 2 -l \\EFI\\gummiboot\\gummibootx64.efi -d /dev/  sdb -p 1

gummiboot will also fail to copy its files. You forgot to add this here, but you must manually copy gummibootx64.efi to the right path.

Finally, you must add mdadm_udev to the list of hooks in /etc/mkinitcpio.conf:
HOOKS="base udev autodetect modconf block filesystems mdadm_udev keyboard fsck"
mkinitcpio -p linux

The order of the hooks matters when you have runtime hooks. Most hooks nowadays simply add udev rules and other magic, so their order does not matter. General rule of thumb: Have base first, and udev before all other hooks that add runtime hooks (in your case, no hook adds a runtime hook).

Gummiboot should then be tricked into booting the system even if one of the disks fails, but you will need to select the correct EFI boot entry

The firmware will fall back to the second entry when the first fails. It should be pretty automatic (however, RAID assembly is probably not automatic when a device is missing, but not marked as failed).

pgoetz · 2014-03-10 18:27:16

brain0 wrote:

Add the '-b internal' option to create an internal bitmap - apparently, this speeds up some raid recovery cases. You can add the bitmap later as well, but I forgot how (mdadm manpage contains is somewhere).

I researched this after you mentioned it the first time, and what I could find seemed to lean towards not using internal write-intent bitmaps; especially for RAID1. What a write-intent bitmap does is that you only have to check dirty bits when a device is removed and then re-attached to a RAID (similar to how a ext4 journal works), which dramatically speeds up the process of re-attaching a large disk. This is much more likely to happen with a large RAID5/6 (i.e. one with many disks), and not so much with a RAID1. If a disk in a RAID fails, the entire disk has to be rebuilt -- the bitmap doesn't buy you anything.

On the other hand, having an internal bitmap can dramatically slow down write performance. See here and here. If you have lots of large disks in a RAID5/6 (I manage one system with 72 3TB disks, for example) an external bitmap seems like a very good idea indeed; particularly since you don't incur any loss of write performance. For storage systems this large, a RAID hardware card with battery backup is probably still the best bet. The big system I mentioned uses an LSI logic controller which does all of this for you.

In any case, I didn't use the -b option when setting up the system, and decided to limit my instructions to exactly what I actually did.

gummiboot will also fail to copy its files. You forgot to add this here, but you must manually copy gummibootx64.efi to the right path.

Good catch. I'm not sure how I overlooked this from my notes, but I edited the instructions to include this. I also forgot to include the step for formatting the /dev/mdX partitions, and added this as well, along with some clarifying comments.

I noticed that you wrote or were involved in writing mkinitcpio? Thanks for doing this. In reading the mkinitcpio Wiki page I realized that this will solve another thorny problem we have at work which we've been scratching our heads over how to solve for about a year. Just one more reason to move all 300+ systems from Ubuntu to Arch!

Last edited by pgoetz (2014-03-10 18:28:07)

ms · 2014-05-02 20:24:30

First, thanks pgoetz for your write up on setting up RAID on UEFI machine, its help me a lot.

Second, in my setup the right setting for mdadm_udev hooks is before filesystems, like this

HOOKS="base udev autodetect modconf block mdadm_udev filesystems keyboard fsck"

Reason: Linux check for RAID partition first and then mount it using based on what file system reside on RAID partitions.

pgoetz · 2014-05-02 21:16:43

ms wrote:

Second, in my setup the right setting for mdadm_udev hooks is before filesystems, like this
HOOKS="base udev autodetect modconf block mdadm_udev filesystems keyboard fsck"
Reason: Linux check for RAID partition first and then mount it using based on what file system reside on RAID partitions.

Question: were you not able to boot with the

HOOKS="base udev autodetect modconf block filesystems mdadm_udev keyboard fsck"

set up like this, with filesystems and mdadm_udev reversed?

ms · 2014-05-02 22:46:23

Question: were you not able to boot with the
HOOKS="base udev autodetect modconf block filesystems mdadm_udev keyboard fsck"
set up like this, with filesystems and mdadm_udev reversed?

Yes, linux can't find the root partititon.

pgoetz · 2014-05-03 09:11:37

ms wrote:

Yes, linux can't find the root partititon.

That's weird. On my system I have it set up as per the original step by step instructions, and it boots fine. Much of the credit for this setup goes to brain0. I ask him about the order of the mkinitcpio hooks, and this was his response:

brain0 wrote:

The order of the hooks matters when you have runtime hooks. Most hooks nowadays simply add udev rules and other magic, so their order does not matter. General rule of thumb: Have base first, and udev before all other hooks that add runtime hooks (in your case, no hook adds a runtime hook).

I'll make a note of this in the step by step instructions above in case someone else runs in to this.

ms · 2014-05-03 09:16:20

pgoetz wrote:

That's weird. On my system I have it set up as per the original step by step instructions, and it boots fine

You can test it by yourself, place "mdadm_udev" after "filesystems", run "mkinitcpio -p linux", and reboot.

Let see if your machine can boot or not, FOR SCIENCE!

edit: oh, obviously you already set it like that. Ignore my message.

Last edited by ms (2014-05-03 09:17:53)

Arch Linux

#1 2014-02-26 15:01:29

[SOLVED] gummiboot + mdadm RAID 1?

#2 2014-02-26 15:54:40

Re: [SOLVED] gummiboot + mdadm RAID 1?

#3 2014-02-26 16:32:59

Re: [SOLVED] gummiboot + mdadm RAID 1?

#4 2014-02-27 09:17:06

Re: [SOLVED] gummiboot + mdadm RAID 1?

#5 2014-03-01 01:18:19

Re: [SOLVED] gummiboot + mdadm RAID 1?

#6 2014-03-07 14:47:40

Re: [SOLVED] gummiboot + mdadm RAID 1?

#7 2014-03-07 16:18:51

Re: [SOLVED] gummiboot + mdadm RAID 1?

#8 2014-03-10 14:16:02

Re: [SOLVED] gummiboot + mdadm RAID 1?

#9 2014-03-10 14:56:17

Re: [SOLVED] gummiboot + mdadm RAID 1?

Step by Step Summary:

#10 2014-03-10 15:28:34

Re: [SOLVED] gummiboot + mdadm RAID 1?

#11 2014-03-10 18:27:16

Re: [SOLVED] gummiboot + mdadm RAID 1?

#12 2014-05-02 20:24:30

Re: [SOLVED] gummiboot + mdadm RAID 1?

#13 2014-05-02 21:16:43

Re: [SOLVED] gummiboot + mdadm RAID 1?

#14 2014-05-02 22:46:23

Re: [SOLVED] gummiboot + mdadm RAID 1?

#15 2014-05-03 09:11:37

Re: [SOLVED] gummiboot + mdadm RAID 1?

#16 2014-05-03 09:16:20

Re: [SOLVED] gummiboot + mdadm RAID 1?

Board footer