[Solved] grub-install on IMSM RAID

nickz · 2014-02-04 09:11:09

Hi I'm having trouble with grub-install on an IMSM (now called IRST) RAID system on an Intel DQ77MK motherboard.

I created a RAID container with 2 x 1TB discs then a RAID 1 volume inside the container with LVM on top of that. I installed Archlinux to the Logical Volume from a USB stick but can't make grub-install install to the boot sector of the RAID drive.

The basic problem is that IMSM RAID will only use whole discs, not partitions, but grub-install will not install on a partition-less disc. Creating partitions then creating the RAID volumes seems to cause irreparable damage to the file-systems.

Here's what I've tried.

usb# fdisk -l /dev/sda

Disk /dev/sda: 931.5 GiB, 1000204886016 bytes, 1953525168 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 4096 bytes
I/O size (minimum/optimal): 4096 bytes / 4096 bytes

usb#

/dev/sde is the same

Start with a clean slate

usb# mdadm --zero-superblock --metadata=imsm /dev/sda
usb# mdadm --zero-superblock --metadata=imsm /dev/sde
usb# mdadm --zero-superblock /dev/sda
mdadm: Unrecognised md component device - /dev/sda
usb# mdadm --zero-superblock /dev/sde
mdadm: Unrecognised md component device - /dev/sde

Maybe not that clean

usb#  mdadm -v --create /dev/md/imsm0 --metadata=imsm --level=c
 ontainer --raid-devices=2 /dev/sda /dev/sde
mdadm: /dev/sda appears to be part of a raid array:
       level=raid0 devices=0 ctime=Thu Jan  1 01:00:00 1970
mdadm: /dev/sde appears to be part of a raid array:
       level=raid0 devices=0 ctime=Thu Jan  1 01:00:00 1970
mdadm: size set to 976759431K
Continue creating array? y
mdadm: container /dev/md/imsm prepared.
usb# mdadm --create /dev/md/md0 --size=900G --level=1 --raid-de
 vices=2 /dev/md/imsm
mdadm: internal bitmaps not supported with imsm metadata
usb# mdadm --create /dev/md/md0 --size=900G --level=1 --raid-de
 vices=2 --bitmap=none /dev/md/imsm
mdadm: array /dev/md/md0 started.
usb# mdadm --create /dev/md/md1 --level=0 --raid-devices=2 /dev
 /md/imsm
mdadm: array /dev/md/md1 started.

usb# ls -l /dev/md/*
lrwxrwxrwx 1 root root 8 Feb  3 11:27 imsm -> ../md127
lrwxrwxrwx 1 root root 8 Feb  3 11:28 md0 -> ../md126
lrwxrwxrwx 1 root root 8 Feb  3 11:28 md1 -> ../md125
usb#

LVM. Array md1 remains unused.

usb# pvcreate /dev/md/md0
  Physical volume "/dev/md/md0" successfully created
usb# vgcreate vg0 /dev/md/md0
  Volume group "vg0" successfully created
usb# lvcreate --size 15G vg0 --name archlinux
  Logical volume "archlinux" created
usb# mkfs -t ext4 /dev/mapper/vg0-archlinux
usb# mount /mnt/vg0-archlinux /dev/mapper/vg0-archlinux

Here I either install Archlinux on /mnt/vg0-archlinux from the USB stick following these instructions https://wiki.archlinux.org/index.php/Installation_Guide or copy an already installed instance.

usb# arch-chroot /mnt/vg0-archlinux/
sh-4.2# egrep '^BINA|^HOOK' /etc/mkinitcpio.conf
BINARIES="mdmon"
HOOKS="base udev mdadm_udev block autodetect modconf lvm2 filesystems keyboard fsck shutdown"
sh-4.2#

This is the filtered output of grub-mkconfig

sh-4.2# grub-mkconfig -o /boot/grub/grub.cfg
Generating grub configuration file ...
Found linux image: /boot/vmlinuz-linux
Found initramfs image: /boot/initramfs-linux.img
Found fallback initramfs image: /boot/initramfs-linux-fallback.img
Found linux image: /boot/vmlinuz-linux
Found initrd image: /boot/initramfs-linux.img
done
sh-4.2#

What was taken out? Dozens of these

  Found duplicate PV qzPdVyPkmcMmipAy88CyrD5eECTibkwP: using /dev/md126 not /dev/sde
  Found duplicate PV qzPdVyPkmcMmipAy88CyrD5eECTibkwP: using /dev/sde not /dev/sda

Dozens of these

  WARNING: Failed to connect to lvmetad: No such file or directory. Falling back to internal scanning.

Thousands of these

  /run/lvm/lvmetad.socket: connect failed: No such file or directory

sh-4.2# mkinitcpio -Mp linux
==> Building image from preset: /etc/mkinitcpio.d/linux.preset: 'default'
  -> -k /boot/vmlinuz-linux -c /etc/mkinitcpio.conf -g /boot/initramfs-linux.img
==> Starting build: 3.12.8-1-ARCH
  -> Running build hook: [base]
  -> Running build hook: [udev]
  -> Running build hook: [mdadm_udev]
Custom /etc/mdadm.conf file will be used in initramfs for assembling arrays.
  -> Running build hook: [autodetect]
  -> Running build hook: [modconf]
  -> Running build hook: [block]
  -> Running build hook: [lvm2]
  -> Running build hook: [filesystems]
  -> Running build hook: [keyboard]
  -> Running build hook: [fsck]
  -> Running build hook: [shutdown]
==> Generating module dependencies
==> Creating gzip initcpio image: /boot/initramfs-linux.img
==> Image generation successful
==> Building image from preset: /etc/mkinitcpio.d/linux.preset: 'fallback'
  -> -k /boot/vmlinuz-linux -c /etc/mkinitcpio.conf -g /boot/initramfs-linux-fallback.img -S autodetect
==> Starting build: 3.12.8-1-ARCH
  -> Running build hook: [base]
  -> Running build hook: [udev]
  -> Running build hook: [mdadm_udev]
Custom /etc/mdadm.conf file will be used in initramfs for assembling arrays.
  -> Running build hook: [modconf]
  -> Running build hook: [block]
==> WARNING: Possibly missing firmware for module: aic94xx
==> WARNING: Possibly missing firmware for module: smsmdtv
  -> Running build hook: [lvm2]
  -> Running build hook: [filesystems]
  -> Running build hook: [keyboard]
  -> Running build hook: [fsck]
  -> Running build hook: [shutdown]
==> Generating module dependencies
==> Creating gzip initcpio image: /boot/initramfs-linux-fallback.img
==> Image generation successful
sh-4.2#

Try to install grub

sh-4.2# grub-install /dev/md/md0 2>&1 |egrep -v 'duplicate|lvmetad'          
Installing for i386-pc platform.
grub-install: error: unable to identify a filesystem in hostdisk//dev/md126; safety check can't be performed.
sh-4.2#
sh-4.2# grub-install /dev/md/imsm 2>&1 |egrep -v 'duplicate|lvmetad'
Installing for i386-pc platform.
grub-install: error: attempt to read or write outside of disk `hostdisk//dev/md127'.
sh-4.2#
sh-4.2# grub-install --skip-fs-probe /dev/md/md0 2>&1 |egrep -v 'duplicate|lvmetad'
Installing for i386-pc platform.
grub-install: warning: Attempting to install GRUB to a partitionless disk or to a partition.  This is a BAD idea..
grub-install: error: embedding is not possible, but this is required for RAID and LVM install.
sh-4.2#
sh-4.2# grub-install --force --skip-fs-probe /dev/md/md0 2>&1 |egrep -v 'duplicate|lvmetad'
Installing for i386-pc platform.
grub-install: warning: Attempting to install GRUB to a partitionless disk or to a partition.  This is a BAD idea..
grub-install: error: embedding is not possible, but this is required for RAID and LVM install.
sh-4.2#

It should be really simple: boot off the RAID disc shown in the BIOS & run the stuff in this LVM /boot directory but I can't make it work.

Any suggestions would be appreciated. I don't care which boot-loader is used. At the moment I want to be able to boot from the Archlinux installation, then once I'm sure it's stable, restore my Fedora installation, home dirs etc. Then I would like to add a couple more hard drives & convert the RAID 1 volume (md0) to RAID 5. After that, some sort of hardware virtualization to run virtual machines under Archlinux.

Some details of the RAID volumes

usb # egrep -v '^$|^#' /etc/mdadm.conf
ARRAY metadata=imsm UUID=d69a760b:857b540c:4af2771b:ab6d6941
ARRAY /dev/md/md0 container=d69a760b:857b540c:4af2771b:ab6d6941 member=0 UUID=b9af4b93:45657783:bdfae4e6:0731ba36
ARRAY /dev/md/md1 container=d69a760b:857b540c:4af2771b:ab6d6941 member=1 UUID=ac140322:14cf8d1c:1b3e1164:19b705cd
usb #

usb # mdadm -E /dev/md/imsm
/dev/md/imsm:
          Magic : Intel Raid ISM Cfg Sig.
        Version : 1.3.00
    Orig Family : 9a3c9ed7
         Family : 9a3c9ed7
     Generation : 00000136
     Attributes : All supported
           UUID : 038e4945:696c1d54:5d0e4160:0a539ffb
       Checksum : a89b83b5 correct
    MPB Sectors : 2
          Disks : 2
   RAID Devices : 2

  Disk00 Serial : S13PJDWS608386
          State : active
             Id : 00000004
    Usable Size : 1953518862 (931.51 GiB 1000.20 GB)

[md0]:
           UUID : 559d862c:7cd2b4bf:d2ceb7ae:1a787744
     RAID Level : 1
        Members : 2
          Slots : [UU]
    Failed disk : none
      This Slot : 0
     Array Size : 1887436800 (900.00 GiB 966.37 GB)
   Per Dev Size : 1887436800 (900.00 GiB 966.37 GB)
  Sector Offset : 0
    Num Stripes : 7372800
     Chunk Size : 64 KiB
       Reserved : 0
  Migrate State : idle
      Map State : normal
    Dirty State : clean

[md1]:
           UUID : 3745ffaf:951b66ac:a99d26e4:7be4c2db
     RAID Level : 0
        Members : 2
          Slots : [UU]
    Failed disk : none
      This Slot : 0
     Array Size : 132155392 (63.02 GiB 67.66 GB)
   Per Dev Size : 66077952 (31.51 GiB 33.83 GB)
  Sector Offset : 1887440896
    Num Stripes : 258117
     Chunk Size : 128 KiB
       Reserved : 0
  Migrate State : idle
      Map State : normal
    Dirty State : clean

  Disk01 Serial : WD-WCC1S5684189
          State : active
             Id : 00000000
    Usable Size : 1953518862 (931.51 GiB 1000.20 GB)
usb #

Unsure how relevant this is but the Intel Rapid Storage Technology enterprise (Intel RSTe) for Linux OS guide offers this sample output, with the ports listed.

mdadm -–detail-platform
Platform : Intel(R) Matrix Storage Manager
Version : 8.9.0.1023
RAID Levels : raid0 raid1 raid10 raid5
Chunk Sizes : 4k 8k 16k 32k 64k 128k
Max Disks : 6
Max Volumes : 2
I/O Controller : /sys/devices/pci0000:00/0000:00:1f.2
Port0 : /dev/sda (3MT0585Z)
Port1 : - non-disk device (ATAPI DVD D DH16D4S) -
Port2 : /dev/sdb (WD-WCANK2850263)
Port3 : /dev/sdc (3MT005ML)
Port4 : /dev/sdd (WD-WCANK2850441)
Port5 : /dev/sde (WD-WCANK2852905)
Port6 : - no device attached –

but this is what I see

usb # mdadm --detail-platform --metadata=imsm -v
mdmon: found Intel(R) SATA RAID controller at 0000:00:1f.2.
       Platform : Intel(R) Matrix Storage Manager
        Version : 11.6.0.1702
    RAID Levels : raid0 raid1 raid10 raid5
    Chunk Sizes : 4k 8k 16k 32k 64k 128k
    2TB volumes : supported
      2TB disks : supported
      Max Disks : 6
    Max Volumes : 2 per array, 4 per controller
 I/O Controller : /sys/devices/pci0000:00/0000:00:1f.2 (SATA)
mdmon: failed to determine port number for /sys/devices/pci0000:00/0000:00:1f.2/ata1/host0/target0:0:0/0:0:0:0
mdmon: failed to enumerate ports on SATA controller at 0000:00:1f.2.
usb #

* I'm marking this as solved though there are some unanswered questions in my two posts.

Last edited by nickz (2014-02-19 20:32:56)

frostschutz · 2014-02-04 14:39:00

If there is no Windows on this machine, why not mdadm SoftwareRAID instead of the IMSM stuff?

And usually you'd partition such a raid, if you want to boot off it. I'm not sure if there is a place for Grub in a LVM PV (assuming any IMSM metadata is at the end). A PV only has a single 512 byte block at the beginning...

Last edited by frostschutz (2014-02-04 14:44:49)

benjamin · 2014-02-04 15:11:06

I was just replying to say the same thing. Pure mdadm would be much more flexible, as well as give you the option to transfer the drives and boot the array in another machine if necessary.

nickz · 2014-02-14 11:37:54

Thank you both (and to lilsirecho) for your replies. Firmware RAID has one advantage over software RAID in that it assembles the volumes before boot, which allows the bootloader to reside on RAID and means you don't have to deal with physical discs from inside the o/s (though you can). The Intel BIOS/OROM, basic though it is, allows you to add and remove RAID discs before any o/s starts. I also wanted to use it just because it was there.

Frostschutz, I followed your suggestion that "usually you'd partition such a raid" and now it looks like the right and probably only way to do it. There's a section in man mdadm about partitions. Currently it's all working so here I've written down the steps to take, using the ARCH201402 iso.

This time I used UEFI boot with a GPT (GUID partition table) because I was unsure whether otherwise the RAID volume would be limited to 2TB (opinions?) but apart from that I expect it would work just as well with BIOS and MBR.

After creating and deleting arrays with every type of metadata in the hope of clearing the phantom metadata shown in the first post, I couldn't get past this error

usb# mdadm -v --create /dev/md/imsm --metadata=imsm --level=container --raid-devices=2 /dev/sda /dev/sde
mdadm: /dev/sda is not suitable for this array.
mdadm: /dev/sde is not suitable for this array.
mdadm: create aborted

so I did the next 2 steps from Fedora, which didn't object (and had the same version 3.3 of mdadm). I expect the Intel OROM could also have been used to create the container.

Create the container

fedora# mdadm --create /dev/md/imsm --metadata=imsm --raid-devices=2 /dev/sda /dev/sde
mdadm: /dev/sda appears to be part of a raid array:
mdadm: /dev/sde appears to be part of a raid array:
mdadm: container /dev/md/imsm prepared.

Create the raid volumes
No "internal bitmaps not supported with imsm metadata" error as in the previous attempt, presumably because it's RAID 0 instead of RAID 1. The route to RAID 5 in the Intel guide is 1 → 0 → 5 so starting from 0 makes more sense. The size is 900GB per raid-device. The 2nd volume uses all the remaining space which is the only way it works. Only 1 or 2 volumes are allowed.

fedora# mdadm --create /dev/md/md0 --size=900G --level=0 --raid-devices=2 /dev/md/imsm
fedora# mdadm --create /dev/md/md1 --level=0 --raid-devices=2 /dev/md/imsm

Partition the first raid volume with gdisk:
Create a GPT.
Make a 512MB EFI partition of type EF00.
Make a second partition to cover the rest of the RAID volume, type 8E00 (LVM2).
This is what it looks like.

usb# gdisk -l /dev/md/md0
GPT fdisk (gdisk) version 0.8.8
...
Total free space is 2014 sectors (1007.0 KiB)

Number  Start (sector)    End (sector)  Size       Code  Name
   1            2048         1050623   512.0 MiB   EF00  EFI System
   2         1050624      3774873566   1.8 TiB     8E00  Linux LVM
usb#  ls -l /dev/md/*
lrwxrwxrwx 1 root root 8 Feb 13 15:24 /dev/md/imsm -> ../md127
lrwxrwxrwx 1 root root 6 Feb 13 15:24 /dev/md/md0 -> ../md0
lrwxrwxrwx 1 root root 8 Feb 13 15:24 /dev/md/md0p1 -> ../md0p1
lrwxrwxrwx 1 root root 8 Feb 13 15:24 /dev/md/md0p2 -> ../md0p2
lrwxrwxrwx 1 root root 6 Feb 13 15:24 /dev/md/md1 -> ../md1

Format the EFI partition as FAT32

usb# mkfs.fat -F32 /dev/md/md0p1

Create logical volumes on the LVM2 partition.

usb# pvcreate /dev/md/md0p2
usb# pvs /dev/md/md0p2
usb# vgcreate vg0 /dev/md/md0p2
usb# lvcreate --size 15G vg0 --name archlinux

Install Arch Linux on a logical volume as in the Installation Guide but use the following steps where necessary. These start after the Arch Linux volume is mounted and you've arch-chrooted into it.

Mount the EFI partition on /boot/efi

arch# mkdir -p /boot/efi
arch# mount /dev/md/md0p1 /boot/efi
arch# mount|grep boot
/dev/md0p1 on /boot/efi type vfat (rw,relatime,fmask=0022,dmask=0022,codepage=437,iocharset=iso8859-1,shortname=mixed,errors=remount-ro)

Add it to /etc/fstab. The safest way is to use the UUID

arch# grep fat /etc/fstab
UUID=FBDE-031F	/boot/efi	vfat	rw,relatime,fmask=0022,dmask=0022,codepage=437,iocharset=iso8859-1,shortname=mixed,errors=remount-ro   0 2

Add the RAID details to mdadm.conf for persistent names.

arch# mdadm --detail --scan >> /etc/mdadm/mdadm.conf

Edit the names in mdadm.conf if required.

Configure /etc/mkinitcpio.conf for mdadm & lvm2. I omitted the "shutdown" keyword because of the following extract from the Arch Linux guide to mkinitcpio; it doesn't say that "shutdown" is not necessary for mdadm but I haven't seen any RAID rebuilds since omitting it.

mdadm_udev
Provides support for assembling RAID arrays via udev. You must have mdadm installed to use this. If you use this hook with a FakeRAID array, it is recommended to include mdmon in the binaries section and add the shutdown hook in order to avoid unnecessary RAID rebuilds on reboot.
...
shutdown
Adds shutdown initramfs support. Usage of this hook was strongly recommended before mkinitcpio 0.16, if you have a separate /usr partition or encrypted root. From mkinitcpio 0.16 onwards, it is deemed not necessary.

arch# mkinitcpio -V
mkinitcpio 16

I added the "mdmon" binary despite reading somewhere (where?) that it's not needed here.

arch# egrep -v '^#|^$' /etc/mkinitcpio.conf
MODULES=""
BINARIES="mdmon"
FILES=""
HOOKS="base udev mdadm_udev autodetect modconf block lvm2 filesystems keymap keyboard fsck"

Create the initial ramdisk

arch# mkinitcpio -Mp linux

Install these packages.

arch# pacman -S grub dosfstools efibootmgr

Install grub to the default /boot/grub directory. grub-install uses efibootmgr to make a new, default boot menu item whose name is set with –-bootloader-id

arch# grub-install --target=x86_64-efi --efi-directory=/boot/efi –-bootloader-id="Archlinux grub" --debug > grub-install.debug.out 2>&1

arch# grub-mkconfig -o /boot/grub/grub.cfg

Exit arch-chroot
Reboot.

The new entry should now be in the UEFI boot menu and you should be able to launch Arch Linux.

No menu item
In my case the menu entry wasn't there because this Q77 motherboard suffers (well I suffer – it doesn't care) from the problem described in the Arch Linux Boot Loaders page where the firmware looks here for the file to load
\EFI\boot\bootx64.efi
but grub-install has put it here
\EFI\grub\grubx64.efi

To fix this you can either copy the .efi file so it enables the menu item

arch# cp -p /boot/efi/EFI/grub/grubx64.efi /boot/efi/EFI/boot/bootx64.efi

or move it

arch# mv /boot/efi/EFI/grub/grubx64.efi /boot/efi/EFI/boot/bootx64.efi

then install a new menu item with efibootmgr

arch# efibootmgr -c -d /dev/md/md0 -p 1 -w -L "ArchLinux grub 2" -l '\EFI\boot\bootx64.efi'

arch# efibootmgr -v|grep ArchLinux
Boot0001* ArchLinux grub        HD(1,800,100000,0e0d813f-2c3e-4963-98ac-24b97247b51a)File(\EFI\grub\grubx64.efi)
Boot0002* ArchLinux grub 2      HD(1,800,100000,0e0d813f-2c3e-4963-98ac-24b97247b51a)File(\EFI\boot\bootx64.efi)

This post from the Ubuntu Forums puts the path to the efi file in the --target option for grub-install but I haven't tried it. Does anyone know where this is documented?

grub-install --bootloader-id ubuntu /dev/sda  --target /boot/efi/EFI/ubuntu --directory=/usr/local/lib/grub/x86_64-efi/

When the thrill of everything working has worn off, I'll find a simpler bootloader. I might also investigate the portability of this RAID, when it will be too late to change.

Finally, looking at the RAID disc devices (/dev/sda etc) with fdisk or gdisk can give worrying results. Don't do it.

Further reading

man pages, especially mdadm
https://wiki.archlinux.org/index.php/Installation_Guide
https://wiki.archlinux.org/index.php/RAID
https://wiki.archlinux.org/index.php/Mkinitcpio#HOOKS
https://raid.wiki.kernel.org/index.php/ … l_Metadata
https://wiki.archlinux.org/index.php/GRUB#UEFI_systems
https://wiki.archlinux.org/index.php/Un … _Interface
http://download.intel.com/support/chips … _linux.pdf

kamz · 2014-07-30 00:13:46

Thank you! You are awesome.

Arch Linux

#1 2014-02-04 09:11:09

[Solved] grub-install on IMSM RAID

#2 2014-02-04 14:39:00

Re: [Solved] grub-install on IMSM RAID

#3 2014-02-04 15:11:06

Re: [Solved] grub-install on IMSM RAID

#4 2014-02-14 11:37:54

Re: [Solved] grub-install on IMSM RAID

#5 2014-07-30 00:13:46

Re: [Solved] grub-install on IMSM RAID

Board footer