[SOLVED] Difficulty with GRUB booting RAID1 on Inspiron 7720se

stubble · 2015-10-06 21:02:49

First post, apologies if I miss information or get the style/location of posting incorrect. Background information which may be pertinent:

This system has been installed and running arch happily for a couple of years now, and I recently added a second SATA drive to expand to a software RAID1 arrangement. The drives are:

/dev/sda -> 1tb sata (msdos partition table, the original SATA drive)
/dev/sdb -> 1tb sata (msdos partition table, the new SATA drive)
/dev/sdc -> 32gb ssd (gpt partition table, the original SSD)

Articles I've been following along with and/or cross-referencing extensively:
https://wiki.archlinux.org/index.php/Co … em_to_RAID
https://wiki.archlinux.org/index.php/Beginners'_guide
https://wiki.archlinux.org/index.php/Installation_guide
https://wiki.archlinux.org/index.php/Partitioning
https://wiki.archlinux.org/index.php/GRUB
https://wiki.archlinux.org/index.php/RAID

Output of lsblk before I started:

NAME          MAJ:MIN RM   SIZE RO TYPE  MOUNTPOINT
sda             8:0    0 931.5G  0 disk
├─sda1          8:1    0   256M  0 part  /boot
├─sda2          8:2    0     1K  0 part
├─sda5          8:5    0   893G  0 part  /home
├─sda6          8:6    0    20G  0 part  /tmp
├─sda7          8:7    0    10G  0 part  /var
└─sda8          8:8    0   8.3G  0 part  [SWAP]
sdb             8:32   0  29.8G  0 disk
└─sdb1          8:33   0  29.8G  0 part  /
sr0            11:0    1  1024M  0 rom

Output of lsblk currently:

NAME          MAJ:MIN RM   SIZE RO TYPE  MOUNTPOINT
sda             8:0    0 931.5G  0 disk
├─sda1          8:1    0   256M  0 part
├─sda2          8:2    0     1K  0 part
├─sda5          8:5    0   893G  0 part
├─sda6          8:6    0    20G  0 part
├─sda7          8:7    0    10G  0 part
└─sda8          8:8    0   8.3G  0 part
sdb             8:16   0 931.5G  0 disk
├─sdb1          8:17   0 923.2G  0 part
│ └─md126       9:127  0 923.1G  0 raid1
│   ├─md126p1 259:0    0   256M  0 md    /boot
│   ├─md126p2 259:1    0 862.8G  0 md    /home
│   ├─md126p3 259:2    0    40G  0 md    /var
│   └─md126p4 259:3    0    20G  0 md    /tmp
└─sdb2          8:18   0   8.3G  0 part
  └─md127       9:126  0   8.3G  0 raid1
    └─md127p1 259:4    0   8.3G  0 md    [SWAP]
sdc             8:32   0  29.8G  0 disk
└─sdc1          8:33   0  29.8G  0 part  /
sr0            11:0    1  1024M  0 rom

Contents of fstab currently:

# /dev/sdc1
UUID=b7ecd501-462f-48e9-b63b-e4aa25321e25 /           btrfs       rw,relatime,ssd,space_cache 0 0

# /dev/md126p2 LABEL=home
UUID=cce3a0ce-1353-46c6-9ad6-b74628d70bdb /home       ext4        rw,relatime,data=ordered  0 2

# /dev/md126p1 LABEL=boot
UUID=ed911b0a-e6ad-46c6-b679-021f18ac6cf0 /boot       ext4        rw,relatime,stripe=4,data=ordered 0 2

# /dev/md126p3 LABEL=var
UUID=4bbf19e6-2dd2-46e2-95ba-6d6be746d391 /var        ext4        rw,relatime,data=ordered  0 2

# /dev/md126p4 LABEL=tmp
UUID=a7e0cf7f-31ad-44af-9382-9c2dd78fa31f /tmp        ext2        rw,relatime 0 2

# /dev/md127p1 LABEL=swap
UUID=89bd8ffa-a9ef-48df-abf6-2ceaa1eefc3c none        swap        defaults    0 0

I've just noticed while researching to write this post that the comments in this file list the partitions for /dev/md126p* and /dev/md127p* the wrong way around, but the UUIDs still seem to match the right partitions so I don't think this is relevant.

Contents of mdadm.conf currently:

DEVICE partitions
ARRAY /dev/md/ruby:0 metadata=1.2 name=ruby:0 UUID=00c162ff:e3b0f060:196a8436:ef9b60d7
ARRAY /dev/md/ruby:1 metadata=1.2 name=ruby:1 UUID=1c5ec78b:a61e05c7:f5584228:d2f19192

Output of cat /proc/mdstat currently:

Personalities : [raid1] 
md126 : active raid1 sdb1[1]
      967917568 blocks super 1.2 [2/1] [_U]
      bitmap: 8/8 pages [32KB], 65536KB chunk

md127 : active (auto-read-only) raid1 sdb2[1]
      8704704 blocks super 1.2 [2/1] [_U]
      
unused devices: <none>

Contents of /etc/default/grub currently:

GRUB_DEFAULT=0
GRUB_TIMEOUT=5
GRUB_DISTRIBUTOR="Arch"
GRUB_CMDLINE_LINUX_DEFAULT=""
GRUB_CMDLINE_LINUX=""
GRUB_DISABLE_SUBMENU=y

# Preload both GPT and MBR modules so that they are not missed
GRUB_PRELOAD_MODULES="part_gpt part_msdos mdraid"

# Uncomment to enable Hidden Menu, and optionally hide the timeout count
#GRUB_HIDDEN_TIMEOUT=5
#GRUB_HIDDEN_TIMEOUT_QUIET=true

# Uncomment to use basic console
GRUB_TERMINAL_INPUT=console

# Uncomment to disable graphical terminal
GRUB_TERMINAL_OUTPUT=console

# The resolution used on graphical terminal
# note that you can use only modes which your graphic card supports via VBE
# you can see them in real GRUB with the command `vbeinfo'
#GRUB_GFXMODE=1920x1080x32

# Uncomment to allow the kernel use the same resolution used by grub
#GRUB_GFXPAYLOAD_LINUX=keep

# Uncomment if you want GRUB to pass to the Linux kernel the old parameter
# format "root=/dev/xxx" instead of "root=/dev/disk/by-uuid/xxx"
#GRUB_DISABLE_LINUX_UUID=true

# Uncomment to disable generation of recovery mode menu entries
GRUB_DISABLE_RECOVERY=true

# Uncomment and set to the desired menu colors.  Used by normal and wallpaper
# modes only.  Entries specified as foreground/background.
#GRUB_COLOR_NORMAL="light-blue/black"
#GRUB_COLOR_HIGHLIGHT="light-cyan/blue"

# Uncomment one of them for the gfx desired, a image background or a gfxtheme
#GRUB_BACKGROUND="/path/to/wallpaper"
#GRUB_THEME="/path/to/gfxtheme"

# Uncomment to get a beep at GRUB start
#GRUB_INIT_TUNE="480 440 1"

#GRUB_SAVEDEFAULT="true"

Contents of "working" grub.cfg on /dev/sda1:

#
# DO NOT EDIT THIS FILE
#
# It is automatically generated by grub-mkconfig using templates
# from /etc/grub.d and settings from /etc/default/grub
#

### BEGIN /etc/grub.d/00_header ###
insmod part_gpt
insmod part_msdos
if [ -s $prefix/grubenv ]; then
  load_env
fi
if [ "${next_entry}" ] ; then
   set default="${next_entry}"
   set next_entry=
   save_env next_entry
   set boot_once=true
else
   set default="0"
fi

if [ x"${feature_menuentry_id}" = xy ]; then
  menuentry_id_option="--id"
else
  menuentry_id_option=""
fi

export menuentry_id_option

if [ "${prev_saved_entry}" ]; then
  set saved_entry="${prev_saved_entry}"
  save_env saved_entry
  set prev_saved_entry=
  save_env prev_saved_entry
  set boot_once=true
fi

function savedefault {
  if [ -z "${boot_once}" ]; then
    saved_entry="${chosen}"
    save_env saved_entry
  fi
}

function load_video {
  if [ x$feature_all_video_module = xy ]; then
    insmod all_video
  else
    insmod efi_gop
    insmod efi_uga
    insmod ieee1275_fb
    insmod vbe
    insmod vga
    insmod video_bochs
    insmod video_cirrus
  fi
}

terminal_input console
terminal_output console
if [ x$feature_timeout_style = xy ] ; then
  set timeout_style=menu
  set timeout=5
# Fallback normal timeout code in case the timeout_style feature is
# unavailable.
else
  set timeout=5
fi
### END /etc/grub.d/00_header ###

### BEGIN /etc/grub.d/10_linux ###
menuentry 'Arch Linux, with Linux linux' --class arch --class gnu-linux --class gnu --class os $menuentry_id_option 'gnulinux-linux-advanced-b7ecd501-462f-48e9-b63b-e4aa25321e25' {
  load_video
  insmod gzio
  insmod part_msdos
  insmod ext2
  set root='hd0,msdos1'
  if [ x$feature_platform_search_hint = xy ]; then
    search --no-floppy --fs-uuid --set=root --hint-bios=hd0,msdos1 --hint-efi=hd0,msdos1 --hint-baremetal=ahci0,msdos1  b24a1fe0-7b08-449d-a14d-3cfe7e5590d3
  else
    search --no-floppy --fs-uuid --set=root b24a1fe0-7b08-449d-a14d-3cfe7e5590d3
  fi
  echo  'Loading Linux linux ...'
  linux /vmlinuz-linux root=UUID=b7ecd501-462f-48e9-b63b-e4aa25321e25 rw
  echo  'Loading initial ramdisk ...'
  initrd  /intel-ucode.img /initramfs-linux.img
}
menuentry 'Arch Linux, with Linux linux (fallback initramfs)' --class arch --class gnu-linux --class gnu --class os $menuentry_id_option 'gnulinux-linux-fallback-b7ecd501-462f-48e9-b63b-e4aa25321e25' {
  load_video
  insmod gzio
  insmod part_msdos
  insmod ext2
  set root='hd0,msdos1'
  if [ x$feature_platform_search_hint = xy ]; then
    search --no-floppy --fs-uuid --set=root --hint-bios=hd0,msdos1 --hint-efi=hd0,msdos1 --hint-baremetal=ahci0,msdos1  b24a1fe0-7b08-449d-a14d-3cfe7e5590d3
  else
    search --no-floppy --fs-uuid --set=root b24a1fe0-7b08-449d-a14d-3cfe7e5590d3
  fi
  echo  'Loading Linux linux ...'
  linux /vmlinuz-linux root=UUID=b7ecd501-462f-48e9-b63b-e4aa25321e25 rw
  echo  'Loading initial ramdisk ...'
  initrd  /intel-ucode.img /initramfs-linux-fallback.img
}

### END /etc/grub.d/10_linux ###

### BEGIN /etc/grub.d/20_linux_xen ###
### END /etc/grub.d/20_linux_xen ###

### BEGIN /etc/grub.d/30_os-prober ###
### END /etc/grub.d/30_os-prober ###

### BEGIN /etc/grub.d/40_custom ###
# This file provides an easy way to add custom menu entries.  Simply type the
# menu entries you want to add after this comment.  Be careful not to change
# the 'exec tail' line above.
### END /etc/grub.d/40_custom ###

### BEGIN /etc/grub.d/41_custom ###
if [ -f  ${config_directory}/custom.cfg ]; then
  source ${config_directory}/custom.cfg
elif [ -z "${config_directory}" -a -f  $prefix/custom.cfg ]; then
  source $prefix/custom.cfg;
fi
### END /etc/grub.d/41_custom ###

### BEGIN /etc/grub.d/60_memtest86+ ###
if [ "${grub_platform}" == "pc" ]; then
    menuentry "Memory Tester (memtest86+)" --class memtest86 --class gnu --class tool {
        search --fs-uuid --no-floppy --set=root --hint-bios=hd0,msdos1 --hint-efi=hd0,msdos1 --hint-baremetal=ahci0,msdos1  b24a1fe0-7b08-449d-a14d-3cfe7e5590d3
        linux16 /memtest86+/memtest.bin
    }
fi
### END /etc/grub.d/60_memtest86+ ###

Contents of not-working (I think) grub.cfg on /dev/md126p1 currently:

#
# DO NOT EDIT THIS FILE
#
# It is automatically generated by grub-mkconfig using templates
# from /etc/grub.d and settings from /etc/default/grub
#

### BEGIN /etc/grub.d/00_header ###
insmod part_gpt
insmod part_msdos
insmod mdraid
if [ -s $prefix/grubenv ]; then
  load_env
fi
if [ "${next_entry}" ] ; then
   set default="${next_entry}"
   set next_entry=
   save_env next_entry
   set boot_once=true
else
   set default="0"
fi

if [ x"${feature_menuentry_id}" = xy ]; then
  menuentry_id_option="--id"
else
  menuentry_id_option=""
fi

export menuentry_id_option

if [ "${prev_saved_entry}" ]; then
  set saved_entry="${prev_saved_entry}"
  save_env saved_entry
  set prev_saved_entry=
  save_env prev_saved_entry
  set boot_once=true
fi

function savedefault {
  if [ -z "${boot_once}" ]; then
    saved_entry="${chosen}"
    save_env saved_entry
  fi
}

function load_video {
  if [ x$feature_all_video_module = xy ]; then
    insmod all_video
  else
    insmod efi_gop
    insmod efi_uga
    insmod ieee1275_fb
    insmod vbe
    insmod vga
    insmod video_bochs
    insmod video_cirrus
  fi
}

terminal_input console
terminal_output console
if [ x$feature_timeout_style = xy ] ; then
  set timeout_style=menu
  set timeout=5
# Fallback normal timeout code in case the timeout_style feature is
# unavailable.
else
  set timeout=5
fi
### END /etc/grub.d/00_header ###

### BEGIN /etc/grub.d/10_linux ###
menuentry 'Arch Linux, with Linux linux' --class arch --class gnu-linux --class gnu --class os $menuentry_id_option 'gnulinux-linux-advanced-b7ecd501-462f-48e9-b63b-e4aa25321e25' {
  load_video
  insmod gzio
  insmod part_msdos msdos
  insmod diskfilter mdraid1x
  insmod ext2
  set root=(md/126,1)
  if [ x$feature_platform_search_hint = xy ]; then
    search --no-floppy --fs-uuid --set=root --hint='mduuid/00c162ffe3b0f060196a8436ef9b60d7,msdos1'  ed911b0a-e6ad-46c6-b679-021f18ac6cf0
  else
    search --no-floppy --fs-uuid --set=root ed911b0a-e6ad-46c6-b679-021f18ac6cf0
  fi
  echo  'Loading Linux linux ...'
  linux /vmlinuz-linux root=/dev/sdc1 rw
  echo  'Loading initial ramdisk ...'
  initrd  /intel-ucode.img /initramfs-linux.img
}
menuentry 'Arch Linux, with Linux linux (fallback initramfs)' --class arch --class gnu-linux --class gnu --class os $menuentry_id_option 'gnulinux-linux-fallback-b7ecd501-462f-48e9-b63b-e4aa25321e25' {
  load_video
  insmod gzio
  insmod part_msdos msdos
  insmod diskfilter mdraid1x
  insmod ext2
  set root=(md/126,1)
  if [ x$feature_platform_search_hint = xy ]; then
    search --no-floppy --fs-uuid --set=root --hint='mduuid/00c162ffe3b0f060196a8436ef9b60d7,msdos1'  ed911b0a-e6ad-46c6-b679-021f18ac6cf0
  else
    search --no-floppy --fs-uuid --set=root ed911b0a-e6ad-46c6-b679-021f18ac6cf0
  fi
  echo  'Loading Linux linux ...'
  linux /vmlinuz-linux root=/dev/sdc1 rw
  echo  'Loading initial ramdisk ...'
  initrd  /intel-ucode.img /initramfs-linux-fallback.img
}

### END /etc/grub.d/10_linux ###

### BEGIN /etc/grub.d/20_linux_xen ###
### END /etc/grub.d/20_linux_xen ###

### BEGIN /etc/grub.d/30_os-prober ###
### END /etc/grub.d/30_os-prober ###

### BEGIN /etc/grub.d/40_custom ###
# This file provides an easy way to add custom menu entries.  Simply type the
# menu entries you want to add after this comment.  Be careful not to change
# the 'exec tail' line above.
### END /etc/grub.d/40_custom ###

### BEGIN /etc/grub.d/41_custom ###
if [ -f  ${config_directory}/custom.cfg ]; then
  source ${config_directory}/custom.cfg
elif [ -z "${config_directory}" -a -f  $prefix/custom.cfg ]; then
  source $prefix/custom.cfg;
fi
### END /etc/grub.d/41_custom ###

### BEGIN /etc/grub.d/60_memtest86+ ###
if [ "${grub_platform}" == "pc" ]; then
    menuentry "Memory Tester (memtest86+)" --class memtest86 --class gnu --class tool {
        search --fs-uuid --no-floppy --set=root --hint='mduuid/00c162ffe3b0f060196a8436ef9b60d7,msdos1'  ed911b0a-e6ad-46c6-b679-021f18ac6cf0
        linux16 /memtest86+/memtest.bin
    }
fi
### END /etc/grub.d/60_memtest86+ ###

Current state of affairs:

This is my best description of "what's happening?" right now - again, apologies if this doesn't give enough or the correct information.

Two mdadm RAID arrays have been created on the new SATA drive (one for the main data currently housed on /dev/sda, and a separate swap array), and mdadm has been configured (in degraded array mode) to find and create both of those arrays automatically. The new partitions have been created inside the arrays (with the same filesystems as present for the equivalent partitions on /dev/sda), and I used cp -vRp (from a livecd) to port the data for each of the /boot, /home, /tmp and /var partitions over to the main array. Swap has also been enabled.

Within the live cd, I've mounted /dev/sdc1 followed by each of the /dev/md126p1, /dev/md126p2, /dev/md126p3, and /dev/md126p4 partitions to their appropriate places (and enabled swap on /dev/md127p1), then arch-chrooted to the mounted root and run grub-install --target=i386-pc --recheck /dev/sdb followed by grub-mkconfig -o /boot/grub/grub.cfg. Both commands complete and appear to generate reasonable output.

At this point, thinking that I've followed every indicated-or-linked-to-in-another-wiki-article step, I rebooted to confirm the install to the new drive was complete, before beginning the process of porting /dev/sda over as a component of the mdadm array.

If /dev/sda is the boot device in the BIOS, then the GRUB menu comes up as normal. Hitting e before any entry can auto-boot I can verify that the content is that of the old /boot/grub/grub.cfg from /dev/sda1. Selecting the main entry boots successfully to the desktop as normal.

If /dev/sdb is the boot device in the BIOS, then the same GRUB menu comes up. Hitting e before any entry can auto-boot I see that I'm still (somehow) loading up the OLD content from /boot/grub/grub.cfg on /dev/sda1. To confirm that this is happening, I disabled /dev/sda completely in the BIOS and the system is unable to find an operating system to boot.

As a diagnostic test, I rebooted back into the livecd and copied the new content of /boot/* from /dev/md126p1 onto the old /dev/sda1 partition and rebooted back to /dev/sda. This causes GRUB to fail with the error "symbol 'grub_real_boot_time' not found".

At this point I'm kinda at the end of my technical knowledge and don't know where I'm going wrong. Any help or pointers to issues you're able to offer would be much appreciated. If you need more information, please ask.

Last edited by stubble (2015-10-19 07:38:38)

boban_dj · 2015-10-07 05:19:05

What is your purpose, to boot from grub in sda,sdb or sdc? explain what you want to boot from each disk, Arch or backup etc.
Because I understand from your story you have installed grub twice, in sda and sdc?

stubble · 2015-10-07 05:45:32

My short-term goal is to get /dev/sdb to be bootable and launch the degraded-RAID1 as the main Arch system drive - to basically completely remove the system's dependence on /dev/sda. Once that is done, I will rebuild /dev/sda so that it is also bootable, and joins the RAID. /dev/sdc will remain unchanged as the root filesystem drive.

In an ideal world, booting from /dev/sdc would be great and make this whole situation much easier, but the BIOS doesn't allow the SSD to be a boot device at all. This limitation is why there is a /boot partition currently on /dev/sda, and why I've added a /boot partition to the degraded array.

branch · 2015-10-07 19:40:46

My guess is that grub-install is using stale info (core.img?) on the new /boot (/dev/md126p1) that was copied from the old boot (/dev/sda1), causing the stub installed on /dev/sdb point back /dev/sda1.

I would try deleting everything from the new /boot (/dev/md126p1) and reinstalling grub as you did before, which should force grub-install to regenerate everything.

stubble · 2015-10-07 20:18:06

That's a fair point and one I hadn't considered.

I just ran through the following steps:
* rebooted into a livecd, remounted the partitions in the appropriate arrangement and arch-chrooted into the mounted directory
* backed up the contents of /boot from /dev/md126p1 to another partition
* nuked the entire contents of the /boot directory on /dev/md126p1
* re-ran grub-install --target=i386-pc --recheck /dev/sdb followed by grub-mkconfig -o /boot/grub/grub.cfg.
* noticed that all the kernel/initramfs images were gone - because I deleted them, duh! - so copied just those files back into place from the backup (everything except the /boot/grub directory, basically)
* ran mkinitcpio -p linux for good measure
* rebooted back to the hard drive, and once again GRUB is running from /boot on /dev/sda1

No errors were generated by any of the above (particularly the grub-*) commands.

Is it possible that something in the pre-partition gap of /dev/sdb (in the MBR area) is instructing the use of /dev/sda1 despite repeated re-running of grub-install? Can grub-install find existing content in that area and not bother to overwrite?
Is there a way to nuke (or any value in nuking) that area of the disk (without killing the partition table!!!!) to remove any persisting boot instructions so grub-install applies fresh data?

Last edited by stubble (2015-10-08 07:22:15)

stubble · 2015-10-19 07:38:21

Solved, thanks to attacking it from a different angle.

I rebuilt /dev/sdb with a GPT (and BIOS boot partition) instead of an MBR table, then followed the exact same process as before and it worked flawlessly first time.

Only 10 hours until the RAID rebuilds onto /dev/sda, think I'll go make a coffee or seven.

Arch Linux

#1 2015-10-06 21:02:49

[SOLVED] Difficulty with GRUB booting RAID1 on Inspiron 7720se

#2 2015-10-07 05:19:05

Re: [SOLVED] Difficulty with GRUB booting RAID1 on Inspiron 7720se

#3 2015-10-07 05:45:32

Re: [SOLVED] Difficulty with GRUB booting RAID1 on Inspiron 7720se

#4 2015-10-07 19:40:46

Re: [SOLVED] Difficulty with GRUB booting RAID1 on Inspiron 7720se

#5 2015-10-07 20:18:06

Re: [SOLVED] Difficulty with GRUB booting RAID1 on Inspiron 7720se

#6 2015-10-19 07:38:21

Re: [SOLVED] Difficulty with GRUB booting RAID1 on Inspiron 7720se

Board footer