Software raid won't boot after updating to "mdadm" in mkinitcpio.conf

chas · 2010-05-01 11:11:04

After a power outage I've discovetred the config I was using (with raid in mkinitcpio.conf) no longer works, it's mdadm now - that's fine. I've updated that and re-run mkinitcpio successfully, however my system is unable to boot from the root filesystem /dev/md2 like so:

Waiting for 10 seconds for device /dev/md2 ...
Root device '/dev/md2' doesn't exist. Attempting to create it.
ERROR: Unable to determine major/minor number of root device '/dev/md2'.
You are being dropped to a recovery shell
Type 'exit' to try and continue booting
/bin/sh: can't access tty; job control turned off
[ramfs /]#

As far as I can see from reading various threads and http://wiki.archlinux.org/index.php/Ins … AID_or_LVM I'm doing the right things now (although I'm not using lvm at all, which makes the installation document a little confusing).

I think I've included all the appropriate bits of config here that should be working. I assume I've missed something fundamental - any ideas?

menu.lst:

# (0) Arch Linux
title Arch Linux [/boot/vmlinuz26]
root (hd0,0)
kernel /vmlinuz26 root=/dev/md2 ro
initrd /kernel26.img

mkinitcpio.conf:

HOOKS="base udev autodetect pata scsi mdadm sata filesystems"

fstab:

/dev/md1 /boot ext3 defaults 0 1
/dev/md2 / ext3 defaults 0 1

mdadm.conf

ARRAY /dev/md1 level=raid1 num-devices=2 metadata=0.90 UUID=7ae70fa6:9f54ba0a:21
47a9fe:d45dbc0c
ARRAY /dev/md2 level=raid1 num-devices=2 metadata=0.90 UUID=20560268:8a089af7:e6
043406:dbdabe38

Thanks!

mikesd · 2010-05-01 11:54:20

Did you recreate your initrd image after generating your mdadm.conf file?

Try putting mdadm after sata in the hooks array.

Your mdadm.conf does look a little weird. Are the ARRAY lines actually split over two lines or are they being wrapped somewhere?

chas · 2010-05-01 15:50:18

mikesd wrote:

Did you recreate your initrd image after generating your mdadm.conf file?

I've recreated the initrd a fair few times since removing raid and adding madam. The mdadm.conf file itself hasn't changed in about a year.

mikesd wrote:

Try putting mdadm after sata in the hooks array.

Ok, have done that and again done a mkinitcpio but the issue remains.

mikesd wrote:

Your mdadm.conf does look a little weird. Are the ARRAY lines actually split over two lines or are they being wrapped somewhere?

Yes, that's bizzare wrapping. There are only two lines and they both start with ARRAY.

mikesd · 2010-05-02 00:54:50

Check the modules line in your mkinitcpio.conf. I have:

MODULES="dm_mod md_mod raid1 raid10"

Not sure if this is required or not. You won't need dm_mod if you are not using lvm2.

Best way to check if the raid modules are being included in your initrd:

[mike@mercury|~] $ zcat /boot/kernel26.img | cpio -t | grep drivers/md/
17636 blocks
/lib/modules/2.6.33-ARCH/kernel/drivers/md/dm-mod.ko
/lib/modules/2.6.33-ARCH/kernel/drivers/md/md-mod.ko
/lib/modules/2.6.33-ARCH/kernel/drivers/md/raid1.ko
/lib/modules/2.6.33-ARCH/kernel/drivers/md/raid10.ko
/lib/modules/2.6.33-ARCH/kernel/drivers/md/dm-snapshot.ko
/lib/modules/2.6.33-ARCH/kernel/drivers/md/dm-log.ko
/lib/modules/2.6.33-ARCH/kernel/drivers/md/dm-region-hash.ko
/lib/modules/2.6.33-ARCH/kernel/drivers/md/dm-mirror.ko

You will need md-mod.ko and raid1.ko at least.

chas · 2010-05-02 07:40:44

My modules conf line in mkinitcpio.conf was actually empty, but even with it like that it seems mkinitcpio was smart enough to work out I need md_mod and raid1:

MODULES=""

zcat /boot/kernel26.img | cpio -t | grep drivers/md/
5915 blocks
/lib/modules/2.6.33-ARCH/kernel/drivers/md/md-mod.ko
/lib/modules/2.6.33-ARCH/kernel/drivers/md/raid1.ko

Adding md_mod and raid1 to MODULES didn't help (nor adding dm_mod md_mod raid1 raid0, just in case).

mikesd · 2010-05-02 08:12:58

chas wrote:

My modules conf line in mkinitcpio.conf was actually empty, but even with it like that it seems mkinitcpio was smart enough to work out I need md_mod and raid1:

Yeah, that's how it should work I think. Can't recall when I added the modules manually to my setup.

Don't know what to try next. Maybe check that your /etc/mdadm.conf matches the output of:

sudo mdadm --detail --scan

perhaps the UUIDs changed for some reason.

You could also check the partition types are set to 0xfd (Linux RAID Autodetect).

Sorry grasping at straws now.

chas · 2010-05-02 08:54:31

Somewhere along the way the output of mdadm -D --scan has changed from this:

ARRAY /dev/md1 level=raid1 num-devices=2 metadata=0.90 UUID=7ae70fa6:9f54ba0a:2147a9fe:d45dbc0c
ARRAY /dev/md2 level=raid1 num-devices=2 metadata=0.90 UUID=20560268:8a089af7:e6043406:dbdabe38

to this:

ARRAY /dev/md1 metadata=0.90 UUID=7ae70fa6:9f54ba0a:2147a9fe:d45dbc0c
ARRAY /dev/md2 metadata=0.90 UUID=20560268:8a089af7:e6043406:dbdabe38

But updating mdadm.conf and re-running mkinitcpio doesn't change the situation.

All the partitions are Linux raid autodetect.

I am also a bit baflled, as far as I can tell this should just work now and well it doesn't

chas · 2010-05-02 09:43:51

chas wrote:

Somewhere along the way the output of mdadm -D --scan has changed from this:

Actually, that's wrong. It shows it without the raid level and the device count if you run mdadm --detail --scan from within a chroot (when booting from the rescue disc) but outside the chroot it includes this detail.

magec · 2010-05-02 10:04:20

I was having the same but not anymore, I guess the problem was related with mkinitcpio running from inside an "incomplete" chroot when using the live iso image.

make sure you mount both proc and sysfs and copy the dev filesystem

mount /dev/your_root_device /mnt
mdadm -A --scan -c /mnt/etc/mdadm.conf
mount -t proc none /mnt/proc
mount -t sysfs none /mnt/sys
mount -n -t ramfs none /mnt/dev
cp -Rp /dev/* /mnt/dev
chroot /mnt

Once in the chroot run mkinitcpio again in mkinitcpio.conf MODULES have to be empty, it autodetects every dependency correctly, to check so try (mkinitcpio -k 2.6.33-ARCH -M). After that you should be able to reboot. No need to set md=0,blah,blah in the menu.lst just with the root=/dev/mdN will do.

It worked for me and was stucked just like you I think the problem was the dev filesystem in the chroot, hope this helps

chas · 2010-05-02 11:02:13

Hi magec, that's quite helfpul - I've certainly got further.

Before I was doing this to set up the chroot (which is what is suggested in the wiki article about setting up software raid):

mdadm -A /dev/md1 /dev/sda1 /dev/sdb1
mdadm -A /dev/md2 /dev/sda2 /dev/sdb2
mount /dev/md2 /mnt
mount /dev/md1 /mnt/boot
mount -o bind /dev /mnt/dev
mount -t proc none /mnt/proc
chroot /mnt /bin/bash

But based on your suggestion it's working better

mdadm -A /dev/md1 /dev/sda1 /dev/sdb1
mdadm -A /dev/md2 /dev/sda2 /dev/sdb2
mount /dev/md2 /mnt
mount /dev/md1 /mnt/boot
mount -t proc none /mnt/proc
mount -t sysfs none /mnt/sys
mount -n -t ramfs none /mnt/dev
cp -Rp /dev/* /mnt/dev
chroot /mnt /bin/bash

The boot is now getting further, but now I'm getting:

md: md2 stopped.
md: bind<sdb2>
md: bind<sda2>
raid1: raid set md2 active with 2 out of 2 mirrors
md2: detected capacity change from 0 to 32218349568
mdadm: /dev/md2 has been started with 2 drives.
  md2: Waiting 10 seconds for device /dev/md2 ...
  unknown partition table
mount: mounting /dev/md2 on /new_root failed: No such device
ERROR: Failed to mount the real root device.
Bailing out, you are on your own. Good luck.

/bin/sh: can't access tty; job contol turned off
[ramfs /]#

The bit that really confuses me is this:

[ramfs /]# cat /proc/mdstat
Personalities : [raid1]
md2 : active raid1 sda2[0] sdb2[1]
         31463232 blocks [2/2] [UU]

md1 : active raid1 sda1[0] sdb1[1]
         208704 blocks [2/2] [UU]

unused devices: <none>
[ramfs /]# mount /dev/md2 /new_root
mount: mounting /dev/md2 on /new_root failed: No such file or directory
[ramfs /]# ls /dev/md2
/dev/md2
[ramfs /]#

So the array is up, the device node is there but it can't be mounted? Very strange.

Last edited by chas (2010-05-02 11:24:09)

magec · 2010-05-02 11:21:37

That was the same I was having before it got solved. I can't do more than telling you my conf.

> cat /etc/mkinitcpio.conf|grep -v ^#
MODULES=""
BINARIES="/usr/bin/strace"
FILES=""
HOOKS="base udev autodetect pata scsi sata usbinput mdadm lvm2 filesystems"

make sure the MODULES is empty and also check the order of the hooks.

>cat /etc/mdadm.conf
ARRAY /dev/md0 level=raid0 num-devices=2 metadata=0.90 UUID=31cc556a:481721c2:32440aa5:8399a9a9
ARRAY /dev/md1 level=raid1 num-devices=2 metadata=0.90 UUID=07ac94cf:344f146c:3d56b11b:c638b737
ARRAY /dev/md2 level=raid1 num-devices=2 metadata=0.90 UUID=fa1ae878:0ccbbb67:08640dd1:958dcc73

Note the level and num-devices ( either way as you said when executed from within the chroot did not showed that info)

menu.lst

title  Arch Linux
root   (hd0,1)
kernel /boot/vmlinuz26 root=/dev/md1 ro 
initrd /boot/kernel26.img

I was having the same issue and mdassemble complained saying that the devices were busy or something (but no process seemed to be accessing) seems that there were two things trying to set the raid up, no idea. Hope this helps.

chas · 2010-05-02 13:07:43

Fixed!

Doing a blkid from the ramfs system I noticed that my /dev/md2 was reporting itself as an LVM volume (which I experiemted with and removed as part of this troubleshooting).

Interestingly in the live cd it reported it to be ext3, so something seemd amiss. I backed up the contents of md2, destroyed and recreated the array, then put the files back, updated mdadam.conf ran mkinicpio.

Now it boots!

Thanks for all the help!

mreschke · 2011-11-05 05:34:25

Old thread I know, but strangely enough I just received this error too. My root partition is a raid1 much like this posts describes. It happened after my last archlinux update around 2011-10-??. I followed #9 above (magec's unique chroot mounting) and ran the mkinitcpio -p linux and still didn't work. The culprit ended up being the linux grub line with root=/dev/disk/by-uuid/xxxxx. I have had the same grub line for a year or so, and it always boots fine, meaning that /dev/disks/by-uuid/xxxx has always existed during boot, until the last update. When the boot failed and I was dropped to the ramfs prompt, a simple ls /dev/disks/by-uuid revealed my guid was not present, very strange. So by setting the grub line to either root=/dev/md1 or root=UUID=xxxx solved my problem.

Anyone know what might have changed in recent updates that would cause the guid to not appear in /dev/disks/by-uuid during boot (note: during boot only, after boot my guid is in /dev)?

big_gie · 2011-11-30 14:38:04

I stumbled upon this post after having problems to boot too. I have two IDE drives in a RAID1 configuration. Two partitions are mirrored on the two drives: / and /boot. On the second drive, there is a third partition to host some data (and is not mirrored). I used this wiki entry to configure this server: https://wiki.archlinux.org/index.php/Co … em_to_RAID

The server had a ~3 months uptime when I rebooted it. I'm using the -lts kernel (2.6.32.48). It wouldn't boot because it couldn't find the root partition. Thanks to your post mreschke, I changed my grub kernel option from:
root=/dev/disk/by-label/raid1_1
to:
root=UUID=ff03ca2b-4475-414d-803e-1cf065621ef6
and it booted without issue. It seems the initrd is now failing to identify drives by either labels or uuid (through /dev/disk/by-* paths)...

Arch Linux

#1 2010-05-01 11:11:04

Software raid won't boot after updating to "mdadm" in mkinitcpio.conf

#2 2010-05-01 11:54:20

Re: Software raid won't boot after updating to "mdadm" in mkinitcpio.conf

#3 2010-05-01 15:50:18

Re: Software raid won't boot after updating to "mdadm" in mkinitcpio.conf

#4 2010-05-02 00:54:50

Re: Software raid won't boot after updating to "mdadm" in mkinitcpio.conf

#5 2010-05-02 07:40:44

Re: Software raid won't boot after updating to "mdadm" in mkinitcpio.conf

#6 2010-05-02 08:12:58

Re: Software raid won't boot after updating to "mdadm" in mkinitcpio.conf

#7 2010-05-02 08:54:31

Re: Software raid won't boot after updating to "mdadm" in mkinitcpio.conf

#8 2010-05-02 09:43:51

Re: Software raid won't boot after updating to "mdadm" in mkinitcpio.conf

#9 2010-05-02 10:04:20

Re: Software raid won't boot after updating to "mdadm" in mkinitcpio.conf

#10 2010-05-02 11:02:13

Re: Software raid won't boot after updating to "mdadm" in mkinitcpio.conf

#11 2010-05-02 11:21:37

Re: Software raid won't boot after updating to "mdadm" in mkinitcpio.conf

#12 2010-05-02 13:07:43

Re: Software raid won't boot after updating to "mdadm" in mkinitcpio.conf

#13 2011-11-05 05:34:25

Re: Software raid won't boot after updating to "mdadm" in mkinitcpio.conf

#14 2011-11-30 14:38:04

Re: Software raid won't boot after updating to "mdadm" in mkinitcpio.conf

Board footer