Mounting array and volume group fails after upgrade

filam · 2012-04-06 21:31:59

I recently did a full upgrade of my system and after reboot the init scripts can't assemble the RAID that includes my root directory.

...
:: Running Hook [mdadm]
mdadm: /dev/md/2 has been started with 3 drives.
mdadm: /dev/md/1 has been started with 3 drives.
mdadm: failed to RUN_ARRAY /dev/md/0: Invalid argument
:: Running Hook [lvm2]
Activating logical volumes...
    No volume groups found
Waiting 10 seconds for device /dev/mapper/VolGroupArray-lvroot ...
ERROR: Unable to find root device '/dev/mapper/VolGroupArray-lvroot'.

And then I'm dropped into a recovery shell (i.e. rootfs) that can't access a tty. I assume that LVM error occurs because mdadm fails to assemble the array holding the root partition.

My system includes three disks (i.e. /dev/sd[bcd][123]) that each include a /boot, /swap, and / partition, which I believe were originally assembled into /dev/md125, /dev/md126, /dev/md127. You'll note that is not where they are being assembled during boot above.

When I boot a Live CD the following occurs:

# mdadm --assemble /dev/md125 /dev/sdb1 /dev/sdc1 /dev/sdd1
mdadm: cannot open device /dev/sdb1: Device or resource busy
mdadm: /dev/sdb1 has no superblock - assembly aborted

Any help would be greatly appreciated.

Edit: As falconindy stated on IRC, all software RAIDs using mdadm should use mdadm_udev instead of mdadm as a hook within mkinitcpio, because incremental assembly via udev is what upstream supports.

Last edited by filam (2012-04-07 03:35:47)

jasonwryan · 2012-04-06 21:41:31

Check that udev hasn't helpfully assembled a couple of phantom arrays based on the superblocks on your drives:

cat /proc/mdstat

you can then stop them with

mdadm --stop /dev/md[xxx]

and then assemble your correct arrays...

filam · 2012-04-06 21:49:54

jasonwryan wrote:

Check that udev hasn't helpfully assembled a couple of phantom arrays based on the superblocks on your drives

When I boot the system with an old Live CD and check /proc/mdstat I only see md125, md126, and md127, which are all [UUU]. I mounted the /dev/md125 (i.e. /boot) to a directory without a problem.

For some reason on boot the system tries to assemble the arrays to md/[012] instead of md12[567].

Edit: lvdisplay lists all the logical volumes (e.g. /dev/VolGroupArray/lvroot), but states that they are not available.

Last edited by filam (2012-04-06 21:54:04)

jasonwryan · 2012-04-06 21:51:59

Was your mdadm.conf overwritten? Sounds like you will have to set everything up manually and then rewrite it.

filam · 2012-04-06 21:55:47

jasonwryan wrote:

Was your mdadm.conf overwritten? Sounds like you will have to set everything up manually and then rewrite it.

That sounds like it to me too. It must have been overwritten during the upgrade, although why would that happen? Shouldn't a new version of mdadm.conf be saved as /etc/mdadm.pacnew?

Edit: And thanks a lot for the help!

Last edited by filam (2012-04-06 21:56:23)

jasonwryan · 2012-04-06 21:59:23

I wouldn't have thought it would have happened as a result of pacman (it certainly hasn't affected me).

Look back through pacman's log and see if there are any messages that will provide a clue. You sure you haven't been doing any tinkering

filam · 2012-04-06 22:11:58

jasonwryan wrote:

You sure you haven't been doing any tinkering

Haha, I tried setting up Solarized. Using xrdb messed up my terminal colors. That's probably it! ;-)

I just noticed looking back at /proc/mdstat that md126 and m127 are "(auto-read-only)", but md125 isn't.[1] And mdadm --examine --scan lists three arrays all with "name=archiso:[012]", but none of my md12[567] arrays. I'm not sure if any of that is meaningful.

I need to check the docs about mounting /dev/md127, which holds my root partition, to the Live CD. It's currently yelling at me because its a LVM2_member.

Edit: Although pvdisplay shows me information about the physical volume created from /dev/md127 the following is not reassuring.

# pvs VolGroupArray
Failed to read physical volume "VolGroupArray"

Sorry, I should have done the following:

# pvs /dev/md127
PV        VG             Fmt  Attr PSize PFree
/dev/md127 VolGroupArray lvm2 a-- 1.82t 0

Edit: And I can't look at the pacman log if I can't mount the logical volume.

[1]: mdadm message (auto-read-only) on "/dev/md1" = swap file system

Last edited by filam (2012-04-06 22:38:29)

jasonwryan · 2012-04-06 22:35:27

filam wrote:

mdadm --examine --scan lists three arrays all with "name=archiso:[012]", but none of my md12[567] arrays. I'm not sure if any of that is meaningful.

That's not right. It should look something like:

ARRAY /dev/md125 UUID=xxyyzz
ARRAY /dev/md/126 UUID=xxyyzz name=archiso:1
etc...

Assemble the arrays manually, make the LVs available and from the chroot regenerate your mdadm.conf

Last edited by jasonwryan (2012-04-06 22:59:22)

filam · 2012-04-06 23:00:47

jasonwryan wrote:

chroot in and assemble the arrays manually, make the LVs available and regenerate your mdadm.conf

I have never used chroot before, could you explain how to "chroot in" with a little more detail? That essentially means that I should change the root of the directory structure from that of the Live CD to the root directory on my system, but how do I do that if I can't mount it?

Is that why my md12[567] arrays are listed in /proc/mdstat, but not with mdadm --examine --scan?

jasonwryan · 2012-04-06 23:05:44

For whatever reason, it appears that your arrays are scrambled. From the live environment, you will need to assemble the arrays in the correct sequence - that should then allow you to access the logical volumes. Once you have done that, you can chroot and fix your mdadm.conf -- after that, I'd also look at the mkinitcpio hooks line (and probably regenerate the image just to be sure) while in there.

This is how I would do it.

jasonwryan · 2012-04-09 01:29:58

What type of array is it? There is a bug opened for Raid 5&6 with a workaround.

Arch Linux

#1 2012-04-06 21:31:59

Mounting array and volume group fails after upgrade

#2 2012-04-06 21:41:31

Re: Mounting array and volume group fails after upgrade

#3 2012-04-06 21:49:54

Re: Mounting array and volume group fails after upgrade

#4 2012-04-06 21:51:59

Re: Mounting array and volume group fails after upgrade

#5 2012-04-06 21:55:47

Re: Mounting array and volume group fails after upgrade

#6 2012-04-06 21:59:23

Re: Mounting array and volume group fails after upgrade

#7 2012-04-06 22:11:58

Re: Mounting array and volume group fails after upgrade

#8 2012-04-06 22:35:27

Re: Mounting array and volume group fails after upgrade

#9 2012-04-06 23:00:47

Re: Mounting array and volume group fails after upgrade

#10 2012-04-06 23:05:44

Re: Mounting array and volume group fails after upgrade

#11 2012-04-09 01:29:58

Re: Mounting array and volume group fails after upgrade

Board footer