Rescue RAID system by booting from CD?

diamondmind · 2008-10-23 03:50:44

As the title suggests, I need to fix my server which is running a RAID setup per the wiki page. Everything was working perfectly until I did a pacman -Syu the other day and managed to render my system unbootable.

I tried to rescue it from the install CD like I've done with my other non-RAID boxes where I can use:

arch root=/dev/sdxx ro

but couldn't figure out how to do that with a RAID setup.

arch root=/dev/array/root

and a few other variations didn't work as I'd hoped.

I (obviously) don't know much about RAID but after Googling for a while I couldn't find anything very applicable so I was hoping the Arch Forum would come through for me ... again.

byte · 2008-10-23 04:22:46

Almost correct. You're only missing the "md=0,/dev/sdaX,/dev/sdbX" stuff on the kernel command line.

diamondmind · 2008-10-23 14:50:44

I tried this:

arch root=/dev/array/root ro  md=1,/dev/sda1,/dev/sdb1 md=0,/dev/sda3,/dev/sdb3

And get the following:

Root device '/dev/array/root/ doesn't exist, attempting to create it
ERROR: Failed to parse block device name for '/dev/array/root'
ERROR: Unable to create/detect root device 'dev/array/root'
Dropping into a recovery shell... type 'exit' to reboot
NOTE: klibc contains no 'ls' binary, use 'echo *' instead

If the device '/dev/array/root' gets created while you are here, 
try adding 'rootdelay=8' or higher to the kernel command-line
sh: no job control in this shell
ramfs$

I think I may have borked it up beyond repair last night while trying to figure it out myself.

What originally broke my system was I moved /usr/lib/klibc to /usr/lib/klibc-old when I got a bunch of conflicting files during a pacman update. I also tried doing a kernel update through pacman while booting off of the CD last night by mounting the root partition of my RAID array and chroot-ing:

chroot /mnt /bin/bash

which I think has really broken things.

I guess what I'm trying to say is that it's probably more than just a problem booting a RAID from CD by now.

Last edited by diamondmind (2008-10-23 14:52:28)

nexus7 · 2008-10-23 19:18:16

You mean you'd like to boot your raid from live-cd?
I wonder, if this easily possible, but of course you can boot from cd, get into your system (as you tried) to repair it, to update it and make it boot nrmally again.
Maybe a look at my recent post would help.

Did you modprobe your raid-module according to your type of raid?

And before chrooting you need to mount some more specific directories.
Just leave the ercyption stuff away and do something like this:

modprobe raid1
mdadm -Ac partitions -m 0 /dev/md0
mdadm -Ac partitions -m 1 /dev/md1

mount /dev/mapper/root /mnt
mount /dev/md1 /mnt/boot
mount -o bind /sys /mnt/sys
mount -o bind /dev /mnt/dev
mount -o bind /proc /mnt/proc

chroot /mnt

And your /etc/mkinitcpio.conf is correctly configured, isn't it?

To boot your raid from cd, the kernel with the initrd's had to be built correctly with all modules before because it seems not be possible to enable/disable modules via boot parameters (like the hooks).

Last edited by nexus7 (2008-10-23 19:31:54)

diamondmind · 2008-10-24 02:42:42

nexus7 wrote:

You mean you'd like to boot your raid from live-cd?

More or less .. I guess that is a better way to describe it.

nexus7 wrote:

Did you modprobe your raid-module according to your type of raid?

I modprobed all the modules I needed before I tried it again tonight but I still can't get it quite right. When I assemble my /

mdadm -Ac partitions -m 0 /dev/md0

I don't get /dev/mapper.

I read through the other post too, but it's for booting the system once it's bootable, right? Everything worked before I broke it so I shouldn't need to update anything else, correct? I'm still trying to wrap my head around all the intricacies of raid and lvm etc.

nexus7 · 2008-10-24 08:10:39

diamondmind wrote:

mdadm -Ac partitions -m 0 /dev/md0
I don't get /dev/mapper.

Sry, my fault, /dev/mapper in this case belongs to the LUKS-container. Yours should be of course s.th. like /dev/mdX for your /-raid-arrray then.

So what modules do you modprobe explictiely, e.g. before chrooting?
And in my case just putting MODULES="dm-mod" into mkinitcpio.conf for opening raid at boot time did the trick.

Because I dont use lvm, I only can point you to this subsection, which you probably already read before.

diamondmind wrote:

I read through the other post too, but it's for booting the system once it's bootable, right? Everything worked before I broke it so I shouldn't need to update anything else, correct? I'm still trying to wrap my head around all the intricacies of raid and lvm etc. hmm

Why not update, if maybe a simple # pacman -Syu would be the solution? If it wouldn't that won't harm you; on the other side, if syu-ing fails, that other error could give you another hint (though it mustn't).---So, e.g. if you used kernel .26 before, then you'll have the new kernel .27 (what implies general improvement), with which you can build test-initrds with your edited config files for booting via grub from Arch's live-cd (if grub on your hd does not work).

diamondmind · 2008-10-24 21:01:19

nexus7 wrote:

diamondmind wrote:
mdadm -Ac partitions -m 0 /dev/md0
I don't get /dev/mapper.
Sry, my fault, /dev/mapper in this case belongs to the LUKS-container. Yours should be of course s.th. like /dev/mdX for your /-raid-arrray then.

I'm ~95% sure I tried mounting /dev/mdx and it didn't work .. I got a message from mount telling me the filesystem type was bad etc. I'll have to try again when I get home.

nexus7 wrote:

So what modules do you modprobe explictiely, e.g. before chrooting?

Both raid1 and raid5.

nexus7 wrote:

Because I dont use lvm, I only can point you to this subsection, which you probably already read before.

Yeah, I briefly looked into it before I realized my problem wasn't with LVM (yet) .. I need to read more on LVM to figure out what exactly I need to do.

nexus7 wrote:

diamondmind wrote:
I read through the other post too, but it's for booting the system once it's bootable, right? Everything worked before I broke it so I shouldn't need to update anything else, correct? I'm still trying to wrap my head around all the intricacies of raid and lvm etc. hmm
Why not update, if maybe a simple # pacman -Syu would be the solution? If it wouldn't that won't harm you; on the other side, if syu-ing fails, that other error could give you another hint (though it mustn't).---So, e.g. if you used kernel .26 before, then you'll have the new kernel .27 (what implies general improvement), with which you can build test-initrds with your edited config files for booting via grub from Arch's live-cd (if grub on your hd does not work).

That was the plan before I realized I couldn't get my / mounted and chrooted into.

diamondmind · 2008-10-25 00:49:24

Now I'm stumped (and a little worried ) since I'm able to mount /boot:

mount -t ext3 /dev/md1 /mnt

but not /

mount -t ext3 /dev/md0 /mnt

mount: wrong fs type, bad option, bad superblock on /dev/md0,
       missing codepage or other error
       In some cases useful info is found in syslog - try
       dmesg | tail  or so

Assembling the drives appears to work:

#mdadm -Ac partitions /dev/md0
mdadm: /dev/md0 has been started with 3 drives.

Do I need to do something different with / since it's RAID5 and /boot is RAID1?

diamondmind · 2008-10-25 03:46:24

I just got it working. To get it where I could mount my / I had to fsck it which found a bunch of errors and then I was able to mount everything and get pacman running which fixed the issue with my unbootable kernel.

For completeness here's a summary of what I did .. I'm sure it's not complete but hopefully it will help someone in the future.

Repair the / files system:

fsck.ext3 -v /dev/md0

Mount the newly repaired / file system:

mount -t ext3 /dev/md0 /mnt

Mount up the appropriate file systems (per the raid lvm wiki page)

# mount -o bind /dev /mnt/dev
# mount -t proc none /mnt/proc

chroot:

# chroot /mnt /bin/bash

And run pacman to get everything worked out from my kernel muss up:

pacman -Syu

I also ran through and made sure my /etc/mkinitcpio.conf and /etc/rc.conf were all sound before exiting chroot and rebooting.

I had all my data backed up thankfully, I'm just glad I was able to troubleshoot it since my backup box is configured identically to this one.

diamondmind · 2008-10-25 05:08:57

Spoke too soon. I booted up just fine but then noticed the partition where all my data is located wasn't mounted and I can't figure out how to get it back.

I booted off the CD again and assembled my drives:

mdadm -Ac partitions -m 3 /dev/md3

Which appears to have succeeded:

mdadm: /dev/md3 has been started with 2 drives (out of 3) and 1 spare.

But when I try mounting it I get:

# mount -t ext3 /dev/md3 /mnt/
mount: wrong fs type, bad option, bad superblock on /dev/md3,
       missing codepage or helper program, or other error
       In some cases useful info is found in syslog - try
       dmesg | tail  or so

And fscking doesn't help much either:

# fsck.ext3 -v /dev/md3
e2fsck 1.41.2 (02-Oct-2008)
fsck.ext3: Superblock invalid, trying backup blocks...
fsck.ext3: Bad magic number in super-block while trying to open /dev/md3

The superblock could not be read or does not describe a correct ext2
filesystem.  If the device is valid and it really contains an ext2
filesystem (and not swap or ufs or something else), then the superblock
is corrupt, and you might try running e2fsck with an alternate superblock:
    e2fsck -b 8193 <device>

I read a couple of places that you cant fsck on RAID devices? But it worked on my / ?

It's late and I've been at this for a good 3-4 hours now so I'm probably missing something obvious.

nexus7 · 2008-10-25 10:45:21

diamondmind wrote:

Spoke too soon. I booted up just fine but then noticed the partition where all my data is located wasn't mounted and I can't figure out how to get it back.

ugh, that's annoying, i feel with you~
Maybe you could post a list with all drives, belonging to what mdX and arrays and to where they should be mounted. Actually I find it a bit confusing...

diamondmind wrote:

mdadm: /dev/md3 has been started with 2 drives (out of 3) and 1 spare.

What does "and 1 spare" mean, s.th. about lvm (what I don't) use?

diamondmind wrote:

But when I try mounting it I get:
# mount -t ext3 /dev/md3 /mnt/

Probably it's a typo and/or it wouldn't make any difference here, but should it not be /mnt (and not /mnt/)?

diamondmind wrote:

I read a couple of places that you cant fsck on RAID devices?

Oh, really? I didn't know, could you plz name one/some? At least it is fsck'ing at boot-time...

diamondmind wrote:

But it worked on my / ?

So it didn't become an mdX/array? Complicated...

Anyhow, t seems not be a harddisk malfunction.
Hmm, as an emergency solution in your place I'd consider to mount the ordninary /dev/sdaX, back up your data and setup a fresh raid

.:B:. · 2008-10-25 14:43:57

It's no real solution - rather a workaround to make it easier - but I always just boot the Arch install CD, then assemble my RAID arrays and mount them, chroot into the installation, after which I proceed with fixing whatever is broken on the system.

nexus7 wrote:

diamondmind wrote:
Spoke too soon. I booted up just fine but then noticed the partition where all my data is located wasn't mounted and I can't figure out how to get it back.
ugh, that's annoying, i feel with you~
Maybe you could post a list with all drives, belonging to what mdX and arrays and to where they should be mounted. Actually I find it a bit confusing...
diamondmind wrote:
mdadm: /dev/md3 has been started with 2 drives (out of 3) and 1 spare.
What does "and 1 spare" mean, s.th. about lvm (what I don't) use?

It just means what it says - a spare drive in the RAID array that just sits idle and will get used when one of the other drives fails. A backup drive, so to say. E.g. RAID 5 minimally requires 3 partitions/HDs, but you can add extra 'spare' devices that will take over in case something goes wrong with one of the three active partitions/HDs.

diamondmind wrote:

What originally broke my system was I moved /usr/lib/klibc to /usr/lib/klibc-old when I got a bunch of conflicting files during a pacman update.

I hope you never forget this anymore: read the front page regularly, warnings concerning updates are posted there .

Also - do you have an encrypted partition or not? That kind of complicates your setup.

Last edited by B (2008-10-25 15:20:41)

diamondmind · 2008-10-25 16:02:02

I think I'm going to scrap this one and start over. This box was really just a mish mash of old hardware I was using to learn about RAID setups and such .. which I think I've accomplished that goal ... even if I can't fix this one.

B wrote:

I hope you never forget this anymore: read the front page regularly, warnings concerning updates are posted there .

*Sigh* .. fortunately(?) it happened on something (1) backup daily and (2) don't *really* need the data back, but I definitely won't be blindly -Syu-ing anymore.

B wrote:

Also - do you have an encrypted partition or not? That kind of complicates your setup.

No, I don't have anything important enough to be that paranoid.

Arch Linux

#1 2008-10-23 03:50:44

Rescue RAID system by booting from CD?

#2 2008-10-23 04:22:46

Re: Rescue RAID system by booting from CD?

#3 2008-10-23 14:50:44

Re: Rescue RAID system by booting from CD?

#4 2008-10-23 19:18:16

Re: Rescue RAID system by booting from CD?

#5 2008-10-24 02:42:42

Re: Rescue RAID system by booting from CD?

#6 2008-10-24 08:10:39

Re: Rescue RAID system by booting from CD?

#7 2008-10-24 21:01:19

Re: Rescue RAID system by booting from CD?

#8 2008-10-25 00:49:24

Re: Rescue RAID system by booting from CD?

#9 2008-10-25 03:46:24

Re: Rescue RAID system by booting from CD?

#10 2008-10-25 05:08:57

Re: Rescue RAID system by booting from CD?

#11 2008-10-25 10:45:21

Re: Rescue RAID system by booting from CD?

#12 2008-10-25 14:43:57

Re: Rescue RAID system by booting from CD?

#13 2008-10-25 16:02:02

Re: Rescue RAID system by booting from CD?

Board footer