Arch Linux

ibid

Member

From: Champaign, IL

Registered: 2009-10-20

Posts: 31

Website

Broke my LVM and RAID

I'm running a RAID0 with two LVM groups /dev/array/alpha and /dev/array/root (they're also known as /dev/mapper/array-{alpha,root})

I created them according the the Arch wiki. It's been working well since I created it at the beginning of the year.

I decided I'd try and shrink down one of the groups so that I could add another partition. Not exactly sure what I did, but it was a combo of lvresize and pvresize. My actions became more erratic and I may have marked one of the drives in the RAID as a fail. I think this was under the impression that I could then boot from just one HD and save the other in case of some catastrophe.

Wrong, now I've got a catastrophe. Arch won't boot, so I'm up in systemrescuecd. lvs and lvdisplay is able to see my LVM groups. I can mount /dev/array/root. But the group that I messed with, alpha, shows "LV Status: Suspended" and all attempts to mount it fail.

Should also note that fsck.ext4 shows a bad superblock warning.

What are my options here? I'm thinking I could reformat one of the drives and find some dd voodoo to copy over the data from the other, but I'm hoping to get away with something less drastic.

mikesd

Member

From: Australia

Registered: 2008-02-01

Posts: 788

Website

Re: Broke my LVM and RAID

RAID0 ??? RAID0 is striping and I don't know why you would be using that as well as LVM. Do you mean RAID1?

---

[ twitter | blog | configs ]

ibid

Member

From: Champaign, IL

Registered: 2009-10-20

Posts: 31

Website

Re: Broke my LVM and RAID

you're right. RAID1

ibid

Member

From: Champaign, IL

Registered: 2009-10-20

Posts: 31

Website

Re: Broke my LVM and RAID

Some more info about the setup:
RAID1. Two 1TB drives.

md0 = / (858G) ext4
md2 = swap (1.86G)
md1 = /boot ext2

md0 is divided into two parts:
/dev/array/root = / (50G)
/dev/array/alpha (everything else)

I ran fsarchiver (Show Filesystems in systemresccd) and md0 displays as 800G, but there is another disk that's called dm-0, which is 50G
I'm also running TestDisk, which displays this disk as 858G

ibid

Member

From: Champaign, IL

Registered: 2009-10-20

Posts: 31

Website

Re: Broke my LVM and RAID

I've been looking in on the RAID side of things
I failed the /dev/sda3 drive and this may be causing the error

cat /proc/mdstat returns:
md0 : active raid1 sdb3[0]
838860800 blocks [2/1] [U_]

md127 : inactive sda3[1](S)
974711680 blocks

...

well, the md127 must have resulted from me 'failing' the sda3 drive improperly. Maybe it's fomed it's own faulty RAID?

Is there a way to get sda3 to rejoin sdb3? I'm going to check now if I can get md0 running with only one drive. Fixing the RAID may be a wild goose chase if my LVM is borked, but what can I do?

ibid

Member

From: Champaign, IL

Registered: 2009-10-20

Posts: 31

Website

Re: Broke my LVM and RAID

I think this is fixable. I just need a little help getting it all the way.

I was about to access the root partition of the LVM /dev/array/root, and on there I found a backup config  in /etc/lvm/archive that I loaded with vgcfgrestore. With this backup config I can now recognize my /dev/array/alpha, but it won't mount and fsck fails.

Here's what I've done:
fsck.ext4 /dev/mapper/array-alpha. I "y" everything, but it aborts with error. I agreed to clear the journal, so I believe this is effectively an ext2 filesystem at this point.

mke2fs -n /dev/mapper/array-alpha gets me a list of superblock backups where before it told me the filesystem was 0 and the partition table was broken and so on.