[SOLVED] mdadm, raid6 and superblock missing

maz · 2011-10-30 11:58:01

I was unfortunate enough to have a hardware failure yesterday. One of my controller cards died for a second which meant that my array stopped.

When trying to assemble it again it was stubborn and I decided to just reboot.
Now it seems that i cant reassemble it.

[root@boll ~]# mdadm --misc --detail /dev/md0
mdadm: md device /dev/md0 does not appear to be active.
[root@boll ~]# cat /proc/mdstat
Personalities :
md0 : inactive sdk1[9](S) sdh1[6](S) sdl1[10](S) sdi1[7](S) sdb1[1](S) sdd1[3](S) sde1[4](S) sdc1[2](S) sdg1[5](S) sdj1[8](S) sda1[0](S)
10744004151 blocks super 1.2
unused devices: <none>
[root@boll ~]# mdadm --assemble /dev/md0 --uuid=263e0a3e:b39b3ee2:c19575d2:b275c773
[root@boll ~]# mdadm --misc --detail /dev/md0
mdadm: md device /dev/md0 does not appear to be active.
[root@boll ~]# mdadm --assemble /dev/md0 /dev/sda1 /dev/sdb1 /dev/sdc1 /dev/sdd1 /dev/sde1 /dev/sdg1 /dev/sdh1 /dev/sdi1 /dev/sdj1 /dev/sdk1 /dev/sdl1 --verbose
mdadm: looking for devices for /dev/md0
mdadm: cannot open device /dev/sda1: Device or resource busy
mdadm: /dev/sda1 has no superblock - assembly aborted
[root@boll ~]# mdadm --assemble --scan --verbose
mdadm: looking for devices for /dev/md/0
mdadm: cannot open device /dev/sdl1: Device or resource busy
mdadm: cannot open device /dev/sdl: Device or resource busy
mdadm: cannot open device /dev/sdk1: Device or resource busy
mdadm: cannot open device /dev/sdk: Device or resource busy
mdadm: cannot open device /dev/sdj1: Device or resource busy
mdadm: cannot open device /dev/sdj: Device or resource busy
mdadm: cannot open device /dev/sdi1: Device or resource busy
mdadm: cannot open device /dev/sdi: Device or resource busy
mdadm: cannot open device /dev/sdh1: Device or resource busy
mdadm: cannot open device /dev/sdh: Device or resource busy
mdadm: cannot open device /dev/sdg1: Device or resource busy
mdadm: cannot open device /dev/sdg: Device or resource busy
mdadm: cannot open device /dev/sdf4: Device or resource busy
mdadm: cannot open device /dev/sdf3: Device or resource busy
mdadm: cannot open device /dev/sdf2: Device or resource busy
mdadm: cannot open device /dev/sdf1: Device or resource busy
mdadm: cannot open device /dev/sdf: Device or resource busy
mdadm: cannot open device /dev/sde1: Device or resource busy
mdadm: cannot open device /dev/sde: Device or resource busy
mdadm: cannot open device /dev/sdd1: Device or resource busy
mdadm: cannot open device /dev/sdd: Device or resource busy
mdadm: cannot open device /dev/sdc1: Device or resource busy
mdadm: cannot open device /dev/sdc: Device or resource busy
mdadm: cannot open device /dev/sdb1: Device or resource busy
mdadm: cannot open device /dev/sdb: Device or resource busy
mdadm: cannot open device /dev/sda1: Device or resource busy
mdadm: cannot open device /dev/sda: Device or resource busy

It says that sda1 doesnt have a superblock so I try to examine

[root@boll ~]# mdadm --examine /dev/sd[abcdeghijkl]1 | egrep 'Update|State'
State : clean
Update Time : Sat Oct 29 23:11:31 2011
Array State : AAAAA...... ('A' == active, '.' == missing)
State : clean
Update Time : Sat Oct 29 23:11:31 2011
Array State : AAAAA...... ('A' == active, '.' == missing)
State : clean
Update Time : Sat Oct 29 23:11:31 2011
Array State : AAAAA...... ('A' == active, '.' == missing)
State : clean
Update Time : Sat Oct 29 23:11:31 2011
Array State : AAAAA...... ('A' == active, '.' == missing)
State : clean
Update Time : Sat Oct 29 23:11:31 2011
Array State : AAAAA...... ('A' == active, '.' == missing)
State : clean
Update Time : Sat Oct 29 15:43:47 2011
Array State : AAAAAAAAAAA ('A' == active, '.' == missing)
State : clean
Update Time : Sat Oct 29 15:43:47 2011
Array State : AAAAAAAAAAA ('A' == active, '.' == missing)
State : clean
Update Time : Sat Oct 29 15:43:47 2011
Array State : AAAAAAAAAAA ('A' == active, '.' == missing)
State : clean
Update Time : Sat Oct 29 15:43:47 2011
Array State : AAAAAAAAAAA ('A' == active, '.' == missing)
State : clean
Update Time : Sat Oct 29 15:43:47 2011
Array State : AAAAAAAAAAA ('A' == active, '.' == missing)
State : clean
Update Time : Sat Oct 29 15:43:47 2011
Array State : AAAAAAAAAAA ('A' == active, '.' == missing)

[root@boll ~]# blkid
/dev/sdb1: UUID="263e0a3e-b39b-3ee2-c195-75d2b275c773" UUID_SUB="cb92ccf9-45b9-0e02-8191-928c457040bb" LABEL="BOLL.EVIL.NU:0" TYPE="linux_raid_member"
/dev/sdk1: UUID="263e0a3e-b39b-3ee2-c195-75d2b275c773" UUID_SUB="6fcd04dd-42e0-e263-3aab-30301fd19671" LABEL="BOLL.EVIL.NU:0" TYPE="linux_raid_member"
/dev/sda1: UUID="263e0a3e-b39b-3ee2-c195-75d2b275c773" UUID_SUB="3c772400-41f8-6077-db19-ebfe4f0e0830" LABEL="BOLL.EVIL.NU:0" TYPE="linux_raid_member"
/dev/sdc1: UUID="263e0a3e-b39b-3ee2-c195-75d2b275c773" UUID_SUB="b3b8d1ed-2c5d-c202-5dca-3a4c1d6c7d96" LABEL="BOLL.EVIL.NU:0" TYPE="linux_raid_member"
/dev/sdd1: UUID="263e0a3e-b39b-3ee2-c195-75d2b275c773" UUID_SUB="cf0dbcfc-6be6-3c58-194a-9ccfb730d35a" LABEL="BOLL.EVIL.NU:0" TYPE="linux_raid_member"
/dev/sde1: UUID="263e0a3e-b39b-3ee2-c195-75d2b275c773" UUID_SUB="619e7521-2e6d-be78-47fd-33738baf97c1" LABEL="BOLL.EVIL.NU:0" TYPE="linux_raid_member"
/dev/sdf1: UUID="90f83e41-c368-42fb-9b61-4d668a0c77a3" TYPE="ext2"
/dev/sdf2: UUID="7f15666a-f415-40a2-bbd7-6d84237f91df" TYPE="swap"
/dev/sdf3: UUID="e8c80933-96b1-48f4-8a12-aacf85d97bbc" TYPE="ext4"
/dev/sdf4: UUID="f781c68e-ac48-4f6e-8220-784bedd640d3" TYPE="ext4"
/dev/sdg1: UUID="263e0a3e-b39b-3ee2-c195-75d2b275c773" UUID_SUB="520d893a-6ccb-b81b-9833-5ab4a75092e6" LABEL="BOLL.EVIL.NU:0" TYPE="linux_raid_member"
/dev/sdh1: UUID="263e0a3e-b39b-3ee2-c195-75d2b275c773" UUID_SUB="cf817fbb-7bd6-cc75-7a69-2604d15292bd" LABEL="BOLL.EVIL.NU:0" TYPE="linux_raid_member"
/dev/sdi1: UUID="263e0a3e-b39b-3ee2-c195-75d2b275c773" UUID_SUB="4d730c77-77df-3cf3-04f0-9462cc5aad19" LABEL="BOLL.EVIL.NU:0" TYPE="linux_raid_member"
/dev/sdj1: UUID="263e0a3e-b39b-3ee2-c195-75d2b275c773" UUID_SUB="152f7633-bc93-23f4-ee47-e0d8545ed6b8" LABEL="BOLL.EVIL.NU:0" TYPE="linux_raid_member"
/dev/sdl1: UUID="263e0a3e-b39b-3ee2-c195-75d2b275c773" UUID_SUB="625d1e4e-7593-4ef4-5f96-27acb69d28e1" LABEL="BOLL.EVIL.NU:0" TYPE="linux_raid_member"

How do I move on? Im kinda lost at this point.
Mdadm 1.2
Vanilla 3.0-ARCH

Appreciate all the help I can get

Last edited by maz (2011-11-02 16:32:19)

karol · 2011-10-30 15:59:28

When pasting code, please use [ code ], not [ quote ] tags https://bbs.archlinux.org/help.php#bbcode

like this

It makes the code more readable and more convenient to scroll through.

lilsirecho · 2011-10-30 17:53:25

Perhaps you can examine the individual drives to establish their status for clues to the solution?

maz · 2011-10-30 18:35:43

Karol: Thank you for that, I will do it the next time

lilseriecho:
I have done that aswell. It all seems fine, the checksums are correct and they are clean according to mdadm.
If I try to create the array again will it rewrite the array or does it try to assemble it again?

According to this link it will work, however I dont know if it will work properly with the whole array when 5 were listed as "faulty spares" because of the controller breakdown.
http://en.wikipedia.org/wiki/Mdadm#Zero … superblock

lilsirecho · 2011-10-30 18:52:12

Don't use create but try :

--assemble --scan /dev/md(x) where X is your raid dev

You may have to use STOP before trying the --assemble.

Then look into faulty in man mdadm to determine how to fix the faults.

You may have to fix the faults on each one at a time,,,,

Haven't had the pleasure!!!

maz · 2011-10-30 19:11:03

lilsirecho wrote:

Don't use create but try :
--assemble --scan /dev/md(x) where X is your raid dev
You may have to use STOP before trying the --assemble.
Then look into faulty in man mdadm to determine how to fix the faults.
You may have to fix the faults on each one at a time,,,,
Haven't had the pleasure!!!

Oh, I totally forgot about stopping the array... feel a bit stupid now.
So I stopped the array and tried to assemble it again and this appears:

[root@boll etc]# mdadm --assemble --scan -v
mdadm: looking for devices for /dev/md/0
mdadm: no RAID superblock on /dev/sdl
mdadm: no RAID superblock on /dev/sdk
mdadm: no RAID superblock on /dev/sdj
mdadm: no RAID superblock on /dev/sdi
mdadm: no RAID superblock on /dev/sdh
mdadm: no RAID superblock on /dev/sdg
mdadm: cannot open device /dev/sdf4: Device or resource busy
mdadm: cannot open device /dev/sdf3: Device or resource busy
mdadm: cannot open device /dev/sdf2: Device or resource busy
mdadm: cannot open device /dev/sdf1: Device or resource busy
mdadm: cannot open device /dev/sdf: Device or resource busy
mdadm: no RAID superblock on /dev/sde
mdadm: no RAID superblock on /dev/sdd
mdadm: no RAID superblock on /dev/sdc
mdadm: no RAID superblock on /dev/sdb
mdadm: no RAID superblock on /dev/sda
mdadm: /dev/sdl1 is identified as a member of /dev/md/0, slot 10.
mdadm: /dev/sdk1 is identified as a member of /dev/md/0, slot 9.
mdadm: /dev/sdj1 is identified as a member of /dev/md/0, slot 8.
mdadm: /dev/sdi1 is identified as a member of /dev/md/0, slot 7.
mdadm: /dev/sdh1 is identified as a member of /dev/md/0, slot 6.
mdadm: /dev/sdg1 is identified as a member of /dev/md/0, slot 5.
mdadm: /dev/sde1 is identified as a member of /dev/md/0, slot 4.
mdadm: /dev/sdd1 is identified as a member of /dev/md/0, slot 3.
mdadm: /dev/sdc1 is identified as a member of /dev/md/0, slot 2.
mdadm: /dev/sdb1 is identified as a member of /dev/md/0, slot 1.
mdadm: /dev/sda1 is identified as a member of /dev/md/0, slot 0.
mdadm: added /dev/sdb1 to /dev/md/0 as 1
mdadm: added /dev/sdc1 to /dev/md/0 as 2
mdadm: added /dev/sdd1 to /dev/md/0 as 3
mdadm: added /dev/sde1 to /dev/md/0 as 4
mdadm: added /dev/sdg1 to /dev/md/0 as 5
mdadm: added /dev/sdh1 to /dev/md/0 as 6
mdadm: added /dev/sdi1 to /dev/md/0 as 7
mdadm: added /dev/sdj1 to /dev/md/0 as 8
mdadm: added /dev/sdk1 to /dev/md/0 as 9
mdadm: added /dev/sdl1 to /dev/md/0 as 10
mdadm: added /dev/sda1 to /dev/md/0 as 0
mdadm: /dev/md/0 assembled from 5 drives - not enough to start the array.

The 6 drives that it cant assemble without are from the controller card that malfuntioned. Does this mean that Im out of luck?

lilsirecho · 2011-10-30 19:59:09

My experience with raid arrays indicates that if I try to create an array that attempts to utilize devices already identified as raid devices will cause the mdadm to report that fact. However, it can then "continue to create array" and it does create a new array as desired.

This may work in your case. although the faulty problem may interfere with a new create.

You may have to run...mdadm -D --scan>>/etc/mdadm.conf after the array is created.

I assume the create activity can be aborted if it doesn't look like it will generate a new array.

Thus you probably aren't at further risk if you try the create approach.

EDIT: I am booting in raid0 linux3.0 with two CF cards in this posting.

Last edited by lilsirecho (2011-10-30 20:09:24)

maz · 2011-10-30 20:15:07

The thing is that I still want to access the data on the array so creating a totally new array is the last thing I'll do.

lilsirecho · 2011-10-30 20:31:58

Agreed:

Therefore, the solution to your dilemma is to search for info that gives you the reason why the devices are busy.

It is usually due to another array name in md.

Try .....ls -l |grep md for instance.

You should be able to assemble an array as I described if the devices are not busy.

Perhaps you can try mounting and unmounting each array device to disarm them.

Perhaps a clue in blkid.

Also, ls /dev |grep md

Or just ...ls /dev....big list!

A different array identified therein would cause the devices to be busy.

You may have to chroot to handle this problem as well. Therein try the assemble --scan approach.

maz · 2011-10-30 20:50:43

/dev/sdf is the system hard drive so thats not a problem itself.

As I stated before i stopped the array (/dev/md0) and then tried to assemble it again and it says mdadm: /dev/md/0 assembled from 5 drives - not enough to start the array.
I have no idea on why it will only try 5 instead of 11 drives or why the other 6 drives wont assemble with them. mdstat still shows that all drives are inactive and there is no sign of the array in /dev

lilsirecho · 2011-10-30 21:10:47

Perhaps the 6 drives not included in /md0 assemble command are the faulty drives and mdadm procedure for faulty needs to be applied to those specific drives.

--assemble --scan /dev/md0 should assemble all drives that are scanned by mdadm and may not include faulty drives.

You may have to ID the faulty drives and add them one by one to md0 assemble.

maz · 2011-10-30 21:59:09

I have identified the 6 drives now. How do I add them again when I cant start the array?

Even if I assemble the manual way:

mdadm --assemble /dev/md0 /dev/sda1 /dev/sdb1 /dev/sdc1 /dev/sdd1 /dev/sde1 /dev/sdg1 /dev/sdh1 /dev/sdi1 /dev/sdj1 /dev/sdk1 /dev/sdl1 --verbose

(dev/sdg1 /dev/sdh1 /dev/sdi1 /dev/sdj1 /dev/sdk1 /dev/sdl1) are the faulty ones

..It wont start. Do I have to stop the array then mark each and every faulty drives as --fail and then readd them? Does that even work since I have 6 "faulty" drives?
Im going through the wiki but I cant pin down the solution

lilsirecho · 2011-10-30 22:06:57

Perhaps man mdadm has more direct info for --add

Thus --assemble --scan /dev/md0 (Drives 1 thru 5 --add /drive 6)

Then do another for 7 and so on.''EDIT: You may need --grow command in trying to --add a drive.

Last edited by lilsirecho (2011-10-30 22:42:22)

laloch · 2011-10-31 09:20:09

mdadm: added /dev/sdb1 to /dev/md/0 as 1
mdadm: added /dev/sdc1 to /dev/md/0 as 2
mdadm: added /dev/sdd1 to /dev/md/0 as 3
mdadm: added /dev/sde1 to /dev/md/0 as 4
mdadm: added /dev/sdg1 to /dev/md/0 as 5
mdadm: added /dev/sdh1 to /dev/md/0 as 6
mdadm: added /dev/sdi1 to /dev/md/0 as 7
mdadm: added /dev/sdj1 to /dev/md/0 as 8
mdadm: added /dev/sdk1 to /dev/md/0 as 9
mdadm: added /dev/sdl1 to /dev/md/0 as 10
mdadm: added /dev/sda1 to /dev/md/0 as 0
mdadm: /dev/md/0 assembled from 5 drives - not enough to start the array.

How does the /proc/mdstat look after that?

maz · 2011-10-31 12:09:34

laloch wrote:

How does the /proc/mdstat look after that?

It says:

mdadm: md device /dev/md0 does not appear to be active.

After searching for a long time yesterday, from the tip I got, I came up with this
http://osdir.com/ml/linux-raid/2009-06/msg00018.html

mdadm --create /dev/md0 --level=6 --chunk=64 --raid-devices=11 /dev/sda1 /dev/sdb1 /dev/sdc1 /dev/sdd1 /dev/sde1 /dev/sdg1 /dev/sdh1 /dev/sdi1 /dev/sdj1 /dev/sdk1 /dev/sdl1

I modified it to my raid6 and my drives and decided to "recreate" but I think it wrecked the whole array. Its resyncing the array now but I think that my data is forever lost.

maz · 2011-11-02 16:31:14

After 2 days the resync is complete and it worked! All my data is still there. Time for backup and disassemble the array.
Thank you guys.

lilsirecho · 2011-11-02 17:28:39

Well done maz!!!!

Arch Linux

#1 2011-10-30 11:58:01

[SOLVED] mdadm, raid6 and superblock missing

#2 2011-10-30 15:59:28

Re: [SOLVED] mdadm, raid6 and superblock missing

#3 2011-10-30 17:53:25

Re: [SOLVED] mdadm, raid6 and superblock missing

#4 2011-10-30 18:35:43

Re: [SOLVED] mdadm, raid6 and superblock missing

#5 2011-10-30 18:52:12

Re: [SOLVED] mdadm, raid6 and superblock missing

#6 2011-10-30 19:11:03

Re: [SOLVED] mdadm, raid6 and superblock missing

#7 2011-10-30 19:59:09

Re: [SOLVED] mdadm, raid6 and superblock missing

#8 2011-10-30 20:15:07

Re: [SOLVED] mdadm, raid6 and superblock missing

#9 2011-10-30 20:31:58

Re: [SOLVED] mdadm, raid6 and superblock missing

#10 2011-10-30 20:50:43

Re: [SOLVED] mdadm, raid6 and superblock missing

#11 2011-10-30 21:10:47

Re: [SOLVED] mdadm, raid6 and superblock missing

#12 2011-10-30 21:59:09

Re: [SOLVED] mdadm, raid6 and superblock missing

#13 2011-10-30 22:06:57

Re: [SOLVED] mdadm, raid6 and superblock missing

#14 2011-10-31 09:20:09

Re: [SOLVED] mdadm, raid6 and superblock missing

#15 2011-10-31 12:09:34

Re: [SOLVED] mdadm, raid6 and superblock missing

#16 2011-11-02 16:31:14

Re: [SOLVED] mdadm, raid6 and superblock missing

#17 2011-11-02 17:28:39

Re: [SOLVED] mdadm, raid6 and superblock missing

Board footer