You are not logged in.

#1 2011-10-30 11:58:01

maz
Member
Registered: 2006-05-05
Posts: 38

[SOLVED] mdadm, raid6 and superblock missing

I was unfortunate enough to have a hardware failure yesterday. One of my controller cards died for a second which meant that my array stopped.

When trying to assemble it again it was stubborn and I decided to just reboot.
Now it seems that i cant reassemble it.

[root@boll ~]# mdadm --misc --detail /dev/md0
mdadm: md device /dev/md0 does not appear to be active.

[root@boll ~]# cat /proc/mdstat
Personalities :
md0 : inactive sdk1[9](S) sdh1[6](S) sdl1[10](S) sdi1[7](S) sdb1[1](S) sdd1[3](S) sde1[4](S) sdc1[2](S) sdg1[5](S) sdj1[8](S) sda1[0](S)
      10744004151 blocks super 1.2

unused devices: <none>
[root@boll ~]# mdadm --assemble /dev/md0 --uuid=263e0a3e:b39b3ee2:c19575d2:b275c773

[root@boll ~]# mdadm --misc --detail /dev/md0
mdadm: md device /dev/md0 does not appear to be active.

[root@boll ~]# mdadm --assemble /dev/md0 /dev/sda1 /dev/sdb1 /dev/sdc1 /dev/sdd1 /dev/sde1 /dev/sdg1 /dev/sdh1 /dev/sdi1 /dev/sdj1 /dev/sdk1 /dev/sdl1 --verbose
mdadm: looking for devices for /dev/md0
mdadm: cannot open device /dev/sda1: Device or resource busy
mdadm: /dev/sda1 has no superblock - assembly aborted

[root@boll ~]# mdadm --assemble --scan --verbose
mdadm: looking for devices for /dev/md/0
mdadm: cannot open device /dev/sdl1: Device or resource busy
mdadm: cannot open device /dev/sdl: Device or resource busy
mdadm: cannot open device /dev/sdk1: Device or resource busy
mdadm: cannot open device /dev/sdk: Device or resource busy
mdadm: cannot open device /dev/sdj1: Device or resource busy
mdadm: cannot open device /dev/sdj: Device or resource busy
mdadm: cannot open device /dev/sdi1: Device or resource busy
mdadm: cannot open device /dev/sdi: Device or resource busy
mdadm: cannot open device /dev/sdh1: Device or resource busy
mdadm: cannot open device /dev/sdh: Device or resource busy
mdadm: cannot open device /dev/sdg1: Device or resource busy
mdadm: cannot open device /dev/sdg: Device or resource busy
mdadm: cannot open device /dev/sdf4: Device or resource busy
mdadm: cannot open device /dev/sdf3: Device or resource busy
mdadm: cannot open device /dev/sdf2: Device or resource busy
mdadm: cannot open device /dev/sdf1: Device or resource busy
mdadm: cannot open device /dev/sdf: Device or resource busy
mdadm: cannot open device /dev/sde1: Device or resource busy
mdadm: cannot open device /dev/sde: Device or resource busy
mdadm: cannot open device /dev/sdd1: Device or resource busy
mdadm: cannot open device /dev/sdd: Device or resource busy
mdadm: cannot open device /dev/sdc1: Device or resource busy
mdadm: cannot open device /dev/sdc: Device or resource busy
mdadm: cannot open device /dev/sdb1: Device or resource busy
mdadm: cannot open device /dev/sdb: Device or resource busy
mdadm: cannot open device /dev/sda1: Device or resource busy
mdadm: cannot open device /dev/sda: Device or resource busy

It says that sda1 doesnt have a superblock so I try to examine

[root@boll ~]# mdadm --examine /dev/sd[abcdeghijkl]1 | egrep 'Update|State'
          State : clean
    Update Time : Sat Oct 29 23:11:31 2011
   Array State : AAAAA...... ('A' == active, '.' == missing)
          State : clean
    Update Time : Sat Oct 29 23:11:31 2011
   Array State : AAAAA...... ('A' == active, '.' == missing)
          State : clean
    Update Time : Sat Oct 29 23:11:31 2011
   Array State : AAAAA...... ('A' == active, '.' == missing)
          State : clean
    Update Time : Sat Oct 29 23:11:31 2011
   Array State : AAAAA...... ('A' == active, '.' == missing)
          State : clean
    Update Time : Sat Oct 29 23:11:31 2011
   Array State : AAAAA...... ('A' == active, '.' == missing)
          State : clean
    Update Time : Sat Oct 29 15:43:47 2011
   Array State : AAAAAAAAAAA ('A' == active, '.' == missing)
          State : clean
    Update Time : Sat Oct 29 15:43:47 2011
   Array State : AAAAAAAAAAA ('A' == active, '.' == missing)
          State : clean
    Update Time : Sat Oct 29 15:43:47 2011
   Array State : AAAAAAAAAAA ('A' == active, '.' == missing)
          State : clean
    Update Time : Sat Oct 29 15:43:47 2011
   Array State : AAAAAAAAAAA ('A' == active, '.' == missing)
          State : clean
    Update Time : Sat Oct 29 15:43:47 2011
   Array State : AAAAAAAAAAA ('A' == active, '.' == missing)
          State : clean
    Update Time : Sat Oct 29 15:43:47 2011
   Array State : AAAAAAAAAAA ('A' == active, '.' == missing)

[root@boll ~]# blkid
/dev/sdb1: UUID="263e0a3e-b39b-3ee2-c195-75d2b275c773" UUID_SUB="cb92ccf9-45b9-0e02-8191-928c457040bb" LABEL="BOLL.EVIL.NU:0" TYPE="linux_raid_member"
/dev/sdk1: UUID="263e0a3e-b39b-3ee2-c195-75d2b275c773" UUID_SUB="6fcd04dd-42e0-e263-3aab-30301fd19671" LABEL="BOLL.EVIL.NU:0" TYPE="linux_raid_member"
/dev/sda1: UUID="263e0a3e-b39b-3ee2-c195-75d2b275c773" UUID_SUB="3c772400-41f8-6077-db19-ebfe4f0e0830" LABEL="BOLL.EVIL.NU:0" TYPE="linux_raid_member"
/dev/sdc1: UUID="263e0a3e-b39b-3ee2-c195-75d2b275c773" UUID_SUB="b3b8d1ed-2c5d-c202-5dca-3a4c1d6c7d96" LABEL="BOLL.EVIL.NU:0" TYPE="linux_raid_member"
/dev/sdd1: UUID="263e0a3e-b39b-3ee2-c195-75d2b275c773" UUID_SUB="cf0dbcfc-6be6-3c58-194a-9ccfb730d35a" LABEL="BOLL.EVIL.NU:0" TYPE="linux_raid_member"
/dev/sde1: UUID="263e0a3e-b39b-3ee2-c195-75d2b275c773" UUID_SUB="619e7521-2e6d-be78-47fd-33738baf97c1" LABEL="BOLL.EVIL.NU:0" TYPE="linux_raid_member"
/dev/sdf1: UUID="90f83e41-c368-42fb-9b61-4d668a0c77a3" TYPE="ext2"
/dev/sdf2: UUID="7f15666a-f415-40a2-bbd7-6d84237f91df" TYPE="swap"
/dev/sdf3: UUID="e8c80933-96b1-48f4-8a12-aacf85d97bbc" TYPE="ext4"
/dev/sdf4: UUID="f781c68e-ac48-4f6e-8220-784bedd640d3" TYPE="ext4"
/dev/sdg1: UUID="263e0a3e-b39b-3ee2-c195-75d2b275c773" UUID_SUB="520d893a-6ccb-b81b-9833-5ab4a75092e6" LABEL="BOLL.EVIL.NU:0" TYPE="linux_raid_member"
/dev/sdh1: UUID="263e0a3e-b39b-3ee2-c195-75d2b275c773" UUID_SUB="cf817fbb-7bd6-cc75-7a69-2604d15292bd" LABEL="BOLL.EVIL.NU:0" TYPE="linux_raid_member"
/dev/sdi1: UUID="263e0a3e-b39b-3ee2-c195-75d2b275c773" UUID_SUB="4d730c77-77df-3cf3-04f0-9462cc5aad19" LABEL="BOLL.EVIL.NU:0" TYPE="linux_raid_member"
/dev/sdj1: UUID="263e0a3e-b39b-3ee2-c195-75d2b275c773" UUID_SUB="152f7633-bc93-23f4-ee47-e0d8545ed6b8" LABEL="BOLL.EVIL.NU:0" TYPE="linux_raid_member"
/dev/sdl1: UUID="263e0a3e-b39b-3ee2-c195-75d2b275c773" UUID_SUB="625d1e4e-7593-4ef4-5f96-27acb69d28e1" LABEL="BOLL.EVIL.NU:0" TYPE="linux_raid_member"


How do I move on? Im kinda lost at this point.
Mdadm 1.2
Vanilla 3.0-ARCH

Appreciate all the help I can get

Last edited by maz (2011-11-02 16:32:19)

Offline

#2 2011-10-30 15:59:28

karol
Archivist
Registered: 2009-05-06
Posts: 25,440

Re: [SOLVED] mdadm, raid6 and superblock missing

When pasting code, please use [ code ], not [ quote ] tags https://bbs.archlinux.org/help.php#bbcode

like this

It makes the code more readable and more convenient to scroll through.

Offline

#3 2011-10-30 17:53:25

lilsirecho
Veteran
Registered: 2003-10-24
Posts: 5,000

Re: [SOLVED] mdadm, raid6 and superblock missing

Perhaps you can examine the individual drives to establish their status for clues to the solution?


Prediction...This year will be a very odd year!
Hard work does not kill people but why risk it: Charlie Mccarthy
A man is not complete until he is married..then..he is finished.
When ALL is lost, what can be found? Even bytes get lonely for a little bit!     X-ray confirms Iam spineless!

Offline

#4 2011-10-30 18:35:43

maz
Member
Registered: 2006-05-05
Posts: 38

Re: [SOLVED] mdadm, raid6 and superblock missing

Karol: Thank you for that, I will do it the next time smile

lilseriecho:
I have done that aswell. It all seems fine, the checksums are correct and they are clean according to mdadm.
If I try to create the array again will it rewrite the array or does it try to assemble it again?

According to this link it will work, however I dont know if it will work properly with the whole array when 5 were listed as "faulty spares" because of the controller breakdown.
http://en.wikipedia.org/wiki/Mdadm#Zero … superblock

Offline

#5 2011-10-30 18:52:12

lilsirecho
Veteran
Registered: 2003-10-24
Posts: 5,000

Re: [SOLVED] mdadm, raid6 and superblock missing

Don't use create but try :

--assemble --scan /dev/md(x)   where X is your raid dev

You may have to use STOP before trying the --assemble.

Then look into faulty in man mdadm to determine how to fix the faults.

You may have to fix the faults on each one at a time,,,,

Haven't had the pleasure!!!


Prediction...This year will be a very odd year!
Hard work does not kill people but why risk it: Charlie Mccarthy
A man is not complete until he is married..then..he is finished.
When ALL is lost, what can be found? Even bytes get lonely for a little bit!     X-ray confirms Iam spineless!

Offline

#6 2011-10-30 19:11:03

maz
Member
Registered: 2006-05-05
Posts: 38

Re: [SOLVED] mdadm, raid6 and superblock missing

lilsirecho wrote:

Don't use create but try :

--assemble --scan /dev/md(x)   where X is your raid dev

You may have to use STOP before trying the --assemble.

Then look into faulty in man mdadm to determine how to fix the faults.

You may have to fix the faults on each one at a time,,,,

Haven't had the pleasure!!!

Oh, I totally forgot about stopping the array... feel a bit stupid now.
So I stopped the array and tried to assemble it again and this appears:

[root@boll etc]# mdadm --assemble --scan -v
mdadm: looking for devices for /dev/md/0
mdadm: no RAID superblock on /dev/sdl
mdadm: no RAID superblock on /dev/sdk
mdadm: no RAID superblock on /dev/sdj
mdadm: no RAID superblock on /dev/sdi
mdadm: no RAID superblock on /dev/sdh
mdadm: no RAID superblock on /dev/sdg
mdadm: cannot open device /dev/sdf4: Device or resource busy
mdadm: cannot open device /dev/sdf3: Device or resource busy
mdadm: cannot open device /dev/sdf2: Device or resource busy
mdadm: cannot open device /dev/sdf1: Device or resource busy
mdadm: cannot open device /dev/sdf: Device or resource busy
mdadm: no RAID superblock on /dev/sde
mdadm: no RAID superblock on /dev/sdd
mdadm: no RAID superblock on /dev/sdc
mdadm: no RAID superblock on /dev/sdb
mdadm: no RAID superblock on /dev/sda
mdadm: /dev/sdl1 is identified as a member of /dev/md/0, slot 10.
mdadm: /dev/sdk1 is identified as a member of /dev/md/0, slot 9.
mdadm: /dev/sdj1 is identified as a member of /dev/md/0, slot 8.
mdadm: /dev/sdi1 is identified as a member of /dev/md/0, slot 7.
mdadm: /dev/sdh1 is identified as a member of /dev/md/0, slot 6.
mdadm: /dev/sdg1 is identified as a member of /dev/md/0, slot 5.
mdadm: /dev/sde1 is identified as a member of /dev/md/0, slot 4.
mdadm: /dev/sdd1 is identified as a member of /dev/md/0, slot 3.
mdadm: /dev/sdc1 is identified as a member of /dev/md/0, slot 2.
mdadm: /dev/sdb1 is identified as a member of /dev/md/0, slot 1.
mdadm: /dev/sda1 is identified as a member of /dev/md/0, slot 0.
mdadm: added /dev/sdb1 to /dev/md/0 as 1
mdadm: added /dev/sdc1 to /dev/md/0 as 2
mdadm: added /dev/sdd1 to /dev/md/0 as 3
mdadm: added /dev/sde1 to /dev/md/0 as 4
mdadm: added /dev/sdg1 to /dev/md/0 as 5
mdadm: added /dev/sdh1 to /dev/md/0 as 6
mdadm: added /dev/sdi1 to /dev/md/0 as 7
mdadm: added /dev/sdj1 to /dev/md/0 as 8
mdadm: added /dev/sdk1 to /dev/md/0 as 9
mdadm: added /dev/sdl1 to /dev/md/0 as 10
mdadm: added /dev/sda1 to /dev/md/0 as 0
mdadm: /dev/md/0 assembled from 5 drives - not enough to start the array.

The 6 drives that it cant assemble without are from the controller card that malfuntioned. Does this mean that Im out of luck? hmm

Offline

#7 2011-10-30 19:59:09

lilsirecho
Veteran
Registered: 2003-10-24
Posts: 5,000

Re: [SOLVED] mdadm, raid6 and superblock missing

My experience with raid arrays indicates that if I try to create an array that attempts to utilize devices already identified as raid devices will cause the mdadm to report that fact.  However, it can then "continue to create array" and it does create a new array as desired.

This may work in your case. although the faulty problem may interfere with a new create. 

You may have to run...mdadm -D --scan>>/etc/mdadm.conf after the array is created.

I assume the create activity can be aborted if it doesn't look like it will generate a new array.

Thus you probably aren't at further risk if you try the create approach.

EDIT:  I am booting in raid0 linux3.0 with two CF cards in this posting.

Last edited by lilsirecho (2011-10-30 20:09:24)


Prediction...This year will be a very odd year!
Hard work does not kill people but why risk it: Charlie Mccarthy
A man is not complete until he is married..then..he is finished.
When ALL is lost, what can be found? Even bytes get lonely for a little bit!     X-ray confirms Iam spineless!

Offline

#8 2011-10-30 20:15:07

maz
Member
Registered: 2006-05-05
Posts: 38

Re: [SOLVED] mdadm, raid6 and superblock missing

The thing is that I still want to access the data on the array so creating a totally new array is the last thing I'll do.

Offline

#9 2011-10-30 20:31:58

lilsirecho
Veteran
Registered: 2003-10-24
Posts: 5,000

Re: [SOLVED] mdadm, raid6 and superblock missing

Agreed:

Therefore, the solution to your dilemma is to search for info that gives you the reason why the devices are busy. 

It is usually due to another array name in md.

Try .....ls -l |grep md for instance.

You should be able to assemble an array as I described if the devices are not busy.

Perhaps you can try mounting and unmounting each array device to disarm them.

Perhaps a clue in blkid.

Also, ls /dev |grep md

Or just ...ls /dev....big list!

A different array identified therein would cause the devices to be busy.

You may have to chroot to handle this problem as well.  Therein try the assemble --scan approach.


Prediction...This year will be a very odd year!
Hard work does not kill people but why risk it: Charlie Mccarthy
A man is not complete until he is married..then..he is finished.
When ALL is lost, what can be found? Even bytes get lonely for a little bit!     X-ray confirms Iam spineless!

Offline

#10 2011-10-30 20:50:43

maz
Member
Registered: 2006-05-05
Posts: 38

Re: [SOLVED] mdadm, raid6 and superblock missing

/dev/sdf is the system hard drive so thats not a problem itself.

As I stated before i stopped the array (/dev/md0) and then tried to assemble it again and it says mdadm: /dev/md/0 assembled from 5 drives - not enough to start the array.
I have no idea on why it will only try 5 instead of 11 drives or why the other 6 drives wont assemble with them. mdstat still shows that all drives are inactive and there is no sign of the array in /dev

Offline

#11 2011-10-30 21:10:47

lilsirecho
Veteran
Registered: 2003-10-24
Posts: 5,000

Re: [SOLVED] mdadm, raid6 and superblock missing

Perhaps the 6 drives not included in /md0 assemble command are the faulty drives and mdadm procedure for faulty needs to be applied to those specific drives.

--assemble --scan /dev/md0 should assemble all drives that are scanned by mdadm and may not include faulty drives.

You may have to ID the faulty drives and add them one by one to md0 assemble.


Prediction...This year will be a very odd year!
Hard work does not kill people but why risk it: Charlie Mccarthy
A man is not complete until he is married..then..he is finished.
When ALL is lost, what can be found? Even bytes get lonely for a little bit!     X-ray confirms Iam spineless!

Offline

#12 2011-10-30 21:59:09

maz
Member
Registered: 2006-05-05
Posts: 38

Re: [SOLVED] mdadm, raid6 and superblock missing

I have identified the 6 drives now. How do I add them again when I cant start the array?

Even if I assemble the manual way:

mdadm --assemble /dev/md0 /dev/sda1 /dev/sdb1 /dev/sdc1 /dev/sdd1 /dev/sde1 /dev/sdg1 /dev/sdh1 /dev/sdi1 /dev/sdj1 /dev/sdk1 /dev/sdl1 --verbose

(dev/sdg1 /dev/sdh1 /dev/sdi1 /dev/sdj1 /dev/sdk1 /dev/sdl1) are the faulty ones

..It wont start. Do I have to stop the array then mark each and every faulty drives as --fail and then readd them? Does that even work since I have 6 "faulty" drives?
Im going through the wiki but I cant pin down the solution sad

Offline

#13 2011-10-30 22:06:57

lilsirecho
Veteran
Registered: 2003-10-24
Posts: 5,000

Re: [SOLVED] mdadm, raid6 and superblock missing

Perhaps man mdadm has more direct info for --add

Thus --assemble --scan /dev/md0 (Drives 1 thru 5 --add /drive 6)

Then do another for 7 and so on.''EDIT:  You may need --grow command in trying to --add a drive.

Last edited by lilsirecho (2011-10-30 22:42:22)


Prediction...This year will be a very odd year!
Hard work does not kill people but why risk it: Charlie Mccarthy
A man is not complete until he is married..then..he is finished.
When ALL is lost, what can be found? Even bytes get lonely for a little bit!     X-ray confirms Iam spineless!

Offline

#14 2011-10-31 09:20:09

laloch
Member
Registered: 2010-02-04
Posts: 186

Re: [SOLVED] mdadm, raid6 and superblock missing

mdadm: added /dev/sdb1 to /dev/md/0 as 1
mdadm: added /dev/sdc1 to /dev/md/0 as 2
mdadm: added /dev/sdd1 to /dev/md/0 as 3
mdadm: added /dev/sde1 to /dev/md/0 as 4
mdadm: added /dev/sdg1 to /dev/md/0 as 5
mdadm: added /dev/sdh1 to /dev/md/0 as 6
mdadm: added /dev/sdi1 to /dev/md/0 as 7
mdadm: added /dev/sdj1 to /dev/md/0 as 8
mdadm: added /dev/sdk1 to /dev/md/0 as 9
mdadm: added /dev/sdl1 to /dev/md/0 as 10
mdadm: added /dev/sda1 to /dev/md/0 as 0
mdadm: /dev/md/0 assembled from 5 drives - not enough to start the array.

How does the /proc/mdstat look after that?

Offline

#15 2011-10-31 12:09:34

maz
Member
Registered: 2006-05-05
Posts: 38

Re: [SOLVED] mdadm, raid6 and superblock missing

laloch wrote:

How does the /proc/mdstat look after that?

It says:

mdadm: md device /dev/md0 does not appear to be active.

After searching for a long time yesterday, from the tip I got, I came up with this
http://osdir.com/ml/linux-raid/2009-06/msg00018.html

mdadm --create /dev/md0 --level=6 --chunk=64 --raid-devices=11 /dev/sda1 /dev/sdb1 /dev/sdc1 /dev/sdd1 /dev/sde1 /dev/sdg1 /dev/sdh1 /dev/sdi1 /dev/sdj1 /dev/sdk1 /dev/sdl1

I modified it to my raid6 and my drives and decided to "recreate" but I think it wrecked the whole array. Its resyncing the array now but I think that my data is forever lost.

Offline

#16 2011-11-02 16:31:14

maz
Member
Registered: 2006-05-05
Posts: 38

Re: [SOLVED] mdadm, raid6 and superblock missing

After 2 days the resync is complete and it worked! All my data is still there. Time for backup and disassemble the array.
Thank you guys.

Offline

#17 2011-11-02 17:28:39

lilsirecho
Veteran
Registered: 2003-10-24
Posts: 5,000

Re: [SOLVED] mdadm, raid6 and superblock missing

Well done maz!!!!


Prediction...This year will be a very odd year!
Hard work does not kill people but why risk it: Charlie Mccarthy
A man is not complete until he is married..then..he is finished.
When ALL is lost, what can be found? Even bytes get lonely for a little bit!     X-ray confirms Iam spineless!

Offline

Board footer

Powered by FluxBB