You are not logged in.

#1 2010-07-08 01:23:38

Studio271
Member
Registered: 2010-05-16
Posts: 17

[SOLVED:User Ignorance] Stuck at GRUB stage1.5 on RAID10 Array

I have four (4) 750-GB SATA2 Western Digital drives setup as a RAID10 array. I followed the instructions found here, http://wiki.archlinux.org/index.php/Ins … AID_or_LVM, and had it all up and running successfully for several weeks with little downtime and multiple on/off cycles. The details are fuzzy, but I think I had a 1.5-TB drive at /dev/md-0. It was partitioned with the Arch "standard" of a
128-MB ext2 /boot
256-MB swap
10-GB ext4 /
and the rest was ext4 /home
It is an x86_64 Arch system, and "hdparm -t /dev/md-0" would show ~140 MB/sec usually, which should indicate that I had the RAID10 setup properly (other than the functioning system, that is tongue )

Well, that was nice and all, but now it won't boot. I had it up for the past 2.5 days, then decided to reboot and go grab a sandwich because I had modified /etc/fstab to put my /tmp as tmpfs (as per http://wiki.archlinux.org/index.php/Max … tmp_to_RAM ). Also, I had used a utility in my XFCE menu: "System - Disk Utility", and selected one of the listed 1.5-TB listings (not sure if/why there were multiple, but it didn't seem strange to me at the time) and performed the "benchmark" function (data was returned, and I seem to remember that it told me I had an access time of 14.7ms). When I came back a few minutes later, I didn't find the console asking me for my username. What I found was:

Booting...
GRUB Loading stage1.5
(blinking cursor)

My only explanation would be if the Disk Utility somehow messed up the drives' partition tables by writing to what it thought was free space; just a thought, though, because the machine was up for at least 10 minutes after that (before being rebooted) and was not indicating any issues AFAIK.

I have tried loading the most recent Arch Live CD, and when it boots, I have in my /dev/ (among other things, obviously):
sda sda1 sda2
sdb
sdc sdc1 sdc2
sdd
sde sde1 (an extra 1-TB drive that I'm absolutely, 100%, completely pissed off that I didn't back anything up to because it is totally blank (seriously, wtf))

# fdisk -l
Warning: invalid flag 0x0000 of partition table 5 will be corrected by w(rite)

Disk /dev/sda: 750.1 GB, 750156374016 bytes
255 heads, 63 sectors/track, 91201 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
Disk identifier: 0x00000000

   Device Boot      Start         End      Blocks   Id  System
/dev/sda1   *           1           4       32098+  83  Linux
/dev/sda2               5      182364  1464806700    5  Extended

Disk /dev/sdb: 750.1 GB, 750156374016 bytes
255 heads, 63 sectors/track, 91201 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
Disk identifier: 0x00000000

Disk /dev/sdb doesn't contain a valid partition table
Warning: invalid flag 0x0000 of partition table 5 will be corrected by w(rite)

Disk /dev/sdc: 750.1 GB, 750156374016 bytes
255 heads, 63 sectors/track, 91201 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
Disk identifier: 0x00000000

   Device Boot      Start         End      Blocks   Id  System
/dev/sdc1   *           1           4       32098+  83  Linux
/dev/sdc2               5      182364  1464806700    5  Extended

Disk /dev/sdd: 750.1 GB, 750156374016 bytes
255 heads, 63 sectors/track, 91201 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
Disk identifier: 0x00000000

Disk /dev/sdd doesn't contain a valid partition table

Disk /dev/sde: 1000.2 GB, 1000204886016 bytes
255 heads, 63 sectors/track, 121601 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
Disk identifier: 0x000bd7b9

   Device Boot      Start         End      Blocks   Id  System
/dev/sde1               1      121602   976760832   83  Linux

Does that look right for a 4-drive soft-RAID10 system? I'm not sure what it came up as before, unfortunately.
Multiple reboots have not changed anything, and neither has letting it sit at the blinking cursor (there is no disk activity, and it doesn't appear to be waiting on anything). I don't think I know what I doing here, so I'm quite afraid to screw around on my own anymore than I have already. ;(

Last edited by Studio271 (2011-02-15 14:46:03)

Offline

#2 2010-07-09 15:34:55

Studio271
Member
Registered: 2010-05-16
Posts: 17

Re: [SOLVED:User Ignorance] Stuck at GRUB stage1.5 on RAID10 Array

I used my spare 1-TB drive to reinstall Arch in order to tide me over for a bit, but I absolutely hope someone familiar with MBR/partition tables can help me out here.

I have quite a lot of personal files that, while entirely my fault for not backing up, I would love to at least TRY to get back. I figure all I need to do is rewrite the partition tables on all four drives and hope I have the sizes correct, but I don't know the best way to go about doing that, or whether I should even try. It may make my data entirely unrecoverable, but I honestly would rather know that I lost everything for sure than sit here in limbo for any longer.

After that, I'll only have to perpetually ponder HOW this happened in the first place.

Offline

#3 2010-07-13 21:54:46

Studio271
Member
Registered: 2010-05-16
Posts: 17

Re: [SOLVED:User Ignorance] Stuck at GRUB stage1.5 on RAID10 Array

I hate to cry for help here, but I have hard drives containing over a decade's worth of personal files, school files, and many other items with immense sentimental/historical value to me. I really don't want to give up on this data; it's there, but the partition table is most likely what is messed up. Yes, I screwed up by not making a backup recently. I had one, but I destroyed it when I converted my system from a single drive to a software raid, which ran successfully for many weeks, and I had not got around to setting up an auto-backup system for myself again. Over a year of daily backups of data to a spare HD, and something goes wrong a month after I stopped them. This sucks.

Short story:
(1) Working RAID10 system: single raid "drive" (4 disks) with all partitions setup on it. Worked for several weeks, no problems.
(2) Casual reboot of system.
(3) Nothing works anymore.
(4) Panicked, tried rebooting several times with nothing different happening.
(5) fdisk reports strange things with partition tables.
(6) Setup new system on a spare HD to keep myself sane (for now).
*(7) Extensive SMART tests on all four drives show nothing is wrong with the drives!
(8) HELP ME!!! PLEASE!

* - recent update to this line of events.

Any input whatsoever would be nice! I don't care if you tell me to shut up and get over it! One desperate human looking for some online empathy here. ;(

Jeez, you can even say you have no idea what I'm even saying is wrong! That would at least let me know that I haven't quite defined the problem well enough for someone to help me!

Last edited by Studio271 (2010-07-13 21:56:19)

Offline

#4 2010-07-13 22:25:10

fsckd
Forum Fellow
Registered: 2009-06-15
Posts: 4,173

Re: [SOLVED:User Ignorance] Stuck at GRUB stage1.5 on RAID10 Array

I have no idea how to help you. All I can offer you is my story when I accidently ran fsck on root.

Looks like sdb and sdd lost their partition tables but sda and sdc retain it.


aur S & M :: forum rules :: Community Ethos
Resources for Women, POC, LGBT*, and allies

Offline

#5 2010-07-13 22:36:36

karol
Archivist
Registered: 2009-05-06
Posts: 25,440

Re: [SOLVED:User Ignorance] Stuck at GRUB stage1.5 on RAID10 Array

<read in a deep voice> It's the year 2020, the year of the petabyte-scale data loss.
Have you tried http://www.sysresccd.org/Main_Page ?

Offline

#6 2010-07-13 22:40:32

Studio271
Member
Registered: 2010-05-16
Posts: 17

Re: [SOLVED:User Ignorance] Stuck at GRUB stage1.5 on RAID10 Array

karol wrote:

<read in a deep voice> It's the year 2020, the year of the petabyte-scale data loss.
Have you tried http://www.sysresccd.org/Main_Page ?

Well, I certainly appreciate the link; it has some nice looking tutorials that I'll have to thumb through. As a direct response to your question, I have not tried that, no. big_smile

Offline

#7 2010-07-15 23:15:46

Studio271
Member
Registered: 2010-05-16
Posts: 17

Re: [SOLVED:User Ignorance] Stuck at GRUB stage1.5 on RAID10 Array

Still no solution, by the way. SysRescCD is just a live cd with programs I can get with Arch, and they still don't help me.
I'm fairly certain I know the problem (partition tables corrupted), I've got no clue what caused it (I blame faulty software), and I know what tools I need to fix it (cfdisk!). I just don't know if I'm right, I'm worried that what caused it is a bug in some software that could effect someone else at some point, and I don't know whether my plan to rewrite the partition tables will work as I think it will, especially since I don't remember for certain what the sizes of my partitions were and I don't know how they were setup for the softRAID.
All around confusion about how to fix this here.

Offline

#8 2010-07-16 01:30:04

cesura
Package Maintainer (PM)
From: Tallinn, Estonia
Registered: 2010-01-23
Posts: 1,867

Re: [SOLVED:User Ignorance] Stuck at GRUB stage1.5 on RAID10 Array

Studio271 wrote:

I used my spare 1-TB drive to reinstall Arch (...)

Spare!?! yikes The biggest hard drive I have ever owned is only 400GB!

I know this may not be directly related, but can you do any of the following?:
1) Boot a live-cd and mount a partition
2) Boot a live-cd and reinstall grub

EDIT: I'm interested in helping you with this, as I know how data loss feels. (One week ago, 8-foot high loft bed, laptop falling, you get the idea)

Last edited by cesura (2010-07-16 01:36:14)

Offline

#9 2010-07-16 01:34:50

Studio271
Member
Registered: 2010-05-16
Posts: 17

Re: [SOLVED:User Ignorance] Stuck at GRUB stage1.5 on RAID10 Array

itsbrad212 wrote:
Studio271 wrote:

I used my spare 1-TB drive to reinstall Arch (...)

Spare!?! yikes The biggest hard drive I have ever owned is only 400GB!

It was intended to backup most of the 1.5TB RAID10 array that I no longer have access to. sad
Gotta spend that student loan money on something! lol

Offline

#10 2010-07-16 01:51:14

Studio271
Member
Registered: 2010-05-16
Posts: 17

Re: [SOLVED:User Ignorance] Stuck at GRUB stage1.5 on RAID10 Array

I know this may not be directly related, but can you do any of the following?:
1) Boot a live-cd and mount a partition
2) Boot a live-cd and reinstall grub

Well, I don't need to use a live-cd to try to mount a partition. There aren't any on /dev/sdb or /dev/sdd, and /dev/sda2 and /dev/sdc2 are extended partition types, so they'll get me nowhere AFAIK. Here's what happens when I try for /dev/sda1 (same results for /dev/sdc1):

drew ~ $ sudo mount /dev/sda1 /mnt/temp/
mount: wrong fs type, bad option, bad superblock on /dev/sda1,
       missing codepage or helper program, or other error
       In some cases useful info is found in syslog - try
       dmesg | tail  or so

drew ~ $ dmesg | tail
...
EXT2-fs (sda1): error: ext2_check_descriptors: Block bitmap for group 0 not in group (block 2598645247)!
EXT2-fs (sda1): group descriptors corrupted

When I installed Arch to the 1-TB drive, I installed just as I had before (meaning I have GRUB on this disk). I have fumbled around with auto-completes in its command-shell, but to no avail. It can't seem to make anymore sense of what is on these four drives than the Arch system can.

Offline

#11 2010-07-16 01:53:24

cesura
Package Maintainer (PM)
From: Tallinn, Estonia
Registered: 2010-01-23
Posts: 1,867

Re: [SOLVED:User Ignorance] Stuck at GRUB stage1.5 on RAID10 Array

Studio271 wrote:

I know this may not be directly related, but can you do any of the following?:
1) Boot a live-cd and mount a partition
2) Boot a live-cd and reinstall grub

Well, I don't need to use a live-cd to try to mount a partition. There aren't any on /dev/sdb or /dev/sdd, and /dev/sda2 and /dev/sdc2 are extended partition types, so they'll get me nowhere AFAIK. Here's what happens when I try for /dev/sda1 (same results for /dev/sdc1):

drew ~ $ sudo mount /dev/sda1 /mnt/temp/
mount: wrong fs type, bad option, bad superblock on /dev/sda1,
       missing codepage or helper program, or other error
       In some cases useful info is found in syslog - try
       dmesg | tail  or so

drew ~ $ dmesg | tail
...
EXT2-fs (sda1): error: ext2_check_descriptors: Block bitmap for group 0 not in group (block 2598645247)!
EXT2-fs (sda1): group descriptors corrupted

When I installed Arch to the 1-TB drive, I installed just as I had before (meaning I have GRUB on this disk). I have fumbled around with auto-completes in its command-shell, but to no avail. It can't seem to make anymore sense of what is on these four drives than the Arch system can.

Can you possibly try the third post from:
http://ubuntuforums.org/showthread.php?t=982699

Offline

#12 2010-07-16 02:27:40

Studio271
Member
Registered: 2010-05-16
Posts: 17

Re: [SOLVED:User Ignorance] Stuck at GRUB stage1.5 on RAID10 Array

itsbrad212 wrote:

Can you possibly try the third post from:
http://ubuntuforums.org/showthread.php?t=982699

drew ~ $ sudo e2fsck -C0 -f -v /dev/sda1
e2fsck 1.41.12 (17-May-2010)
e2fsck: Group descriptors look bad... trying backup blocks...
e2fsck: Bad magic number in super-block when using the backup blocks
e2fsck: going back to original superblock
e2fsck: Device or resource busy while trying to open /dev/sda1
Filesystem mounted or opened exclusively by another program?

drew ~ $ sudo e2fsck -C0 -f -v /dev/sda2
e2fsck 1.41.12 (17-May-2010)
e2fsck: Attempt to read block from filesystem resulted in short read while trying to open /dev/sda2
Could this be a zero-length partition?

...and, just for the hell of it; doesn't make sense to run a partition checker on a disk... tongue

drew ~ $ sudo e2fsck -C0 -f -v /dev/sda
e2fsck 1.41.12 (17-May-2010)
e2fsck: Superblock invalid, trying backup blocks...
e2fsck: Bad magic number in super-block while trying to open /dev/sda

The superblock could not be read or does not describe a correct ext2
filesystem.  If the device is valid and it really contains an ext2
filesystem (and not swap or ufs or something else), then the superblock
is corrupt, and you might try running e2fsck with an alternate superblock:
    e2fsck -b 8193 <device>

Same results occur when done in an Arch 2010.05 live-cd. However, it got weird when I did the same for /dev/sdc1 - it started doing what it did in post #4 of that thread. However, eventually (I used the -y option) it said it would start over, and then it stopped and spat out the error it gives on /dev/sda:

drew ~ $ sudo e2fsck -C0 -f -v /dev/sdc1
e2fsck 1.41.12 (17-May-2010)
e2fsck: Superblock invalid, trying backup blocks...
e2fsck: The ext2 superblock is corrupt while trying to open /dev/sdc1

The superblock could not be read or does not describe a correct ext2
filesystem.  If the device is valid and it really contains an ext2
filesystem (and not swap or ufs or something else), then the superblock
is corrupt, and you might try running e2fsck with an alternate superblock:
    e2fsck -b 8193 <device>

It continues to do so now; I guess that indicates that I may have made it worse. roll

Offline

#13 2010-07-16 02:31:41

cesura
Package Maintainer (PM)
From: Tallinn, Estonia
Registered: 2010-01-23
Posts: 1,867

Re: [SOLVED:User Ignorance] Stuck at GRUB stage1.5 on RAID10 Array

Is /dev/sda1 mounted?

cat /etc/mtab

Last edited by cesura (2010-07-16 02:32:04)

Offline

#14 2010-07-16 02:35:06

Studio271
Member
Registered: 2010-05-16
Posts: 17

Re: [SOLVED:User Ignorance] Stuck at GRUB stage1.5 on RAID10 Array

itsbrad212 wrote:

Is /dev/sda1 mounted?

Nope; like I said, it says the exact same thing when using an Arch live-cd. Something else is making it think that is the case.

drew ~ $ cat /etc/mtab
proc /proc proc rw,relatime
sys /sys sysfs rw,relatime
udev /dev devtmpfs rw,nosuid,relatime,size=10240k,nr_inodes=497414,mode=755
/dev/disk/by-uuid/236cacf1-36c6-4f60-b388-a7f5e6516662 / ext4 rw,relatime,barrier=1,data=ordered
devpts /dev/pts devpts rw 0 0
shm /dev/shm tmpfs rw,nosuid,nodev 0 0
/dev/sde4 /home ext4 rw 0 0
/dev/sde1 /boot ext2 rw 0 0
fusectl /sys/fs/fuse/connections fusectl rw 0 0
gvfs-fuse-daemon /home/drew/.gvfs fuse.gvfs-fuse-daemon rw,nosuid,nodev,user=drew 0 0

...and for clarity...

lrwxrwxrwx 1 root root 10 Jul 15 21:17 236cacf1-36c6-4f60-b388-a7f5e6516662 -> ../../sde3

Last edited by Studio271 (2010-07-16 02:56:40)

Offline

#15 2010-07-16 02:55:55

cesura
Package Maintainer (PM)
From: Tallinn, Estonia
Registered: 2010-01-23
Posts: 1,867

Re: [SOLVED:User Ignorance] Stuck at GRUB stage1.5 on RAID10 Array

What is:

dev/disk/by-uuid/236cacf1-36c6-4f60-b388-a7f5e6516662 / ext4 rw,relatime,barrier=1,data=ordered

?

Check to see the uuid of sda1:

ls -l /dev/disk/by-uuid/

EDIT: Or is that a partition on your currently working drive? *bashes self in head*

Last edited by cesura (2010-07-16 02:57:39)

Offline

#16 2010-07-16 02:59:33

Studio271
Member
Registered: 2010-05-16
Posts: 17

Re: [SOLVED:User Ignorance] Stuck at GRUB stage1.5 on RAID10 Array

itsbrad212 wrote:

What is:

dev/disk/by-uuid/236cacf1-36c6-4f60-b388-a7f5e6516662 / ext4 rw,relatime,barrier=1,data=ordered

?

Check to see the uuid of sda1:

ls -l /dev/disk/by-uuid/

EDIT: Or is that a partition on your currently working drive?

Sorry, already added that to an edit, but:

drew ~ $ la /dev/disk/by-uuid/
total 0
lrwxrwxrwx 1 root root 10 Jul 15 21:17 236cacf1-36c6-4f60-b388-a7f5e6516662 -> ../../sde3
lrwxrwxrwx 1 root root 10 Jul 15 21:17 688dd506-d9e8-4a20-8c1e-43c972943e02 -> ../../sde2
lrwxrwxrwx 1 root root 10 Jul 15 21:25 a38e4f70-81a9-4960-9dde-7f8cbce7040b -> ../../sdc1
lrwxrwxrwx 1 root root 10 Jul 15 21:17 b1b79dbd-1b33-4c4d-b452-31cd4f7d24a7 -> ../../sde4
lrwxrwxrwx 1 root root 10 Jul 15 21:17 dff710d2-bddd-4f31-8b48-4b9da316439a -> ../../sde1

However, I did notice that /dev/sdc1 is in there now... suggestions on what to do to investigate that?
I tried to mount it and it seemed to be churning the disk for about 5 seconds, but still got an error. Why would it show up there, but nothing else (such as /dev/sda1)?

drew ~ $ sudo mount /dev/sdc1 /mnt/temp
mount: wrong fs type, bad option, bad superblock on /dev/sdc1,
       missing codepage or helper program, or other error
       In some cases useful info is found in syslog - try
       dmesg | tail  or so

drew ~ $ dmesg | tail
...
EXT2-fs (sdc1): error: ext2_check_descriptors: Block bitmap for group 4 not in group (block 0)!
EXT2-fs (sdc1): group descriptors corrupted

Last edited by Studio271 (2010-07-16 03:02:37)

Offline

#17 2010-07-16 03:05:56

cesura
Package Maintainer (PM)
From: Tallinn, Estonia
Registered: 2010-01-23
Posts: 1,867

Re: [SOLVED:User Ignorance] Stuck at GRUB stage1.5 on RAID10 Array

This is bizarre, /dev/sdc2 is showing up, but nothing else? Wow! I'm fairly incompetent when it comes to RAID, so sorry wink

What does:

sudo mdadm --assemble --scan

show?

Offline

#18 2010-07-16 03:17:24

Studio271
Member
Registered: 2010-05-16
Posts: 17

Re: [SOLVED:User Ignorance] Stuck at GRUB stage1.5 on RAID10 Array

itsbrad212 wrote:

This is bizarre, /dev/sdc2 is showing up, but nothing else? Wow! I'm fairly incompetent when it comes to RAID, so sorry wink

What does:

sudo mdadm --assemble --scan

show?

That's /dev/sdc1 that is being provided a uuid, and that command outputs nothing useful:

drew ~ $ sudo mdadm --assemble --scan
mdadm: No arrays found in config file or automatically

Also, I'd like to slip in a biiiiig THANK YOU for giving me some time with this! big_smile

Last edited by Studio271 (2010-07-16 03:18:51)

Offline

#19 2010-07-16 18:20:02

Studio271
Member
Registered: 2010-05-16
Posts: 17

Re: [SOLVED:User Ignorance] Stuck at GRUB stage1.5 on RAID10 Array

Nothing enlightening has occurred with this problem so far; anyone else have some ideas to toss out here?

Offline

#20 2010-07-16 20:34:57

George.Harmony
Member
From: Off The Wire
Registered: 2010-04-30
Posts: 97

Re: [SOLVED:User Ignorance] Stuck at GRUB stage1.5 on RAID10 Array

Just to toss it out there.

If it was a hardware raid I would just rebuild it being sure not to format anything or touch the partition tables. To me it sounds like the system has lost track of the raid. Is there a way to just rebuild the software raid without touching the partitions?

I think if you boot with a live cd and dont make it aware of your software raid you would get the errors you have mentioned when you run the diagnostics.

Again since you asked for people to throw things out there that is what I am doing. Maybe it will send you in the right direction or give you some things to think about that you hadn't considered before.


Desktop:  Compiz Stand Alone w/ Cairo Dock.
Laptop:    Pekwm w/ Tint2
Jukebox:   MPD w/ cli
Gateway: Vuurmuur w/dialog

Offline

#21 2010-07-16 20:42:41

cesura
Package Maintainer (PM)
From: Tallinn, Estonia
Registered: 2010-01-23
Posts: 1,867

Re: [SOLVED:User Ignorance] Stuck at GRUB stage1.5 on RAID10 Array

George.Harmony wrote:

Just to toss it out there.

If it was a hardware raid I would just rebuild it being sure not to format anything or touch the partition tables. To me it sounds like the system has lost track of the raid. Is there a way to just rebuild the software raid without touching the partitions?

I think if you boot with a live cd and dont make it aware of your software raid you would get the errors you have mentioned when you run the diagnostics.

Again since you asked for people to throw things out there that is what I am doing. Maybe it will send you in the right direction or give you some things to think about that you hadn't considered before.

I was kind of thinking in that direction as well. Although I have never had a RAID setup, that seems like a logical thing to do.

Offline

#22 2011-02-15 14:44:25

Studio271
Member
Registered: 2010-05-16
Posts: 17

Re: [SOLVED:User Ignorance] Stuck at GRUB stage1.5 on RAID10 Array

Studio271 wrote:

Also, I had used a utility in my XFCE menu: "System - Disk Utility", and selected one of the listed 1.5-TB listings (not sure if/why there were multiple, but it didn't seem strange to me at the time) and performed the "benchmark" function (data was returned, and I seem to remember that it told me I had an access time of 14.7ms).

Ok, I'm pretty sure this is what did it. It has been a while, but I must have done a write benchmark and ignored (if there were any) any warnings that it would be overwriting my data. Which it did. Good show! Oh well, I've moved on.

Offline

Board footer

Powered by FluxBB