raid5 issue after upgrade

insanum · 2012-04-09 02:50:58

TLDR = scared to death to answer 'yes' to the following from e2fsck:

Superblock has an invalid journal (inode 8).                                                                                                                          
Clear<y>? cancelled!

After upgrading to linux 3.2.14-1 (along with others important packages like udev, mkinitcpio, etc) my raid5 array would not assemble. I don't have the original message but after two days have passed and watching the forums I'm sure it was related to this: https://bugs.archlinux.org/task/29344

Anyways I'm past that issue and am now having filesystem problems with my raid array after recreating. Note I'm currently using

filesystem 2012.2-2
linux 3.2.14-1
mkinitcpio 0.8.5-1
udev 181-5

During the past two days of tinkering I've never let fsck touch the filesystem nor have I done anything like mkfs on it either. Only thing I've done is re-create the array using mdadm and then seeing if I can mount it.

The raid5 array contains /dev/sdb1 /dev/sdc1 and /dev/sdd1. My initial though was thinking that somehow maybe the drive labels got renamed during the kernel upgrade so I've tried re-creating the array will all six possible permutations of the disks. Nothing worked. I know the original order was sdb1/sdc1/sdd1.

Here is the dumpe2fs info for each of the disks:

# dumpe2fs /dev/sdb1
dumpe2fs 1.42.1 (17-Feb-2012)
Filesystem volume name:   <none>
Last mounted on:          /mnt/raid
Filesystem UUID:          08bb1bfd-4932-494a-93ba-fe8fd37ea22c
Filesystem magic number:  0xEF53
Filesystem revision #:    1 (dynamic)
Filesystem features:      has_journal ext_attr resize_inode dir_index filetype sparse_super large_file
Filesystem flags:         signed_directory_hash
Default mount options:    (none)
Filesystem state:         clean
Errors behavior:          Continue
Filesystem OS type:       Linux
Inode count:              122101760
Block count:              488379968
Reserved block count:     488379
Free blocks:              106448913
Free inodes:              121612868
First block:              0
Block size:               4096
Fragment size:            4096
Reserved GDT blocks:      907
Blocks per group:         32768
Fragments per group:      32768
Inodes per group:         8192
Inode blocks per group:   512
RAID stride:              16
RAID stripe width:        32
Filesystem created:       Thu Oct  8 23:09:57 2009
Last mount time:          Mon Apr  2 10:19:36 2012
Last write time:          Fri Apr  6 19:15:06 2012
Mount count:              13
Maximum mount count:      38
Last checked:             Sun Oct 30 06:54:03 2011
Check interval:           15552000 (6 months)
Next check after:         Fri Apr 27 06:54:03 2012
Lifetime writes:          12 GB
Reserved blocks uid:      0 (user root)
Reserved blocks gid:      0 (group root)
First inode:              11
Inode size:               256
Required extra isize:     28
Desired extra isize:      28
Journal inode:            8
Default directory hash:   half_md4
Directory Hash Seed:      84ce591f-34ae-43d0-846c-d8dae91ef4b9
Journal backup:           inode blocks
dumpe2fs: A block group is missing an inode table while reading journal inode

# dumpe2fs /dev/sdc1
dumpe2fs 1.42.1 (17-Feb-2012)
dumpe2fs: Bad magic number in super-block while trying to open /dev/sdc1
Couldn't find valid filesystem superblock.

# dumpe2fs /dev/sdd1
dumpe2fs 1.42.1 (17-Feb-2012)
dumpe2fs: Filesystem revision too high while trying to open /dev/sdd1
Couldn't find valid filesystem superblock.

When I create the array I see the following from mdadm. Which is expected as it sees the existing filesystem:

# mdadm --create --verbose /dev/md0 --level=5 --raid-devices=3 --metadata=0.90 /dev/sdb1 /dev/sdc1 /dev/sdd1
mdadm: layout defaults to left-symmetric
mdadm: chunk size defaults to 512K
mdadm: layout defaults to left-symmetric
mdadm: /dev/sdb1 appears to contain an ext2fs file system
    size=1953519872K  mtime=Mon Apr  2 10:19:36 2012
mdadm: /dev/sdb1 appears to be part of a raid array:
    level=raid5 devices=3 ctime=Sun Apr  8 10:01:02 2012
mdadm: layout defaults to left-symmetric
mdadm: /dev/sdc1 appears to be part of a raid array:
    level=raid5 devices=3 ctime=Sun Apr  8 10:01:02 2012
mdadm: layout defaults to left-symmetric
mdadm: /dev/sdd1 appears to contain an ext2fs file system
    size=2066852100K  mtime=Sat Apr  7 05:08:08 2029
mdadm: /dev/sdd1 appears to be part of a raid array:
    level=raid5 devices=3 ctime=Sun Apr  8 10:01:02 2012
mdadm: size set to 976759808K
Continue creating array? y
mdadm: array /dev/md0 started.

The raid array is created without issue:

# cat /proc/mdstat
Personalities : [raid6] [raid5] [raid4]
md0 : active raid5 sdb1[0] sdd1[2] sdc1[1]
      1953519616 blocks level 5, 512k chunk, algorithm 2 [3/3] [UUU]

# mdadm --detail /dev/md0
/dev/md0:
        Version : 0.90
  Creation Time : Sun Apr  8 14:56:14 2012
     Raid Level : raid5
     Array Size : 1953519616 (1863.02 GiB 2000.40 GB)
  Used Dev Size : 976759808 (931.51 GiB 1000.20 GB)
   Raid Devices : 3
  Total Devices : 3
Preferred Minor : 0
    Persistence : Superblock is persistent

    Update Time : Sun Apr  8 18:07:19 2012
          State : clean
 Active Devices : 3
Working Devices : 3
 Failed Devices : 0
  Spare Devices : 0

         Layout : left-symmetric
     Chunk Size : 512K

           UUID : fe541eb6:764599e3:5d64d1b1:6d03292e (local to host jackshrimp)
         Events : 0.19

    Number   Major   Minor   RaidDevice State
       0       8       17        0      active sync   /dev/sdb1
       1       8       33        1      active sync   /dev/sdc1
       2       8       49        2      active sync   /dev/sdd1

# mdadm --examine --scan
ARRAY /dev/md0 UUID=00000000:00000000:00000000:00000000
   spares=1
ARRAY /dev/md0 UUID=fe541eb6:764599e3:5d64d1b1:6d03292e

This is how I originally created the filesystem on the array (back in 10/2009). Here with "-n" to show the superblock locations:

# mkfs.ext3 -n -v -m .1 -b 4096 /dev/md0
mke2fs 1.42.1 (17-Feb-2012)
fs_types for mke2fs.conf resolution: 'ext3'
Filesystem label=
OS type: Linux
Block size=4096 (log=2)
Fragment size=4096 (log=2)
Stride=16 blocks, Stripe width=32 blocks
122101760 inodes, 488379904 blocks
488379 blocks (0.10%) reserved for the super user
First data block=0
Maximum filesystem blocks=0
14905 block groups
32768 blocks per group, 32768 fragments per group
8192 inodes per group
Superblock backups stored on blocks:
        32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632, 2654208,
        4096000, 7962624, 11239424, 20480000, 23887872, 71663616, 78675968,
        102400000, 214990848

I've tried running e2fsck with (-b) each of those blocks and none worked. Here is what I get from e2fsck:

# e2fsck /dev/md0
e2fsck 1.42.1 (17-Feb-2012)
e2fsck: Group descriptors look bad... trying backup blocks...
e2fsck: Filesystem revision too high when using the backup blocks
e2fsck: going back to original superblock
e2fsck: Group descriptors look bad... trying backup blocks...
e2fsck: Filesystem revision too high when using the backup blocks
e2fsck: going back to original superblock
Superblock has an invalid journal (inode 8).
Clear<y>? cancelled!

e2fsck: Illegal inode number while checking ext3 journal for /dev/md0

/dev/md0: ********** WARNING: Filesystem still has errors **********

While I'm quite sure the data is still there I'm freaking out about letting fsck do anything!

Should I let fsck do what it want to do? Any thoughts on how I can recover this array? Any suggestions on other things I should try?

Thanks for any and all help.

Arch Linux

#1 2012-04-09 02:50:58

raid5 issue after upgrade

Board footer