Arch Linux

corpulentrobin

Member

Registered: 2025-05-14

Posts: 3

Any mdadm experts that can help me with a raid 5 I can't reassemble.

I have a raid 5 and I'm unable to reassemble it. I think it was unmounted in an unclean way. I was able to plug everything in and it looked like it was doing a rebuild on one of the drives, one of my drives isn't properly showing up and it was earlier. I think the rebuild failed and I'm in a weird state. Have I lost everything or is there a way to recover this?

Here's some data of what I'm seeing. Please let me know if there's anything else that I'm missing that would be helpful.  This mdadm array was originally plugged into another desktop "f-desktop" it's now on "d-desktop" not sure if that matters.

---
[root@d-desktop ~]# cat /proc/mdstat
Personalities : [raid1]
md126 : active raid1 sdf1[1] sde1[0]
      488254464 blocks super 1.2 [2/2] [UU]
      bitmap: 0/4 pages [0KB], 65536KB chunk

md127 : inactive sdg[4](S) sdb[5](S) sdc[2](S)
      4395022534 blocks super 1.2

---
[root@d-desktop ~]# mount /dev/md127 /mnt
mount: /mnt: can't read superblock on /dev/md127.
       dmesg(1) may have more information after failed mount system call.

---
[root@d-desktop ~]# lsblk | filtered
sdb             8:16   0   1.4T  0 disk 
└─md127         9:127  0     0B  0       
sdc             8:32   0   1.4T  0 disk 
└─md127         9:127  0     0B  0       
sdd             8:48   0   1.4T  0 disk 
└─sdd1          8:49   0   1.4T  0 part 
└─sde1          8:65   0 465.8G  0 part 
  └─md126       9:126  0 465.6G  0 raid1
    └─md126p1 259:5    0 465.6G  0 part  /run/media/HOME
--
└─sdf1          8:81   0 465.8G  0 part 
  └─md126       9:126  0 465.6G  0 raid1
    └─md126p1 259:5    0 465.6G  0 part  /run/media/HOME
sdg             8:96   0   1.4T  0 disk 
└─md127         9:127  0     0B  0     

---
/etc/mdadm.conf
# This is the only non commented line that I have added.
ARRAY /dev/md/2 metadata=1.2 UUID=b5bcbf9d:891bbcec:e693ec04:81662a21

---
[root@d-desktop ~]# mdadm --examine --scan
ARRAY /dev/md/0  metadata=1.2 UUID=3131dd8a:df49c8c4:9b5255d7:f9566319
ARRAY /dev/md/1  metadata=1.2 UUID=cae29611:35d8290f:33f9c465:8193448f
ARRAY /dev/md/0  metadata=1.2 UUID=b5bcbf9d:891bbcec:e693ec04:81662a21

---
[root@d-desktop ~]# mdadm -E /dev/sdb
/dev/sdb:
          Magic : a92b4efc
        Version : 1.2
    Feature Map : 0x1
     Array UUID : b5bcbf9d:891bbcec:e693ec04:81662a21
           Name : f-desktop:0
  Creation Time : Sat Feb 28 17:08:35 2015
     Raid Level : raid5
   Raid Devices : 4

Avail Dev Size : 2930015023 sectors (1397.14 GiB 1500.17 GB)
     Array Size : 4395021312 KiB (4.09 TiB 4.50 TB)
  Used Dev Size : 2930014208 sectors (1397.14 GiB 1500.17 GB)
    Data Offset : 262144 sectors
   Super Offset : 8 sectors
   Unused Space : before=262064 sectors, after=815 sectors
          State : clean
    Device UUID : 9200b67a:32389760:73246d05:e4c1b790

Internal Bitmap : 8 sectors from superblock
    Update Time : Mon Feb 26 10:54:20 2024
  Bad Block Log : 512 entries available at offset 16 sectors
       Checksum : b85fa551 - correct
         Events : 232628

         Layout : left-symmetric
     Chunk Size : 512K

   Device Role : Active device 0
   Array State : A.A. ('A' == active, '.' == missing, 'R' == replacing)
[root@d-desktop ~]# mdadm -E /dev/sdc
/dev/sdc:
          Magic : a92b4efc
        Version : 1.2
    Feature Map : 0x1
     Array UUID : b5bcbf9d:891bbcec:e693ec04:81662a21
           Name : f-desktop:0
  Creation Time : Sat Feb 28 17:08:35 2015
     Raid Level : raid5
   Raid Devices : 4

Avail Dev Size : 2930015023 sectors (1397.14 GiB 1500.17 GB)
     Array Size : 4395021312 KiB (4.09 TiB 4.50 TB)
  Used Dev Size : 2930014208 sectors (1397.14 GiB 1500.17 GB)
    Data Offset : 262144 sectors
   Super Offset : 8 sectors
   Unused Space : before=262056 sectors, after=815 sectors
          State : clean
    Device UUID : 7c2cbc41:4ef5265f:14cabdc9:adaa4037

Internal Bitmap : 8 sectors from superblock
    Update Time : Mon Feb 26 10:54:20 2024
  Bad Block Log : 512 entries available at offset 72 sectors
       Checksum : e8cf1cf6 - correct
         Events : 232628

         Layout : left-symmetric
     Chunk Size : 512K

   Device Role : Active device 2
   Array State : A.A. ('A' == active, '.' == missing, 'R' == replacing)
[root@d-desktop ~]# mdadm -E /dev/sdd
/dev/sdd:
   MBR Magic : aa55
Partition[0] :   2930274304 sectors at         2048 (type 83)
[root@d-desktop ~]# mdadm -E /dev/sdg
/dev/sdg:
          Magic : a92b4efc
        Version : 1.2
    Feature Map : 0x1
     Array UUID : b5bcbf9d:891bbcec:e693ec04:81662a21
           Name : f-desktop:0
  Creation Time : Sat Feb 28 17:08:35 2015
     Raid Level : raid5
   Raid Devices : 4

Avail Dev Size : 2930015023 sectors (1397.14 GiB 1500.17 GB)
     Array Size : 4395021312 KiB (4.09 TiB 4.50 TB)
  Used Dev Size : 2930014208 sectors (1397.14 GiB 1500.17 GB)
    Data Offset : 262144 sectors
   Super Offset : 8 sectors
   Unused Space : before=262056 sectors, after=815 sectors
          State : clean
    Device UUID : 11ea6780:9fa43438:fcb2751e:9505a238

Internal Bitmap : 8 sectors from superblock
    Update Time : Tue Mar  2 16:35:15 2021
  Bad Block Log : 512 entries available at offset 72 sectors
       Checksum : 5104ec70 - correct
         Events : 176140

         Layout : left-symmetric
     Chunk Size : 512K

   Device Role : Active device 3
   Array State : AAAA ('A' == active, '.' == missing, 'R' == replacing)

corpulentrobin

Member

Registered: 2025-05-14

Posts: 3

Re: Any mdadm experts that can help me with a raid 5 I can't reassemble.

[root@d-desktop ~]# mdadm --detail /dev/md127
/dev/md127:
           Version : 1.2
        Raid Level : raid5
     Total Devices : 3
       Persistence : Superblock is persistent

             State : inactive
    Active Devices : 0
   Working Devices : 3
    Failed Devices : 0
     Spare Devices : 3

              Name : f-desktop:0
              UUID : 3131dd8a:df49c8c4:9b5255d7:f9566319
            Events : 3525

    Number   Major   Minor   RaidDevice

       -       8       97        -        /dev/sdg1
       -       8       33        -        /dev/sdc1
       -       8       17        -        /dev/sdb1

frostschutz

Member

Registered: 2013-11-15

Posts: 1,545

Re: Any mdadm experts that can help me with a raid 5 I can't reassemble.

RAID 5 with 4 devices, of which you have showed 3. But the 3rd drive seems to not have been part of the array for a long time (Update time Mar 2 2021, when the other have Feb 2024). Event counters completely off too.

You pretty much need the missing drive back. Is it sdd? Then it seems it was wiped by a partition table (and that's why you don't use unpartitioned drives in a raid).

You'd have to re-create it with correct offset and settings. See for example https://unix.stackexchange.com/a/131927/30851

Drive order in your example would be sdb (Role 0) sdd (Role 1) sdc (Role 2) sdg (Role 3 but it's outdated, so put it as missing when you re-create the array).

PS: if you get it working, check with wipefs if there is any GPT backup header at the end of your drive(s), and get rid of it. If anything restores the GPT backup, mdadm metadata will be wiped again.

Last edited by frostschutz (2025-05-14 20:19:38)

corpulentrobin

Member

Registered: 2025-05-14

Posts: 3

Re: Any mdadm experts that can help me with a raid 5 I can't reassemble.

Thank you, yes you are correct. /dev/sdd is the missing drive, oddly enough it appeared recently. I'll need to figure out an option to image these drives while I attempt recovery.
I believe my drive order is
/dev/sdc1
/dev/sdd1
/dev/sdb1
/dev/sdg1
based on the Device Role: Active device <x> lines from examine.

[root@d-desktop ~]# mdadm --examine /dev/sdd1
/dev/sdd1:
          Magic : a92b4efc
        Version : 1.2
    Feature Map : 0x1
     Array UUID : 3131dd8a:df49c8c4:9b5255d7:f9566319
           Name : f-desktop:0
  Creation Time : Fri Feb 27 19:50:10 2015
     Raid Level : raid5
   Raid Devices : 4

Avail Dev Size : 2930012160 sectors (1397.14 GiB 1500.17 GB)
     Array Size : 4395018240 KiB (4.09 TiB 4.50 TB)
    Data Offset : 262144 sectors
   Super Offset : 8 sectors
   Unused Space : before=262056 sectors, after=0 sectors
          State : clean
    Device UUID : 6408bee5:5f2d6a85:ac7bba0e:ac118711

Internal Bitmap : 8 sectors from superblock
    Update Time : Wed May 14 10:45:56 2025
  Bad Block Log : 512 entries available at offset 72 sectors
       Checksum : 6467d63b - correct
         Events : 3525

         Layout : left-symmetric
     Chunk Size : 512K

   Device Role : Active device 1
   Array State : AA.. ('A' == active, '.' == missing, 'R' == replacing)

frostschutz

Member

Registered: 2013-11-15

Posts: 1,545

Re: Any mdadm experts that can help me with a raid 5 I can't reassemble.

That has an earlier creation date than your other arrays. With any luck you created RAID on partitions first, before changing your mind and switching to whole drives, and what you're seeing there (after partition table was restored) is old metadata. Which survived because mdadm has large data offsets so it doesn't actually overwrite anything much for the first 128M of the drive.

You have to re-create with sdd (not sdd1). Or create partition tables on all drives, with partition offset 1MiB, and reduce mdadm data offset by 1MiB, so you come out at the offset where your actual data starts. But if you do that, some sectors at the end of your array will be missing. Depending how your drive size aligns to chunk size, the array may be smaller than before which will cause problems with your filesystems.

The partition issue is something to worry about later, get your data back first.