raid10 resync hangs when mounting a lvm volume

emes · 2014-06-25 21:32:38

Hi,

This is very strange behavior and I've been experiencing it with a couple of recent kernel releases.
I have the following setup:

1. two 1GB disks equally partitioned
2. raid10 on /dev/sda3 and /dev/sdb3
3. lvm on top of raid (/dev/grzybnia)
4. xfs volumes for / (/dev/grzybnia/arch) and /home (/dev/grzybnia/users)

If the raid becomes faulty, which happens too often on restarts with no apparent reason, I can mount / with the recovery process going in the background.
However, when I try to mount /home, the recovery stalls and any process accessing /home freezes as well.
Neither console nor dmesg show any error. It just freezes forever.
There are no reports of physical problems with any of the drives.

This means I'm forced to boot from rescue usb stick each time the raid becomes faulty and run 4h long recovery before I'm able to mount /home.
It is unacceptable for obvious reasons.

Why does it freeze and why does that happen only on the particular LVM volume?

excerpts follow:

cortado% cat /proc/mdstat
Personalities : [raid1] [raid10] 
md126 : active raid10 sdb3[1] sda3[0]
      974510848 blocks 64K chunks 2 far-copies [2/2] [UU]
      [>....................]  resync =  1.9% (18592576/974510848) finish=229.7min speed=69327K/sec
      
md127 : active raid1 sdb1[1] sda1[0]
      248768 blocks [2/2] [UU]
      
unused devices: <none>

cortado% lvdisplay
  --- Logical volume ---
  LV Path                /dev/grzybnia/users
  LV Name                users
  VG Name                grzybnia
  LV UUID                dsc7gG-rxNn-mHQy-xFU8-jHi1-cWQb-6pZFPs
  LV Write Access        read/write
  LV Creation host, time , 
  LV Status              available
  # open                 0
  LV Size                500.00 GiB
  Current LE             128000
  Segments               4
  Allocation             inherit
  Read ahead sectors     auto
  - currently set to     512
  Block device           254:0
   
  --- Logical volume ---
  LV Path                /dev/grzybnia/arch
  LV Name                arch
  VG Name                grzybnia
  LV UUID                lbqoyV-KRXD-kkcL-4J0z-d37z-FyWX-nty7e2
  LV Write Access        read/write
  LV Creation host, time cortado, 2014-03-20 00:42:03 +0100
  LV Status              available
  # open                 1
  LV Size                10.00 GiB
  Current LE             2560
  Segments               1
  Allocation             inherit
  Read ahead sectors     auto
  - currently set to     512
  Block device           254:1

cortado% uname -a
Linux cortado 3.15.1-1-ARCH #1 SMP PREEMPT Tue Jun 17 09:32:20 CEST 2014 x86_64 GNU/Linux

frostschutz · 2014-06-26 12:39:09

Two 1GB disks, the LV is 500GiB, the RAID is more like 100GiB?

Can you post mdadm --examine /dev/sd[ab]* as well as lvs -o +devices?

Also, why do you have raid10 for two drives?

emes · 2014-06-26 13:06:49

Well, perhaps I should have used simple raid1. I'm thinking about moving, but that means a time-consuming backup&restore operation of all my data.

Here's the output of the commands you suggested. The array is now clean and running.

~:# mdadm --examine /dev/sd[ab]*
/dev/sda:
   MBR Magic : aa55
Partition[0] :       497765 sectors at          250 (type fd)
Partition[1] :      4000185 sectors at       498015 (type 82)
Partition[2] :   1949021865 sectors at      4498200 (type fd)
/dev/sda1:
          Magic : a92b4efc
        Version : 0.90.00
           UUID : 064edd94:1013e485:7850132b:de26eb7b (local to host cortado)
  Creation Time : Mon Dec 24 02:46:52 2012
     Raid Level : raid1
  Used Dev Size : 248768 (242.98 MiB 254.74 MB)
     Array Size : 248768 (242.98 MiB 254.74 MB)
   Raid Devices : 2
  Total Devices : 2
Preferred Minor : 127

    Update Time : Thu Jun 26 11:31:57 2014
          State : clean
 Active Devices : 2
Working Devices : 2
 Failed Devices : 0
  Spare Devices : 0
       Checksum : ba8c7b2e - correct
         Events : 251


      Number   Major   Minor   RaidDevice State
this     0       8        1        0      active sync   /dev/sda1

   0     0       8        1        0      active sync   /dev/sda1
   1     1       8       17        1      active sync   /dev/sdb1
mdadm: No md superblock detected on /dev/sda2.
/dev/sda3:
          Magic : a92b4efc
        Version : 0.90.00
           UUID : a449cf06:91c6289d:e43ba3c5:b59fda4a
  Creation Time : Tue Sep 29 05:24:27 2009
     Raid Level : raid10
  Used Dev Size : 974510848 (929.37 GiB 997.90 GB)
     Array Size : 974510848 (929.37 GiB 997.90 GB)
   Raid Devices : 2
  Total Devices : 2
Preferred Minor : 126

    Update Time : Thu Jun 26 11:33:15 2014
          State : active
 Active Devices : 2
Working Devices : 2
 Failed Devices : 0
  Spare Devices : 0
       Checksum : 51e932d2 - correct
         Events : 5122346

         Layout : far=2
     Chunk Size : 64K

      Number   Major   Minor   RaidDevice State
this     0       8        3        0      active sync   /dev/sda3

   0     0       8        3        0      active sync   /dev/sda3
   1     1       8       19        1      active sync   /dev/sdb3
/dev/sdb:
   MBR Magic : aa55
Partition[0] :       497765 sectors at          250 (type fd)
Partition[1] :      4000185 sectors at       498015 (type 82)
Partition[2] :   1949021865 sectors at      4498200 (type fd)
/dev/sdb1:
          Magic : a92b4efc
        Version : 0.90.00
           UUID : 064edd94:1013e485:7850132b:de26eb7b (local to host cortado)
  Creation Time : Mon Dec 24 02:46:52 2012
     Raid Level : raid1
  Used Dev Size : 248768 (242.98 MiB 254.74 MB)
     Array Size : 248768 (242.98 MiB 254.74 MB)
   Raid Devices : 2
  Total Devices : 2
Preferred Minor : 127

    Update Time : Thu Jun 26 11:31:57 2014
          State : clean
 Active Devices : 2
Working Devices : 2
 Failed Devices : 0
  Spare Devices : 0
       Checksum : ba8c7b40 - correct
         Events : 251


      Number   Major   Minor   RaidDevice State
this     1       8       17        1      active sync   /dev/sdb1

   0     0       8        1        0      active sync   /dev/sda1
   1     1       8       17        1      active sync   /dev/sdb1
mdadm: No md superblock detected on /dev/sdb2.
/dev/sdb3:
          Magic : a92b4efc
        Version : 0.90.00
           UUID : a449cf06:91c6289d:e43ba3c5:b59fda4a
  Creation Time : Tue Sep 29 05:24:27 2009
     Raid Level : raid10
  Used Dev Size : 974510848 (929.37 GiB 997.90 GB)
     Array Size : 974510848 (929.37 GiB 997.90 GB)
   Raid Devices : 2
  Total Devices : 2
Preferred Minor : 126

    Update Time : Thu Jun 26 11:33:15 2014
          State : active
 Active Devices : 2
Working Devices : 2
 Failed Devices : 0
  Spare Devices : 0
       Checksum : 51e932e4 - correct
         Events : 5122346

         Layout : far=2
     Chunk Size : 64K

      Number   Major   Minor   RaidDevice State
this     1       8       19        1      active sync   /dev/sdb3

   0     0       8        3        0      active sync   /dev/sda3
   1     1       8       19        1      active sync   /dev/sdb3

~:# lvs -o +devices 
  LV    VG       Attr       LSize   Pool Origin Data%  Move Log Cpy%Sync Convert Devices           
  arch  grzybnia -wi-ao----  10,00g                                              /dev/md126(0)     
  aur   grzybnia -wi-ao----  10,00g                                              /dev/md126(109904)
  users grzybnia -wi-ao---- 500,00g                                              /dev/md126(4944)  
  users grzybnia -wi-ao---- 500,00g                                              /dev/md126(58704) 
  users grzybnia -wi-ao---- 500,00g                                              /dev/md126(113152)
  users grzybnia -wi-ao---- 500,00g                                              /dev/md126(123437)

Arch Linux

#1 2014-06-25 21:32:38

raid10 resync hangs when mounting a lvm volume

#2 2014-06-26 12:39:09

Re: raid10 resync hangs when mounting a lvm volume

#3 2014-06-26 13:06:49

Re: raid10 resync hangs when mounting a lvm volume

Board footer