You are not logged in.

#1 2016-05-02 19:37:52

Guidobelix
Member
From: Italy
Registered: 2013-06-21
Posts: 31

Raid5 stuck in reshape operation

Looking for help on the following issue:
I have decided to add a disk to my RAID5 array, from 4x2TB to 5x2TB, so I have partitioned the new disk as the others and then added it to the array.

┌─[root@Archlinux]─[~]
└──> mdadm --grow --raid-devices=5 /dev/md0

The reshape operation started successfully but it then got stuch at 67.6% and it is like that since 24 hours.

┌─[root@Archlinux]─[~]
└──> cat /proc/mdstat
Personalities : [raid6] [raid5] [raid4] 
md0 : active raid5 sdf1[6] sde1[4] sdc1[1] sdd1[5] sdb1[0]
      5859840000 blocks super 1.2 level 5, 256k chunk, algorithm 2 [5/5] [UUUUU]
      [=============>.......]  reshape = 67.6% (1320482400/1953280000) finish=2074327.9min speed=5K/sec
      bitmap: 0/15 pages [0KB], 65536KB chunk

unused devices: <none>

I see only one occurrence of this message in journactl:

mag 01 22:30:49 Archlinux kernel: INFO: task md0_reshape:637 blocked for more than 120 seconds.
mag 01 22:30:49 Archlinux kernel:       Tainted: G           O    4.5.1-1-ARCH #1
mag 01 22:30:49 Archlinux kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
mag 01 22:30:49 Archlinux kernel: md0_reshape     D c0670318     0   637      2 0x00000000

The disk is still mounted, but If I cannot list or access files.

What can I do now? Is there a way to restart the operation without loosing the data?

Thanks a lot for your help!

Offline

#2 2016-05-03 08:13:57

Polyatomic
Member
Registered: 2014-05-31
Posts: 7

Re: Raid5 stuck in reshape operation

For what its worth, I've spotted a similar raid drama here:
https://forums.gentoo.org/viewtopic-t-1043706.html

Offline

#3 2016-05-03 20:35:51

Guidobelix
Member
From: Italy
Registered: 2013-06-21
Posts: 31

Re: Raid5 stuck in reshape operation

Indeed, I saw that one while looking for some help on the Internet.
Update from today: I have rebooted the server, but mdadm is still stuck, and I cannot restart it.
I really don' know what to do...

Last edited by Guidobelix (2016-05-03 20:36:13)

Offline

#4 2016-05-03 21:09:00

frostschutz
Member
Registered: 2013-11-15
Posts: 1,409

Re: Raid5 stuck in reshape operation

Can you show mdadm --detail /dev/md*, mdadm --examine /dev/sd*, and smartctl -a /dev/sdX for all disks?

Are you using hardened, secure linux, app armor, or similar?

If you suspect a kernel bug you could also try and see if it just works with SystemRescueCD or similar...

Last edited by frostschutz (2016-05-04 10:25:29)

Offline

#5 2016-05-04 19:51:36

Guidobelix
Member
From: Italy
Registered: 2013-06-21
Posts: 31

Re: Raid5 stuck in reshape operation

Starting from the end: I am using standard version, updated to Kernel 4.5.1.
Then, disk1:

┌─[root@Archlinux]─[~]
└──> smartctl -a /dev/sdb
smartctl 6.4 2015-06-04 r4109 [x86_64-linux-4.5.1-1-ARCH] (local build)
Copyright (C) 2002-15, Bruce Allen, Christian Franke, www.smartmontools.org

=== START OF INFORMATION SECTION ===
Model Family:     Western Digital Red
Device Model:     WDC WD20EFRX-68EUZN0
Serial Number:    WD-WCC4M0405843
LU WWN Device Id: 5 0014ee 25e94cd2a
Firmware Version: 80.00A80
User Capacity:    2,000,398,934,016 bytes [2,00 TB]
Sector Sizes:     512 bytes logical, 4096 bytes physical
Rotation Rate:    5400 rpm
Device is:        In smartctl database [for details use: -P show]
ATA Version is:   ACS-2 (minor revision not indicated)
SATA Version is:  SATA 3.0, 6.0 Gb/s (current: 3.0 Gb/s)
Local Time is:    Wed May  4 21:29:02 2016 CEST
SMART support is: Available - device has SMART capability.
SMART support is: Enabled

=== START OF READ SMART DATA SECTION ===
SMART overall-health self-assessment test result: PASSED

General SMART Values:
Offline data collection status:  (0x82) Offline data collection activity
                                        was completed without error.
                                        Auto Offline Data Collection: Enabled.
Self-test execution status:      (   0) The previous self-test routine completed
                                        without error or no self-test has ever 
                                        been run.
Total time to complete Offline 
data collection:                (25500) seconds.
Offline data collection
capabilities:                    (0x7b) SMART execute Offline immediate.
                                        Auto Offline data collection on/off support.
                                        Suspend Offline collection upon new
                                        command.
                                        Offline surface scan supported.
                                        Self-test supported.
                                        Conveyance Self-test supported.
                                        Selective Self-test supported.
SMART capabilities:            (0x0003) Saves SMART data before entering
                                        power-saving mode.
                                        Supports SMART auto save timer.
Error logging capability:        (0x01) Error logging supported.
                                        General Purpose Logging supported.
Short self-test routine 
recommended polling time:        (   2) minutes.
Extended self-test routine
recommended polling time:        ( 258) minutes.
Conveyance self-test routine
recommended polling time:        (   5) minutes.
SCT capabilities:              (0x703d) SCT Status supported.
                                        SCT Error Recovery Control supported.
                                        SCT Feature Control supported.
                                        SCT Data Table supported.

SMART Attributes Data Structure revision number: 16
Vendor Specific SMART Attributes with Thresholds:
ID# ATTRIBUTE_NAME          FLAG     VALUE WORST THRESH TYPE      UPDATED  WHEN_FAILED RAW_VALUE
  1 Raw_Read_Error_Rate     0x002f   200   200   051    Pre-fail  Always       -       0
  3 Spin_Up_Time            0x0027   175   172   021    Pre-fail  Always       -       4225
  4 Start_Stop_Count        0x0032   100   100   000    Old_age   Always       -       112
  5 Reallocated_Sector_Ct   0x0033   200   200   140    Pre-fail  Always       -       0
  7 Seek_Error_Rate         0x002e   200   200   000    Old_age   Always       -       0
  9 Power_On_Hours          0x0032   074   074   000    Old_age   Always       -       19649
 10 Spin_Retry_Count        0x0032   100   100   000    Old_age   Always       -       0
 11 Calibration_Retry_Count 0x0032   100   100   000    Old_age   Always       -       0
 12 Power_Cycle_Count       0x0032   100   100   000    Old_age   Always       -       103
192 Power-Off_Retract_Count 0x0032   200   200   000    Old_age   Always       -       21
193 Load_Cycle_Count        0x0032   001   001   000    Old_age   Always       -       878595
194 Temperature_Celsius     0x0022   110   103   000    Old_age   Always       -       37
196 Reallocated_Event_Count 0x0032   200   200   000    Old_age   Always       -       0
197 Current_Pending_Sector  0x0032   200   200   000    Old_age   Always       -       0
198 Offline_Uncorrectable   0x0030   200   200   000    Old_age   Offline      -       0
199 UDMA_CRC_Error_Count    0x0032   200   200   000    Old_age   Always       -       0
200 Multi_Zone_Error_Rate   0x0008   200   200   000    Old_age   Offline      -       0

SMART Error Log Version: 1
No Errors Logged

SMART Self-test log structure revision number 1
Num  Test_Description    Status                  Remaining  LifeTime(hours)  LBA_of_first_error
# 1  Extended offline    Completed without error       00%     19583         -
# 2  Short offline       Completed without error       00%     19413         -
# 3  Short offline       Completed without error       00%     19246         -
# 4  Short offline       Completed without error       00%     19080         -
# 5  Short offline       Completed without error       00%     18912         -
# 6  Extended offline    Completed without error       00%     18846         -
# 7  Short offline       Completed without error       00%     18603         -
# 8  Short offline       Completed without error       00%     18435         -
# 9  Short offline       Completed without error       00%     18268         -
#10  Short offline       Completed without error       00%     18133         -
#11  Short offline       Completed without error       00%     17965         -
#12  Short offline       Completed without error       00%     17797         -
#13  Short offline       Completed without error       00%     17630         -
#14  Short offline       Completed without error       00%     17462         -
#15  Short offline       Completed without error       00%     17294         -
#16  Short offline       Completed without error       00%     17126         -
#17  Short offline       Completed without error       00%     16958         -
#18  Short offline       Completed without error       00%     16790         -
#19  Short offline       Completed without error       00%     16623         -
#20  Short offline       Completed without error       00%     16455         -
#21  Short offline       Completed without error       00%     16287         -

SMART Selective self-test log data structure revision number 1
 SPAN  MIN_LBA  MAX_LBA  CURRENT_TEST_STATUS
    1        0        0  Not_testing
    2        0        0  Not_testing
    3        0        0  Not_testing
    4        0        0  Not_testing
    5        0        0  Not_testing
Selective self-test flags (0x0):
  After scanning selected spans, do NOT read-scan remainder of disk.
If Selective self-test is pending on power-up, resume after 0 minute delay.

Disk2:

┌─[root@Archlinux]─[~]
└──> smartctl -a /dev/sdc
smartctl 6.4 2015-06-04 r4109 [x86_64-linux-4.5.1-1-ARCH] (local build)
Copyright (C) 2002-15, Bruce Allen, Christian Franke, www.smartmontools.org

=== START OF INFORMATION SECTION ===
Model Family:     Western Digital Red
Device Model:     WDC WD20EFRX-68EUZN0
Serial Number:    WD-WCC4M0389308
LU WWN Device Id: 5 0014ee 2093ce7b4
Firmware Version: 80.00A80
User Capacity:    2,000,398,934,016 bytes [2,00 TB]
Sector Sizes:     512 bytes logical, 4096 bytes physical
Rotation Rate:    5400 rpm
Device is:        In smartctl database [for details use: -P show]
ATA Version is:   ACS-2 (minor revision not indicated)
SATA Version is:  SATA 3.0, 6.0 Gb/s (current: 3.0 Gb/s)
Local Time is:    Wed May  4 21:31:26 2016 CEST
SMART support is: Available - device has SMART capability.
SMART support is: Enabled

=== START OF READ SMART DATA SECTION ===
SMART overall-health self-assessment test result: PASSED

General SMART Values:
Offline data collection status:  (0x82) Offline data collection activity
                                        was completed without error.
                                        Auto Offline Data Collection: Enabled.
Self-test execution status:      (   0) The previous self-test routine completed
                                        without error or no self-test has ever 
                                        been run.
Total time to complete Offline 
data collection:                (25980) seconds.
Offline data collection
capabilities:                    (0x7b) SMART execute Offline immediate.
                                        Auto Offline data collection on/off support.
                                        Suspend Offline collection upon new
                                        command.
                                        Offline surface scan supported.
                                        Self-test supported.
                                        Conveyance Self-test supported.
                                        Selective Self-test supported.
SMART capabilities:            (0x0003) Saves SMART data before entering
                                        power-saving mode.
                                        Supports SMART auto save timer.
Error logging capability:        (0x01) Error logging supported.
                                        General Purpose Logging supported.
Short self-test routine 
recommended polling time:        (   2) minutes.
Extended self-test routine
recommended polling time:        ( 263) minutes.
Conveyance self-test routine
recommended polling time:        (   5) minutes.
SCT capabilities:              (0x703d) SCT Status supported.
                                        SCT Error Recovery Control supported.
                                        SCT Feature Control supported.
                                        SCT Data Table supported.

SMART Attributes Data Structure revision number: 16
Vendor Specific SMART Attributes with Thresholds:
ID# ATTRIBUTE_NAME          FLAG     VALUE WORST THRESH TYPE      UPDATED  WHEN_FAILED RAW_VALUE
  1 Raw_Read_Error_Rate     0x002f   200   200   051    Pre-fail  Always       -       0
  3 Spin_Up_Time            0x0027   175   174   021    Pre-fail  Always       -       4216
  4 Start_Stop_Count        0x0032   100   100   000    Old_age   Always       -       108
  5 Reallocated_Sector_Ct   0x0033   200   200   140    Pre-fail  Always       -       0
  7 Seek_Error_Rate         0x002e   200   200   000    Old_age   Always       -       0
  9 Power_On_Hours          0x0032   074   074   000    Old_age   Always       -       19655
 10 Spin_Retry_Count        0x0032   100   100   000    Old_age   Always       -       0
 11 Calibration_Retry_Count 0x0032   100   253   000    Old_age   Always       -       0
 12 Power_Cycle_Count       0x0032   100   100   000    Old_age   Always       -       99
192 Power-Off_Retract_Count 0x0032   200   200   000    Old_age   Always       -       18
193 Load_Cycle_Count        0x0032   200   200   000    Old_age   Always       -       89
194 Temperature_Celsius     0x0022   108   101   000    Old_age   Always       -       39
196 Reallocated_Event_Count 0x0032   200   200   000    Old_age   Always       -       0
197 Current_Pending_Sector  0x0032   200   200   000    Old_age   Always       -       0
198 Offline_Uncorrectable   0x0030   200   200   000    Old_age   Offline      -       0
199 UDMA_CRC_Error_Count    0x0032   200   200   000    Old_age   Always       -       0
200 Multi_Zone_Error_Rate   0x0008   200   200   000    Old_age   Offline      -       0

SMART Error Log Version: 1
No Errors Logged

SMART Self-test log structure revision number 1
Num  Test_Description    Status                  Remaining  LifeTime(hours)  LBA_of_first_error
# 1  Short offline       Completed without error       00%     19611         -
# 2  Extended offline    Completed without error       00%     19592         -
# 3  Short offline       Completed without error       00%     19443         -
# 4  Short offline       Completed without error       00%     19275         -
# 5  Short offline       Completed without error       00%     19110         -
# 6  Short offline       Completed without error       00%     18942         -
# 7  Extended offline    Completed without error       00%     18876         -
# 8  Short offline       Completed without error       00%     18774         -
# 9  Short offline       Completed without error       00%     18632         -
#10  Short offline       Completed without error       00%     18465         -
#11  Short offline       Completed without error       00%     18298         -
#12  Short offline       Completed without error       00%     18163         -
#13  Short offline       Completed without error       00%     17995         -
#14  Short offline       Completed without error       00%     17827         -
#15  Short offline       Completed without error       00%     17659         -
#16  Short offline       Completed without error       00%     17491         -
#17  Short offline       Completed without error       00%     17323         -
#18  Short offline       Completed without error       00%     17156         -
#19  Short offline       Completed without error       00%     16988         -
#20  Short offline       Completed without error       00%     16820         -
#21  Short offline       Completed without error       00%     16652         -

SMART Selective self-test log data structure revision number 1
 SPAN  MIN_LBA  MAX_LBA  CURRENT_TEST_STATUS
    1        0        0  Not_testing
    2        0        0  Not_testing
    3        0        0  Not_testing
    4        0        0  Not_testing
    5        0        0  Not_testing
Selective self-test flags (0x0):
  After scanning selected spans, do NOT read-scan remainder of disk.
If Selective self-test is pending on power-up, resume after 0 minute delay.

Disk3:

┌─[root@Archlinux]─[~]
└──> smartctl -a /dev/sdd
smartctl 6.4 2015-06-04 r4109 [x86_64-linux-4.5.1-1-ARCH] (local build)
Copyright (C) 2002-15, Bruce Allen, Christian Franke, www.smartmontools.org

=== START OF INFORMATION SECTION ===
Model Family:     Western Digital Red
Device Model:     WDC WD20EFRX-68EUZN0
Serial Number:    WD-WCC4M0VVZVTH
LU WWN Device Id: 5 0014ee 262372fa6
Firmware Version: 82.00A82
User Capacity:    2,000,398,934,016 bytes [2,00 TB]
Sector Sizes:     512 bytes logical, 4096 bytes physical
Rotation Rate:    5400 rpm
Device is:        In smartctl database [for details use: -P show]
ATA Version is:   ACS-2 (minor revision not indicated)
SATA Version is:  SATA 3.0, 6.0 Gb/s (current: 3.0 Gb/s)
Local Time is:    Wed May  4 21:32:15 2016 CEST
SMART support is: Available - device has SMART capability.
SMART support is: Enabled

=== START OF READ SMART DATA SECTION ===
SMART overall-health self-assessment test result: PASSED

General SMART Values:
Offline data collection status:  (0x82) Offline data collection activity
                                        was completed without error.
                                        Auto Offline Data Collection: Enabled.
Self-test execution status:      (   0) The previous self-test routine completed
                                        without error or no self-test has ever 
                                        been run.
Total time to complete Offline 
data collection:                (25680) seconds.
Offline data collection
capabilities:                    (0x7b) SMART execute Offline immediate.
                                        Auto Offline data collection on/off support.
                                        Suspend Offline collection upon new
                                        command.
                                        Offline surface scan supported.
                                        Self-test supported.
                                        Conveyance Self-test supported.
                                        Selective Self-test supported.
SMART capabilities:            (0x0003) Saves SMART data before entering
                                        power-saving mode.
                                        Supports SMART auto save timer.
Error logging capability:        (0x01) Error logging supported.
                                        General Purpose Logging supported.
Short self-test routine 
recommended polling time:        (   2) minutes.
Extended self-test routine
recommended polling time:        ( 260) minutes.
Conveyance self-test routine
recommended polling time:        (   5) minutes.
SCT capabilities:              (0x703d) SCT Status supported.
                                        SCT Error Recovery Control supported.
                                        SCT Feature Control supported.
                                        SCT Data Table supported.

SMART Attributes Data Structure revision number: 16
Vendor Specific SMART Attributes with Thresholds:
ID# ATTRIBUTE_NAME          FLAG     VALUE WORST THRESH TYPE      UPDATED  WHEN_FAILED RAW_VALUE
  1 Raw_Read_Error_Rate     0x002f   200   200   051    Pre-fail  Always       -       0
  3 Spin_Up_Time            0x0027   100   253   021    Pre-fail  Always       -       0
  4 Start_Stop_Count        0x0032   100   100   000    Old_age   Always       -       3
  5 Reallocated_Sector_Ct   0x0033   200   200   140    Pre-fail  Always       -       0
  7 Seek_Error_Rate         0x002e   200   200   000    Old_age   Always       -       0
  9 Power_On_Hours          0x0032   099   099   000    Old_age   Always       -       861
 10 Spin_Retry_Count        0x0032   100   253   000    Old_age   Always       -       0
 11 Calibration_Retry_Count 0x0032   100   253   000    Old_age   Always       -       0
 12 Power_Cycle_Count       0x0032   100   100   000    Old_age   Always       -       3
192 Power-Off_Retract_Count 0x0032   200   200   000    Old_age   Always       -       0
193 Load_Cycle_Count        0x0032   200   200   000    Old_age   Always       -       115
194 Temperature_Celsius     0x0022   108   102   000    Old_age   Always       -       39
196 Reallocated_Event_Count 0x0032   200   200   000    Old_age   Always       -       0
197 Current_Pending_Sector  0x0032   200   200   000    Old_age   Always       -       0
198 Offline_Uncorrectable   0x0030   200   200   000    Old_age   Offline      -       0
199 UDMA_CRC_Error_Count    0x0032   200   200   000    Old_age   Always       -       0
200 Multi_Zone_Error_Rate   0x0008   200   200   000    Old_age   Offline      -       0

SMART Error Log Version: 1
No Errors Logged

SMART Self-test log structure revision number 1
Num  Test_Description    Status                  Remaining  LifeTime(hours)  LBA_of_first_error
# 1  Extended offline    Completed without error       00%       822         -
# 2  Short offline       Completed without error       00%       673         -
# 3  Short offline       Completed without error       00%       505         -
# 4  Short offline       Completed without error       00%       339         -
# 5  Short offline       Completed without error       00%       171         -
# 6  Extended offline    Completed without error       00%       105         -
# 7  Short offline       Completed without error       00%        14         -

SMART Selective self-test log data structure revision number 1
 SPAN  MIN_LBA  MAX_LBA  CURRENT_TEST_STATUS
    1        0        0  Not_testing
    2        0        0  Not_testing
    3        0        0  Not_testing
    4        0        0  Not_testing
    5        0        0  Not_testing
Selective self-test flags (0x0):
  After scanning selected spans, do NOT read-scan remainder of disk.
If Selective self-test is pending on power-up, resume after 0 minute delay.

Disk4:

┌─[root@Archlinux]─[~]
└──> smartctl -a /dev/sde
smartctl 6.4 2015-06-04 r4109 [x86_64-linux-4.5.1-1-ARCH] (local build)
Copyright (C) 2002-15, Bruce Allen, Christian Franke, www.smartmontools.org

=== START OF INFORMATION SECTION ===
Model Family:     Western Digital Red
Device Model:     WDC WD20EFRX-68EUZN0
Serial Number:    WD-WCC4M0418450
LU WWN Device Id: 5 0014ee 25e95481f
Firmware Version: 80.00A80
User Capacity:    2,000,398,934,016 bytes [2,00 TB]
Sector Sizes:     512 bytes logical, 4096 bytes physical
Rotation Rate:    5400 rpm
Device is:        In smartctl database [for details use: -P show]
ATA Version is:   ACS-2 (minor revision not indicated)
SATA Version is:  SATA 3.0, 6.0 Gb/s (current: 3.0 Gb/s)
Local Time is:    Wed May  4 21:33:29 2016 CEST
SMART support is: Available - device has SMART capability.
SMART support is: Enabled

=== START OF READ SMART DATA SECTION ===
SMART overall-health self-assessment test result: PASSED

General SMART Values:
Offline data collection status:  (0x82) Offline data collection activity
                                        was completed without error.
                                        Auto Offline Data Collection: Enabled.
Self-test execution status:      (   0) The previous self-test routine completed
                                        without error or no self-test has ever 
                                        been run.
Total time to complete Offline 
data collection:                (27240) seconds.
Offline data collection
capabilities:                    (0x7b) SMART execute Offline immediate.
                                        Auto Offline data collection on/off support.
                                        Suspend Offline collection upon new
                                        command.
                                        Offline surface scan supported.
                                        Self-test supported.
                                        Conveyance Self-test supported.
                                        Selective Self-test supported.
SMART capabilities:            (0x0003) Saves SMART data before entering
                                        power-saving mode.
                                        Supports SMART auto save timer.
Error logging capability:        (0x01) Error logging supported.
                                        General Purpose Logging supported.
Short self-test routine 
recommended polling time:        (   2) minutes.
Extended self-test routine
recommended polling time:        ( 275) minutes.
Conveyance self-test routine
recommended polling time:        (   5) minutes.
SCT capabilities:              (0x703d) SCT Status supported.
                                        SCT Error Recovery Control supported.
                                        SCT Feature Control supported.
                                        SCT Data Table supported.

SMART Attributes Data Structure revision number: 16
Vendor Specific SMART Attributes with Thresholds:
ID# ATTRIBUTE_NAME          FLAG     VALUE WORST THRESH TYPE      UPDATED  WHEN_FAILED RAW_VALUE
  1 Raw_Read_Error_Rate     0x002f   200   200   051    Pre-fail  Always       -       1
  3 Spin_Up_Time            0x0027   174   172   021    Pre-fail  Always       -       4275
  4 Start_Stop_Count        0x0032   100   100   000    Old_age   Always       -       100
  5 Reallocated_Sector_Ct   0x0033   200   200   140    Pre-fail  Always       -       0
  7 Seek_Error_Rate         0x002e   200   200   000    Old_age   Always       -       0
  9 Power_On_Hours          0x0032   074   074   000    Old_age   Always       -       19646
 10 Spin_Retry_Count        0x0032   100   253   000    Old_age   Always       -       0
 11 Calibration_Retry_Count 0x0032   100   253   000    Old_age   Always       -       0
 12 Power_Cycle_Count       0x0032   100   100   000    Old_age   Always       -       91
192 Power-Off_Retract_Count 0x0032   200   200   000    Old_age   Always       -       15
193 Load_Cycle_Count        0x0032   001   001   000    Old_age   Always       -       888318
194 Temperature_Celsius     0x0022   109   099   000    Old_age   Always       -       38
196 Reallocated_Event_Count 0x0032   200   200   000    Old_age   Always       -       0
197 Current_Pending_Sector  0x0032   200   200   000    Old_age   Always       -       0
198 Offline_Uncorrectable   0x0030   200   200   000    Old_age   Offline      -       0
199 UDMA_CRC_Error_Count    0x0032   200   200   000    Old_age   Always       -       0
200 Multi_Zone_Error_Rate   0x0008   200   200   000    Old_age   Offline      -       0

SMART Error Log Version: 1
No Errors Logged

SMART Self-test log structure revision number 1
Num  Test_Description    Status                  Remaining  LifeTime(hours)  LBA_of_first_error
# 1  Short offline       Completed without error       00%     19482         -
# 2  Short offline       Completed without error       00%     19314         -
# 3  Short offline       Completed without error       00%     19148         -
# 4  Short offline       Completed without error       00%     18980         -
# 5  Extended offline    Completed without error       00%     18914         -
# 6  Short offline       Completed without error       00%     18813         -
# 7  Short offline       Completed without error       00%     18744         -
# 8  Short offline       Completed without error       00%     18575         -
# 9  Short offline       Completed without error       00%     18407         -
#10  Short offline       Completed without error       00%     18241         -
#11  Short offline       Completed without error       00%     18105         -
#12  Short offline       Completed without error       00%     17938         -
#13  Short offline       Completed without error       00%     17770         -
#14  Short offline       Completed without error       00%     17602         -
#15  Short offline       Completed without error       00%     17434         -
#16  Short offline       Completed without error       00%     17266         -
#17  Short offline       Completed without error       00%     17098         -
#18  Short offline       Completed without error       00%     16931         -
#19  Short offline       Completed without error       00%     16763         -
#20  Short offline       Completed without error       00%     16595         -
#21  Short offline       Completed without error       00%     16427         -

SMART Selective self-test log data structure revision number 1
 SPAN  MIN_LBA  MAX_LBA  CURRENT_TEST_STATUS
    1        0        0  Not_testing
    2        0        0  Not_testing
    3        0        0  Not_testing
    4        0        0  Not_testing
    5        0        0  Not_testing
Selective self-test flags (0x0):
  After scanning selected spans, do NOT read-scan remainder of disk.
If Selective self-test is pending on power-up, resume after 0 minute delay.

Disk5:

┌─[root@Archlinux]─[~]
└──> smartctl -a /dev/sdf
smartctl 6.4 2015-06-04 r4109 [x86_64-linux-4.5.1-1-ARCH] (local build)
Copyright (C) 2002-15, Bruce Allen, Christian Franke, www.smartmontools.org

=== START OF INFORMATION SECTION ===
Model Family:     Western Digital Red
Device Model:     WDC WD20EFRX-68EUZN0
Serial Number:    WD-WCC4M0PT9ZFF
LU WWN Device Id: 5 0014ee 20bc2286c
Firmware Version: 82.00A82
User Capacity:    2,000,398,934,016 bytes [2,00 TB]
Sector Sizes:     512 bytes logical, 4096 bytes physical
Rotation Rate:    5400 rpm
Device is:        In smartctl database [for details use: -P show]
ATA Version is:   ACS-2 (minor revision not indicated)
SATA Version is:  SATA 3.0, 6.0 Gb/s (current: 6.0 Gb/s)
Local Time is:    Wed May  4 21:33:58 2016 CEST
SMART support is: Available - device has SMART capability.
SMART support is: Enabled

=== START OF READ SMART DATA SECTION ===
SMART overall-health self-assessment test result: PASSED

General SMART Values:
Offline data collection status:  (0x00) Offline data collection activity
                                        was never started.
                                        Auto Offline Data Collection: Disabled.
Self-test execution status:      (   0) The previous self-test routine completed
                                        without error or no self-test has ever 
                                        been run.
Total time to complete Offline 
data collection:                (28800) seconds.
Offline data collection
capabilities:                    (0x7b) SMART execute Offline immediate.
                                        Auto Offline data collection on/off support.
                                        Suspend Offline collection upon new
                                        command.
                                        Offline surface scan supported.
                                        Self-test supported.
                                        Conveyance Self-test supported.
                                        Selective Self-test supported.
SMART capabilities:            (0x0003) Saves SMART data before entering
                                        power-saving mode.
                                        Supports SMART auto save timer.
Error logging capability:        (0x01) Error logging supported.
                                        General Purpose Logging supported.
Short self-test routine 
recommended polling time:        (   2) minutes.
Extended self-test routine
recommended polling time:        ( 290) minutes.
Conveyance self-test routine
recommended polling time:        (   5) minutes.
SCT capabilities:              (0x703d) SCT Status supported.
                                        SCT Error Recovery Control supported.
                                        SCT Feature Control supported.
                                        SCT Data Table supported.

SMART Attributes Data Structure revision number: 16
Vendor Specific SMART Attributes with Thresholds:
ID# ATTRIBUTE_NAME          FLAG     VALUE WORST THRESH TYPE      UPDATED  WHEN_FAILED RAW_VALUE
  1 Raw_Read_Error_Rate     0x002f   100   253   051    Pre-fail  Always       -       0
  3 Spin_Up_Time            0x0027   180   180   021    Pre-fail  Always       -       3958
  4 Start_Stop_Count        0x0032   100   100   000    Old_age   Always       -       6
  5 Reallocated_Sector_Ct   0x0033   200   200   140    Pre-fail  Always       -       0
  7 Seek_Error_Rate         0x002e   200   200   000    Old_age   Always       -       0
  9 Power_On_Hours          0x0032   100   100   000    Old_age   Always       -       99
 10 Spin_Retry_Count        0x0032   100   253   000    Old_age   Always       -       0
 11 Calibration_Retry_Count 0x0032   100   253   000    Old_age   Always       -       0
 12 Power_Cycle_Count       0x0032   100   100   000    Old_age   Always       -       3
192 Power-Off_Retract_Count 0x0032   200   200   000    Old_age   Always       -       0
193 Load_Cycle_Count        0x0032   200   200   000    Old_age   Always       -       82
194 Temperature_Celsius     0x0022   118   112   000    Old_age   Always       -       29
196 Reallocated_Event_Count 0x0032   200   200   000    Old_age   Always       -       0
197 Current_Pending_Sector  0x0032   200   200   000    Old_age   Always       -       0
198 Offline_Uncorrectable   0x0030   100   253   000    Old_age   Offline      -       0
199 UDMA_CRC_Error_Count    0x0032   200   200   000    Old_age   Always       -       0
200 Multi_Zone_Error_Rate   0x0008   100   253   000    Old_age   Offline      -       0

SMART Error Log Version: 1
No Errors Logged

SMART Self-test log structure revision number 1
No self-tests have been logged.  [To run self-tests, use: smartctl -t]

SMART Selective self-test log data structure revision number 1
 SPAN  MIN_LBA  MAX_LBA  CURRENT_TEST_STATUS
    1        0        0  Not_testing
    2        0        0  Not_testing
    3        0        0  Not_testing
    4        0        0  Not_testing
    5        0        0  Not_testing
Selective self-test flags (0x0):
  After scanning selected spans, do NOT read-scan remainder of disk.
If Selective self-test is pending on power-up, resume after 0 minute delay.
┌─[root@Archlinux]─[~]
└──> mdadm --examine /dev/sdb
/dev/sdb:
   MBR Magic : aa55
Partition[0] :   3907029167 sectors at            1 (type ee)
┌─[root@Archlinux]─[~]
└──> mdadm --examine /dev/sdc
/dev/sdc:
   MBR Magic : aa55
Partition[0] :   3907029167 sectors at            1 (type ee)
┌─[root@Archlinux]─[~]
└──> mdadm --examine /dev/sdd
/dev/sdd:
   MBR Magic : aa55
Partition[0] :   3907029167 sectors at            1 (type ee)
┌─[root@Archlinux]─[~]
└──> mdadm --examine /dev/sde
/dev/sde:
   MBR Magic : aa55
Partition[0] :   3907029167 sectors at            1 (type ee)
┌─[root@Archlinux]─[~]
└──> mdadm --examine /dev/sdf
/dev/sdf:
   MBR Magic : aa55
Partition[0] :   3907029167 sectors at            1 (type ee)

And finally:

┌─[root@Archlinux]─[~]
└──> mdadm --detail /dev/md0
/dev/md0:
        Version : 1.2
  Creation Time : Sat Jan 25 00:09:35 2014
     Raid Level : raid5
     Array Size : 5859840000 (5588.38 GiB 6000.48 GB)
  Used Dev Size : 1953280000 (1862.79 GiB 2000.16 GB)
   Raid Devices : 5
  Total Devices : 5
    Persistence : Superblock is persistent

  Intent Bitmap : Internal

    Update Time : Tue May  3 22:14:13 2016
          State : clean, reshaping 
 Active Devices : 5
Working Devices : 5
 Failed Devices : 0
  Spare Devices : 0

         Layout : left-symmetric
     Chunk Size : 256K

 Reshape Status : 67% complete
  Delta Devices : 1, (4->5)

           Name : archlinuxNAS:0
           UUID : 82e58d60:774f61d7:7e816c49:de093881
         Events : 416429

    Number   Major   Minor   RaidDevice State
       0       8       17        0      active sync   /dev/sdb1
       1       8       33        1      active sync   /dev/sdc1
       5       8       49        2      active sync   /dev/sdd1
       4       8       65        3      active sync   /dev/sde1
       6       8       81        4      active sync   /dev/sdf1

Every time I reboot (i did it only twice) the reshape restart automatically and everything hangs up and I cannot access the disk. Is there a way to prevent the reshaping to start automatically at boot? Maybe in that way I can access the RAID and backup that 20% that it is not backed up already (I know I am an idiot, please don't tell me that :-( )

Thanks a lot for your help!

Last edited by Guidobelix (2016-05-04 19:52:20)

Offline

#6 2016-05-04 20:47:22

frostschutz
Member
Registered: 2013-11-15
Posts: 1,409

Re: Raid5 stuck in reshape operation

mdadm has a --freeze-reshape option to prevent reshape resume at boot/assembly time. (and a --continue option to resume it at a later time)

I don't know whether arch mkinitcpio is making use of those options somehow.

You forgot the * for the examine (should be on the actual raid members wink i.e. sd[bcdef]1)

Your disks look fine (except for the one with raw read error 1, but as long as that value doesn't change and passes selftest, it might be a one time thing) and you seem to be running selftests regularly.

Maybe in that way I can access the RAID and backup that 20% that it is not backed up already (I know I am an idiot, please don't tell me that :-( )

There should be nothing wrong with your data (as long as no more than one disk failed). I'm not sure why yours got stuck, there seem to be other such cases recently so maybe it is actually a kernel problem. Is there anything in dmesg?

Even if it can't be kicked back into operation, with a grow you have the situation that the first 67% of your capacity is a 5 disk raid5, and the remaining 33% is a 4 disk raid5, and there is a region in between that has duplicate data (in 4 disk raid5 format that already got relocated to lower sectors of 5 disk raid5). So it should definitely be possible to recover even if you had to do it from scratch using overlays https://raid.wiki.kernel.org/index.php/ … erlay_file by building two raid5 and finding a point in the duplicate data region somewhere below the 67% mark...

If you're willing to share your RAID metadata (first 64K or so of each disk) I can play around with it, but the last time someone gave me their "stuck" metadata it simply resumed reshaping for me... so not sure why this problem happens for some people but not others.

If all else fails you could also take it to the linux-raid mailing list and see if the people there have any other ideas. The Gentoo thread linked earlier has also a thread on the mailing list, subject "mdadm - stuck reshape operation" but the only other suggestion I see there is to compile mdadm from git://neil.brown.name/mdadm (which I'm not sure will help since gentoo/archlinux tend to use recent versions already)

Offline

#7 2016-05-05 07:40:29

Guidobelix
Member
From: Italy
Registered: 2013-06-21
Posts: 31

Re: Raid5 stuck in reshape operation

Thank you very much once again for your help :-) What a great community this is!
I thought of using overlays, but I am afraid that's a bit other my current skills.
Before I explore that way, I have a question: As long as no disks fail, a RAID5 should be accessible during reshape, shouldn't it?
In such a case, what about the following plan:

  1. Start with a Live CD (so the RAID will not be assembled, I think).

  2. Assembly manually the array with the option --freeze-reshape.

  3. Try to backup the 30-40% of the data which don't have a backup yet.

  4. Erase everything and start from the beginning with a 5-disk array.

Is there a risk of damaging data by trying to assemble the array from a Live CD?
Last question: how do I extract the RAID metadata you mentioned?

Thank you once more :-)
Guido

Offline

#8 2016-05-14 15:50:53

Guidobelix
Member
From: Italy
Registered: 2013-06-21
Posts: 31

Re: Raid5 stuck in reshape operation

I have tried several times to start the machine from a live CD and then start the array with the --freze-reshape option, but I had no success. the array is assembled but the reshape operation starts anyway and everything get stuck.
I really need some guidance :-(

Offline

#9 2016-05-14 16:18:24

frostschutz
Member
Registered: 2013-11-15
Posts: 1,409

Re: Raid5 stuck in reshape operation

Guidobelix wrote:

Last question: how do I extract the RAID metadata you mentioned?

With dd, something like

dd bs=64K count=1 if=/dev/thing of=thing.header

(and that for each member of your raid)

mdadm itself also has a --dump=directory/ option, never tried that one before.

Offline

#10 2016-06-05 12:44:16

Guidobelix
Member
From: Italy
Registered: 2013-06-21
Posts: 31

Re: Raid5 stuck in reshape operation

@frostschutz
Apologies for the long silence, I have been busy with "real" work and had no access to my machine.
Anyway, I have just sent you an email with the the headers I have extracted, following your instructions.

Anyway, for me it would be good enough to be able to stop reshaping (or, better, to prevent reshape from starting when I reboot the machine), so that I can update my backup.
Once that is done, I can simply kill the raid and restart from zero.
But with the reshaping running and stuck, I cannot access the files on the disk (any attempt to mount or access the file-system hangs as well).

Thanks everybody for help/hints/suggestions

Last edited by Guidobelix (2016-06-05 12:47:07)

Offline

#11 2016-06-06 10:11:39

frostschutz
Member
Registered: 2013-11-15
Posts: 1,409

Re: Raid5 stuck in reshape operation

What you sent me was just a partition table...

Offline

#12 2016-06-06 10:22:36

Guidobelix
Member
From: Italy
Registered: 2013-06-21
Posts: 31

Re: Raid5 stuck in reshape operation

Apologies, I might have misunderstood your instructions.
Haven't you told me to run
dd bs=64K count=1 if=/dev/sdX of=sdX.header
for each member of the raid?
I will double check the commands I have run once I have access to the machine again.

Offline

#13 2016-06-06 10:26:38

frostschutz
Member
Registered: 2013-11-15
Posts: 1,409

Re: Raid5 stuck in reshape operation

Yes, for each member of the raid... member = partition in your case. Seems you ran it on the whole disk instead...

Offline

#14 2016-06-06 21:44:02

Guidobelix
Member
From: Italy
Registered: 2013-06-21
Posts: 31

Re: Raid5 stuck in reshape operation

yikes shame on me!
I've misread your instructions, sorry sad
I'll redo that.

Offline

#15 2016-06-12 17:52:55

Guidobelix
Member
From: Italy
Registered: 2013-06-21
Posts: 31

Re: Raid5 stuck in reshape operation

@frostschutz
I have tried again and sent you the result. I hpope this time I have understood your instructions in the right way.

Does anybody know what happen if I take one disk out and the start the system? Will I be able to start the raid in degradated mode (mdadm --manage /dev/md0 --run) and then access it? Or will I simpèly destroy the data on the raid?

Thanks a lot once more for your help.

Guido

Offline

#16 2016-06-12 22:07:57

frostschutz
Member
Registered: 2013-11-15
Posts: 1,409

Re: Raid5 stuck in reshape operation

It would still have to continue grow even with a disk missing. Very risky as you'd have no redundancy this way.

I've used your metadata to build your array out of loop devices... the strange thing is, the grow resumes normally for me. I tested this on a Gentoo box, kernel 4.6.1, mdadm 3.4. On my side it seems there is no problem.

The only oddity is that the md metadata reports bad blocks:

# mdadm --examine-badblocks sd*1.header 
Bad-blocks list is empty in sdb1.header
Bad-blocks list is empty in sdc1.header
Bad-blocks on sdd1.header:
          2641039808 for 320 sectors
Bad-blocks on sde1.header:
          2641039808 for 320 sectors
Bad-blocks list is empty in sdf1.header

You could try reading these, should be

dd bs=512 if=/dev/sdd1 skip=2641039808 count=320 | wc -c
dd bs=512 if=/dev/sde1 skip=2641039808 count=320 | wc -c

If there are no read errors, it seems md has recorded bad blocks as a fluke, maybe due to a controller/kernel problem... The bad block log in theory is a nice idea but my experience with it is not good. Once it has entries you can never get rid of them...

Offline

#17 2016-06-13 20:27:57

Guidobelix
Member
From: Italy
Registered: 2013-06-21
Posts: 31

Re: Raid5 stuck in reshape operation

┌─[root@Archlinux]─[~]
└──> dd bs=512 if=/dev/sdd1 skip=2641039808 count=320 | wc -c
163840
320+0 record dentro
320+0 record fuori
163840 bytes (164 kB, 160 KiB) copied, 0,559506 s, 293 kB/s
┌─[root@Archlinux]─[~]
└──> dd bs=512 if=/dev/sde1 skip=2641039808 count=320 | wc -c
163840
320+0 record dentro
320+0 record fuori
163840 bytes (164 kB, 160 KiB) copied, 0,681416 s, 240 kB/s
┌─[root@Archlinux]─[~]
└──> 

It seems I can read both blocks without errors.

Do you have any suggestion for the next move?

Offline

Board footer

Powered by FluxBB