You are not logged in.
- Topics: Active | Unanswered
Pages: 1
#1 2016-05-02 19:37:52
- Guidobelix
- Member
- From: Italy
- Registered: 2013-06-21
- Posts: 31
Raid5 stuck in reshape operation
Looking for help on the following issue:
I have decided to add a disk to my RAID5 array, from 4x2TB to 5x2TB, so I have partitioned the new disk as the others and then added it to the array.
┌─[root@Archlinux]─[~]
└──> mdadm --grow --raid-devices=5 /dev/md0The reshape operation started successfully but it then got stuch at 67.6% and it is like that since 24 hours.
┌─[root@Archlinux]─[~]
└──> cat /proc/mdstat
Personalities : [raid6] [raid5] [raid4]
md0 : active raid5 sdf1[6] sde1[4] sdc1[1] sdd1[5] sdb1[0]
5859840000 blocks super 1.2 level 5, 256k chunk, algorithm 2 [5/5] [UUUUU]
[=============>.......] reshape = 67.6% (1320482400/1953280000) finish=2074327.9min speed=5K/sec
bitmap: 0/15 pages [0KB], 65536KB chunk
unused devices: <none>I see only one occurrence of this message in journactl:
mag 01 22:30:49 Archlinux kernel: INFO: task md0_reshape:637 blocked for more than 120 seconds.
mag 01 22:30:49 Archlinux kernel: Tainted: G O 4.5.1-1-ARCH #1
mag 01 22:30:49 Archlinux kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
mag 01 22:30:49 Archlinux kernel: md0_reshape D c0670318 0 637 2 0x00000000The disk is still mounted, but If I cannot list or access files.
What can I do now? Is there a way to restart the operation without loosing the data?
Thanks a lot for your help!
Offline
#2 2016-05-03 08:13:57
- Polyatomic
- Member
- Registered: 2014-05-31
- Posts: 7
Re: Raid5 stuck in reshape operation
For what its worth, I've spotted a similar raid drama here:
https://forums.gentoo.org/viewtopic-t-1043706.html
Offline
#3 2016-05-03 20:35:51
- Guidobelix
- Member
- From: Italy
- Registered: 2013-06-21
- Posts: 31
Re: Raid5 stuck in reshape operation
Indeed, I saw that one while looking for some help on the Internet.
Update from today: I have rebooted the server, but mdadm is still stuck, and I cannot restart it.
I really don' know what to do...
Last edited by Guidobelix (2016-05-03 20:36:13)
Offline
#4 2016-05-03 21:09:00
- frostschutz
- Member
- Registered: 2013-11-15
- Posts: 1,475
Re: Raid5 stuck in reshape operation
Can you show mdadm --detail /dev/md*, mdadm --examine /dev/sd*, and smartctl -a /dev/sdX for all disks?
Are you using hardened, secure linux, app armor, or similar?
If you suspect a kernel bug you could also try and see if it just works with SystemRescueCD or similar...
Last edited by frostschutz (2016-05-04 10:25:29)
Offline
#5 2016-05-04 19:51:36
- Guidobelix
- Member
- From: Italy
- Registered: 2013-06-21
- Posts: 31
Re: Raid5 stuck in reshape operation
Starting from the end: I am using standard version, updated to Kernel 4.5.1.
Then, disk1:
┌─[root@Archlinux]─[~]
└──> smartctl -a /dev/sdb
smartctl 6.4 2015-06-04 r4109 [x86_64-linux-4.5.1-1-ARCH] (local build)
Copyright (C) 2002-15, Bruce Allen, Christian Franke, www.smartmontools.org
=== START OF INFORMATION SECTION ===
Model Family: Western Digital Red
Device Model: WDC WD20EFRX-68EUZN0
Serial Number: WD-WCC4M0405843
LU WWN Device Id: 5 0014ee 25e94cd2a
Firmware Version: 80.00A80
User Capacity: 2,000,398,934,016 bytes [2,00 TB]
Sector Sizes: 512 bytes logical, 4096 bytes physical
Rotation Rate: 5400 rpm
Device is: In smartctl database [for details use: -P show]
ATA Version is: ACS-2 (minor revision not indicated)
SATA Version is: SATA 3.0, 6.0 Gb/s (current: 3.0 Gb/s)
Local Time is: Wed May 4 21:29:02 2016 CEST
SMART support is: Available - device has SMART capability.
SMART support is: Enabled
=== START OF READ SMART DATA SECTION ===
SMART overall-health self-assessment test result: PASSED
General SMART Values:
Offline data collection status: (0x82) Offline data collection activity
was completed without error.
Auto Offline Data Collection: Enabled.
Self-test execution status: ( 0) The previous self-test routine completed
without error or no self-test has ever
been run.
Total time to complete Offline
data collection: (25500) seconds.
Offline data collection
capabilities: (0x7b) SMART execute Offline immediate.
Auto Offline data collection on/off support.
Suspend Offline collection upon new
command.
Offline surface scan supported.
Self-test supported.
Conveyance Self-test supported.
Selective Self-test supported.
SMART capabilities: (0x0003) Saves SMART data before entering
power-saving mode.
Supports SMART auto save timer.
Error logging capability: (0x01) Error logging supported.
General Purpose Logging supported.
Short self-test routine
recommended polling time: ( 2) minutes.
Extended self-test routine
recommended polling time: ( 258) minutes.
Conveyance self-test routine
recommended polling time: ( 5) minutes.
SCT capabilities: (0x703d) SCT Status supported.
SCT Error Recovery Control supported.
SCT Feature Control supported.
SCT Data Table supported.
SMART Attributes Data Structure revision number: 16
Vendor Specific SMART Attributes with Thresholds:
ID# ATTRIBUTE_NAME FLAG VALUE WORST THRESH TYPE UPDATED WHEN_FAILED RAW_VALUE
1 Raw_Read_Error_Rate 0x002f 200 200 051 Pre-fail Always - 0
3 Spin_Up_Time 0x0027 175 172 021 Pre-fail Always - 4225
4 Start_Stop_Count 0x0032 100 100 000 Old_age Always - 112
5 Reallocated_Sector_Ct 0x0033 200 200 140 Pre-fail Always - 0
7 Seek_Error_Rate 0x002e 200 200 000 Old_age Always - 0
9 Power_On_Hours 0x0032 074 074 000 Old_age Always - 19649
10 Spin_Retry_Count 0x0032 100 100 000 Old_age Always - 0
11 Calibration_Retry_Count 0x0032 100 100 000 Old_age Always - 0
12 Power_Cycle_Count 0x0032 100 100 000 Old_age Always - 103
192 Power-Off_Retract_Count 0x0032 200 200 000 Old_age Always - 21
193 Load_Cycle_Count 0x0032 001 001 000 Old_age Always - 878595
194 Temperature_Celsius 0x0022 110 103 000 Old_age Always - 37
196 Reallocated_Event_Count 0x0032 200 200 000 Old_age Always - 0
197 Current_Pending_Sector 0x0032 200 200 000 Old_age Always - 0
198 Offline_Uncorrectable 0x0030 200 200 000 Old_age Offline - 0
199 UDMA_CRC_Error_Count 0x0032 200 200 000 Old_age Always - 0
200 Multi_Zone_Error_Rate 0x0008 200 200 000 Old_age Offline - 0
SMART Error Log Version: 1
No Errors Logged
SMART Self-test log structure revision number 1
Num Test_Description Status Remaining LifeTime(hours) LBA_of_first_error
# 1 Extended offline Completed without error 00% 19583 -
# 2 Short offline Completed without error 00% 19413 -
# 3 Short offline Completed without error 00% 19246 -
# 4 Short offline Completed without error 00% 19080 -
# 5 Short offline Completed without error 00% 18912 -
# 6 Extended offline Completed without error 00% 18846 -
# 7 Short offline Completed without error 00% 18603 -
# 8 Short offline Completed without error 00% 18435 -
# 9 Short offline Completed without error 00% 18268 -
#10 Short offline Completed without error 00% 18133 -
#11 Short offline Completed without error 00% 17965 -
#12 Short offline Completed without error 00% 17797 -
#13 Short offline Completed without error 00% 17630 -
#14 Short offline Completed without error 00% 17462 -
#15 Short offline Completed without error 00% 17294 -
#16 Short offline Completed without error 00% 17126 -
#17 Short offline Completed without error 00% 16958 -
#18 Short offline Completed without error 00% 16790 -
#19 Short offline Completed without error 00% 16623 -
#20 Short offline Completed without error 00% 16455 -
#21 Short offline Completed without error 00% 16287 -
SMART Selective self-test log data structure revision number 1
SPAN MIN_LBA MAX_LBA CURRENT_TEST_STATUS
1 0 0 Not_testing
2 0 0 Not_testing
3 0 0 Not_testing
4 0 0 Not_testing
5 0 0 Not_testing
Selective self-test flags (0x0):
After scanning selected spans, do NOT read-scan remainder of disk.
If Selective self-test is pending on power-up, resume after 0 minute delay.Disk2:
┌─[root@Archlinux]─[~]
└──> smartctl -a /dev/sdc
smartctl 6.4 2015-06-04 r4109 [x86_64-linux-4.5.1-1-ARCH] (local build)
Copyright (C) 2002-15, Bruce Allen, Christian Franke, www.smartmontools.org
=== START OF INFORMATION SECTION ===
Model Family: Western Digital Red
Device Model: WDC WD20EFRX-68EUZN0
Serial Number: WD-WCC4M0389308
LU WWN Device Id: 5 0014ee 2093ce7b4
Firmware Version: 80.00A80
User Capacity: 2,000,398,934,016 bytes [2,00 TB]
Sector Sizes: 512 bytes logical, 4096 bytes physical
Rotation Rate: 5400 rpm
Device is: In smartctl database [for details use: -P show]
ATA Version is: ACS-2 (minor revision not indicated)
SATA Version is: SATA 3.0, 6.0 Gb/s (current: 3.0 Gb/s)
Local Time is: Wed May 4 21:31:26 2016 CEST
SMART support is: Available - device has SMART capability.
SMART support is: Enabled
=== START OF READ SMART DATA SECTION ===
SMART overall-health self-assessment test result: PASSED
General SMART Values:
Offline data collection status: (0x82) Offline data collection activity
was completed without error.
Auto Offline Data Collection: Enabled.
Self-test execution status: ( 0) The previous self-test routine completed
without error or no self-test has ever
been run.
Total time to complete Offline
data collection: (25980) seconds.
Offline data collection
capabilities: (0x7b) SMART execute Offline immediate.
Auto Offline data collection on/off support.
Suspend Offline collection upon new
command.
Offline surface scan supported.
Self-test supported.
Conveyance Self-test supported.
Selective Self-test supported.
SMART capabilities: (0x0003) Saves SMART data before entering
power-saving mode.
Supports SMART auto save timer.
Error logging capability: (0x01) Error logging supported.
General Purpose Logging supported.
Short self-test routine
recommended polling time: ( 2) minutes.
Extended self-test routine
recommended polling time: ( 263) minutes.
Conveyance self-test routine
recommended polling time: ( 5) minutes.
SCT capabilities: (0x703d) SCT Status supported.
SCT Error Recovery Control supported.
SCT Feature Control supported.
SCT Data Table supported.
SMART Attributes Data Structure revision number: 16
Vendor Specific SMART Attributes with Thresholds:
ID# ATTRIBUTE_NAME FLAG VALUE WORST THRESH TYPE UPDATED WHEN_FAILED RAW_VALUE
1 Raw_Read_Error_Rate 0x002f 200 200 051 Pre-fail Always - 0
3 Spin_Up_Time 0x0027 175 174 021 Pre-fail Always - 4216
4 Start_Stop_Count 0x0032 100 100 000 Old_age Always - 108
5 Reallocated_Sector_Ct 0x0033 200 200 140 Pre-fail Always - 0
7 Seek_Error_Rate 0x002e 200 200 000 Old_age Always - 0
9 Power_On_Hours 0x0032 074 074 000 Old_age Always - 19655
10 Spin_Retry_Count 0x0032 100 100 000 Old_age Always - 0
11 Calibration_Retry_Count 0x0032 100 253 000 Old_age Always - 0
12 Power_Cycle_Count 0x0032 100 100 000 Old_age Always - 99
192 Power-Off_Retract_Count 0x0032 200 200 000 Old_age Always - 18
193 Load_Cycle_Count 0x0032 200 200 000 Old_age Always - 89
194 Temperature_Celsius 0x0022 108 101 000 Old_age Always - 39
196 Reallocated_Event_Count 0x0032 200 200 000 Old_age Always - 0
197 Current_Pending_Sector 0x0032 200 200 000 Old_age Always - 0
198 Offline_Uncorrectable 0x0030 200 200 000 Old_age Offline - 0
199 UDMA_CRC_Error_Count 0x0032 200 200 000 Old_age Always - 0
200 Multi_Zone_Error_Rate 0x0008 200 200 000 Old_age Offline - 0
SMART Error Log Version: 1
No Errors Logged
SMART Self-test log structure revision number 1
Num Test_Description Status Remaining LifeTime(hours) LBA_of_first_error
# 1 Short offline Completed without error 00% 19611 -
# 2 Extended offline Completed without error 00% 19592 -
# 3 Short offline Completed without error 00% 19443 -
# 4 Short offline Completed without error 00% 19275 -
# 5 Short offline Completed without error 00% 19110 -
# 6 Short offline Completed without error 00% 18942 -
# 7 Extended offline Completed without error 00% 18876 -
# 8 Short offline Completed without error 00% 18774 -
# 9 Short offline Completed without error 00% 18632 -
#10 Short offline Completed without error 00% 18465 -
#11 Short offline Completed without error 00% 18298 -
#12 Short offline Completed without error 00% 18163 -
#13 Short offline Completed without error 00% 17995 -
#14 Short offline Completed without error 00% 17827 -
#15 Short offline Completed without error 00% 17659 -
#16 Short offline Completed without error 00% 17491 -
#17 Short offline Completed without error 00% 17323 -
#18 Short offline Completed without error 00% 17156 -
#19 Short offline Completed without error 00% 16988 -
#20 Short offline Completed without error 00% 16820 -
#21 Short offline Completed without error 00% 16652 -
SMART Selective self-test log data structure revision number 1
SPAN MIN_LBA MAX_LBA CURRENT_TEST_STATUS
1 0 0 Not_testing
2 0 0 Not_testing
3 0 0 Not_testing
4 0 0 Not_testing
5 0 0 Not_testing
Selective self-test flags (0x0):
After scanning selected spans, do NOT read-scan remainder of disk.
If Selective self-test is pending on power-up, resume after 0 minute delay.Disk3:
┌─[root@Archlinux]─[~]
└──> smartctl -a /dev/sdd
smartctl 6.4 2015-06-04 r4109 [x86_64-linux-4.5.1-1-ARCH] (local build)
Copyright (C) 2002-15, Bruce Allen, Christian Franke, www.smartmontools.org
=== START OF INFORMATION SECTION ===
Model Family: Western Digital Red
Device Model: WDC WD20EFRX-68EUZN0
Serial Number: WD-WCC4M0VVZVTH
LU WWN Device Id: 5 0014ee 262372fa6
Firmware Version: 82.00A82
User Capacity: 2,000,398,934,016 bytes [2,00 TB]
Sector Sizes: 512 bytes logical, 4096 bytes physical
Rotation Rate: 5400 rpm
Device is: In smartctl database [for details use: -P show]
ATA Version is: ACS-2 (minor revision not indicated)
SATA Version is: SATA 3.0, 6.0 Gb/s (current: 3.0 Gb/s)
Local Time is: Wed May 4 21:32:15 2016 CEST
SMART support is: Available - device has SMART capability.
SMART support is: Enabled
=== START OF READ SMART DATA SECTION ===
SMART overall-health self-assessment test result: PASSED
General SMART Values:
Offline data collection status: (0x82) Offline data collection activity
was completed without error.
Auto Offline Data Collection: Enabled.
Self-test execution status: ( 0) The previous self-test routine completed
without error or no self-test has ever
been run.
Total time to complete Offline
data collection: (25680) seconds.
Offline data collection
capabilities: (0x7b) SMART execute Offline immediate.
Auto Offline data collection on/off support.
Suspend Offline collection upon new
command.
Offline surface scan supported.
Self-test supported.
Conveyance Self-test supported.
Selective Self-test supported.
SMART capabilities: (0x0003) Saves SMART data before entering
power-saving mode.
Supports SMART auto save timer.
Error logging capability: (0x01) Error logging supported.
General Purpose Logging supported.
Short self-test routine
recommended polling time: ( 2) minutes.
Extended self-test routine
recommended polling time: ( 260) minutes.
Conveyance self-test routine
recommended polling time: ( 5) minutes.
SCT capabilities: (0x703d) SCT Status supported.
SCT Error Recovery Control supported.
SCT Feature Control supported.
SCT Data Table supported.
SMART Attributes Data Structure revision number: 16
Vendor Specific SMART Attributes with Thresholds:
ID# ATTRIBUTE_NAME FLAG VALUE WORST THRESH TYPE UPDATED WHEN_FAILED RAW_VALUE
1 Raw_Read_Error_Rate 0x002f 200 200 051 Pre-fail Always - 0
3 Spin_Up_Time 0x0027 100 253 021 Pre-fail Always - 0
4 Start_Stop_Count 0x0032 100 100 000 Old_age Always - 3
5 Reallocated_Sector_Ct 0x0033 200 200 140 Pre-fail Always - 0
7 Seek_Error_Rate 0x002e 200 200 000 Old_age Always - 0
9 Power_On_Hours 0x0032 099 099 000 Old_age Always - 861
10 Spin_Retry_Count 0x0032 100 253 000 Old_age Always - 0
11 Calibration_Retry_Count 0x0032 100 253 000 Old_age Always - 0
12 Power_Cycle_Count 0x0032 100 100 000 Old_age Always - 3
192 Power-Off_Retract_Count 0x0032 200 200 000 Old_age Always - 0
193 Load_Cycle_Count 0x0032 200 200 000 Old_age Always - 115
194 Temperature_Celsius 0x0022 108 102 000 Old_age Always - 39
196 Reallocated_Event_Count 0x0032 200 200 000 Old_age Always - 0
197 Current_Pending_Sector 0x0032 200 200 000 Old_age Always - 0
198 Offline_Uncorrectable 0x0030 200 200 000 Old_age Offline - 0
199 UDMA_CRC_Error_Count 0x0032 200 200 000 Old_age Always - 0
200 Multi_Zone_Error_Rate 0x0008 200 200 000 Old_age Offline - 0
SMART Error Log Version: 1
No Errors Logged
SMART Self-test log structure revision number 1
Num Test_Description Status Remaining LifeTime(hours) LBA_of_first_error
# 1 Extended offline Completed without error 00% 822 -
# 2 Short offline Completed without error 00% 673 -
# 3 Short offline Completed without error 00% 505 -
# 4 Short offline Completed without error 00% 339 -
# 5 Short offline Completed without error 00% 171 -
# 6 Extended offline Completed without error 00% 105 -
# 7 Short offline Completed without error 00% 14 -
SMART Selective self-test log data structure revision number 1
SPAN MIN_LBA MAX_LBA CURRENT_TEST_STATUS
1 0 0 Not_testing
2 0 0 Not_testing
3 0 0 Not_testing
4 0 0 Not_testing
5 0 0 Not_testing
Selective self-test flags (0x0):
After scanning selected spans, do NOT read-scan remainder of disk.
If Selective self-test is pending on power-up, resume after 0 minute delay.Disk4:
┌─[root@Archlinux]─[~]
└──> smartctl -a /dev/sde
smartctl 6.4 2015-06-04 r4109 [x86_64-linux-4.5.1-1-ARCH] (local build)
Copyright (C) 2002-15, Bruce Allen, Christian Franke, www.smartmontools.org
=== START OF INFORMATION SECTION ===
Model Family: Western Digital Red
Device Model: WDC WD20EFRX-68EUZN0
Serial Number: WD-WCC4M0418450
LU WWN Device Id: 5 0014ee 25e95481f
Firmware Version: 80.00A80
User Capacity: 2,000,398,934,016 bytes [2,00 TB]
Sector Sizes: 512 bytes logical, 4096 bytes physical
Rotation Rate: 5400 rpm
Device is: In smartctl database [for details use: -P show]
ATA Version is: ACS-2 (minor revision not indicated)
SATA Version is: SATA 3.0, 6.0 Gb/s (current: 3.0 Gb/s)
Local Time is: Wed May 4 21:33:29 2016 CEST
SMART support is: Available - device has SMART capability.
SMART support is: Enabled
=== START OF READ SMART DATA SECTION ===
SMART overall-health self-assessment test result: PASSED
General SMART Values:
Offline data collection status: (0x82) Offline data collection activity
was completed without error.
Auto Offline Data Collection: Enabled.
Self-test execution status: ( 0) The previous self-test routine completed
without error or no self-test has ever
been run.
Total time to complete Offline
data collection: (27240) seconds.
Offline data collection
capabilities: (0x7b) SMART execute Offline immediate.
Auto Offline data collection on/off support.
Suspend Offline collection upon new
command.
Offline surface scan supported.
Self-test supported.
Conveyance Self-test supported.
Selective Self-test supported.
SMART capabilities: (0x0003) Saves SMART data before entering
power-saving mode.
Supports SMART auto save timer.
Error logging capability: (0x01) Error logging supported.
General Purpose Logging supported.
Short self-test routine
recommended polling time: ( 2) minutes.
Extended self-test routine
recommended polling time: ( 275) minutes.
Conveyance self-test routine
recommended polling time: ( 5) minutes.
SCT capabilities: (0x703d) SCT Status supported.
SCT Error Recovery Control supported.
SCT Feature Control supported.
SCT Data Table supported.
SMART Attributes Data Structure revision number: 16
Vendor Specific SMART Attributes with Thresholds:
ID# ATTRIBUTE_NAME FLAG VALUE WORST THRESH TYPE UPDATED WHEN_FAILED RAW_VALUE
1 Raw_Read_Error_Rate 0x002f 200 200 051 Pre-fail Always - 1
3 Spin_Up_Time 0x0027 174 172 021 Pre-fail Always - 4275
4 Start_Stop_Count 0x0032 100 100 000 Old_age Always - 100
5 Reallocated_Sector_Ct 0x0033 200 200 140 Pre-fail Always - 0
7 Seek_Error_Rate 0x002e 200 200 000 Old_age Always - 0
9 Power_On_Hours 0x0032 074 074 000 Old_age Always - 19646
10 Spin_Retry_Count 0x0032 100 253 000 Old_age Always - 0
11 Calibration_Retry_Count 0x0032 100 253 000 Old_age Always - 0
12 Power_Cycle_Count 0x0032 100 100 000 Old_age Always - 91
192 Power-Off_Retract_Count 0x0032 200 200 000 Old_age Always - 15
193 Load_Cycle_Count 0x0032 001 001 000 Old_age Always - 888318
194 Temperature_Celsius 0x0022 109 099 000 Old_age Always - 38
196 Reallocated_Event_Count 0x0032 200 200 000 Old_age Always - 0
197 Current_Pending_Sector 0x0032 200 200 000 Old_age Always - 0
198 Offline_Uncorrectable 0x0030 200 200 000 Old_age Offline - 0
199 UDMA_CRC_Error_Count 0x0032 200 200 000 Old_age Always - 0
200 Multi_Zone_Error_Rate 0x0008 200 200 000 Old_age Offline - 0
SMART Error Log Version: 1
No Errors Logged
SMART Self-test log structure revision number 1
Num Test_Description Status Remaining LifeTime(hours) LBA_of_first_error
# 1 Short offline Completed without error 00% 19482 -
# 2 Short offline Completed without error 00% 19314 -
# 3 Short offline Completed without error 00% 19148 -
# 4 Short offline Completed without error 00% 18980 -
# 5 Extended offline Completed without error 00% 18914 -
# 6 Short offline Completed without error 00% 18813 -
# 7 Short offline Completed without error 00% 18744 -
# 8 Short offline Completed without error 00% 18575 -
# 9 Short offline Completed without error 00% 18407 -
#10 Short offline Completed without error 00% 18241 -
#11 Short offline Completed without error 00% 18105 -
#12 Short offline Completed without error 00% 17938 -
#13 Short offline Completed without error 00% 17770 -
#14 Short offline Completed without error 00% 17602 -
#15 Short offline Completed without error 00% 17434 -
#16 Short offline Completed without error 00% 17266 -
#17 Short offline Completed without error 00% 17098 -
#18 Short offline Completed without error 00% 16931 -
#19 Short offline Completed without error 00% 16763 -
#20 Short offline Completed without error 00% 16595 -
#21 Short offline Completed without error 00% 16427 -
SMART Selective self-test log data structure revision number 1
SPAN MIN_LBA MAX_LBA CURRENT_TEST_STATUS
1 0 0 Not_testing
2 0 0 Not_testing
3 0 0 Not_testing
4 0 0 Not_testing
5 0 0 Not_testing
Selective self-test flags (0x0):
After scanning selected spans, do NOT read-scan remainder of disk.
If Selective self-test is pending on power-up, resume after 0 minute delay.Disk5:
┌─[root@Archlinux]─[~]
└──> smartctl -a /dev/sdf
smartctl 6.4 2015-06-04 r4109 [x86_64-linux-4.5.1-1-ARCH] (local build)
Copyright (C) 2002-15, Bruce Allen, Christian Franke, www.smartmontools.org
=== START OF INFORMATION SECTION ===
Model Family: Western Digital Red
Device Model: WDC WD20EFRX-68EUZN0
Serial Number: WD-WCC4M0PT9ZFF
LU WWN Device Id: 5 0014ee 20bc2286c
Firmware Version: 82.00A82
User Capacity: 2,000,398,934,016 bytes [2,00 TB]
Sector Sizes: 512 bytes logical, 4096 bytes physical
Rotation Rate: 5400 rpm
Device is: In smartctl database [for details use: -P show]
ATA Version is: ACS-2 (minor revision not indicated)
SATA Version is: SATA 3.0, 6.0 Gb/s (current: 6.0 Gb/s)
Local Time is: Wed May 4 21:33:58 2016 CEST
SMART support is: Available - device has SMART capability.
SMART support is: Enabled
=== START OF READ SMART DATA SECTION ===
SMART overall-health self-assessment test result: PASSED
General SMART Values:
Offline data collection status: (0x00) Offline data collection activity
was never started.
Auto Offline Data Collection: Disabled.
Self-test execution status: ( 0) The previous self-test routine completed
without error or no self-test has ever
been run.
Total time to complete Offline
data collection: (28800) seconds.
Offline data collection
capabilities: (0x7b) SMART execute Offline immediate.
Auto Offline data collection on/off support.
Suspend Offline collection upon new
command.
Offline surface scan supported.
Self-test supported.
Conveyance Self-test supported.
Selective Self-test supported.
SMART capabilities: (0x0003) Saves SMART data before entering
power-saving mode.
Supports SMART auto save timer.
Error logging capability: (0x01) Error logging supported.
General Purpose Logging supported.
Short self-test routine
recommended polling time: ( 2) minutes.
Extended self-test routine
recommended polling time: ( 290) minutes.
Conveyance self-test routine
recommended polling time: ( 5) minutes.
SCT capabilities: (0x703d) SCT Status supported.
SCT Error Recovery Control supported.
SCT Feature Control supported.
SCT Data Table supported.
SMART Attributes Data Structure revision number: 16
Vendor Specific SMART Attributes with Thresholds:
ID# ATTRIBUTE_NAME FLAG VALUE WORST THRESH TYPE UPDATED WHEN_FAILED RAW_VALUE
1 Raw_Read_Error_Rate 0x002f 100 253 051 Pre-fail Always - 0
3 Spin_Up_Time 0x0027 180 180 021 Pre-fail Always - 3958
4 Start_Stop_Count 0x0032 100 100 000 Old_age Always - 6
5 Reallocated_Sector_Ct 0x0033 200 200 140 Pre-fail Always - 0
7 Seek_Error_Rate 0x002e 200 200 000 Old_age Always - 0
9 Power_On_Hours 0x0032 100 100 000 Old_age Always - 99
10 Spin_Retry_Count 0x0032 100 253 000 Old_age Always - 0
11 Calibration_Retry_Count 0x0032 100 253 000 Old_age Always - 0
12 Power_Cycle_Count 0x0032 100 100 000 Old_age Always - 3
192 Power-Off_Retract_Count 0x0032 200 200 000 Old_age Always - 0
193 Load_Cycle_Count 0x0032 200 200 000 Old_age Always - 82
194 Temperature_Celsius 0x0022 118 112 000 Old_age Always - 29
196 Reallocated_Event_Count 0x0032 200 200 000 Old_age Always - 0
197 Current_Pending_Sector 0x0032 200 200 000 Old_age Always - 0
198 Offline_Uncorrectable 0x0030 100 253 000 Old_age Offline - 0
199 UDMA_CRC_Error_Count 0x0032 200 200 000 Old_age Always - 0
200 Multi_Zone_Error_Rate 0x0008 100 253 000 Old_age Offline - 0
SMART Error Log Version: 1
No Errors Logged
SMART Self-test log structure revision number 1
No self-tests have been logged. [To run self-tests, use: smartctl -t]
SMART Selective self-test log data structure revision number 1
SPAN MIN_LBA MAX_LBA CURRENT_TEST_STATUS
1 0 0 Not_testing
2 0 0 Not_testing
3 0 0 Not_testing
4 0 0 Not_testing
5 0 0 Not_testing
Selective self-test flags (0x0):
After scanning selected spans, do NOT read-scan remainder of disk.
If Selective self-test is pending on power-up, resume after 0 minute delay.┌─[root@Archlinux]─[~]
└──> mdadm --examine /dev/sdb
/dev/sdb:
MBR Magic : aa55
Partition[0] : 3907029167 sectors at 1 (type ee)┌─[root@Archlinux]─[~]
└──> mdadm --examine /dev/sdc
/dev/sdc:
MBR Magic : aa55
Partition[0] : 3907029167 sectors at 1 (type ee)┌─[root@Archlinux]─[~]
└──> mdadm --examine /dev/sdd
/dev/sdd:
MBR Magic : aa55
Partition[0] : 3907029167 sectors at 1 (type ee)┌─[root@Archlinux]─[~]
└──> mdadm --examine /dev/sde
/dev/sde:
MBR Magic : aa55
Partition[0] : 3907029167 sectors at 1 (type ee)┌─[root@Archlinux]─[~]
└──> mdadm --examine /dev/sdf
/dev/sdf:
MBR Magic : aa55
Partition[0] : 3907029167 sectors at 1 (type ee)And finally:
┌─[root@Archlinux]─[~]
└──> mdadm --detail /dev/md0
/dev/md0:
Version : 1.2
Creation Time : Sat Jan 25 00:09:35 2014
Raid Level : raid5
Array Size : 5859840000 (5588.38 GiB 6000.48 GB)
Used Dev Size : 1953280000 (1862.79 GiB 2000.16 GB)
Raid Devices : 5
Total Devices : 5
Persistence : Superblock is persistent
Intent Bitmap : Internal
Update Time : Tue May 3 22:14:13 2016
State : clean, reshaping
Active Devices : 5
Working Devices : 5
Failed Devices : 0
Spare Devices : 0
Layout : left-symmetric
Chunk Size : 256K
Reshape Status : 67% complete
Delta Devices : 1, (4->5)
Name : archlinuxNAS:0
UUID : 82e58d60:774f61d7:7e816c49:de093881
Events : 416429
Number Major Minor RaidDevice State
0 8 17 0 active sync /dev/sdb1
1 8 33 1 active sync /dev/sdc1
5 8 49 2 active sync /dev/sdd1
4 8 65 3 active sync /dev/sde1
6 8 81 4 active sync /dev/sdf1Every time I reboot (i did it only twice) the reshape restart automatically and everything hangs up and I cannot access the disk. Is there a way to prevent the reshaping to start automatically at boot? Maybe in that way I can access the RAID and backup that 20% that it is not backed up already (I know I am an idiot, please don't tell me that :-( )
Thanks a lot for your help!
Last edited by Guidobelix (2016-05-04 19:52:20)
Offline
#6 2016-05-04 20:47:22
- frostschutz
- Member
- Registered: 2013-11-15
- Posts: 1,475
Re: Raid5 stuck in reshape operation
mdadm has a --freeze-reshape option to prevent reshape resume at boot/assembly time. (and a --continue option to resume it at a later time)
I don't know whether arch mkinitcpio is making use of those options somehow.
You forgot the * for the examine (should be on the actual raid members 
i.e. sd[bcdef]1)
Your disks look fine (except for the one with raw read error 1, but as long as that value doesn't change and passes selftest, it might be a one time thing) and you seem to be running selftests regularly.
Maybe in that way I can access the RAID and backup that 20% that it is not backed up already (I know I am an idiot, please don't tell me that :-( )
There should be nothing wrong with your data (as long as no more than one disk failed). I'm not sure why yours got stuck, there seem to be other such cases recently so maybe it is actually a kernel problem. Is there anything in dmesg?
Even if it can't be kicked back into operation, with a grow you have the situation that the first 67% of your capacity is a 5 disk raid5, and the remaining 33% is a 4 disk raid5, and there is a region in between that has duplicate data (in 4 disk raid5 format that already got relocated to lower sectors of 5 disk raid5). So it should definitely be possible to recover even if you had to do it from scratch using overlays https://raid.wiki.kernel.org/index.php/ … erlay_file by building two raid5 and finding a point in the duplicate data region somewhere below the 67% mark...
If you're willing to share your RAID metadata (first 64K or so of each disk) I can play around with it, but the last time someone gave me their "stuck" metadata it simply resumed reshaping for me... so not sure why this problem happens for some people but not others.
If all else fails you could also take it to the linux-raid mailing list and see if the people there have any other ideas. The Gentoo thread linked earlier has also a thread on the mailing list, subject "mdadm - stuck reshape operation" but the only other suggestion I see there is to compile mdadm from git://neil.brown.name/mdadm (which I'm not sure will help since gentoo/archlinux tend to use recent versions already)
Offline
#7 2016-05-05 07:40:29
- Guidobelix
- Member
- From: Italy
- Registered: 2013-06-21
- Posts: 31
Re: Raid5 stuck in reshape operation
Thank you very much once again for your help :-) What a great community this is!
I thought of using overlays, but I am afraid that's a bit other my current skills.
Before I explore that way, I have a question: As long as no disks fail, a RAID5 should be accessible during reshape, shouldn't it?
In such a case, what about the following plan:
Start with a Live CD (so the RAID will not be assembled, I think).
Assembly manually the array with the option --freeze-reshape.
Try to backup the 30-40% of the data which don't have a backup yet.
Erase everything and start from the beginning with a 5-disk array.
Is there a risk of damaging data by trying to assemble the array from a Live CD?
Last question: how do I extract the RAID metadata you mentioned?
Thank you once more :-)
Guido
Offline
#8 2016-05-14 15:50:53
- Guidobelix
- Member
- From: Italy
- Registered: 2013-06-21
- Posts: 31
Re: Raid5 stuck in reshape operation
I have tried several times to start the machine from a live CD and then start the array with the --freze-reshape option, but I had no success. the array is assembled but the reshape operation starts anyway and everything get stuck.
I really need some guidance :-(
Offline
#9 2016-05-14 16:18:24
- frostschutz
- Member
- Registered: 2013-11-15
- Posts: 1,475
Re: Raid5 stuck in reshape operation
Last question: how do I extract the RAID metadata you mentioned?
With dd, something like
dd bs=64K count=1 if=/dev/thing of=thing.header
(and that for each member of your raid)
mdadm itself also has a --dump=directory/ option, never tried that one before.
Offline
#10 2016-06-05 12:44:16
- Guidobelix
- Member
- From: Italy
- Registered: 2013-06-21
- Posts: 31
Re: Raid5 stuck in reshape operation
@frostschutz
Apologies for the long silence, I have been busy with "real" work and had no access to my machine.
Anyway, I have just sent you an email with the the headers I have extracted, following your instructions.
Anyway, for me it would be good enough to be able to stop reshaping (or, better, to prevent reshape from starting when I reboot the machine), so that I can update my backup.
Once that is done, I can simply kill the raid and restart from zero.
But with the reshaping running and stuck, I cannot access the files on the disk (any attempt to mount or access the file-system hangs as well).
Thanks everybody for help/hints/suggestions
Last edited by Guidobelix (2016-06-05 12:47:07)
Offline
#11 2016-06-06 10:11:39
- frostschutz
- Member
- Registered: 2013-11-15
- Posts: 1,475
Re: Raid5 stuck in reshape operation
What you sent me was just a partition table...
Offline
#12 2016-06-06 10:22:36
- Guidobelix
- Member
- From: Italy
- Registered: 2013-06-21
- Posts: 31
Re: Raid5 stuck in reshape operation
Apologies, I might have misunderstood your instructions.
Haven't you told me to run
dd bs=64K count=1 if=/dev/sdX of=sdX.header
for each member of the raid?
I will double check the commands I have run once I have access to the machine again.
Offline
#13 2016-06-06 10:26:38
- frostschutz
- Member
- Registered: 2013-11-15
- Posts: 1,475
Re: Raid5 stuck in reshape operation
Yes, for each member of the raid... member = partition in your case. Seems you ran it on the whole disk instead...
Offline
#14 2016-06-06 21:44:02
- Guidobelix
- Member
- From: Italy
- Registered: 2013-06-21
- Posts: 31
Re: Raid5 stuck in reshape operation
shame on me!
I've misread your instructions, sorry 
I'll redo that.
Offline
#15 2016-06-12 17:52:55
- Guidobelix
- Member
- From: Italy
- Registered: 2013-06-21
- Posts: 31
Re: Raid5 stuck in reshape operation
@frostschutz
I have tried again and sent you the result. I hpope this time I have understood your instructions in the right way.
Does anybody know what happen if I take one disk out and the start the system? Will I be able to start the raid in degradated mode (mdadm --manage /dev/md0 --run) and then access it? Or will I simpèly destroy the data on the raid?
Thanks a lot once more for your help.
Guido
Offline
#16 2016-06-12 22:07:57
- frostschutz
- Member
- Registered: 2013-11-15
- Posts: 1,475
Re: Raid5 stuck in reshape operation
It would still have to continue grow even with a disk missing. Very risky as you'd have no redundancy this way.
I've used your metadata to build your array out of loop devices... the strange thing is, the grow resumes normally for me. I tested this on a Gentoo box, kernel 4.6.1, mdadm 3.4. On my side it seems there is no problem.
The only oddity is that the md metadata reports bad blocks:
# mdadm --examine-badblocks sd*1.header
Bad-blocks list is empty in sdb1.header
Bad-blocks list is empty in sdc1.header
Bad-blocks on sdd1.header:
2641039808 for 320 sectors
Bad-blocks on sde1.header:
2641039808 for 320 sectors
Bad-blocks list is empty in sdf1.headerYou could try reading these, should be
dd bs=512 if=/dev/sdd1 skip=2641039808 count=320 | wc -c
dd bs=512 if=/dev/sde1 skip=2641039808 count=320 | wc -cIf there are no read errors, it seems md has recorded bad blocks as a fluke, maybe due to a controller/kernel problem... The bad block log in theory is a nice idea but my experience with it is not good. Once it has entries you can never get rid of them...
Offline
#17 2016-06-13 20:27:57
- Guidobelix
- Member
- From: Italy
- Registered: 2013-06-21
- Posts: 31
Re: Raid5 stuck in reshape operation
┌─[root@Archlinux]─[~]
└──> dd bs=512 if=/dev/sdd1 skip=2641039808 count=320 | wc -c
163840
320+0 record dentro
320+0 record fuori
163840 bytes (164 kB, 160 KiB) copied, 0,559506 s, 293 kB/s
┌─[root@Archlinux]─[~]
└──> dd bs=512 if=/dev/sde1 skip=2641039808 count=320 | wc -c
163840
320+0 record dentro
320+0 record fuori
163840 bytes (164 kB, 160 KiB) copied, 0,681416 s, 240 kB/s
┌─[root@Archlinux]─[~]
└──> It seems I can read both blocks without errors.
Do you have any suggestion for the next move?
Offline
Pages: 1