You are not logged in.
Last night rebooted the system and started seeing this:
https://ibb.co/K9t7LgH
I have 2 drives connected as RAID1. The setup was working without me touching it in any way for the last few years. I need some help diagnosing and fixing the issue without wiping / resetting the drives. I have some data there that I need.
Can someone help?
Here's what I've done so far.
The drives show up as /dev/sdb and /dev/sdc respectivelly.
# fdisk -l
Disk /dev/sda: 465.76 GiB, 500107862016 bytes, 976773168 sectors
Disk model: Samsung SSD 840
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disklabel type: gpt
Disk identifier: 99623ADA-7DA8-4339-92AD-0908E521692A
Device Start End Sectors Size Type
/dev/sda1 2048 1050623 1048576 512M EFI System
/dev/sda2 1050624 32507903 31457280 15G Linux filesystem
/dev/sda3 32507904 34605055 2097152 1G Linux swap
/dev/sda4 34605056 244320255 209715200 100G Linux filesystem
/dev/sda5 244320256 349177855 104857600 50G Linux root (x86-64)
/dev/sda6 349177856 976773134 627595279 299.3G Linux filesystem
Disk /dev/sdb: 2.73 TiB, 3000592982016 bytes, 5860533168 sectors
Disk model: ST3000DM001-1ER1
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 4096 bytes
I/O size (minimum/optimal): 4096 bytes / 4096 bytes
Disk /dev/sdc: 2.73 TiB, 3000592982016 bytes, 5860533168 sectors
Disk model: ST3000DM001-1ER1
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 4096 bytes
I/O size (minimum/optimal): 4096 bytes / 4096 bytes
Disk /dev/loop0: 100 GiB, 107374182400 bytes, 209715200 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk /dev/loop1: 2 GiB, 2147483648 bytes, 4194304 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes# mdadm --detail --scan
ARRAY /dev/md/Multimedia metadata=imsm UUID=3d3554d8:ed96b029:3d234f2a:fc2df477
ARRAY /dev/md/Multimedia metadata=imsm UUID=3d3554d8:ed96b029:3d234f2a:fc2df477I'm not sure why I have 2 arrays detected, under the same UUID though...
# mdadm --examine --scan
ARRAY metadata=imsm UUID=00000000:00000000:00000000:00000000
ARRAY metadata=imsm UUID=3d3554d8:ed96b029:3d234f2a:fc2df477
ARRAY /dev/md/Multimedia container=3d3554d8:ed96b029:3d234f2a:fc2df477 member=0 UUID=c7316a94:0762f78f:4a988985:e2fa45cdand this is even more confusing
# cat /proc/mdstat
Personalities :
md126 : inactive sdc[0](S)
2520 blocks super external:imsm
md127 : inactive sdb[0](S)
1105 blocks super external:imsm
unused devices: <none>trying to get as much info as possible and run a diagnostic on the drives:
# hdparm -I /dev/sdb
/dev/sdb:
ATA device, with non-removable media
Model Number: ST3000DM001-1ER166
Serial Number: Z500J25S
Firmware Revision: CC25
Transport: Serial, SATA 1.0a, SATA II Extensions, SATA Rev 2.5, SATA Rev 2.6, SATA Rev 3.0
Standards:
Used: unknown (minor revision code 0x001f)
Supported: 9 8 7 6 5
Likely used: 9
Configuration:
Logical max current
cylinders 16383 16383
heads 16 16
sectors/track 63 63
--
CHS current addressable sectors: 16514064
LBA user addressable sectors: 268435455
LBA48 user addressable sectors: 5860533168
Logical Sector size: 512 bytes
Physical Sector size: 4096 bytes
Logical Sector-0 offset: 0 bytes
device size with M = 1024*1024: 2861588 MBytes
device size with M = 1000*1000: 3000592 MBytes (3000 GB)
cache/buffer size = unknown
Form Factor: 3.5 inch
Nominal Media Rotation Rate: 7200
Capabilities:
LBA, IORDY(can be disabled)
Queue depth: 32
Standby timer values: spec'd by Standard, no device specific minimum
R/W multiple sector transfer: Max = 16 Current = 16
Advanced power management level: 128
Recommended acoustic management value: 208, current value: 208
DMA: mdma0 mdma1 mdma2 udma0 udma1 udma2 udma3 udma4 udma5 *udma6
Cycle time: min=120ns recommended=120ns
PIO: pio0 pio1 pio2 pio3 pio4
Cycle time: no flow control=120ns IORDY flow control=120ns
Commands/features:
Enabled Supported:
* SMART feature set
Security Mode feature set
* Power Management feature set
* Write cache
* Look-ahead
* Host Protected Area feature set
* WRITE_BUFFER command
* READ_BUFFER command
* DOWNLOAD_MICROCODE
* Advanced Power Management feature set
Power-Up In Standby feature set
* SET_FEATURES required to spinup after power up
SET_MAX security extension
* 48-bit Address feature set
* Device Configuration Overlay feature set
* Mandatory FLUSH_CACHE
* FLUSH_CACHE_EXT
* SMART error logging
* SMART self-test
* General Purpose Logging feature set
* WRITE_{DMA|MULTIPLE}_FUA_EXT
* 64-bit World wide name
Write-Read-Verify feature set
* WRITE_UNCORRECTABLE_EXT command
* {READ,WRITE}_DMA_EXT_GPL commands
* Segmented DOWNLOAD_MICROCODE
* unknown 119[6]
* Gen1 signaling speed (1.5Gb/s)
* Gen2 signaling speed (3.0Gb/s)
* Gen3 signaling speed (6.0Gb/s)
* Native Command Queueing (NCQ)
* Phy event counters
* READ_LOG_DMA_EXT equivalent to READ_LOG_EXT
* DMA Setup Auto-Activate optimization
Device-initiated interface power management
* Software settings preservation
unknown 78[7]
* SMART Command Transport (SCT) feature set
* SCT Write Same (AC2)
unknown 206[7]
unknown 206[12] (vendor specific)
* SANITIZE feature set
* OVERWRITE_EXT command
Security:
Master password revision code = 65534
supported
not enabled
not locked
frozen
not expired: security count
supported: enhanced erase
308min for SECURITY ERASE UNIT. 308min for ENHANCED SECURITY ERASE UNIT.
Logical Unit WWN Device Identifier: 5000c50079872169
NAA : 5
IEEE OUI : 000c50
Unique ID : 079872169
Checksum: correct# smartctl --info /dev/sdb
smartctl 7.3 2022-02-28 r5338 [x86_64-linux-5.17.5-arch1-1] (local build)
Copyright (C) 2002-22, Bruce Allen, Christian Franke, www.smartmontools.org
=== START OF INFORMATION SECTION ===
Model Family: Seagate Barracuda 7200.14 (AF)
Device Model: ST3000DM001-1ER166
Serial Number: Z500J25S
LU WWN Device Id: 5 000c50 079872169
Firmware Version: CC25
User Capacity: 3,000,592,982,016 bytes [3.00 TB]
Sector Sizes: 512 bytes logical, 4096 bytes physical
Rotation Rate: 7200 rpm
Form Factor: 3.5 inches
Device is: In smartctl database 7.3/5319
ATA Version is: ACS-2, ACS-3 T13/2161-D revision 3b
SATA Version is: SATA 3.1, 6.0 Gb/s (current: 6.0 Gb/s)
Local Time is: Tue May 3 11:01:21 2022 EDT
SMART support is: Available - device has SMART capability.
SMART support is: Enabled
# smartctl -t short /dev/sdb
# smartctl -t conveyance /dev/sdb and results:
# smartctl -a /dev/sdb
smartctl 7.3 2022-02-28 r5338 [x86_64-linux-5.17.5-arch1-1] (local build)
Copyright (C) 2002-22, Bruce Allen, Christian Franke, www.smartmontools.org
=== START OF INFORMATION SECTION ===
Model Family: Seagate Barracuda 7200.14 (AF)
Device Model: ST3000DM001-1ER166
Serial Number: Z500J25S
LU WWN Device Id: 5 000c50 079872169
Firmware Version: CC25
User Capacity: 3,000,592,982,016 bytes [3.00 TB]
Sector Sizes: 512 bytes logical, 4096 bytes physical
Rotation Rate: 7200 rpm
Form Factor: 3.5 inches
Device is: In smartctl database 7.3/5319
ATA Version is: ACS-2, ACS-3 T13/2161-D revision 3b
SATA Version is: SATA 3.1, 6.0 Gb/s (current: 6.0 Gb/s)
Local Time is: Tue May 3 11:55:33 2022 EDT
SMART support is: Available - device has SMART capability.
SMART support is: Enabled
=== START OF READ SMART DATA SECTION ===
SMART overall-health self-assessment test result: PASSED
General SMART Values:
Offline data collection status: (0x00) Offline data collection activity
was never started.
Auto Offline Data Collection: Disabled.
Self-test execution status: ( 0) The previous self-test routine completed
without error or no self-test has ever
been run.
Total time to complete Offline
data collection: ( 89) seconds.
Offline data collection
capabilities: (0x73) SMART execute Offline immediate.
Auto Offline data collection on/off support.
Suspend Offline collection upon new
command.
No Offline surface scan supported.
Self-test supported.
Conveyance Self-test supported.
Selective Self-test supported.
SMART capabilities: (0x0003) Saves SMART data before entering
power-saving mode.
Supports SMART auto save timer.
Error logging capability: (0x01) Error logging supported.
General Purpose Logging supported.
Short self-test routine
recommended polling time: ( 1) minutes.
Extended self-test routine
recommended polling time: ( 322) minutes.
Conveyance self-test routine
recommended polling time: ( 2) minutes.
SCT capabilities: (0x1085) SCT Status supported.
SMART Attributes Data Structure revision number: 10
Vendor Specific SMART Attributes with Thresholds:
ID# ATTRIBUTE_NAME FLAG VALUE WORST THRESH TYPE UPDATED WHEN_FAILED RAW_VALUE
1 Raw_Read_Error_Rate 0x000f 112 099 006 Pre-fail Always - 48842296
3 Spin_Up_Time 0x0003 094 094 000 Pre-fail Always - 0
4 Start_Stop_Count 0x0032 100 100 020 Old_age Always - 75
5 Reallocated_Sector_Ct 0x0033 100 100 010 Pre-fail Always - 0
7 Seek_Error_Rate 0x000f 060 060 030 Pre-fail Always - 1041427
9 Power_On_Hours 0x0032 035 035 000 Old_age Always - 57679
10 Spin_Retry_Count 0x0013 100 100 097 Pre-fail Always - 0
12 Power_Cycle_Count 0x0032 100 100 020 Old_age Always - 74
183 Runtime_Bad_Block 0x0032 072 072 000 Old_age Always - 28
184 End-to-End_Error 0x0032 100 100 099 Old_age Always - 0
187 Reported_Uncorrect 0x0032 100 100 000 Old_age Always - 0
188 Command_Timeout 0x0032 100 100 000 Old_age Always - 0 0 1
189 High_Fly_Writes 0x003a 092 092 000 Old_age Always - 8
190 Airflow_Temperature_Cel 0x0022 065 054 045 Old_age Always - 35 (Min/Max 21/35)
191 G-Sense_Error_Rate 0x0032 100 100 000 Old_age Always - 0
192 Power-Off_Retract_Count 0x0032 100 100 000 Old_age Always - 3
193 Load_Cycle_Count 0x0032 001 001 000 Old_age Always - 312371
194 Temperature_Celsius 0x0022 035 046 000 Old_age Always - 35 (0 15 0 0 0)
197 Current_Pending_Sector 0x0012 100 100 000 Old_age Always - 0
198 Offline_Uncorrectable 0x0010 100 100 000 Old_age Offline - 0
199 UDMA_CRC_Error_Count 0x003e 200 197 000 Old_age Always - 129
240 Head_Flying_Hours 0x0000 100 253 000 Old_age Offline - 3854h+15m+12.831s
241 Total_LBAs_Written 0x0000 100 253 000 Old_age Offline - 35254087362
242 Total_LBAs_Read 0x0000 100 253 000 Old_age Offline - 5039165697
SMART Error Log Version: 1
No Errors Logged
SMART Self-test log structure revision number 1
Num Test_Description Status Remaining LifeTime(hours) LBA_of_first_error
# 1 Conveyance offline Completed without error 00% 57678 -
# 2 Short offline Completed without error 00% 57678 -
SMART Selective self-test log data structure revision number 1
SPAN MIN_LBA MAX_LBA CURRENT_TEST_STATUS
1 0 0 Not_testing
2 0 0 Not_testing
3 0 0 Not_testing
4 0 0 Not_testing
5 0 0 Not_testing
Selective self-test flags (0x0):
After scanning selected spans, do NOT read-scan remainder of disk.
If Selective self-test is pending on power-up, resume after 0 minute delay.no error on /dev/sdb, but there are a few on /dev/sdc:
# smartctl -a /dev/sdc
smartctl 7.3 2022-02-28 r5338 [x86_64-linux-5.17.5-arch1-1] (local build)
Copyright (C) 2002-22, Bruce Allen, Christian Franke, www.smartmontools.org
=== START OF INFORMATION SECTION ===
Model Family: Seagate Barracuda 7200.14 (AF)
Device Model: ST3000DM001-1ER166
Serial Number: Z500K1WW
LU WWN Device Id: 5 000c50 0798760be
Firmware Version: CC25
User Capacity: 3,000,592,982,016 bytes [3.00 TB]
Sector Sizes: 512 bytes logical, 4096 bytes physical
Rotation Rate: 7200 rpm
Form Factor: 3.5 inches
Device is: In smartctl database 7.3/5319
ATA Version is: ACS-2, ACS-3 T13/2161-D revision 3b
SATA Version is: SATA 3.1, 6.0 Gb/s (current: 6.0 Gb/s)
Local Time is: Tue May 3 11:56:19 2022 EDT
SMART support is: Available - device has SMART capability.
SMART support is: Enabled
=== START OF READ SMART DATA SECTION ===
SMART overall-health self-assessment test result: PASSED
General SMART Values:
Offline data collection status: (0x00) Offline data collection activity
was never started.
Auto Offline Data Collection: Disabled.
Self-test execution status: ( 0) The previous self-test routine completed
without error or no self-test has ever
been run.
Total time to complete Offline
data collection: ( 80) seconds.
Offline data collection
capabilities: (0x73) SMART execute Offline immediate.
Auto Offline data collection on/off support.
Suspend Offline collection upon new
command.
No Offline surface scan supported.
Self-test supported.
Conveyance Self-test supported.
Selective Self-test supported.
SMART capabilities: (0x0003) Saves SMART data before entering
power-saving mode.
Supports SMART auto save timer.
Error logging capability: (0x01) Error logging supported.
General Purpose Logging supported.
Short self-test routine
recommended polling time: ( 1) minutes.
Extended self-test routine
recommended polling time: ( 330) minutes.
Conveyance self-test routine
recommended polling time: ( 2) minutes.
SCT capabilities: (0x1085) SCT Status supported.
SMART Attributes Data Structure revision number: 10
Vendor Specific SMART Attributes with Thresholds:
ID# ATTRIBUTE_NAME FLAG VALUE WORST THRESH TYPE UPDATED WHEN_FAILED RAW_VALUE
1 Raw_Read_Error_Rate 0x000f 117 099 006 Pre-fail Always - 144462424
3 Spin_Up_Time 0x0003 094 094 000 Pre-fail Always - 0
4 Start_Stop_Count 0x0032 100 100 020 Old_age Always - 74
5 Reallocated_Sector_Ct 0x0033 100 100 010 Pre-fail Always - 0
7 Seek_Error_Rate 0x000f 062 060 030 Pre-fail Always - 1604581
9 Power_On_Hours 0x0032 035 035 000 Old_age Always - 57795
10 Spin_Retry_Count 0x0013 100 100 097 Pre-fail Always - 0
12 Power_Cycle_Count 0x0032 100 100 020 Old_age Always - 75
183 Runtime_Bad_Block 0x0032 099 099 000 Old_age Always - 1
184 End-to-End_Error 0x0032 100 100 099 Old_age Always - 0
187 Reported_Uncorrect 0x0032 098 098 000 Old_age Always - 2
188 Command_Timeout 0x0032 100 100 000 Old_age Always - 0 0 0
189 High_Fly_Writes 0x003a 097 097 000 Old_age Always - 3
190 Airflow_Temperature_Cel 0x0022 065 054 045 Old_age Always - 35 (Min/Max 21/35)
191 G-Sense_Error_Rate 0x0032 100 100 000 Old_age Always - 0
192 Power-Off_Retract_Count 0x0032 100 100 000 Old_age Always - 3
193 Load_Cycle_Count 0x0032 001 001 000 Old_age Always - 309033
194 Temperature_Celsius 0x0022 035 046 000 Old_age Always - 35 (0 15 0 0 0)
197 Current_Pending_Sector 0x0012 092 092 000 Old_age Always - 1472
198 Offline_Uncorrectable 0x0010 092 092 000 Old_age Offline - 1472
199 UDMA_CRC_Error_Count 0x003e 200 200 000 Old_age Always - 4
240 Head_Flying_Hours 0x0000 100 253 000 Old_age Offline - 4737h+30m+05.426s
241 Total_LBAs_Written 0x0000 100 253 000 Old_age Offline - 11034445113
242 Total_LBAs_Read 0x0000 100 253 000 Old_age Offline - 35044522939
SMART Error Log Version: 1
ATA Error Count: 2
CR = Command Register [HEX]
FR = Features Register [HEX]
SC = Sector Count Register [HEX]
SN = Sector Number Register [HEX]
CL = Cylinder Low Register [HEX]
CH = Cylinder High Register [HEX]
DH = Device/Head Register [HEX]
DC = Device Command Register [HEX]
ER = Error register [HEX]
ST = Status register [HEX]
Powered_Up_Time is measured from power on, and printed as
DDd+hh:mm:SS.sss where DD=days, hh=hours, mm=minutes,
SS=sec, and sss=millisec. It "wraps" after 49.710 days.
Error 2 occurred at disk power-on lifetime: 46209 hours (1925 days + 9 hours)
When the command that caused the error occurred, the device was active or idle.
After command completion occurred, registers were:
ER ST SC SN CL CH DH
-- -- -- -- -- -- --
40 53 00 ff ff ff 0f Error: UNC at LBA = 0x0fffffff = 268435455
Commands leading to the command that caused the error were:
CR FR SC SN CL CH DH DC Powered_Up_Time Command/Feature_Name
-- -- -- -- -- -- -- -- ---------------- --------------------
60 00 08 ff ff ff 4f 00 11d+21:59:43.290 READ FPDMA QUEUED
60 00 08 ff ff ff 4f 00 11d+21:59:43.290 READ FPDMA QUEUED
60 00 08 ff ff ff 4f 00 11d+21:59:43.289 READ FPDMA QUEUED
60 00 08 ff ff ff 4f 00 11d+21:59:43.282 READ FPDMA QUEUED
60 00 00 ff ff ff 4f 00 11d+21:59:43.274 READ FPDMA QUEUED
Error 1 occurred at disk power-on lifetime: 46209 hours (1925 days + 9 hours)
When the command that caused the error occurred, the device was active or idle.
After command completion occurred, registers were:
ER ST SC SN CL CH DH
-- -- -- -- -- -- --
40 53 00 ff ff ff 0f Error: UNC at LBA = 0x0fffffff = 268435455
Commands leading to the command that caused the error were:
CR FR SC SN CL CH DH DC Powered_Up_Time Command/Feature_Name
-- -- -- -- -- -- -- -- ---------------- --------------------
60 00 00 ff ff ff 4f 00 11d+21:59:39.406 READ FPDMA QUEUED
60 00 00 ff ff ff 4f 00 11d+21:59:39.406 READ FPDMA QUEUED
60 00 00 ff ff ff 4f 00 11d+21:59:36.149 READ FPDMA QUEUED
60 00 a0 ff ff ff 4f 00 11d+21:59:36.147 READ FPDMA QUEUED
60 00 00 ff ff ff 4f 00 11d+21:59:35.548 READ FPDMA QUEUED
SMART Self-test log structure revision number 1
Num Test_Description Status Remaining LifeTime(hours) LBA_of_first_error
# 1 Conveyance offline Completed without error 00% 57794 -
# 2 Short offline Completed without error 00% 57794 -
SMART Selective self-test log data structure revision number 1
SPAN MIN_LBA MAX_LBA CURRENT_TEST_STATUS
1 0 0 Not_testing
2 0 0 Not_testing
3 0 0 Not_testing
4 0 0 Not_testing
5 0 0 Not_testing
Selective self-test flags (0x0):
After scanning selected spans, do NOT read-scan remainder of disk.
If Selective self-test is pending on power-up, resume after 0 minute delay.Can anyone help? Thank you!
Last edited by megabeaver (2022-05-06 01:52:03)
Offline
actually, I disconnected one of the drives and the other ones shows as active! Thank God!
I will back everything up and then remove and re-add to the array. I'll update here how it went.
Thank you
Offline
looks like removing the drive and re-adding does the trick as well. I'll mark this as solved!
Offline