You are not logged in.
- Topics: Active | Unanswered
#1 2019-03-11 19:12:00
- Senshi
- Member
- Registered: 2019-01-15
- Posts: 17
[SOLVED]LVM / RAID fails to load - HDD damaged (superblock?)
I set up a home/mediaserver using 3 1 TB HDDs of different age, using the LVM+RAID tutorial on the Archwiki. The only exception: I have the boot partition completely independent from that on a small SSD, together with the swapfile.
Now, I had some trouble accessing some files on it (not all of them).
After a reboot, the system would no longer boot or be accessible by ssh. Upon hooking it back up to a GPU and display so I can see what's happening, I see it fails to load/mount my VolGroupArray and lands in emergency mode.
As I don't have ssh access anymore, so I can only copy stuff by typing it off that screen...
Timed out waiting for device /dev/VolGroupArray/lvmedia
Dependency failed for File System Check on /dev/VolGrouPArray/lvmedia
Dependency failed for /mnt/media
Dependency failed for Local File Systems
fdisk -l shows me some concerning info regarding one hard drive /dev/sdc
It still shows up, but some things are off:
The disk model is correctly shown.
A big red warning of "GPT PMBR size mismatch (1953525167 != 1953516910) will be corrected by write."
Disklabel type is dos instead of gpt as the other LVM/RAID drives.
Disk identifier is 0x00000000 instead of a meaningful ID.
The single partition on it (/dev/sdc1) shows starting at sector 1 (unlike the others which start at 2048), has an ID of 'ee' and its Type is 'GPT' instead of 'Linux RAID'.
I have no idea how I go about fixing this. It seems there's some significant damage to that HDD if it's not even recognized as the correct type? Is there a way to recover the data and simply kick out that particular drive until I get a replacement? I don't want to just fiddle without a clue, for fear of making irreversible mistakes.
Last edited by Senshi (2019-03-22 12:50:52)
Offline
#2 2019-03-11 20:16:54
- loqs
- Member
- Registered: 2014-03-06
- Posts: 18,975
Re: [SOLVED]LVM / RAID fails to load - HDD damaged (superblock?)
What RAID level is the array using? If it is 1 or 5 then the data should still be available on the remaining devices.
Offline
#3 2019-03-11 22:04:53
- Senshi
- Member
- Registered: 2019-01-15
- Posts: 17
Re: [SOLVED]LVM / RAID fails to load - HDD damaged (superblock?)
Heh, missed the most obvious info, sorry for that.
It's RAID 5. The data should be around, yes, but I'm not sure how to recover it. I don't care about the broken HDD, it's gonna get tossed either way.
Offline
#4 2019-03-12 21:09:45
- Swiggles
- Member
- Registered: 2014-08-02
- Posts: 266
Re: [SOLVED]LVM / RAID fails to load - HDD damaged (superblock?)
I think I know this one. Please try the workaround from this bug report: https://bugs.archlinux.org/task/57275#comment167331
Offline
#5 2019-03-16 07:26:31
- Senshi
- Member
- Registered: 2019-01-15
- Posts: 17
Re: [SOLVED]LVM / RAID fails to load - HDD damaged (superblock?)
That did not change anything, sadly. I'm not sure if it even relates to my issue, as my boot is not on an LVM? It's just my HDD media cluster that forms the broken LVM.
Offline
#6 2019-03-16 08:33:11
- frostschutz
- Member
- Registered: 2013-11-15
- Posts: 1,649
Re: [SOLVED]LVM / RAID fails to load - HDD damaged (superblock?)
need more info (boot a live cd)
lsblk
parted -l
cat /proc/mdstat
mdadm --examine /dev/sd*
pvs
vgs
lvsand perhaps smartctl -a for each disk
Last edited by frostschutz (2019-03-16 08:34:43)
Offline
#7 2019-03-16 13:21:02
- Senshi
- Member
- Registered: 2019-01-15
- Posts: 17
Re: [SOLVED]LVM / RAID fails to load - HDD damaged (superblock?)
By now I managed to restore the partition on /dev/sdc using testdisk (quick scan discovered it), but am stuck on getting the RAID back up and running at all.
Got around maintenance mode by mounting the LVM VolGroupArray with the 'nofail' option in fstab, so I can give those sweet logs.
lsblk
NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT
sda 8:0 0 232.9G 0 disk
├─sda1 8:1 0 1M 0 part
└─sda2 8:2 0 232.9G 0 part /
sdb 8:16 0 1.4T 0 disk
└─sdb1 8:17 0 931.5G 0 part
sdc 8:32 0 931.5G 0 disk
└─sdc1 8:33 0 931.5G 0 part
sdd 8:48 0 931.5G 0 disk
└─sdd1 8:49 0 931.5G 0 partparted -l
Model: ATA Samsung SSD 840 (scsi)
Disk /dev/sda: 250GB
Sector size (logical/physical): 512B/512B
Partition Table: gpt
Disk Flags:
Number Start End Size File system Name Flags
1 1049kB 2097kB 1049kB bios_grub
2 2097kB 250GB 250GB ext4
Model: ATA ST1500DL003-9VT1 (scsi)
Disk /dev/sdb: 1500GB
Sector size (logical/physical): 512B/512B
Partition Table: gpt
Disk Flags:
Number Start End Size File system Name Flags
1 1049kB 1000GB 1000GB raid
Model: ATA WDC WD10EALS-00Z (scsi)
Disk /dev/sdc: 1000GB
Sector size (logical/physical): 512B/512B
Partition Table: gpt
Disk Flags:
Number Start End Size File system Name Flags
1 1049kB 1000GB 1000GB raid
Model: ATA SAMSUNG HD103UJ (scsi)
Disk /dev/sdd: 1000GB
Sector size (logical/physical): 512B/512B
Partition Table: gpt
Disk Flags:
Number Start End Size File system Name Flags
1 1049kB 1000GB 1000GB raidmdadm --examine /dev/sd*
/dev/sda:
MBR Magic : aa55
Partition[0] : 488388910 sectors at 1 (type ee)
mdadm: No md superblock detected on /dev/sda1.
mdadm: No md superblock detected on /dev/sda2.
/dev/sdb:
MBR Magic : aa55
Partition[0] : 2930268910 sectors at 1 (type ee)
/dev/sdb1:
Magic : a92b4efc
Version : 1.2
Feature Map : 0x9
Array UUID : 95708db9:5b2e4a9f:c5063a32:714c2003
Name : Baikonur:0 (local to host Baikonur)
Creation Time : Tue Jan 15 21:33:32 2019
Raid Level : raid5
Raid Devices : 3
Avail Dev Size : 1953258830 (931.39 GiB 1000.07 GB)
Array Size : 1953258496 (1862.77 GiB 2000.14 GB)
Used Dev Size : 1953258496 (931.39 GiB 1000.07 GB)
Data Offset : 262144 sectors
Super Offset : 8 sectors
Unused Space : before=262064 sectors, after=334 sectors
State : clean
Device UUID : 4f6eed13:b1d6ca56:effad9df:082bf380
Internal Bitmap : 8 sectors from superblock
Update Time : Mon Mar 11 11:51:10 2019
Bad Block Log : 512 entries available at offset 16 sectors - bad blocks present.
Checksum : 8269b6b6 - correct
Events : 52076
Layout : left-symmetric
Chunk Size : 512K
Device Role : Active device 0
Array State : A.A ('A' == active, '.' == missing, 'R' == replacing)
/dev/sdc:
MBR Magic : aa55
Partition[0] : 1953516910 sectors at 1 (type ee)
/dev/sdc1:
Magic : a92b4efc
Version : 1.2
Feature Map : 0x1
Array UUID : 95708db9:5b2e4a9f:c5063a32:714c2003
Name : Baikonur:0 (local to host Baikonur)
Creation Time : Tue Jan 15 21:33:32 2019
Raid Level : raid5
Raid Devices : 3
Avail Dev Size : 1953260943 (931.39 GiB 1000.07 GB)
Array Size : 1953258496 (1862.77 GiB 2000.14 GB)
Used Dev Size : 1953258496 (931.39 GiB 1000.07 GB)
Data Offset : 262144 sectors
Super Offset : 8 sectors
Unused Space : before=262064 sectors, after=18446744073709544950 sectors
State : clean
Device UUID : 84a34f99:1b3b71b8:3b134c7c:07557ebb
Internal Bitmap : 8 sectors from superblock
Update Time : Tue Jan 15 22:52:55 2019
Bad Block Log : 512 entries available at offset 16 sectors
Checksum : 4028e809 - correct
Events : 2
Layout : left-symmetric
Chunk Size : 512K
Device Role : Active device 1
Array State : AAA ('A' == active, '.' == missing, 'R' == replacing)
/dev/sdd:
MBR Magic : aa55
Partition[0] : 1953523054 sectors at 1 (type ee)
/dev/sdd1:
Magic : a92b4efc
Version : 1.2
Feature Map : 0x1
Array UUID : 95708db9:5b2e4a9f:c5063a32:714c2003
Name : Baikonur:0 (local to host Baikonur)
Creation Time : Tue Jan 15 21:33:32 2019
Raid Level : raid5
Raid Devices : 3
Avail Dev Size : 1953258830 (931.39 GiB 1000.07 GB)
Array Size : 1953258496 (1862.77 GiB 2000.14 GB)
Used Dev Size : 1953258496 (931.39 GiB 1000.07 GB)
Data Offset : 262144 sectors
Super Offset : 8 sectors
Unused Space : before=262064 sectors, after=334 sectors
State : clean
Device UUID : be43780b:5e1cd732:f7233486:f2fd6177
Internal Bitmap : 8 sectors from superblock
Update Time : Mon Mar 11 17:48:44 2019
Bad Block Log : 512 entries available at offset 16 sectors
Checksum : f2ca218b - correct
Events : 52076
Layout : left-symmetric
Chunk Size : 512K
Device Role : Active device 2
Array State : A.A ('A' == active, '.' == missing, 'R' == replacing)cat /proc/mdstat
Personalities : [raid6] [raid5] [raid4]
md0 : active raid5 sdd1[2] sdb1[0]
1953258496 blocks super 1.2 level 5, 512k chunk, algorithm 2 [3/2] [U_U]
bitmap: 8/8 pages [32KB], 65536KB chunk
unused devices: <none>After trying to run RAID manually using
mdadm --run /dev/md0
the result is
cat /proc/mdstat
Personalities : [raid6] [raid5] [raid4]
md0 : active raid5 sdd1[2] sdb1[0]
1953258496 blocks super 1.2 level 5, 512k chunk, algorithm 2 [3/2] [U_U]
bitmap: 8/8 pages [32KB], 65536KB chunk
unused devices: <none>pvs, vgs and lvs all return nothing
smartctl -a /dev/sda
smartctl 7.0 2018-12-30 r4883 [x86_64-linux-5.0.2-arch1-1-ARCH] (local build)
Copyright (C) 2002-18, Bruce Allen, Christian Franke, www.smartmontools.org
=== START OF INFORMATION SECTION ===
Model Family: Samsung based SSDs
Device Model: Samsung SSD 840 Series
Serial Number: S14GNEAD224687Z
LU WWN Device Id: 5 002538 5501f7dc5
Firmware Version: DXT0AB0Q
User Capacity: 250,055,122,432 bytes [250 GB]
Sector Size: 512 bytes logical/physical
Rotation Rate: Solid State Device
Device is: In smartctl database [for details use: -P show]
ATA Version is: ACS-2, ATA8-ACS T13/1699-D revision 4c
SATA Version is: SATA 3.1, 6.0 Gb/s (current: 3.0 Gb/s)
Local Time is: Sat Mar 16 14:16:45 2019 CET
SMART support is: Available - device has SMART capability.
SMART support is: Enabled
=== START OF READ SMART DATA SECTION ===
SMART overall-health self-assessment test result: PASSED
General SMART Values:
Offline data collection status: (0x00) Offline data collection activity
was never started.
Auto Offline Data Collection: Disabled.
Self-test execution status: ( 0) The previous self-test routine completed
without error or no self-test has ever
been run.
Total time to complete Offline
data collection: (53956) seconds.
Offline data collection
capabilities: (0x53) SMART execute Offline immediate.
Auto Offline data collection on/off support.
Suspend Offline collection upon new
command.
No Offline surface scan supported.
Self-test supported.
No Conveyance Self-test supported.
Selective Self-test supported.
SMART capabilities: (0x0003) Saves SMART data before entering
power-saving mode.
Supports SMART auto save timer.
Error logging capability: (0x01) Error logging supported.
General Purpose Logging supported.
Short self-test routine
recommended polling time: ( 2) minutes.
Extended self-test routine
recommended polling time: ( 40) minutes.
SCT capabilities: (0x003d) SCT Status supported.
SCT Error Recovery Control supported.
SCT Feature Control supported.
SCT Data Table supported.
SMART Attributes Data Structure revision number: 1
Vendor Specific SMART Attributes with Thresholds:
ID# ATTRIBUTE_NAME FLAG VALUE WORST THRESH TYPE UPDATED WHEN_FAILED RAW_VALUE
5 Reallocated_Sector_Ct 0x0033 100 100 010 Pre-fail Always - 0
9 Power_On_Hours 0x0032 096 096 000 Old_age Always - 17865
12 Power_Cycle_Count 0x0032 096 096 000 Old_age Always - 3174
177 Wear_Leveling_Count 0x0013 079 079 000 Pre-fail Always - 250
179 Used_Rsvd_Blk_Cnt_Tot 0x0013 100 100 010 Pre-fail Always - 0
181 Program_Fail_Cnt_Total 0x0032 100 100 010 Old_age Always - 0
182 Erase_Fail_Count_Total 0x0032 100 100 010 Old_age Always - 0
183 Runtime_Bad_Block 0x0013 100 100 010 Pre-fail Always - 0
187 Uncorrectable_Error_Cnt 0x0032 100 100 000 Old_age Always - 0
190 Airflow_Temperature_Cel 0x0032 075 039 000 Old_age Always - 25
195 ECC_Error_Rate 0x001a 200 200 000 Old_age Always - 0
199 CRC_Error_Count 0x003e 100 100 000 Old_age Always - 0
235 POR_Recovery_Count 0x0012 099 099 000 Old_age Always - 773
241 Total_LBAs_Written 0x0032 099 099 000 Old_age Always - 67005708275
SMART Error Log Version: 1
No Errors Logged
SMART Self-test log structure revision number 1
No self-tests have been logged. [To run self-tests, use: smartctl -t]
SMART Selective self-test log data structure revision number 1
SPAN MIN_LBA MAX_LBA CURRENT_TEST_STATUS
1 0 0 Not_testing
2 0 0 Not_testing
3 0 0 Not_testing
4 0 0 Not_testing
5 0 0 Not_testing
255 0 65535 Read_scanning was never started
Selective self-test flags (0x0):
After scanning selected spans, do NOT read-scan remainder of disk.
If Selective self-test is pending on power-up, resume after 0 minute delay.smartctl -a /dev/sdb
smartctl 7.0 2018-12-30 r4883 [x86_64-linux-5.0.2-arch1-1-ARCH] (local build)
Copyright (C) 2002-18, Bruce Allen, Christian Franke, www.smartmontools.org
=== START OF INFORMATION SECTION ===
Model Family: Seagate Barracuda Green (AF)
Device Model: ST1500DL003-9VT16L
Serial Number: 5YD1A0SA
LU WWN Device Id: 5 000c50 02f346ee0
Firmware Version: CC32
User Capacity: 1,500,297,682,432 bytes [1.50 TB]
Sector Size: 512 bytes logical/physical
Rotation Rate: 5900 rpm
Device is: In smartctl database [for details use: -P show]
ATA Version is: ATA8-ACS T13/1699-D revision 4
SATA Version is: SATA 3.0, 6.0 Gb/s (current: 3.0 Gb/s)
Local Time is: Sat Mar 16 14:18:13 2019 CET
SMART support is: Available - device has SMART capability.
SMART support is: Enabled
=== START OF READ SMART DATA SECTION ===
SMART overall-health self-assessment test result: FAILED!
Drive failure expected in less than 24 hours. SAVE ALL DATA.
See vendor-specific Attribute list for failed Attributes.
General SMART Values:
Offline data collection status: (0x82) Offline data collection activity
was completed without error.
Auto Offline Data Collection: Enabled.
Self-test execution status: ( 0) The previous self-test routine completed
without error or no self-test has ever
been run.
Total time to complete Offline
data collection: ( 623) seconds.
Offline data collection
capabilities: (0x7b) SMART execute Offline immediate.
Auto Offline data collection on/off support.
Suspend Offline collection upon new
command.
Offline surface scan supported.
Self-test supported.
Conveyance Self-test supported.
Selective Self-test supported.
SMART capabilities: (0x0003) Saves SMART data before entering
power-saving mode.
Supports SMART auto save timer.
Error logging capability: (0x01) Error logging supported.
General Purpose Logging supported.
Short self-test routine
recommended polling time: ( 1) minutes.
Extended self-test routine
recommended polling time: ( 268) minutes.
Conveyance self-test routine
recommended polling time: ( 2) minutes.
SCT capabilities: (0x30b7) SCT Status supported.
SCT Feature Control supported.
SCT Data Table supported.
SMART Attributes Data Structure revision number: 10
Vendor Specific SMART Attributes with Thresholds:
ID# ATTRIBUTE_NAME FLAG VALUE WORST THRESH TYPE UPDATED WHEN_FAILED RAW_VALUE
1 Raw_Read_Error_Rate 0x000f 068 067 006 Pre-fail Always - 27336
3 Spin_Up_Time 0x0003 093 092 000 Pre-fail Always - 0
4 Start_Stop_Count 0x0032 081 081 020 Old_age Always - 20295
5 Reallocated_Sector_Ct 0x0033 008 008 036 Pre-fail Always FAILING_NOW 60400
7 Seek_Error_Rate 0x000f 084 060 030 Pre-fail Always - 4590353948
9 Power_On_Hours 0x0032 072 072 000 Old_age Always - 25173
10 Spin_Retry_Count 0x0013 100 100 097 Pre-fail Always - 0
12 Power_Cycle_Count 0x0032 096 096 020 Old_age Always - 5077
183 Runtime_Bad_Block 0x0032 088 088 000 Old_age Always - 12
184 End-to-End_Error 0x0032 100 100 099 Old_age Always - 0
187 Reported_Uncorrect 0x0032 001 001 000 Old_age Always - 189
188 Command_Timeout 0x0032 100 095 000 Old_age Always - 17180197119
189 High_Fly_Writes 0x003a 050 050 000 Old_age Always - 50
190 Airflow_Temperature_Cel 0x0022 050 040 045 Old_age Always In_the_past 50 (199 56 50 32 0)
191 G-Sense_Error_Rate 0x0032 100 100 000 Old_age Always - 0
192 Power-Off_Retract_Count 0x0032 100 100 000 Old_age Always - 780
193 Load_Cycle_Count 0x0032 090 090 000 Old_age Always - 21854
194 Temperature_Celsius 0x0022 050 060 000 Old_age Always - 50 (0 8 0 0 0)
195 Hardware_ECC_Recovered 0x001a 034 010 000 Old_age Always - 27336
197 Current_Pending_Sector 0x0012 047 047 000 Old_age Always - 4408
198 Offline_Uncorrectable 0x0010 047 047 000 Old_age Offline - 4408
199 UDMA_CRC_Error_Count 0x003e 200 200 000 Old_age Always - 36
240 Head_Flying_Hours 0x0000 100 253 000 Old_age Offline - 24822 (98 224 0)
241 Total_LBAs_Written 0x0000 100 253 000 Old_age Offline - 3171940275
242 Total_LBAs_Read 0x0000 100 253 000 Old_age Offline - 1132349633
SMART Error Log Version: 1
ATA Error Count: 378 (device log contains only the most recent five errors)
CR = Command Register [HEX]
FR = Features Register [HEX]
SC = Sector Count Register [HEX]
SN = Sector Number Register [HEX]
CL = Cylinder Low Register [HEX]
CH = Cylinder High Register [HEX]
DH = Device/Head Register [HEX]
DC = Device Command Register [HEX]
ER = Error register [HEX]
ST = Status register [HEX]
Powered_Up_Time is measured from power on, and printed as
DDd+hh:mm:SS.sss where DD=days, hh=hours, mm=minutes,
SS=sec, and sss=millisec. It "wraps" after 49.710 days.
Error 378 occurred at disk power-on lifetime: 25135 hours (1047 days + 7 hours)
When the command that caused the error occurred, the device was active or idle.
After command completion occurred, registers were:
ER ST SC SN CL CH DH
-- -- -- -- -- -- --
40 51 00 ff ff ff 0f Error: UNC at LBA = 0x0fffffff = 268435455
Commands leading to the command that caused the error were:
CR FR SC SN CL CH DH DC Powered_Up_Time Command/Feature_Name
-- -- -- -- -- -- -- -- ---------------- --------------------
60 00 08 ff ff ff 4f 00 19d+19:18:13.758 READ FPDMA QUEUED
60 00 08 ff ff ff 4f 00 19d+19:18:13.758 READ FPDMA QUEUED
61 00 28 ff ff ff 4f 00 19d+19:18:13.758 WRITE FPDMA QUEUED
27 00 00 00 00 00 e0 00 19d+19:18:13.758 READ NATIVE MAX ADDRESS EXT [OBS-ACS-3]
ec 00 00 00 00 00 a0 00 19d+19:18:13.757 IDENTIFY DEVICE
Error 377 occurred at disk power-on lifetime: 25135 hours (1047 days + 7 hours)
When the command that caused the error occurred, the device was active or idle.
After command completion occurred, registers were:
ER ST SC SN CL CH DH
-- -- -- -- -- -- --
40 51 00 ff ff ff 0f Error: UNC at LBA = 0x0fffffff = 268435455
Commands leading to the command that caused the error were:
CR FR SC SN CL CH DH DC Powered_Up_Time Command/Feature_Name
-- -- -- -- -- -- -- -- ---------------- --------------------
60 00 20 ff ff ff 4f 00 19d+19:18:10.501 READ FPDMA QUEUED
60 00 60 ff ff ff 4f 00 19d+19:18:10.500 READ FPDMA QUEUED
60 00 08 ff ff ff 4f 00 19d+19:18:10.498 READ FPDMA QUEUED
60 00 20 ff ff ff 4f 00 19d+19:18:10.497 READ FPDMA QUEUED
ea 00 00 00 00 00 a0 00 19d+19:18:10.482 FLUSH CACHE EXT
Error 376 occurred at disk power-on lifetime: 25135 hours (1047 days + 7 hours)
When the command that caused the error occurred, the device was active or idle.
After command completion occurred, registers were:
ER ST SC SN CL CH DH
-- -- -- -- -- -- --
40 51 00 ff ff ff 0f Error: WP at LBA = 0x0fffffff = 268435455
Commands leading to the command that caused the error were:
CR FR SC SN CL CH DH DC Powered_Up_Time Command/Feature_Name
-- -- -- -- -- -- -- -- ---------------- --------------------
61 00 28 ff ff ff 4f 00 19d+19:14:14.040 WRITE FPDMA QUEUED
60 00 08 ff ff ff 4f 00 19d+19:14:14.040 READ FPDMA QUEUED
60 00 08 ff ff ff 4f 00 19d+19:14:14.040 READ FPDMA QUEUED
60 00 08 ff ff ff 4f 00 19d+19:14:14.040 READ FPDMA QUEUED
60 00 08 ff ff ff 4f 00 19d+19:14:14.040 READ FPDMA QUEUED
Error 375 occurred at disk power-on lifetime: 25135 hours (1047 days + 7 hours)
When the command that caused the error occurred, the device was active or idle.
After command completion occurred, registers were:
ER ST SC SN CL CH DH
-- -- -- -- -- -- --
40 51 00 ff ff ff 0f Error: WP at LBA = 0x0fffffff = 268435455
Commands leading to the command that caused the error were:
CR FR SC SN CL CH DH DC Powered_Up_Time Command/Feature_Name
-- -- -- -- -- -- -- -- ---------------- --------------------
61 00 08 ff ff ff 4f 00 19d+19:14:10.749 WRITE FPDMA QUEUED
61 00 08 ff ff ff 4f 00 19d+19:14:10.731 WRITE FPDMA QUEUED
61 00 08 ff ff ff 4f 00 19d+19:14:10.588 WRITE FPDMA QUEUED
61 00 08 ff ff ff 4f 00 19d+19:14:10.458 WRITE FPDMA QUEUED
61 00 10 ff ff ff 4f 00 19d+19:14:10.346 WRITE FPDMA QUEUED
Error 374 occurred at disk power-on lifetime: 25119 hours (1046 days + 15 hours)
When the command that caused the error occurred, the device was active or idle.
After command completion occurred, registers were:
ER ST SC SN CL CH DH
-- -- -- -- -- -- --
40 51 00 ff ff ff 0f Error: UNC at LBA = 0x0fffffff = 268435455
Commands leading to the command that caused the error were:
CR FR SC SN CL CH DH DC Powered_Up_Time Command/Feature_Name
-- -- -- -- -- -- -- -- ---------------- --------------------
60 00 08 ff ff ff 4f 00 19d+02:46:35.185 READ FPDMA QUEUED
61 00 08 ff ff ff 4f 00 19d+02:46:35.185 WRITE FPDMA QUEUED
60 00 c0 ff ff ff 4f 00 19d+02:46:35.185 READ FPDMA QUEUED
60 00 a8 ff ff ff 4f 00 19d+02:46:35.184 READ FPDMA QUEUED
27 00 00 00 00 00 e0 00 19d+02:46:35.184 READ NATIVE MAX ADDRESS EXT [OBS-ACS-3]
SMART Self-test log structure revision number 1
No self-tests have been logged. [To run self-tests, use: smartctl -t]
SMART Selective self-test log data structure revision number 1
SPAN MIN_LBA MAX_LBA CURRENT_TEST_STATUS
1 0 0 Not_testing
2 0 0 Not_testing
3 0 0 Not_testing
4 0 0 Not_testing
5 0 0 Not_testing
Selective self-test flags (0x0):
After scanning selected spans, do NOT read-scan remainder of disk.
If Selective self-test is pending on power-up, resume after 0 minute delay.smartctl -a /dev/sdc
smartctl 7.0 2018-12-30 r4883 [x86_64-linux-5.0.2-arch1-1-ARCH] (local build)
Copyright (C) 2002-18, Bruce Allen, Christian Franke, www.smartmontools.org
=== START OF INFORMATION SECTION ===
Model Family: Western Digital Caviar Blue (SATA)
Device Model: WDC WD10EALS-00Z8A0
Serial Number: WD-WCATR0382024
LU WWN Device Id: 5 0014ee 2aed5c80f
Firmware Version: 05.01D05
User Capacity: 1,000,200,658,432 bytes [1.00 TB]
Sector Size: 512 bytes logical/physical
Device is: In smartctl database [for details use: -P show]
ATA Version is: ATA8-ACS (minor revision not indicated)
SATA Version is: SATA 2.6, 3.0 Gb/s
Local Time is: Sat Mar 16 14:18:46 2019 CET
SMART support is: Available - device has SMART capability.
SMART support is: Enabled
=== START OF READ SMART DATA SECTION ===
SMART overall-health self-assessment test result: PASSED
General SMART Values:
Offline data collection status: (0x05) Offline data collection activity
was aborted by an interrupting command from host.
Auto Offline Data Collection: Disabled.
Self-test execution status: ( 0) The previous self-test routine completed
without error or no self-test has ever
been run.
Total time to complete Offline
data collection: (16560) seconds.
Offline data collection
capabilities: (0x7b) SMART execute Offline immediate.
Auto Offline data collection on/off support.
Suspend Offline collection upon new
command.
Offline surface scan supported.
Self-test supported.
Conveyance Self-test supported.
Selective Self-test supported.
SMART capabilities: (0x0003) Saves SMART data before entering
power-saving mode.
Supports SMART auto save timer.
Error logging capability: (0x01) Error logging supported.
General Purpose Logging supported.
Short self-test routine
recommended polling time: ( 2) minutes.
Extended self-test routine
recommended polling time: ( 192) minutes.
Conveyance self-test routine
recommended polling time: ( 5) minutes.
SCT capabilities: (0x3037) SCT Status supported.
SCT Feature Control supported.
SCT Data Table supported.
SMART Attributes Data Structure revision number: 16
Vendor Specific SMART Attributes with Thresholds:
ID# ATTRIBUTE_NAME FLAG VALUE WORST THRESH TYPE UPDATED WHEN_FAILED RAW_VALUE
1 Raw_Read_Error_Rate 0x002f 200 200 051 Pre-fail Always - 21
3 Spin_Up_Time 0x0027 181 171 021 Pre-fail Always - 3908
4 Start_Stop_Count 0x0032 089 089 000 Old_age Always - 11769
5 Reallocated_Sector_Ct 0x0033 200 200 140 Pre-fail Always - 0
7 Seek_Error_Rate 0x002e 200 200 000 Old_age Always - 0
9 Power_On_Hours 0x0032 052 052 000 Old_age Always - 35499
10 Spin_Retry_Count 0x0032 100 100 000 Old_age Always - 0
11 Calibration_Retry_Count 0x0032 100 100 000 Old_age Always - 0
12 Power_Cycle_Count 0x0032 095 095 000 Old_age Always - 5985
192 Power-Off_Retract_Count 0x0032 199 199 000 Old_age Always - 969
193 Load_Cycle_Count 0x0032 197 197 000 Old_age Always - 10799
194 Temperature_Celsius 0x0022 101 091 000 Old_age Always - 46
196 Reallocated_Event_Count 0x0032 200 200 000 Old_age Always - 0
197 Current_Pending_Sector 0x0032 200 200 000 Old_age Always - 0
198 Offline_Uncorrectable 0x0030 200 200 000 Old_age Offline - 0
199 UDMA_CRC_Error_Count 0x0032 200 200 000 Old_age Always - 21
200 Multi_Zone_Error_Rate 0x0008 200 200 000 Old_age Offline - 1
SMART Error Log Version: 1
No Errors Logged
SMART Self-test log structure revision number 1
Num Test_Description Status Remaining LifeTime(hours) LBA_of_first_error
# 1 Short offline Completed without error 00% 35495 -
# 2 Extended offline Interrupted (host reset) 90% 35493 -
SMART Selective self-test log data structure revision number 1
SPAN MIN_LBA MAX_LBA CURRENT_TEST_STATUS
1 0 0 Not_testing
2 0 0 Not_testing
3 0 0 Not_testing
4 0 0 Not_testing
5 0 0 Not_testing
Selective self-test flags (0x0):
After scanning selected spans, do NOT read-scan remainder of disk.
If Selective self-test is pending on power-up, resume after 0 minute delay.smartctl -a /dev/sdd
smartctl 7.0 2018-12-30 r4883 [x86_64-linux-5.0.2-arch1-1-ARCH] (local build)
Copyright (C) 2002-18, Bruce Allen, Christian Franke, www.smartmontools.org
=== START OF INFORMATION SECTION ===
Model Family: SAMSUNG SpinPoint F1 DT
Device Model: SAMSUNG HD103UJ
Serial Number: S1PVJ1NQ906284
LU WWN Device Id: 5 0000f0 00051f152
Firmware Version: 1AA01113
User Capacity: 1,000,203,804,160 bytes [1.00 TB]
Sector Size: 512 bytes logical/physical
Device is: In smartctl database [for details use: -P show]
ATA Version is: ATA/ATAPI-7, ATA8-ACS T13/1699-D revision 3b
Local Time is: Sat Mar 16 14:19:05 2019 CET
SMART support is: Available - device has SMART capability.
SMART support is: Enabled
=== START OF READ SMART DATA SECTION ===
SMART overall-health self-assessment test result: PASSED
General SMART Values:
Offline data collection status: (0x00) Offline data collection activity
was never started.
Auto Offline Data Collection: Disabled.
Self-test execution status: ( 0) The previous self-test routine completed
without error or no self-test has ever
been run.
Total time to complete Offline
data collection: (11684) seconds.
Offline data collection
capabilities: (0x7b) SMART execute Offline immediate.
Auto Offline data collection on/off support.
Suspend Offline collection upon new
command.
Offline surface scan supported.
Self-test supported.
Conveyance Self-test supported.
Selective Self-test supported.
SMART capabilities: (0x0003) Saves SMART data before entering
power-saving mode.
Supports SMART auto save timer.
Error logging capability: (0x01) Error logging supported.
General Purpose Logging supported.
Short self-test routine
recommended polling time: ( 2) minutes.
Extended self-test routine
recommended polling time: ( 196) minutes.
Conveyance self-test routine
recommended polling time: ( 21) minutes.
SCT capabilities: (0x003f) SCT Status supported.
SCT Error Recovery Control supported.
SCT Feature Control supported.
SCT Data Table supported.
SMART Attributes Data Structure revision number: 16
Vendor Specific SMART Attributes with Thresholds:
ID# ATTRIBUTE_NAME FLAG VALUE WORST THRESH TYPE UPDATED WHEN_FAILED RAW_VALUE
1 Raw_Read_Error_Rate 0x000f 100 095 051 Pre-fail Always - 0
3 Spin_Up_Time 0x0007 077 077 011 Pre-fail Always - 7820
4 Start_Stop_Count 0x0032 088 088 000 Old_age Always - 12182
5 Reallocated_Sector_Ct 0x0033 100 100 010 Pre-fail Always - 0
7 Seek_Error_Rate 0x000f 100 100 051 Pre-fail Always - 0
8 Seek_Time_Performance 0x0025 100 100 015 Pre-fail Offline - 10150
9 Power_On_Hours 0x0032 092 092 000 Old_age Always - 37977
10 Spin_Retry_Count 0x0033 100 100 051 Pre-fail Always - 1
11 Calibration_Retry_Count 0x0012 100 100 000 Old_age Always - 1
12 Power_Cycle_Count 0x0032 094 094 000 Old_age Always - 6244
13 Read_Soft_Error_Rate 0x000e 100 095 000 Old_age Always - 0
183 Runtime_Bad_Block 0x0032 100 100 000 Old_age Always - 0
184 End-to-End_Error 0x0033 100 100 000 Pre-fail Always - 0
187 Reported_Uncorrect 0x0032 100 100 000 Old_age Always - 489
188 Command_Timeout 0x0032 100 100 000 Old_age Always - 0
190 Airflow_Temperature_Cel 0x0022 061 035 000 Old_age Always - 39 (Min/Max 28/39)
194 Temperature_Celsius 0x0022 059 033 000 Old_age Always - 41 (Min/Max 28/41 #7288)
195 Hardware_ECC_Recovered 0x001a 100 100 000 Old_age Always - 86908
196 Reallocated_Event_Count 0x0032 100 100 000 Old_age Always - 0
197 Current_Pending_Sector 0x0012 100 100 000 Old_age Always - 0
198 Offline_Uncorrectable 0x0030 100 100 000 Old_age Offline - 0
199 UDMA_CRC_Error_Count 0x003e 100 100 000 Old_age Always - 1
200 Multi_Zone_Error_Rate 0x000a 100 100 000 Old_age Always - 0
201 Soft_Read_Error_Rate 0x000a 253 253 000 Old_age Always - 0
SMART Error Log Version: 1
No Errors Logged
SMART Self-test log structure revision number 1
Num Test_Description Status Remaining LifeTime(hours) LBA_of_first_error
# 1 Extended offline Completed without error 00% 27610 -
# 2 Short offline Completed without error 00% 27599 -
# 3 Short offline Completed without error 00% 15046 -
SMART Selective self-test log data structure revision number 1
SPAN MIN_LBA MAX_LBA CURRENT_TEST_STATUS
1 0 0 Not_testing
2 0 0 Not_testing
3 0 0 Not_testing
4 0 0 Not_testing
5 0 0 Not_testing
Selective self-test flags (0x0):
After scanning selected spans, do NOT read-scan remainder of disk.
If Selective self-test is pending on power-up, resume after 0 minute delay.Last edited by Senshi (2019-03-16 13:21:30)
Offline
#8 2019-03-22 12:50:32
- Senshi
- Member
- Registered: 2019-01-15
- Posts: 17
Re: [SOLVED]LVM / RAID fails to load - HDD damaged (superblock?)
After some more fiddling and attempting the fixes at the correct level, I managed to reassemble/fix the RAID with the drives, forcing the recheck I needed. I also used the opportunity to swap out all three drives with much newer ones, as the smartctl output of all is pretty scary.
I'll mark this as solved, although I do have a follow-up issue here, revolving around recovering the logical volume running on said RAID:
https://bbs.archlinux.org/viewtopic.php?pid=1838349
Offline