You are not logged in.

#1 2019-04-17 10:45:05

Natanji
Member
Registered: 2009-09-22
Posts: 133

[solved] Server crash -> mdadm RAID superblocks missing from disks?!

My home server unexpectedly crashed two hours ago; it completely wasn't reachable anymore and so I decided to turn it off and on again. After that, I didn't come up anymore and dropped me into a rescue shell, where I soon found out that my mdadm RAID couldn't be assembled. It usually looks like this:

 # cat /proc/mdstat
Personalities : [raid10] 
md0 : active raid10 sdc2[0] sdb2[1] sda1[5] sde1[6] sdd1[4] sdf[7]
      8788816896 blocks super 1.2 512K chunks 2 near-copies [6/6] [UUUUUU]
      bitmap: 4/17 pages [16KB], 262144KB chunk

unused devices: <none>

But now, it curiously looked like this...?!

 # cat /proc/mdstat
Personalities :
md0 : inactive sdb2[1](S) sda1[5](S)
      5859742168 blocks super 1.2

unused devices: <none>

So as the title says, four of the disks are no longer considered to be part of the array. The problem seems to be that the mdadm superblock can't be found. (This output is the same for all other disks that were part of the array, except sdb2 and sda1)

 mdadm --examine /dev/sdc2
mdadm: No md superblock detected on /dev/sdc2.

It's curious that this affects specifically all the disks which were connected to my PCIe SATA controller (the two that still work are connected to the two internal SATA ports). But swapping the ports around between that controller and the internal one didn't change anything, so it's not the controller reporting something weird but the actual content of the disks having been changed somehow. I suspect it's more likely because the RAID first consisted only of sdb2 and sda1, and afterwards was grown twice, each time adding two more disks to it. (That's also why this is a raid10, and not some other RAID level that would've been more useful if I had had 6 disks from the start)

More detailed output for the assembly:

# mdadm --assemble --scan -v
mdadm: looking for devices for further assembly
mdadm: no recogniseable superblock on /dev/loop0
mdadm: Cannot assemble mbr metadata on /dev/sdg2
mdadm: Cannot assemble mbr metadata on /dev/sdg1
mdadm: Cannot assemble mbr metadata on /dev/sdg
mdadm: no recogniseable superblock on /dev/sdc2
mdadm: Cannot assemble mbr metadata on /dev/sdc1
mdadm: Cannot assemble mbr metadata on /dev/sdc
mdadm: no recogniseable superblock on /dev/sde1
mdadm: Cannot assemble mbr metadata on /dev/sde
mdadm: no recogniseable superblock on /dev/sdf
mdadm: No super block found on /dev/sdb1 (Expected magic a92b4efc, got 00000041)
mdadm: no RAID superblock on /dev/sdb1
mdadm: No super block found on /dev/sdb (Expected magic a92b4efc, got 00000000)
mdadm: no RAID superblock on /dev/sdb
mdadm: No super block found on /dev/sda (Expected magic a92b4efc, got 00000000)
mdadm: no RAID superblock on /dev/sda
mdadm: No super block found on /dev/sdd1 (Expected magic a92b4efc, got 00008340)
mdadm: no RAID superblock on /dev/sdd1
mdadm: No super block found on /dev/sdd (Expected magic a92b4efc, got 00000000)
mdadm: no RAID superblock on /dev/sdd
mdadm: /dev/sdb2 is identified as a member of /dev/md/ant-16.11:0, slot 1.
mdadm: /dev/sda1 is identified as a member of /dev/md/ant-16.11:0, slot 2.
mdadm: no uptodate device for slot 0 of /dev/md/ant-16.11:0
mdadm: added /dev/sda1 to /dev/md/ant-16.11:0 as 2
mdadm: no uptodate device for slot 3 of /dev/md/ant-16.11:0
mdadm: no uptodate device for slot 4 of /dev/md/ant-16.11:0
mdadm: no uptodate device for slot 5 of /dev/md/ant-16.11:0
mdadm: added /dev/sdb2 to /dev/md/ant-16.11:0 as 1
mdadm: /dev/md/ant-16.11:0 assembled from 2 drives - not enough to start the array.
mdadm: looking for devices for further assembly
mdadm: No arrays found in config file or automatically

What also worries me is that the checksum of both working devices is wrong (off by one bit each). Can this be fixed?

# mdadm --examine /dev/sda1
/dev/sda1:
          Magic : a92b4efc
        Version : 1.2
    Feature Map : 0x0
     Array UUID : 70ae8214:9e4a0f4f:de955df3:db5c9f39
           Name : ant-16.11:0
  Creation Time : Tue Dec 20 17:49:08 2016
     Raid Level : raid10
   Raid Devices : 6

 Avail Dev Size : 5860270080 (2794.39 GiB 3000.46 GB)
     Array Size : 8788816896 (8381.67 GiB 8999.75 GB)
  Used Dev Size : 5859211264 (2793.89 GiB 2999.92 GB)
    Data Offset : 260096 sectors
   Super Offset : 8 sectors
   Unused Space : before=260008 sectors, after=1058816 sectors
          State : clean
    Device UUID : 29fbdfdf:bffc414e:51f2f7fc:8a31246f

    Update Time : Wed Apr 17 08:21:01 2019
  Bad Block Log : 512 entries available at offset 72 sectors
       Checksum : 5cdd9451 - expected 5cdd9450
         Events : 95476

         Layout : near=2
     Chunk Size : 512K

   Device Role : Active device 2
   Array State : AAAAAA ('A' == active, '.' == missing, 'R' == replacing)
mdadm --examine /dev/sdb2
/dev/sdb2:
          Magic : a92b4efc
        Version : 1.2
    Feature Map : 0x0
     Array UUID : 70ae8214:9e4a0f4f:de955df3:db5c9f39
           Name : ant-16.11:0
  Creation Time : Tue Dec 20 17:49:08 2016
     Raid Level : raid10
   Raid Devices : 6

 Avail Dev Size : 5859214257 (2793.89 GiB 2999.92 GB)
     Array Size : 8788816896 (8381.67 GiB 8999.75 GB)
  Used Dev Size : 5859211264 (2793.89 GiB 2999.92 GB)
    Data Offset : 260096 sectors
   Super Offset : 8 sectors
   Unused Space : before=260008 sectors, after=2993 sectors
          State : clean
    Device UUID : 38ada061:261c01f7:9989a3ee:6b9e118d

    Update Time : Wed Apr 17 08:21:01 2019
  Bad Block Log : 512 entries available at offset 72 sectors
       Checksum : 96e64d9d - expected 96e64d9c
         Events : 95476

         Layout : near=2
     Chunk Size : 512K

   Device Role : Active device 1
   Array State : AAAAAA ('A' == active, '.' == missing, 'R' == replacing)

So, I'm now at a loss of what to do here. I know I could try to --create the array again, but I'm not certain whether that would induce data loss, so I'd first like to understand better what happened and/or how I can perhaps recreate these superblocks? I tried to check what was written at the start of each of the RAID partitions, and I indeed don't find the RAID label there. Compare the working RAID disks...

head /dev/sda1 | strings
9ant-16.11:0

<some garbage after this>

...to the non-working ones (again, this output is the same :

head /dev/sdc2 | strings
DK2P1215W6 L      S4
.190    AMVRLE LIVTRAULL

<some garbage after this>

I don't know what this means and if this was always the case, but in my understanding, the superblock label should be written to every single disk, and now something else is on there somehow? Or was the superblock style/location changed in recent years (since I added the four later disks at a later point in time and grew the array?). In any case, the search results for those above strings aren't very helpful: https://www.google.com/search?q=DK2P1215W6

I'm at a total loss of what to do and how this could have happened. Does anybody have any clue what could have happened and what the best next plan of action would be here? Would --create recreate a valid superblock where it is missing, and then find that valid RAID data is already on the disk, preventing data loss? Or would create a completely new array instead, thereby wiping all data? What are my options here?

PS: Before anyone complains about the label of the RAID - yes, when I originally created the RAID array it was under Antergos. After having trouble with that, I switched my system over to Arch. So the label is just a remnant of that.

Last edited by Natanji (2019-05-03 23:20:40)

Offline

#2 2019-04-17 10:51:21

Slithery
Administrator
From: Norfolk, UK
Registered: 2013-12-01
Posts: 5,776

Re: [solved] Server crash -> mdadm RAID superblocks missing from disks?!

My guess would be that the PCIe adapter had a wobbly and written corrupt data to the drives when you cut the power.

My recommended solution would be to check the SMART status on the drives to rule out any problems there before just recreating the array from scratch and restoring from your backup.


No, it didn't "fix" anything. It just shifted the brokeness one space to the right. - jasonwryan
Closing -- for deletion; Banning -- for muppetry. - jasonwryan

aur - dotfiles

Offline

#3 2019-04-17 11:07:14

frostschutz
Member
Registered: 2013-11-15
Posts: 1,409

Re: [solved] Server crash -> mdadm RAID superblocks missing from disks?!

full output for mdadm --examine /dev/sd* ?

the partitions are of correct size? lsblk, blockdev --getsize64 ?

also check smartctl -a for all drives ?

mdadm checksums are not random for some reason, so off-by-1 can happen due to a metadata change that did not get updated correctly.

mdadm --create is dangerous and should not be attempted without overlay  https://unix.stackexchange.com/a/131927/30851  (of full disk copy if disk have bad sectors)

Last edited by frostschutz (2019-04-17 11:07:33)

Offline

#4 2019-04-17 13:30:35

Natanji
Member
Registered: 2009-09-22
Posts: 133

Re: [solved] Server crash -> mdadm RAID superblocks missing from disks?!

 # mdadm --examine /dev/sd*
/dev/sda:
   MBR Magic : aa55
Partition[0] :   4294967295 sectors at            1 (type ee)
/dev/sda1:
          Magic : a92b4efc
        Version : 1.2
    Feature Map : 0x0
     Array UUID : 70ae8214:9e4a0f4f:de955df3:db5c9f39
           Name : ant-16.11:0
  Creation Time : Tue Dec 20 17:49:08 2016
     Raid Level : raid10
   Raid Devices : 6

 Avail Dev Size : 5860270080 (2794.39 GiB 3000.46 GB)
     Array Size : 8788816896 (8381.67 GiB 8999.75 GB)
  Used Dev Size : 5859211264 (2793.89 GiB 2999.92 GB)
    Data Offset : 260096 sectors
   Super Offset : 8 sectors
   Unused Space : before=260008 sectors, after=1058816 sectors
          State : clean
    Device UUID : 29fbdfdf:bffc414e:51f2f7fc:8a31246f

    Update Time : Wed Apr 17 08:21:01 2019
  Bad Block Log : 512 entries available at offset 72 sectors
       Checksum : 5cdd9451 - expected 5cdd9450
         Events : 95476

         Layout : near=2
     Chunk Size : 512K

   Device Role : Active device 2
   Array State : AAAAAA ('A' == active, '.' == missing, 'R' == replacing)
/dev/sdb:
   MBR Magic : aa55
Partition[0] :   4294967295 sectors at            1 (type ee)
mdadm: No md superblock detected on /dev/sdb1.
/dev/sdb2:
          Magic : a92b4efc
        Version : 1.2
    Feature Map : 0x0
     Array UUID : 70ae8214:9e4a0f4f:de955df3:db5c9f39
           Name : ant-16.11:0
  Creation Time : Tue Dec 20 17:49:08 2016
     Raid Level : raid10
   Raid Devices : 6

 Avail Dev Size : 5859214257 (2793.89 GiB 2999.92 GB)
     Array Size : 8788816896 (8381.67 GiB 8999.75 GB)
  Used Dev Size : 5859211264 (2793.89 GiB 2999.92 GB)
    Data Offset : 260096 sectors
   Super Offset : 8 sectors
   Unused Space : before=260008 sectors, after=2993 sectors
          State : clean
    Device UUID : 38ada061:261c01f7:9989a3ee:6b9e118d

    Update Time : Wed Apr 17 08:21:01 2019
  Bad Block Log : 512 entries available at offset 72 sectors
       Checksum : 96e64d9d - expected 96e64d9c
         Events : 95476

         Layout : near=2
     Chunk Size : 512K

   Device Role : Active device 1
   Array State : AAAAAA ('A' == active, '.' == missing, 'R' == replacing)
/dev/sdc:
   MBR Magic : aa55
Partition[0] :   4294967295 sectors at            1 (type ee)
/dev/sdc1:
   MBR Magic : aa55
mdadm: No md superblock detected on /dev/sdc2.
/dev/sdd:
   MBR Magic : aa55
Partition[0] :   4294967295 sectors at            1 (type ee)
mdadm: No md superblock detected on /dev/sdd1.
/dev/sde:
   MBR Magic : aa55
Partition[0] :   4294967295 sectors at            1 (type ee)
mdadm: No md superblock detected on /dev/sde1.
mdadm: No md superblock detected on /dev/sdf.
/dev/sdg:
   MBR Magic : aa55
Partition[0] :      1146880 sectors at            0 (type 00)
Partition[1] :       131072 sectors at          164 (type ef)
/dev/sdg1:
   MBR Magic : aa55
Partition[0] :      1146880 sectors at            0 (type 00)
Partition[1] :       131072 sectors at          164 (type ef)
/dev/sdg2:
   MBR Magic : aa55

The partitions should have the the correct sizes, yes. It might seem weird that one of the RAID members is not on a partition but on sdf directly, that was an oversight during my setup which I noticed too late, but the partition sizes are as they should be:

# lsblk
NAME   MAJ:MIN RM   SIZE RO TYPE MOUNTPOINT
loop0    7:0    0 450.4M  1 loop /run/archiso/sfs/airootfs
sda      8:0    0   2.7T  0 disk 
└─sda1   8:1    0   2.7T  0 part 
sdb      8:16   0   2.7T  0 disk 
├─sdb1   8:17   0   512M  0 part 
└─sdb2   8:18   0   2.7T  0 part 
sdc      8:32   0   2.7T  0 disk 
├─sdc1   8:33   0   512M  0 part 
└─sdc2   8:34   0   2.7T  0 part 
sdd      8:48   0   2.7T  0 disk 
└─sdd1   8:49   0   2.7T  0 part 
sde      8:64   0   2.7T  0 disk 
└─sde1   8:65   0   2.7T  0 part 
sdf      8:80   0   2.7T  0 disk 
sdg      8:96   1   3.8G  0 disk 
├─sdg1   8:97   1   560M  0 part /run/archiso/bootmnt
└─sdg2   8:98   1    64M  0 part 

(As you see I'm now on the arch install disk, obviously, to have more options for recovery than in the rescue shell)

The exact partitions sizes are tad bit different from each other, but that was always the case: I set up sdb1 and sdc1 as /boot partitions, so sd[bc]2 are a bit smaller than sd[aed]1. For sdf I forgot to format it with a GPT table and so it's even a tiny bit larger than the others.

root@archiso ~ # blockdev --getsize64 /dev/sdb2
3000050868736
root@archiso ~ # blockdev --getsize64 /dev/sda1 
3000591450112
root@archiso ~ # blockdev --getsize64 /dev/sdc2 
3000050868736
root@archiso ~ # blockdev --getsize64 /dev/sdd1 
3000591450112
root@archiso ~ # blockdev --getsize64 /dev/sde1 
3000591450112
root@archiso ~ # blockdev --getsize64 /dev/sdf 
3000592982016

The SMART status seems to be fine:

root@archiso ~ # smartctl -a /dev/sda
smartctl 6.6 2017-11-05 r4594 [x86_64-linux-4.16.5-1-ARCH] (local build)
Copyright (C) 2002-17, Bruce Allen, Christian Franke, www.smartmontools.org

=== START OF INFORMATION SECTION ===
Model Family:     Western Digital Red
Device Model:     WDC WD30EFRX-68AX9N0
Serial Number:    WD-WCC1T0634599
LU WWN Device Id: 5 0014ee 25d9235c8
Firmware Version: 80.00A80
User Capacity:    3,000,592,982,016 bytes [3.00 TB]
Sector Sizes:     512 bytes logical, 4096 bytes physical
Device is:        In smartctl database [for details use: -P show]
ATA Version is:   ACS-2 (minor revision not indicated)
SATA Version is:  SATA 3.0, 6.0 Gb/s (current: 6.0 Gb/s)
Local Time is:    Wed Apr 17 12:59:32 2019 UTC
SMART support is: Available - device has SMART capability.
SMART support is: Enabled

=== START OF READ SMART DATA SECTION ===
SMART overall-health self-assessment test result: PASSED

General SMART Values:
Offline data collection status:  (0x00)	Offline data collection activity
					was never started.
					Auto Offline Data Collection: Disabled.
Self-test execution status:      (   0)	The previous self-test routine completed
					without error or no self-test has ever 
					been run.
Total time to complete Offline 
data collection: 		(41160) seconds.
Offline data collection
capabilities: 			 (0x7b) SMART execute Offline immediate.
					Auto Offline data collection on/off support.
					Suspend Offline collection upon new
					command.
					Offline surface scan supported.
					Self-test supported.
					Conveyance Self-test supported.
					Selective Self-test supported.
SMART capabilities:            (0x0003)	Saves SMART data before entering
					power-saving mode.
					Supports SMART auto save timer.
Error logging capability:        (0x01)	Error logging supported.
					General Purpose Logging supported.
Short self-test routine 
recommended polling time: 	 (   2) minutes.
Extended self-test routine
recommended polling time: 	 ( 413) minutes.
Conveyance self-test routine
recommended polling time: 	 (   5) minutes.
SCT capabilities: 	       (0x70bd)	SCT Status supported.
					SCT Error Recovery Control supported.
					SCT Feature Control supported.
					SCT Data Table supported.

SMART Attributes Data Structure revision number: 16
Vendor Specific SMART Attributes with Thresholds:
ID# ATTRIBUTE_NAME          FLAG     VALUE WORST THRESH TYPE      UPDATED  WHEN_FAILED RAW_VALUE
  1 Raw_Read_Error_Rate     0x002f   200   200   051    Pre-fail  Always       -       0
  3 Spin_Up_Time            0x0027   188   179   021    Pre-fail  Always       -       5566
  4 Start_Stop_Count        0x0032   098   098   000    Old_age   Always       -       2451
  5 Reallocated_Sector_Ct   0x0033   200   200   140    Pre-fail  Always       -       0
  7 Seek_Error_Rate         0x002e   200   200   000    Old_age   Always       -       0
  9 Power_On_Hours          0x0032   060   060   000    Old_age   Always       -       29448
 10 Spin_Retry_Count        0x0032   100   100   000    Old_age   Always       -       0
 11 Calibration_Retry_Count 0x0032   100   100   000    Old_age   Always       -       0
 12 Power_Cycle_Count       0x0032   098   098   000    Old_age   Always       -       2435
192 Power-Off_Retract_Count 0x0032   200   200   000    Old_age   Always       -       501
193 Load_Cycle_Count        0x0032   200   200   000    Old_age   Always       -       1949
194 Temperature_Celsius     0x0022   112   098   000    Old_age   Always       -       38
196 Reallocated_Event_Count 0x0032   200   200   000    Old_age   Always       -       0
197 Current_Pending_Sector  0x0032   200   200   000    Old_age   Always       -       0
198 Offline_Uncorrectable   0x0030   100   253   000    Old_age   Offline      -       0
199 UDMA_CRC_Error_Count    0x0032   200   200   000    Old_age   Always       -       1
200 Multi_Zone_Error_Rate   0x0008   100   253   000    Old_age   Offline      -       0

SMART Error Log Version: 1
No Errors Logged

SMART Self-test log structure revision number 1
No self-tests have been logged.  [To run self-tests, use: smartctl -t]

SMART Selective self-test log data structure revision number 1
 SPAN  MIN_LBA  MAX_LBA  CURRENT_TEST_STATUS
    1        0        0  Not_testing
    2        0        0  Not_testing
    3        0        0  Not_testing
    4        0        0  Not_testing
    5        0        0  Not_testing
Selective self-test flags (0x0):
  After scanning selected spans, do NOT read-scan remainder of disk.
If Selective self-test is pending on power-up, resume after 0 minute delay.

root@archiso ~ # smartctl -a /dev/sdb
smartctl 6.6 2017-11-05 r4594 [x86_64-linux-4.16.5-1-ARCH] (local build)
Copyright (C) 2002-17, Bruce Allen, Christian Franke, www.smartmontools.org

=== START OF INFORMATION SECTION ===
Model Family:     Western Digital Red
Device Model:     WDC WD30EFRX-68EUZN0
Serial Number:    WD-WCC4N4ARTA67
LU WWN Device Id: 5 0014ee 26324fc7d
Firmware Version: 82.00A82
User Capacity:    3,000,592,982,016 bytes [3.00 TB]
Sector Sizes:     512 bytes logical, 4096 bytes physical
Rotation Rate:    5400 rpm
Device is:        In smartctl database [for details use: -P show]
ATA Version is:   ACS-2 (minor revision not indicated)
SATA Version is:  SATA 3.0, 6.0 Gb/s (current: 6.0 Gb/s)
Local Time is:    Wed Apr 17 12:59:33 2019 UTC
SMART support is: Available - device has SMART capability.
SMART support is: Enabled

=== START OF READ SMART DATA SECTION ===
SMART overall-health self-assessment test result: PASSED

General SMART Values:
Offline data collection status:  (0x00)	Offline data collection activity
					was never started.
					Auto Offline Data Collection: Disabled.
Self-test execution status:      (   0)	The previous self-test routine completed
					without error or no self-test has ever 
					been run.
Total time to complete Offline 
data collection: 		(39840) seconds.
Offline data collection
capabilities: 			 (0x7b) SMART execute Offline immediate.
					Auto Offline data collection on/off support.
					Suspend Offline collection upon new
					command.
					Offline surface scan supported.
					Self-test supported.
					Conveyance Self-test supported.
					Selective Self-test supported.
SMART capabilities:            (0x0003)	Saves SMART data before entering
					power-saving mode.
					Supports SMART auto save timer.
Error logging capability:        (0x01)	Error logging supported.
					General Purpose Logging supported.
Short self-test routine 
recommended polling time: 	 (   2) minutes.
Extended self-test routine
recommended polling time: 	 ( 399) minutes.
Conveyance self-test routine
recommended polling time: 	 (   5) minutes.
SCT capabilities: 	       (0x703d)	SCT Status supported.
					SCT Error Recovery Control supported.
					SCT Feature Control supported.
					SCT Data Table supported.

SMART Attributes Data Structure revision number: 16
Vendor Specific SMART Attributes with Thresholds:
ID# ATTRIBUTE_NAME          FLAG     VALUE WORST THRESH TYPE      UPDATED  WHEN_FAILED RAW_VALUE
  1 Raw_Read_Error_Rate     0x002f   200   200   051    Pre-fail  Always       -       0
  3 Spin_Up_Time            0x0027   188   178   021    Pre-fail  Always       -       5558
  4 Start_Stop_Count        0x0032   100   100   000    Old_age   Always       -       175
  5 Reallocated_Sector_Ct   0x0033   200   200   140    Pre-fail  Always       -       0
  7 Seek_Error_Rate         0x002e   200   200   000    Old_age   Always       -       0
  9 Power_On_Hours          0x0032   077   077   000    Old_age   Always       -       17467
 10 Spin_Retry_Count        0x0032   100   100   000    Old_age   Always       -       0
 11 Calibration_Retry_Count 0x0032   100   100   000    Old_age   Always       -       0
 12 Power_Cycle_Count       0x0032   100   100   000    Old_age   Always       -       170
192 Power-Off_Retract_Count 0x0032   200   200   000    Old_age   Always       -       145
193 Load_Cycle_Count        0x0032   200   200   000    Old_age   Always       -       756
194 Temperature_Celsius     0x0022   113   102   000    Old_age   Always       -       37
196 Reallocated_Event_Count 0x0032   200   200   000    Old_age   Always       -       0
197 Current_Pending_Sector  0x0032   200   200   000    Old_age   Always       -       0
198 Offline_Uncorrectable   0x0030   100   253   000    Old_age   Offline      -       0
199 UDMA_CRC_Error_Count    0x0032   200   200   000    Old_age   Always       -       1
200 Multi_Zone_Error_Rate   0x0008   100   253   000    Old_age   Offline      -       0

SMART Error Log Version: 1
No Errors Logged

SMART Self-test log structure revision number 1
No self-tests have been logged.  [To run self-tests, use: smartctl -t]

SMART Selective self-test log data structure revision number 1
 SPAN  MIN_LBA  MAX_LBA  CURRENT_TEST_STATUS
    1        0        0  Not_testing
    2        0        0  Not_testing
    3        0        0  Not_testing
    4        0        0  Not_testing
    5        0        0  Not_testing
Selective self-test flags (0x0):
  After scanning selected spans, do NOT read-scan remainder of disk.
If Selective self-test is pending on power-up, resume after 0 minute delay.

root@archiso ~ # smartctl -a /dev/sdc
smartctl 6.6 2017-11-05 r4594 [x86_64-linux-4.16.5-1-ARCH] (local build)
Copyright (C) 2002-17, Bruce Allen, Christian Franke, www.smartmontools.org

=== START OF INFORMATION SECTION ===
Model Family:     Western Digital Red
Device Model:     WDC WD30EFRX-68EUZN0
Serial Number:    WD-WCC4N6EXPL3H
LU WWN Device Id: 5 0014ee 26324c0d0
Firmware Version: 82.00A82
User Capacity:    3,000,592,982,016 bytes [3.00 TB]
Sector Sizes:     512 bytes logical, 4096 bytes physical
Rotation Rate:    5400 rpm
Device is:        In smartctl database [for details use: -P show]
ATA Version is:   ACS-2 (minor revision not indicated)
SATA Version is:  SATA 3.0, 6.0 Gb/s (current: 6.0 Gb/s)
Local Time is:    Wed Apr 17 12:59:34 2019 UTC
SMART support is: Available - device has SMART capability.
SMART support is: Enabled

=== START OF READ SMART DATA SECTION ===
SMART overall-health self-assessment test result: PASSED

General SMART Values:
Offline data collection status:  (0x00)	Offline data collection activity
					was never started.
					Auto Offline Data Collection: Disabled.
Self-test execution status:      (   0)	The previous self-test routine completed
					without error or no self-test has ever 
					been run.
Total time to complete Offline 
data collection: 		(39840) seconds.
Offline data collection
capabilities: 			 (0x7b) SMART execute Offline immediate.
					Auto Offline data collection on/off support.
					Suspend Offline collection upon new
					command.
					Offline surface scan supported.
					Self-test supported.
					Conveyance Self-test supported.
					Selective Self-test supported.
SMART capabilities:            (0x0003)	Saves SMART data before entering
					power-saving mode.
					Supports SMART auto save timer.
Error logging capability:        (0x01)	Error logging supported.
					General Purpose Logging supported.
Short self-test routine 
recommended polling time: 	 (   2) minutes.
Extended self-test routine
recommended polling time: 	 ( 399) minutes.
Conveyance self-test routine
recommended polling time: 	 (   5) minutes.
SCT capabilities: 	       (0x703d)	SCT Status supported.
					SCT Error Recovery Control supported.
					SCT Feature Control supported.
					SCT Data Table supported.

SMART Attributes Data Structure revision number: 16
Vendor Specific SMART Attributes with Thresholds:
ID# ATTRIBUTE_NAME          FLAG     VALUE WORST THRESH TYPE      UPDATED  WHEN_FAILED RAW_VALUE
  1 Raw_Read_Error_Rate     0x002f   200   200   051    Pre-fail  Always       -       0
  3 Spin_Up_Time            0x0027   186   176   021    Pre-fail  Always       -       5666
  4 Start_Stop_Count        0x0032   100   100   000    Old_age   Always       -       176
  5 Reallocated_Sector_Ct   0x0033   200   200   140    Pre-fail  Always       -       0
  7 Seek_Error_Rate         0x002e   200   200   000    Old_age   Always       -       0
  9 Power_On_Hours          0x0032   073   073   000    Old_age   Always       -       19873
 10 Spin_Retry_Count        0x0032   100   100   000    Old_age   Always       -       0
 11 Calibration_Retry_Count 0x0032   100   100   000    Old_age   Always       -       0
 12 Power_Cycle_Count       0x0032   100   100   000    Old_age   Always       -       171
192 Power-Off_Retract_Count 0x0032   200   200   000    Old_age   Always       -       146
193 Load_Cycle_Count        0x0032   200   200   000    Old_age   Always       -       792
194 Temperature_Celsius     0x0022   112   098   000    Old_age   Always       -       38
196 Reallocated_Event_Count 0x0032   200   200   000    Old_age   Always       -       0
197 Current_Pending_Sector  0x0032   200   200   000    Old_age   Always       -       0
198 Offline_Uncorrectable   0x0030   100   253   000    Old_age   Offline      -       0
199 UDMA_CRC_Error_Count    0x0032   200   200   000    Old_age   Always       -       1
200 Multi_Zone_Error_Rate   0x0008   100   253   000    Old_age   Offline      -       0

SMART Error Log Version: 1
No Errors Logged

SMART Self-test log structure revision number 1
No self-tests have been logged.  [To run self-tests, use: smartctl -t]

SMART Selective self-test log data structure revision number 1
 SPAN  MIN_LBA  MAX_LBA  CURRENT_TEST_STATUS
    1        0        0  Not_testing
    2        0        0  Not_testing
    3        0        0  Not_testing
    4        0        0  Not_testing
    5        0        0  Not_testing
Selective self-test flags (0x0):
  After scanning selected spans, do NOT read-scan remainder of disk.
If Selective self-test is pending on power-up, resume after 0 minute delay.

root@archiso ~ # smartctl -a /dev/sdd
smartctl 6.6 2017-11-05 r4594 [x86_64-linux-4.16.5-1-ARCH] (local build)
Copyright (C) 2002-17, Bruce Allen, Christian Franke, www.smartmontools.org

=== START OF INFORMATION SECTION ===
Model Family:     Western Digital Red
Device Model:     WDC WD30EFRX-68EUZN0
Serial Number:    WD-WCC4N7CDXVST
LU WWN Device Id: 5 0014ee 20df859f0
Firmware Version: 82.00A82
User Capacity:    3,000,592,982,016 bytes [3.00 TB]
Sector Sizes:     512 bytes logical, 4096 bytes physical
Rotation Rate:    5400 rpm
Device is:        In smartctl database [for details use: -P show]
ATA Version is:   ACS-2 (minor revision not indicated)
SATA Version is:  SATA 3.0, 6.0 Gb/s (current: 6.0 Gb/s)
Local Time is:    Wed Apr 17 12:59:36 2019 UTC
SMART support is: Available - device has SMART capability.
SMART support is: Enabled

=== START OF READ SMART DATA SECTION ===
SMART overall-health self-assessment test result: PASSED

General SMART Values:
Offline data collection status:  (0x00)	Offline data collection activity
					was never started.
					Auto Offline Data Collection: Disabled.
Self-test execution status:      (   0)	The previous self-test routine completed
					without error or no self-test has ever 
					been run.
Total time to complete Offline 
data collection: 		(39840) seconds.
Offline data collection
capabilities: 			 (0x7b) SMART execute Offline immediate.
					Auto Offline data collection on/off support.
					Suspend Offline collection upon new
					command.
					Offline surface scan supported.
					Self-test supported.
					Conveyance Self-test supported.
					Selective Self-test supported.
SMART capabilities:            (0x0003)	Saves SMART data before entering
					power-saving mode.
					Supports SMART auto save timer.
Error logging capability:        (0x01)	Error logging supported.
					General Purpose Logging supported.
Short self-test routine 
recommended polling time: 	 (   2) minutes.
Extended self-test routine
recommended polling time: 	 ( 399) minutes.
Conveyance self-test routine
recommended polling time: 	 (   5) minutes.
SCT capabilities: 	       (0x703d)	SCT Status supported.
					SCT Error Recovery Control supported.
					SCT Feature Control supported.
					SCT Data Table supported.

SMART Attributes Data Structure revision number: 16
Vendor Specific SMART Attributes with Thresholds:
ID# ATTRIBUTE_NAME          FLAG     VALUE WORST THRESH TYPE      UPDATED  WHEN_FAILED RAW_VALUE
  1 Raw_Read_Error_Rate     0x002f   200   200   051    Pre-fail  Always       -       0
  3 Spin_Up_Time            0x0027   187   181   021    Pre-fail  Always       -       5625
  4 Start_Stop_Count        0x0032   100   100   000    Old_age   Always       -       50
  5 Reallocated_Sector_Ct   0x0033   200   200   140    Pre-fail  Always       -       0
  7 Seek_Error_Rate         0x002e   200   200   000    Old_age   Always       -       1
  9 Power_On_Hours          0x0032   088   088   000    Old_age   Always       -       9373
 10 Spin_Retry_Count        0x0032   100   253   000    Old_age   Always       -       0
 11 Calibration_Retry_Count 0x0032   100   253   000    Old_age   Always       -       0
 12 Power_Cycle_Count       0x0032   100   100   000    Old_age   Always       -       45
192 Power-Off_Retract_Count 0x0032   200   200   000    Old_age   Always       -       41
193 Load_Cycle_Count        0x0032   199   199   000    Old_age   Always       -       4277
194 Temperature_Celsius     0x0022   108   098   000    Old_age   Always       -       42
196 Reallocated_Event_Count 0x0032   200   200   000    Old_age   Always       -       0
197 Current_Pending_Sector  0x0032   200   200   000    Old_age   Always       -       0
198 Offline_Uncorrectable   0x0030   100   253   000    Old_age   Offline      -       0
199 UDMA_CRC_Error_Count    0x0032   200   200   000    Old_age   Always       -       0
200 Multi_Zone_Error_Rate   0x0008   100   253   000    Old_age   Offline      -       0

SMART Error Log Version: 1
No Errors Logged

SMART Self-test log structure revision number 1
No self-tests have been logged.  [To run self-tests, use: smartctl -t]

SMART Selective self-test log data structure revision number 1
 SPAN  MIN_LBA  MAX_LBA  CURRENT_TEST_STATUS
    1        0        0  Not_testing
    2        0        0  Not_testing
    3        0        0  Not_testing
    4        0        0  Not_testing
    5        0        0  Not_testing
Selective self-test flags (0x0):
  After scanning selected spans, do NOT read-scan remainder of disk.
If Selective self-test is pending on power-up, resume after 0 minute delay.

root@archiso ~ # smartctl -a /dev/sde
smartctl 6.6 2017-11-05 r4594 [x86_64-linux-4.16.5-1-ARCH] (local build)
Copyright (C) 2002-17, Bruce Allen, Christian Franke, www.smartmontools.org

=== START OF INFORMATION SECTION ===
Model Family:     Western Digital Red
Device Model:     WDC WD30EFRX-68EUZN0
Serial Number:    WD-WCC4N2FKD0K8
LU WWN Device Id: 5 0014ee 2b9032f81
Firmware Version: 82.00A82
User Capacity:    3,000,592,982,016 bytes [3.00 TB]
Sector Sizes:     512 bytes logical, 4096 bytes physical
Rotation Rate:    5400 rpm
Device is:        In smartctl database [for details use: -P show]
ATA Version is:   ACS-2 (minor revision not indicated)
SATA Version is:  SATA 3.0, 6.0 Gb/s (current: 6.0 Gb/s)
Local Time is:    Wed Apr 17 12:59:38 2019 UTC
SMART support is: Available - device has SMART capability.
SMART support is: Enabled

=== START OF READ SMART DATA SECTION ===
SMART overall-health self-assessment test result: PASSED

General SMART Values:
Offline data collection status:  (0x00)	Offline data collection activity
					was never started.
					Auto Offline Data Collection: Disabled.
Self-test execution status:      (   0)	The previous self-test routine completed
					without error or no self-test has ever 
					been run.
Total time to complete Offline 
data collection: 		(39240) seconds.
Offline data collection
capabilities: 			 (0x7b) SMART execute Offline immediate.
					Auto Offline data collection on/off support.
					Suspend Offline collection upon new
					command.
					Offline surface scan supported.
					Self-test supported.
					Conveyance Self-test supported.
					Selective Self-test supported.
SMART capabilities:            (0x0003)	Saves SMART data before entering
					power-saving mode.
					Supports SMART auto save timer.
Error logging capability:        (0x01)	Error logging supported.
					General Purpose Logging supported.
Short self-test routine 
recommended polling time: 	 (   2) minutes.
Extended self-test routine
recommended polling time: 	 ( 394) minutes.
Conveyance self-test routine
recommended polling time: 	 (   5) minutes.
SCT capabilities: 	       (0x703d)	SCT Status supported.
					SCT Error Recovery Control supported.
					SCT Feature Control supported.
					SCT Data Table supported.

SMART Attributes Data Structure revision number: 16
Vendor Specific SMART Attributes with Thresholds:
ID# ATTRIBUTE_NAME          FLAG     VALUE WORST THRESH TYPE      UPDATED  WHEN_FAILED RAW_VALUE
  1 Raw_Read_Error_Rate     0x002f   200   200   051    Pre-fail  Always       -       0
  3 Spin_Up_Time            0x0027   187   178   021    Pre-fail  Always       -       5625
  4 Start_Stop_Count        0x0032   100   100   000    Old_age   Always       -       104
  5 Reallocated_Sector_Ct   0x0033   200   200   140    Pre-fail  Always       -       0
  7 Seek_Error_Rate         0x002e   200   200   000    Old_age   Always       -       0
  9 Power_On_Hours          0x0032   079   079   000    Old_age   Always       -       15908
 10 Spin_Retry_Count        0x0032   100   100   000    Old_age   Always       -       0
 11 Calibration_Retry_Count 0x0032   100   253   000    Old_age   Always       -       0
 12 Power_Cycle_Count       0x0032   100   100   000    Old_age   Always       -       99
192 Power-Off_Retract_Count 0x0032   200   200   000    Old_age   Always       -       89
193 Load_Cycle_Count        0x0032   200   200   000    Old_age   Always       -       723
194 Temperature_Celsius     0x0022   112   102   000    Old_age   Always       -       38
196 Reallocated_Event_Count 0x0032   200   200   000    Old_age   Always       -       0
197 Current_Pending_Sector  0x0032   200   200   000    Old_age   Always       -       0
198 Offline_Uncorrectable   0x0030   100   253   000    Old_age   Offline      -       0
199 UDMA_CRC_Error_Count    0x0032   200   200   000    Old_age   Always       -       2
200 Multi_Zone_Error_Rate   0x0008   100   253   000    Old_age   Offline      -       0

SMART Error Log Version: 1
No Errors Logged

SMART Self-test log structure revision number 1
No self-tests have been logged.  [To run self-tests, use: smartctl -t]

SMART Selective self-test log data structure revision number 1
 SPAN  MIN_LBA  MAX_LBA  CURRENT_TEST_STATUS
    1        0        0  Not_testing
    2        0        0  Not_testing
    3        0        0  Not_testing
    4        0        0  Not_testing
    5        0        0  Not_testing
Selective self-test flags (0x0):
  After scanning selected spans, do NOT read-scan remainder of disk.
If Selective self-test is pending on power-up, resume after 0 minute delay.

root@archiso ~ # smartctl -a /dev/sdf
smartctl 6.6 2017-11-05 r4594 [x86_64-linux-4.16.5-1-ARCH] (local build)
Copyright (C) 2002-17, Bruce Allen, Christian Franke, www.smartmontools.org

=== START OF INFORMATION SECTION ===
Model Family:     Western Digital Red
Device Model:     WDC WD30EFRX-68N32N0
Serial Number:    WD-WCC7K6FT392L
LU WWN Device Id: 5 0014ee 2644a9812
Firmware Version: 82.00A82
User Capacity:    3,000,592,982,016 bytes [3.00 TB]
Sector Sizes:     512 bytes logical, 4096 bytes physical
Rotation Rate:    5400 rpm
Form Factor:      3.5 inches
Device is:        In smartctl database [for details use: -P show]
ATA Version is:   ACS-3 T13/2161-D revision 5
SATA Version is:  SATA 3.1, 6.0 Gb/s (current: 6.0 Gb/s)
Local Time is:    Wed Apr 17 12:59:39 2019 UTC
SMART support is: Available - device has SMART capability.
SMART support is: Enabled

=== START OF READ SMART DATA SECTION ===
SMART overall-health self-assessment test result: PASSED

General SMART Values:
Offline data collection status:  (0x00)	Offline data collection activity
					was never started.
					Auto Offline Data Collection: Disabled.
Self-test execution status:      (   0)	The previous self-test routine completed
					without error or no self-test has ever 
					been run.
Total time to complete Offline 
data collection: 		(31560) seconds.
Offline data collection
capabilities: 			 (0x7b) SMART execute Offline immediate.
					Auto Offline data collection on/off support.
					Suspend Offline collection upon new
					command.
					Offline surface scan supported.
					Self-test supported.
					Conveyance Self-test supported.
					Selective Self-test supported.
SMART capabilities:            (0x0003)	Saves SMART data before entering
					power-saving mode.
					Supports SMART auto save timer.
Error logging capability:        (0x01)	Error logging supported.
					General Purpose Logging supported.
Short self-test routine 
recommended polling time: 	 (   2) minutes.
Extended self-test routine
recommended polling time: 	 ( 337) minutes.
Conveyance self-test routine
recommended polling time: 	 (   5) minutes.
SCT capabilities: 	       (0x303d)	SCT Status supported.
					SCT Error Recovery Control supported.
					SCT Feature Control supported.
					SCT Data Table supported.

SMART Attributes Data Structure revision number: 16
Vendor Specific SMART Attributes with Thresholds:
ID# ATTRIBUTE_NAME          FLAG     VALUE WORST THRESH TYPE      UPDATED  WHEN_FAILED RAW_VALUE
  1 Raw_Read_Error_Rate     0x002f   200   200   051    Pre-fail  Always       -       0
  3 Spin_Up_Time            0x0027   199   184   021    Pre-fail  Always       -       5025
  4 Start_Stop_Count        0x0032   100   100   000    Old_age   Always       -       42
  5 Reallocated_Sector_Ct   0x0033   200   200   140    Pre-fail  Always       -       0
  7 Seek_Error_Rate         0x002e   200   200   000    Old_age   Always       -       0
  9 Power_On_Hours          0x0032   090   090   000    Old_age   Always       -       7876
 10 Spin_Retry_Count        0x0032   100   253   000    Old_age   Always       -       0
 11 Calibration_Retry_Count 0x0032   100   253   000    Old_age   Always       -       0
 12 Power_Cycle_Count       0x0032   100   100   000    Old_age   Always       -       37
192 Power-Off_Retract_Count 0x0032   200   200   000    Old_age   Always       -       33
193 Load_Cycle_Count        0x0032   200   200   000    Old_age   Always       -       62
194 Temperature_Celsius     0x0022   115   101   000    Old_age   Always       -       35
196 Reallocated_Event_Count 0x0032   200   200   000    Old_age   Always       -       0
197 Current_Pending_Sector  0x0032   200   200   000    Old_age   Always       -       0
198 Offline_Uncorrectable   0x0030   100   253   000    Old_age   Offline      -       0
199 UDMA_CRC_Error_Count    0x0032   200   200   000    Old_age   Always       -       0
200 Multi_Zone_Error_Rate   0x0008   100   253   000    Old_age   Offline      -       0

SMART Error Log Version: 1
No Errors Logged

SMART Self-test log structure revision number 1
No self-tests have been logged.  [To run self-tests, use: smartctl -t]

SMART Selective self-test log data structure revision number 1
 SPAN  MIN_LBA  MAX_LBA  CURRENT_TEST_STATUS
    1        0        0  Not_testing
    2        0        0  Not_testing
    3        0        0  Not_testing
    4        0        0  Not_testing
    5        0        0  Not_testing
Selective self-test flags (0x0):
  After scanning selected spans, do NOT read-scan remainder of disk.
If Selective self-test is pending on power-up, resume after 0 minute delay.

I didn't know about the overlay option, I will try to get that to work, that sounds good for recovery!

Do I understand correctly that I should just use --build instead of --create in my situation, since that will make the device not care about the missing metadata (but I then need to manually provide that in the command)?

Do I provide the disks in the order as returned by the (original) mdstat output (md0 : active raid10 sdc2[0] sdb2[1] sda1[5] sde1[6] sdd1[4] sdf[7]) or by the order of the numbers in brackets [] behind them?

If the overlay should run full unexpectedly, it won't start writing to the device but just block the writes, correct?

EDIT: Ah, --build is probably the wrong option for me, since my raid after all *does* contain metadata. What I tried was (with the sd* being overlay devices):

# mdadm --create /dev/md0 --assume-clean --level=10 --chunk=512 --metadata=1.2 --data-offset=260096 --raid-devices=6 sdc2 sdb2 sda1 sde1 sdd1 sdf

But that doesn't seem to have worked:

 # cryptsetup -vv luksOpen /dev/md0 lvm
Command failed with code -1 (wrong or missing parameters).

Was it the wrong order of disks?

Last edited by Natanji (2019-04-17 13:44:50)

Offline

#5 2019-04-17 14:12:08

frostschutz
Member
Registered: 2013-11-15
Posts: 1,409

Re: [solved] Server crash -> mdadm RAID superblocks missing from disks?!

Wrong data offset for sure, cause --data-offset=260096 gives you Data Offset : 520192 sectors in --examine. (always check your --examine)

The problem with mdadm is, it is not very consistent in the units it uses

try --data-offset=127M ( how you got this number I am not sure, maybe it started out with 128M, then growing at a later date ate 1M offset )

disk order etc. remains to be seen ( even if it mounts, find a large file of at least number of drives times chunksize, and verify its contents. )

your cryptsetup command I'm not sure, it should say " ... is not a valid LUKS device "

Last edited by frostschutz (2019-04-17 14:14:23)

Offline

#6 2019-04-30 11:33:11

Natanji
Member
Registered: 2009-09-22
Posts: 133

Re: [solved] Server crash -> mdadm RAID superblocks missing from disks?!

Thansk a bunch! It turned out that the order of disks posted above was in fact correct, so the order of disks shown in /proc/mdstart is indeed the correct one.  When switching out the order of the later disks, luksOpen would still work and give me /dev/mapper/lvm but that lvm would not contain any volume groups, so it was obviously the wrong order. I'm still somewhat curious what the numbers behind the devices indicate in the mdstat output, in case anyone knows that? What is their meaning?

On my first attempt of copying a backup the data off of the server, the devices crashed again mid-way. I have now replaced the faulty Syba SI-PEX40064 controller with a Dell PERC H310 running in IT mode, which isn't actually more expensive when you get it off eBay, but is supposed to be way more reliable.

It's still bizarre to me that the controller specifically destroyed the RAID superblock, but not the partition table (which is also at the start of the device, after all), and I couldn't find any other corrupted data so far either. I've almost finished copying a backup the data off of the server and it seems completely fine, even large video files of 8GB+ size play without a problem.

What's left to do is to remove the overlay and re-create the RAID on the actual partition. Should I use assume-clean there, too, or does that risk data corruption? What's the upside and downside of assume-clean in my scenario?

Offline

#7 2019-04-30 20:13:37

frostschutz
Member
Registered: 2013-11-15
Posts: 1,409

Re: [solved] Server crash -> mdadm RAID superblocks missing from disks?!

the risk of assume clean is that there may be hidden parity mismatches, which may result in corrupt data after a rebuild.

(mismatches in "free space" are fine, mismatches in actual file data are not fine, unfortunately mdadm is unable to tell the difference)

you can run a mdadm --action=check to get a counter in /sys/block/md*/md/mismatch_cnt, and investigate further if mismatch_cnt != 0

there are two ways to deal with parity mismatches... --action=repair will recalculate parity (data prevails). otherwise, assembling the array with a drive missing then syncing that drive back in, will overwrite that drive with data calculated from parity on the other drives.

if you know that one of your drives is bound to be worse / outdated than others, it makes sense to leave it as missing (if you have redundancy to allow for it to be missing)

read man md, scrubbing and mismatches section

Offline

#8 2019-05-02 21:50:04

Natanji
Member
Registered: 2009-09-22
Posts: 133

Re: [solved] Server crash -> mdadm RAID superblocks missing from disks?!

While doing the backup and/or doing --check (don't remember which one), there were actually lots of messages about unrcovered read errors on one of the disks, and it eventually failed in the array:

[97056.674975] sd 0:0:2:0: [sdc] tag#0 CDB: Read(16) 88 00 00 00 00 00 07 49 a0 00 00 00 01 00 00 00
[97056.674985] print_req_error: critical medium error, dev sdc, sector 122265600
[97060.663822] sd 0:0:2:0: [sdc] tag#1 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE
[97060.663848] sd 0:0:2:0: [sdc] tag#1 Sense Key : Medium Error [current] 
[97060.663881] sd 0:0:2:0: [sdc] tag#1 Add. Sense: Unrecovered read error
[97060.663890] sd 0:0:2:0: [sdc] tag#1 CDB: Read(16) 88 00 00 00 00 00 07 49 a1 00 00 00 00 80 00 00
[97060.663896] print_req_error: critical medium error, dev sdc, sector 122265856
[97064.686078] sd 0:0:2:0: [sdc] tag#1 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE
[97064.686112] sd 0:0:2:0: [sdc] tag#1 Sense Key : Medium Error [current] 
[97064.686123] sd 0:0:2:0: [sdc] tag#1 Add. Sense: Unrecovered read error
[97064.686136] sd 0:0:2:0: [sdc] tag#1 CDB: Read(16) 88 00 00 00 00 00 07 49 a1 80 00 00 00 80 00 00
[97064.686145] print_req_error: critical medium error, dev sdc, sector 122265984
[97068.675010] sd 0:0:2:0: [sdc] tag#1 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE
[97068.675042] sd 0:0:2:0: [sdc] tag#1 Sense Key : Medium Error [current] 
[97068.675053] sd 0:0:2:0: [sdc] tag#1 Add. Sense: Unrecovered read error
[97068.675065] sd 0:0:2:0: [sdc] tag#1 CDB: Read(16) 88 00 00 00 00 00 07 49 a2 00 00 00 01 00 00 00
[97068.675075] print_req_error: critical medium error, dev sdc, sector 122266112
[97072.663944] sd 0:0:2:0: [sdc] tag#1 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE
[97072.663978] sd 0:0:2:0: [sdc] tag#1 Sense Key : Medium Error [current] 
[97072.663989] sd 0:0:2:0: [sdc] tag#1 Add. Sense: Unrecovered read error
[97072.664002] sd 0:0:2:0: [sdc] tag#1 CDB: Read(16) 88 00 00 00 00 00 07 49 a3 00 00 00 01 00 00 00
[97072.664011] print_req_error: critical medium error, dev sdc, sector 122266368
[97076.730576] mpt2sas_cm0: log_info(0x31080000): originator(PL), code(0x08), sub_code(0x0000)
[97076.730588] mpt2sas_cm0: log_info(0x31080000): originator(PL), code(0x08), sub_code(0x0000)
[97076.730595] mpt2sas_cm0: log_info(0x31080000): originator(PL), code(0x08), sub_code(0x0000)
[97076.730602] mpt2sas_cm0: log_info(0x31080000): originator(PL), code(0x08), sub_code(0x0000)
[97076.730609] mpt2sas_cm0: log_info(0x31080000): originator(PL), code(0x08), sub_code(0x0000)
[97076.730616] mpt2sas_cm0: log_info(0x31080000): originator(PL), code(0x08), sub_code(0x0000)
[97076.730624] mpt2sas_cm0: log_info(0x31080000): originator(PL), code(0x08), sub_code(0x0000)
[97076.730631] mpt2sas_cm0: log_info(0x31080000): originator(PL), code(0x08), sub_code(0x0000)
[97076.730637] mpt2sas_cm0: log_info(0x31080000): originator(PL), code(0x08), sub_code(0x0000)
[97076.730644] mpt2sas_cm0: log_info(0x31080000): originator(PL), code(0x08), sub_code(0x0000)
[97076.730651] mpt2sas_cm0: log_info(0x31080000): originator(PL), code(0x08), sub_code(0x0000)
[97076.730658] mpt2sas_cm0: log_info(0x31080000): originator(PL), code(0x08), sub_code(0x0000)
[97076.730665] mpt2sas_cm0: log_info(0x31080000): originator(PL), code(0x08), sub_code(0x0000)
[97076.730672] mpt2sas_cm0: log_info(0x31080000): originator(PL), code(0x08), sub_code(0x0000)
[97076.730679] mpt2sas_cm0: log_info(0x31080000): originator(PL), code(0x08), sub_code(0x0000)
[97076.730687] mpt2sas_cm0: log_info(0x31080000): originator(PL), code(0x08), sub_code(0x0000)
[97076.730695] mpt2sas_cm0: log_info(0x31080000): originator(PL), code(0x08), sub_code(0x0000)
[97076.730701] mpt2sas_cm0: log_info(0x31080000): originator(PL), code(0x08), sub_code(0x0000)
[97076.730708] mpt2sas_cm0: log_info(0x31080000): originator(PL), code(0x08), sub_code(0x0000)
[97076.730716] mpt2sas_cm0: log_info(0x31080000): originator(PL), code(0x08), sub_code(0x0000)
[97076.730723] mpt2sas_cm0: log_info(0x31080000): originator(PL), code(0x08), sub_code(0x0000)
[97076.730730] mpt2sas_cm0: log_info(0x31080000): originator(PL), code(0x08), sub_code(0x0000)
[97076.730737] mpt2sas_cm0: log_info(0x31080000): originator(PL), code(0x08), sub_code(0x0000)
[97076.730745] mpt2sas_cm0: log_info(0x31080000): originator(PL), code(0x08), sub_code(0x0000)
[97076.730753] mpt2sas_cm0: log_info(0x31080000): originator(PL), code(0x08), sub_code(0x0000)
[97076.730760] mpt2sas_cm0: log_info(0x31080000): originator(PL), code(0x08), sub_code(0x0000)
[97076.730766] mpt2sas_cm0: log_info(0x31080000): originator(PL), code(0x08), sub_code(0x0000)
[97076.730774] mpt2sas_cm0: log_info(0x31080000): originator(PL), code(0x08), sub_code(0x0000)
[97076.730781] mpt2sas_cm0: log_info(0x31080000): originator(PL), code(0x08), sub_code(0x0000)
[97076.730788] mpt2sas_cm0: log_info(0x31080000): originator(PL), code(0x08), sub_code(0x0000)
[97076.730795] mpt2sas_cm0: log_info(0x31080000): originator(PL), code(0x08), sub_code(0x0000)
[97076.730875] sd 0:0:2:0: [sdc] tag#2 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE
[97076.730887] sd 0:0:2:0: [sdc] tag#2 Sense Key : Medium Error [current] 
[97076.730895] sd 0:0:2:0: [sdc] tag#2 Add. Sense: Unrecovered read error
[97076.730904] sd 0:0:2:0: [sdc] tag#2 CDB: Read(16) 88 00 00 00 00 00 07 49 a4 00 00 00 04 00 00 00
[97076.730910] print_req_error: critical medium error, dev sdc, sector 122266624
[97080.708384] mpt2sas_cm0: log_info(0x31080000): originator(PL), code(0x08), sub_code(0x0000)
[97080.708395] mpt2sas_cm0: log_info(0x31080000): originator(PL), code(0x08), sub_code(0x0000)
[97080.708402] mpt2sas_cm0: log_info(0x31080000): originator(PL), code(0x08), sub_code(0x0000)
[97080.708408] mpt2sas_cm0: log_info(0x31080000): originator(PL), code(0x08), sub_code(0x0000)
[97080.708414] mpt2sas_cm0: log_info(0x31080000): originator(PL), code(0x08), sub_code(0x0000)
[97080.708420] mpt2sas_cm0: log_info(0x31080000): originator(PL), code(0x08), sub_code(0x0000)
[97080.708427] mpt2sas_cm0: log_info(0x31080000): originator(PL), code(0x08), sub_code(0x0000)
[97080.708433] mpt2sas_cm0: log_info(0x31080000): originator(PL), code(0x08), sub_code(0x0000)
[97080.708439] mpt2sas_cm0: log_info(0x31080000): originator(PL), code(0x08), sub_code(0x0000)
[97080.708445] mpt2sas_cm0: log_info(0x31080000): originator(PL), code(0x08), sub_code(0x0000)
[97080.708451] mpt2sas_cm0: log_info(0x31080000): originator(PL), code(0x08), sub_code(0x0000)
[97080.708458] mpt2sas_cm0: log_info(0x31080000): originator(PL), code(0x08), sub_code(0x0000)
[97080.708465] mpt2sas_cm0: log_info(0x31080000): originator(PL), code(0x08), sub_code(0x0000)
[97080.708471] mpt2sas_cm0: log_info(0x31080000): originator(PL), code(0x08), sub_code(0x0000)
[97080.708477] mpt2sas_cm0: log_info(0x31080000): originator(PL), code(0x08), sub_code(0x0000)
[97080.708484] mpt2sas_cm0: log_info(0x31080000): originator(PL), code(0x08), sub_code(0x0000)
[97080.708490] mpt2sas_cm0: log_info(0x31080000): originator(PL), code(0x08), sub_code(0x0000)
[97080.708524] mpt2sas_cm0: log_info(0x31080000): originator(PL), code(0x08), sub_code(0x0000)
[97080.708532] mpt2sas_cm0: log_info(0x31080000): originator(PL), code(0x08), sub_code(0x0000)
[97080.708539] mpt2sas_cm0: log_info(0x31080000): originator(PL), code(0x08), sub_code(0x0000)
[97080.708545] mpt2sas_cm0: log_info(0x31080000): originator(PL), code(0x08), sub_code(0x0000)
[97080.708552] mpt2sas_cm0: log_info(0x31080000): originator(PL), code(0x08), sub_code(0x0000)
[97080.708559] mpt2sas_cm0: log_info(0x31080000): originator(PL), code(0x08), sub_code(0x0000)
[97080.708566] mpt2sas_cm0: log_info(0x31080000): originator(PL), code(0x08), sub_code(0x0000)
[97080.708574] mpt2sas_cm0: log_info(0x31080000): originator(PL), code(0x08), sub_code(0x0000)
[97080.708581] mpt2sas_cm0: log_info(0x31080000): originator(PL), code(0x08), sub_code(0x0000)
[97080.708588] mpt2sas_cm0: log_info(0x31080000): originator(PL), code(0x08), sub_code(0x0000)
[97080.708595] mpt2sas_cm0: log_info(0x31080000): originator(PL), code(0x08), sub_code(0x0000)
[97080.708604] mpt2sas_cm0: log_info(0x31080000): originator(PL), code(0x08), sub_code(0x0000)
[97080.708610] mpt2sas_cm0: log_info(0x31080000): originator(PL), code(0x08), sub_code(0x0000)
[97080.708618] mpt2sas_cm0: log_info(0x31080000): originator(PL), code(0x08), sub_code(0x0000)
[97080.708705] sd 0:0:2:0: [sdc] tag#24 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE
[97080.708717] sd 0:0:2:0: [sdc] tag#24 Sense Key : Medium Error [current] 
[97080.708725] sd 0:0:2:0: [sdc] tag#24 Add. Sense: Unrecovered read error
[97080.708735] sd 0:0:2:0: [sdc] tag#24 CDB: Read(16) 88 00 00 00 00 00 07 49 c0 00 00 00 04 00 00 00
[97080.708741] print_req_error: critical medium error, dev sdc, sector 122273792
[97084.686187] mpt2sas_cm0: log_info(0x31080000): originator(PL), code(0x08), sub_code(0x0000)
[97084.686207] mpt2sas_cm0: log_info(0x31080000): originator(PL), code(0x08), sub_code(0x0000)
[97084.686211] mpt2sas_cm0: log_info(0x31080000): originator(PL), code(0x08), sub_code(0x0000)
[97084.686214] mpt2sas_cm0: log_info(0x31080000): originator(PL), code(0x08), sub_code(0x0000)
[97084.686217] mpt2sas_cm0: log_info(0x31080000): originator(PL), code(0x08), sub_code(0x0000)
[97084.686220] mpt2sas_cm0: log_info(0x31080000): originator(PL), code(0x08), sub_code(0x0000)
[97084.686223] mpt2sas_cm0: log_info(0x31080000): originator(PL), code(0x08), sub_code(0x0000)
[97084.686226] mpt2sas_cm0: log_info(0x31080000): originator(PL), code(0x08), sub_code(0x0000)
[97084.686229] mpt2sas_cm0: log_info(0x31080000): originator(PL), code(0x08), sub_code(0x0000)
[97084.686232] mpt2sas_cm0: log_info(0x31080000): originator(PL), code(0x08), sub_code(0x0000)
[97084.686236] mpt2sas_cm0: log_info(0x31080000): originator(PL), code(0x08), sub_code(0x0000)
[97084.686239] mpt2sas_cm0: log_info(0x31080000): originator(PL), code(0x08), sub_code(0x0000)
[97084.686242] mpt2sas_cm0: log_info(0x31080000): originator(PL), code(0x08), sub_code(0x0000)
[97084.686245] mpt2sas_cm0: log_info(0x31080000): originator(PL), code(0x08), sub_code(0x0000)
[97084.686249] mpt2sas_cm0: log_info(0x31080000): originator(PL), code(0x08), sub_code(0x0000)
[97084.686252] mpt2sas_cm0: log_info(0x31080000): originator(PL), code(0x08), sub_code(0x0000)
[97084.686308] mpt2sas_cm0: log_info(0x31080000): originator(PL), code(0x08), sub_code(0x0000)
[97084.686314] mpt2sas_cm0: log_info(0x31080000): originator(PL), code(0x08), sub_code(0x0000)
[97084.686319] mpt2sas_cm0: log_info(0x31080000): originator(PL), code(0x08), sub_code(0x0000)
[97084.686324] mpt2sas_cm0: log_info(0x31080000): originator(PL), code(0x08), sub_code(0x0000)
[97084.686329] mpt2sas_cm0: log_info(0x31080000): originator(PL), code(0x08), sub_code(0x0000)
[97084.686334] mpt2sas_cm0: log_info(0x31080000): originator(PL), code(0x08), sub_code(0x0000)
[97084.686340] mpt2sas_cm0: log_info(0x31080000): originator(PL), code(0x08), sub_code(0x0000)
[97084.686346] mpt2sas_cm0: log_info(0x31080000): originator(PL), code(0x08), sub_code(0x0000)
[97084.686351] mpt2sas_cm0: log_info(0x31080000): originator(PL), code(0x08), sub_code(0x0000)
[97084.686356] mpt2sas_cm0: log_info(0x31080000): originator(PL), code(0x08), sub_code(0x0000)
[97084.686362] mpt2sas_cm0: log_info(0x31080000): originator(PL), code(0x08), sub_code(0x0000)
[97084.686367] mpt2sas_cm0: log_info(0x31080000): originator(PL), code(0x08), sub_code(0x0000)
[97084.686372] mpt2sas_cm0: log_info(0x31080000): originator(PL), code(0x08), sub_code(0x0000)
[97084.686378] mpt2sas_cm0: log_info(0x31080000): originator(PL), code(0x08), sub_code(0x0000)
[97084.686390] mpt2sas_cm0: log_info(0x31080000): originator(PL), code(0x08), sub_code(0x0000)
[97084.686460] sd 0:0:2:0: [sdc] tag#1 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE
[97084.686475] sd 0:0:2:0: [sdc] tag#1 Sense Key : Medium Error [current] 
[97084.686482] sd 0:0:2:0: [sdc] tag#1 Add. Sense: Unrecovered read error
[97084.686490] sd 0:0:2:0: [sdc] tag#1 CDB: Read(16) 88 00 00 00 00 00 07 49 a8 00 00 00 04 00 00 00
[97084.686496] print_req_error: critical medium error, dev sdc, sector 122267648
[97088.697344] mpt2sas_cm0: log_info(0x31080000): originator(PL), code(0x08), sub_code(0x0000)
[97088.697354] mpt2sas_cm0: log_info(0x31080000): originator(PL), code(0x08), sub_code(0x0000)
[97088.697361] mpt2sas_cm0: log_info(0x31080000): originator(PL), code(0x08), sub_code(0x0000)
[97088.697367] mpt2sas_cm0: log_info(0x31080000): originator(PL), code(0x08), sub_code(0x0000)
[97088.697373] mpt2sas_cm0: log_info(0x31080000): originator(PL), code(0x08), sub_code(0x0000)
[97088.697379] mpt2sas_cm0: log_info(0x31080000): originator(PL), code(0x08), sub_code(0x0000)
[97088.697386] mpt2sas_cm0: log_info(0x31080000): originator(PL), code(0x08), sub_code(0x0000)
[97088.697392] mpt2sas_cm0: log_info(0x31080000): originator(PL), code(0x08), sub_code(0x0000)
[97088.697397] mpt2sas_cm0: log_info(0x31080000): originator(PL), code(0x08), sub_code(0x0000)
[97088.697404] mpt2sas_cm0: log_info(0x31080000): originator(PL), code(0x08), sub_code(0x0000)
[97088.697410] mpt2sas_cm0: log_info(0x31080000): originator(PL), code(0x08), sub_code(0x0000)
[97088.697417] mpt2sas_cm0: log_info(0x31080000): originator(PL), code(0x08), sub_code(0x0000)
[97088.697424] mpt2sas_cm0: log_info(0x31080000): originator(PL), code(0x08), sub_code(0x0000)
[97088.697430] mpt2sas_cm0: log_info(0x31080000): originator(PL), code(0x08), sub_code(0x0000)
[97088.697436] mpt2sas_cm0: log_info(0x31080000): originator(PL), code(0x08), sub_code(0x0000)
[97088.697443] mpt2sas_cm0: log_info(0x31080000): originator(PL), code(0x08), sub_code(0x0000)
[97088.697479] mpt2sas_cm0: log_info(0x31080000): originator(PL), code(0x08), sub_code(0x0000)
[97088.697486] mpt2sas_cm0: log_info(0x31080000): originator(PL), code(0x08), sub_code(0x0000)
[97088.697492] mpt2sas_cm0: log_info(0x31080000): originator(PL), code(0x08), sub_code(0x0000)
[97088.697500] mpt2sas_cm0: log_info(0x31080000): originator(PL), code(0x08), sub_code(0x0000)
[97088.697506] mpt2sas_cm0: log_info(0x31080000): originator(PL), code(0x08), sub_code(0x0000)
[97088.697513] mpt2sas_cm0: log_info(0x31080000): originator(PL), code(0x08), sub_code(0x0000)
[97088.697520] mpt2sas_cm0: log_info(0x31080000): originator(PL), code(0x08), sub_code(0x0000)
[97088.697527] mpt2sas_cm0: log_info(0x31080000): originator(PL), code(0x08), sub_code(0x0000)
[97088.697534] mpt2sas_cm0: log_info(0x31080000): originator(PL), code(0x08), sub_code(0x0000)
[97088.697542] mpt2sas_cm0: log_info(0x31080000): originator(PL), code(0x08), sub_code(0x0000)
[97088.697549] mpt2sas_cm0: log_info(0x31080000): originator(PL), code(0x08), sub_code(0x0000)
[97088.697557] mpt2sas_cm0: log_info(0x31080000): originator(PL), code(0x08), sub_code(0x0000)
[97088.697564] mpt2sas_cm0: log_info(0x31080000): originator(PL), code(0x08), sub_code(0x0000)
[97088.697571] mpt2sas_cm0: log_info(0x31080000): originator(PL), code(0x08), sub_code(0x0000)
[97088.697579] mpt2sas_cm0: log_info(0x31080000): originator(PL), code(0x08), sub_code(0x0000)
[97088.697670] sd 0:0:2:0: [sdc] tag#29 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE
[97088.697682] sd 0:0:2:0: [sdc] tag#29 Sense Key : Medium Error [current] 
[97088.697690] sd 0:0:2:0: [sdc] tag#29 Add. Sense: Unrecovered read error
[97088.697699] sd 0:0:2:0: [sdc] tag#29 CDB: Read(16) 88 00 00 00 00 00 07 49 ac 00 00 00 04 00 00 00
[97088.697705] print_req_error: critical medium error, dev sdc, sector 122268672
[97092.697399] mpt2sas_cm0: log_info(0x31080000): originator(PL), code(0x08), sub_code(0x0000)
[97092.697411] mpt2sas_cm0: log_info(0x31080000): originator(PL), code(0x08), sub_code(0x0000)
[97092.697419] mpt2sas_cm0: log_info(0x31080000): originator(PL), code(0x08), sub_code(0x0000)
[97092.697425] mpt2sas_cm0: log_info(0x31080000): originator(PL), code(0x08), sub_code(0x0000)
[97092.697432] mpt2sas_cm0: log_info(0x31080000): originator(PL), code(0x08), sub_code(0x0000)
[97092.697439] mpt2sas_cm0: log_info(0x31080000): originator(PL), code(0x08), sub_code(0x0000)
[97092.697446] mpt2sas_cm0: log_info(0x31080000): originator(PL), code(0x08), sub_code(0x0000)
[97092.697452] mpt2sas_cm0: log_info(0x31080000): originator(PL), code(0x08), sub_code(0x0000)
[97092.697459] mpt2sas_cm0: log_info(0x31080000): originator(PL), code(0x08), sub_code(0x0000)
[97092.697466] mpt2sas_cm0: log_info(0x31080000): originator(PL), code(0x08), sub_code(0x0000)
[97092.697472] mpt2sas_cm0: log_info(0x31080000): originator(PL), code(0x08), sub_code(0x0000)
[97092.697480] mpt2sas_cm0: log_info(0x31080000): originator(PL), code(0x08), sub_code(0x0000)
[97092.697487] mpt2sas_cm0: log_info(0x31080000): originator(PL), code(0x08), sub_code(0x0000)
[97092.697495] mpt2sas_cm0: log_info(0x31080000): originator(PL), code(0x08), sub_code(0x0000)
[97092.697503] mpt2sas_cm0: log_info(0x31080000): originator(PL), code(0x08), sub_code(0x0000)
[97092.697510] mpt2sas_cm0: log_info(0x31080000): originator(PL), code(0x08), sub_code(0x0000)
[97092.697517] mpt2sas_cm0: log_info(0x31080000): originator(PL), code(0x08), sub_code(0x0000)
[97092.697524] mpt2sas_cm0: log_info(0x31080000): originator(PL), code(0x08), sub_code(0x0000)
[97092.697531] mpt2sas_cm0: log_info(0x31080000): originator(PL), code(0x08), sub_code(0x0000)
[97092.697537] mpt2sas_cm0: log_info(0x31080000): originator(PL), code(0x08), sub_code(0x0000)
[97092.697545] mpt2sas_cm0: log_info(0x31080000): originator(PL), code(0x08), sub_code(0x0000)
[97092.697552] mpt2sas_cm0: log_info(0x31080000): originator(PL), code(0x08), sub_code(0x0000)
[97092.697560] mpt2sas_cm0: log_info(0x31080000): originator(PL), code(0x08), sub_code(0x0000)
[97092.697567] mpt2sas_cm0: log_info(0x31080000): originator(PL), code(0x08), sub_code(0x0000)
[97092.697574] mpt2sas_cm0: log_info(0x31080000): originator(PL), code(0x08), sub_code(0x0000)
[97092.697582] mpt2sas_cm0: log_info(0x31080000): originator(PL), code(0x08), sub_code(0x0000)
[97092.697589] mpt2sas_cm0: log_info(0x31080000): originator(PL), code(0x08), sub_code(0x0000)
[97092.697596] mpt2sas_cm0: log_info(0x31080000): originator(PL), code(0x08), sub_code(0x0000)
[97092.697603] mpt2sas_cm0: log_info(0x31080000): originator(PL), code(0x08), sub_code(0x0000)
[97092.697610] mpt2sas_cm0: log_info(0x31080000): originator(PL), code(0x08), sub_code(0x0000)
[97092.697617] mpt2sas_cm0: log_info(0x31080000): originator(PL), code(0x08), sub_code(0x0000)
[97092.697739] sd 0:0:2:0: [sdc] tag#2 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE
[97092.697756] sd 0:0:2:0: [sdc] tag#2 Sense Key : Medium Error [current] 
[97092.697763] sd 0:0:2:0: [sdc] tag#2 Add. Sense: Unrecovered read error
[97092.697773] sd 0:0:2:0: [sdc] tag#2 CDB: Read(16) 88 00 00 00 00 00 07 49 b0 00 00 00 04 00 00 00
[97092.697779] print_req_error: critical medium error, dev sdc, sector 122269696
[97096.664096] mpt2sas_cm0: log_info(0x31080000): originator(PL), code(0x08), sub_code(0x0000)
[97096.664108] mpt2sas_cm0: log_info(0x31080000): originator(PL), code(0x08), sub_code(0x0000)
[97096.664114] mpt2sas_cm0: log_info(0x31080000): originator(PL), code(0x08), sub_code(0x0000)
[97096.664120] mpt2sas_cm0: log_info(0x31080000): originator(PL), code(0x08), sub_code(0x0000)
[97096.664126] mpt2sas_cm0: log_info(0x31080000): originator(PL), code(0x08), sub_code(0x0000)
[97096.664133] mpt2sas_cm0: log_info(0x31080000): originator(PL), code(0x08), sub_code(0x0000)
[97096.664139] mpt2sas_cm0: log_info(0x31080000): originator(PL), code(0x08), sub_code(0x0000)
[97096.664145] mpt2sas_cm0: log_info(0x31080000): originator(PL), code(0x08), sub_code(0x0000)
[97096.664151] mpt2sas_cm0: log_info(0x31080000): originator(PL), code(0x08), sub_code(0x0000)
[97096.664158] mpt2sas_cm0: log_info(0x31080000): originator(PL), code(0x08), sub_code(0x0000)
[97096.664162] sd 0:0:2:0: [sdc] tag#1 FAILED Result: hostbyte=DID_SOFT_ERROR driverbyte=DRIVER_OK
[97096.664170] mpt2sas_cm0: log_info(0x31080000): originator(PL), code(0x08), sub_code(0x0000)
[97096.664173] sd 0:0:2:0: [sdc] tag#1 CDB: Read(16) 88 00 00 00 00 00 07 4a 1c 00 00 00 01 00 00 00
[97096.664176] mpt2sas_cm0: log_info(0x31080000): originator(PL), code(0x08), sub_code(0x0000)
[97096.664182] mpt2sas_cm0: log_info(0x31080000): originator(PL), code(0x08), sub_code(0x0000)
[97096.664189] mpt2sas_cm0: log_info(0x31080000): originator(PL), code(0x08), sub_code(0x0000)
[97096.664215] mpt2sas_cm0: log_info(0x31080000): originator(PL), code(0x08), sub_code(0x0000)
[97096.664223] mpt2sas_cm0: log_info(0x31080000): originator(PL), code(0x08), sub_code(0x0000)
[97096.664227] print_req_error: I/O error, dev sdc, sector 122297344
[97096.664237] sd 0:0:2:0: [sdc] tag#3 FAILED Result: hostbyte=DID_SOFT_ERROR driverbyte=DRIVER_OK
[97096.666934] mpt2sas_cm0: log_info(0x31080000): originator(PL), code(0x08), sub_code(0x0000)
[97096.666941] mpt2sas_cm0: log_info(0x31080000): originator(PL), code(0x08), sub_code(0x0000)
[97096.666946] sd 0:0:2:0: [sdc] tag#3 CDB: Read(16) 88 00 00 00 00 00 07 4a 1d 00 00 00 01 00 00 00
[97096.666949] mpt2sas_cm0: log_info(0x31080000): originator(PL), code(0x08), sub_code(0x0000)
[97096.666957] mpt2sas_cm0: log_info(0x31080000): originator(PL), code(0x08), sub_code(0x0000)
[97096.666961] print_req_error: I/O error, dev sdc, sector 122297600
[97096.667096] sd 0:0:2:0: [sdc] tag#4 FAILED Result: hostbyte=DID_SOFT_ERROR driverbyte=DRIVER_OK
[97096.669582] mpt2sas_cm0: log_info(0x31080000): originator(PL), code(0x08), sub_code(0x0000)
[97096.669589] mpt2sas_cm0: log_info(0x31080000): originator(PL), code(0x08), sub_code(0x0000)
[97096.669593] sd 0:0:2:0: [sdc] tag#4 CDB: Read(16) 88 00 00 00 00 00 07 4a 18 00 00 00 04 00 00 00
[97096.669596] mpt2sas_cm0: log_info(0x31080000): originator(PL), code(0x08), sub_code(0x0000)
[97096.669604] mpt2sas_cm0: log_info(0x31080000): originator(PL), code(0x08), sub_code(0x0000)
[97096.669607] print_req_error: I/O error, dev sdc, sector 122296320
[97096.669958] sd 0:0:2:0: [sdc] tag#5 FAILED Result: hostbyte=DID_SOFT_ERROR driverbyte=DRIVER_OK
[97096.672178] mpt2sas_cm0: log_info(0x31080000): originator(PL), code(0x08), sub_code(0x0000)
[97096.672186] mpt2sas_cm0: log_info(0x31080000): originator(PL), code(0x08), sub_code(0x0000)
[97096.672189] sd 0:0:2:0: [sdc] tag#5 CDB: Read(16) 88 00 00 00 00 00 07 4a 14 00 00 00 04 00 00 00
[97096.672197] mpt2sas_cm0: log_info(0x31080000): originator(PL), code(0x08), sub_code(0x0000)
[97096.672200] print_req_error: I/O error, dev sdc, sector 122295296
[97096.672553] sd 0:0:2:0: [sdc] tag#6 FAILED Result: hostbyte=DID_SOFT_ERROR driverbyte=DRIVER_OK
[97096.674740] mpt2sas_cm0: log_info(0x31080000): originator(PL), code(0x08), sub_code(0x0000)
[97096.674748] mpt2sas_cm0: log_info(0x31080000): originator(PL), code(0x08), sub_code(0x0000)
[97096.674750] sd 0:0:2:0: [sdc] tag#6 CDB: Read(16) 88 00 00 00 00 00 07 4a 10 00 00 00 04 00 00 00
[97096.674755] print_req_error: I/O error, dev sdc, sector 122294272
[97096.674758] mpt2sas_cm0: log_info(0x31080000): originator(PL), code(0x08), sub_code(0x0000)
[97096.674767] mpt2sas_cm0: log_info(0x31080000): originator(PL), code(0x08), sub_code(0x0000)
[97096.677239] sd 0:0:2:0: [sdc] tag#0 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE
[97096.677244] sd 0:0:2:0: [sdc] tag#0 Sense Key : Medium Error [current] 
[97096.677247] sd 0:0:2:0: [sdc] tag#0 Add. Sense: Unrecovered read error
[97096.677251] sd 0:0:2:0: [sdc] tag#0 CDB: Read(16) 88 00 00 00 00 00 07 4a 20 00 00 00 04 00 00 00
[97096.677253] print_req_error: critical medium error, dev sdc, sector 122298368
[97096.677408] sd 0:0:2:0: [sdc] tag#7 FAILED Result: hostbyte=DID_SOFT_ERROR driverbyte=DRIVER_OK
[97096.678231] sd 0:0:2:0: [sdc] tag#7 CDB: Read(16) 88 00 00 00 00 00 07 4a 0c 00 00 00 04 00 00 00
[97096.678234] print_req_error: I/O error, dev sdc, sector 122293248
[97096.679375] sd 0:0:2:0: [sdc] tag#8 FAILED Result: hostbyte=DID_SOFT_ERROR driverbyte=DRIVER_OK
[97096.679382] sd 0:0:2:0: [sdc] tag#8 CDB: Read(16) 88 00 00 00 00 00 07 4a 08 00 00 00 04 00 00 00
[97096.679386] print_req_error: I/O error, dev sdc, sector 122292224
[97096.680522] sd 0:0:2:0: [sdc] tag#9 FAILED Result: hostbyte=DID_SOFT_ERROR driverbyte=DRIVER_OK
[97096.680529] sd 0:0:2:0: [sdc] tag#9 CDB: Read(16) 88 00 00 00 00 00 07 4a 04 00 00 00 04 00 00 00
[97096.680532] print_req_error: I/O error, dev sdc, sector 122291200
[97100.675243] mpt2sas_cm0: log_info(0x31080000): originator(PL), code(0x08), sub_code(0x0000)
[97100.675254] mpt2sas_cm0: log_info(0x31080000): originator(PL), code(0x08), sub_code(0x0000)
[97100.675261] mpt2sas_cm0: log_info(0x31080000): originator(PL), code(0x08), sub_code(0x0000)
[97100.675268] mpt2sas_cm0: log_info(0x31080000): originator(PL), code(0x08), sub_code(0x0000)
[97100.675275] mpt2sas_cm0: log_info(0x31080000): originator(PL), code(0x08), sub_code(0x0000)
[97100.675282] mpt2sas_cm0: log_info(0x31080000): originator(PL), code(0x08), sub_code(0x0000)
[97100.675289] mpt2sas_cm0: log_info(0x31080000): originator(PL), code(0x08), sub_code(0x0000)
[97100.675296] mpt2sas_cm0: log_info(0x31080000): originator(PL), code(0x08), sub_code(0x0000)
[97100.675303] mpt2sas_cm0: log_info(0x31080000): originator(PL), code(0x08), sub_code(0x0000)
[97100.675310] mpt2sas_cm0: log_info(0x31080000): originator(PL), code(0x08), sub_code(0x0000)
[97100.675317] mpt2sas_cm0: log_info(0x31080000): originator(PL), code(0x08), sub_code(0x0000)
[97100.675324] mpt2sas_cm0: log_info(0x31080000): originator(PL), code(0x08), sub_code(0x0000)
[97100.675332] mpt2sas_cm0: log_info(0x31080000): originator(PL), code(0x08), sub_code(0x0000)
[97100.675339] mpt2sas_cm0: log_info(0x31080000): originator(PL), code(0x08), sub_code(0x0000)
[97100.675346] mpt2sas_cm0: log_info(0x31080000): originator(PL), code(0x08), sub_code(0x0000)
[97100.675353] mpt2sas_cm0: log_info(0x31080000): originator(PL), code(0x08), sub_code(0x0000)
[97100.675360] mpt2sas_cm0: log_info(0x31080000): originator(PL), code(0x08), sub_code(0x0000)
[97100.675367] mpt2sas_cm0: log_info(0x31080000): originator(PL), code(0x08), sub_code(0x0000)
[97100.675375] mpt2sas_cm0: log_info(0x31080000): originator(PL), code(0x08), sub_code(0x0000)
[97100.675382] mpt2sas_cm0: log_info(0x31080000): originator(PL), code(0x08), sub_code(0x0000)
[97100.675390] mpt2sas_cm0: log_info(0x31080000): originator(PL), code(0x08), sub_code(0x0000)
[97100.675466] scsi_io_completion: 19 callbacks suppressed
[97100.675481] sd 0:0:2:0: [sdc] tag#2 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE
[97100.675492] sd 0:0:2:0: [sdc] tag#2 Sense Key : Medium Error [current] 
[97100.675500] sd 0:0:2:0: [sdc] tag#2 Add. Sense: Unrecovered read error
[97100.675510] sd 0:0:2:0: [sdc] tag#2 CDB: Read(16) 88 00 00 00 00 00 07 4a 30 00 00 00 04 00 00 00
[97100.675513] print_req_error: 19 callbacks suppressed
[97100.675517] print_req_error: critical medium error, dev sdc, sector 122302464
[97104.675259] mpt2sas_cm0: log_info(0x31080000): originator(PL), code(0x08), sub_code(0x0000)
[97104.675269] mpt2sas_cm0: log_info(0x31080000): originator(PL), code(0x08), sub_code(0x0000)
[97104.675276] mpt2sas_cm0: log_info(0x31080000): originator(PL), code(0x08), sub_code(0x0000)
[97104.675282] mpt2sas_cm0: log_info(0x31080000): originator(PL), code(0x08), sub_code(0x0000)
[97104.675288] mpt2sas_cm0: log_info(0x31080000): originator(PL), code(0x08), sub_code(0x0000)
[97104.675295] mpt2sas_cm0: log_info(0x31080000): originator(PL), code(0x08), sub_code(0x0000)
[97104.675302] mpt2sas_cm0: log_info(0x31080000): originator(PL), code(0x08), sub_code(0x0000)
[97104.675309] mpt2sas_cm0: log_info(0x31080000): originator(PL), code(0x08), sub_code(0x0000)
[97104.675315] mpt2sas_cm0: log_info(0x31080000): originator(PL), code(0x08), sub_code(0x0000)
[97104.675321] mpt2sas_cm0: log_info(0x31080000): originator(PL), code(0x08), sub_code(0x0000)
[97104.675326] mpt2sas_cm0: log_info(0x31080000): originator(PL), code(0x08), sub_code(0x0000)
[97104.675332] mpt2sas_cm0: log_info(0x31080000): originator(PL), code(0x08), sub_code(0x0000)
[97104.675339] mpt2sas_cm0: log_info(0x31080000): originator(PL), code(0x08), sub_code(0x0000)
[97104.675345] mpt2sas_cm0: log_info(0x31080000): originator(PL), code(0x08), sub_code(0x0000)
[97104.675351] mpt2sas_cm0: log_info(0x31080000): originator(PL), code(0x08), sub_code(0x0000)
[97104.675357] mpt2sas_cm0: log_info(0x31080000): originator(PL), code(0x08), sub_code(0x0000)
[97104.675364] mpt2sas_cm0: log_info(0x31080000): originator(PL), code(0x08), sub_code(0x0000)
[97104.675370] mpt2sas_cm0: log_info(0x31080000): originator(PL), code(0x08), sub_code(0x0000)
[97104.675377] mpt2sas_cm0: log_info(0x31080000): originator(PL), code(0x08), sub_code(0x0000)
[97104.675383] mpt2sas_cm0: log_info(0x31080000): originator(PL), code(0x08), sub_code(0x0000)
[97104.675389] mpt2sas_cm0: log_info(0x31080000): originator(PL), code(0x08), sub_code(0x0000)
[97104.675396] mpt2sas_cm0: log_info(0x31080000): originator(PL), code(0x08), sub_code(0x0000)
[97104.675460] sd 0:0:2:0: [sdc] tag#22 FAILED Result: hostbyte=DID_SOFT_ERROR driverbyte=DRIVER_OK
[97104.675472] sd 0:0:2:0: [sdc] tag#22 CDB: Read(16) 88 00 00 00 00 00 07 4a 1e 00 00 00 02 00 00 00
[97104.675477] print_req_error: I/O error, dev sdc, sector 122297856
[97104.675529] sd 0:0:2:0: [sdc] tag#14 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE
[97104.677764] sd 0:0:2:0: [sdc] tag#14 Sense Key : Medium Error [current] 
[97104.677773] sd 0:0:2:0: [sdc] tag#14 Add. Sense: Unrecovered read error
[97104.677783] sd 0:0:2:0: [sdc] tag#14 CDB: Read(16) 88 00 00 00 00 00 07 4a 48 00 00 00 04 00 00 00
[97104.677790] print_req_error: critical medium error, dev sdc, sector 122308608
[97108.675328] mpt2sas_cm0: log_info(0x31080000): originator(PL), code(0x08), sub_code(0x0000)
[97108.675340] mpt2sas_cm0: log_info(0x31080000): originator(PL), code(0x08), sub_code(0x0000)
[97108.675347] mpt2sas_cm0: log_info(0x31080000): originator(PL), code(0x08), sub_code(0x0000)
[97108.675353] mpt2sas_cm0: log_info(0x31080000): originator(PL), code(0x08), sub_code(0x0000)
[97108.675362] mpt2sas_cm0: log_info(0x31080000): originator(PL), code(0x08), sub_code(0x0000)
[97108.675368] mpt2sas_cm0: log_info(0x31080000): originator(PL), code(0x08), sub_code(0x0000)
[97108.675375] mpt2sas_cm0: log_info(0x31080000): originator(PL), code(0x08), sub_code(0x0000)
[97108.675381] mpt2sas_cm0: log_info(0x31080000): originator(PL), code(0x08), sub_code(0x0000)
[97108.675389] mpt2sas_cm0: log_info(0x31080000): originator(PL), code(0x08), sub_code(0x0000)
[97108.675395] mpt2sas_cm0: log_info(0x31080000): originator(PL), code(0x08), sub_code(0x0000)
[97108.675402] mpt2sas_cm0: log_info(0x31080000): originator(PL), code(0x08), sub_code(0x0000)
[97108.675409] mpt2sas_cm0: log_info(0x31080000): originator(PL), code(0x08), sub_code(0x0000)
[97108.675415] mpt2sas_cm0: log_info(0x31080000): originator(PL), code(0x08), sub_code(0x0000)
[97108.675422] mpt2sas_cm0: log_info(0x31080000): originator(PL), code(0x08), sub_code(0x0000)
[97108.675430] mpt2sas_cm0: log_info(0x31080000): originator(PL), code(0x08), sub_code(0x0000)
[97108.675437] mpt2sas_cm0: log_info(0x31080000): originator(PL), code(0x08), sub_code(0x0000)
[97108.675444] mpt2sas_cm0: log_info(0x31080000): originator(PL), code(0x08), sub_code(0x0000)
[97108.675452] mpt2sas_cm0: log_info(0x31080000): originator(PL), code(0x08), sub_code(0x0000)
[97108.675459] mpt2sas_cm0: log_info(0x31080000): originator(PL), code(0x08), sub_code(0x0000)
[97108.675467] mpt2sas_cm0: log_info(0x31080000): originator(PL), code(0x08), sub_code(0x0000)
[97108.675493] sd 0:0:2:0: [sdc] tag#9 FAILED Result: hostbyte=DID_SOFT_ERROR driverbyte=DRIVER_OK
[97108.675516] sd 0:0:2:0: [sdc] tag#9 CDB: Read(16) 88 00 00 00 00 00 07 4a 24 00 00 00 04 00 00 00
[97108.675526] print_req_error: I/O error, dev sdc, sector 122299392
[97108.675580] sd 0:0:2:0: [sdc] tag#2 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE
[97108.678156] sd 0:0:2:0: [sdc] tag#2 Sense Key : Medium Error [current] 
[97108.678167] sd 0:0:2:0: [sdc] tag#2 Add. Sense: Unrecovered read error
[97108.678179] sd 0:0:2:0: [sdc] tag#2 CDB: Read(16) 88 00 00 00 00 00 07 4a 68 00 00 00 04 00 00 00
[97108.678188] print_req_error: critical medium error, dev sdc, sector 122316800
[97112.653128] mpt2sas_cm0: log_info(0x31080000): originator(PL), code(0x08), sub_code(0x0000)
[97112.653140] mpt2sas_cm0: log_info(0x31080000): originator(PL), code(0x08), sub_code(0x0000)
[97112.653147] mpt2sas_cm0: log_info(0x31080000): originator(PL), code(0x08), sub_code(0x0000)
[97112.653155] mpt2sas_cm0: log_info(0x31080000): originator(PL), code(0x08), sub_code(0x0000)
[97112.653162] mpt2sas_cm0: log_info(0x31080000): originator(PL), code(0x08), sub_code(0x0000)
[97112.653168] mpt2sas_cm0: log_info(0x31080000): originator(PL), code(0x08), sub_code(0x0000)
[97112.653175] mpt2sas_cm0: log_info(0x31080000): originator(PL), code(0x08), sub_code(0x0000)
[97112.653181] mpt2sas_cm0: log_info(0x31080000): originator(PL), code(0x08), sub_code(0x0000)
[97112.653188] mpt2sas_cm0: log_info(0x31080000): originator(PL), code(0x08), sub_code(0x0000)
[97112.653194] mpt2sas_cm0: log_info(0x31080000): originator(PL), code(0x08), sub_code(0x0000)
[97112.653201] mpt2sas_cm0: log_info(0x31080000): originator(PL), code(0x08), sub_code(0x0000)
[97112.653208] mpt2sas_cm0: log_info(0x31080000): originator(PL), code(0x08), sub_code(0x0000)
[97112.653323] sd 0:0:2:0: [sdc] tag#4 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE
[97112.653329] sd 0:0:2:0: [sdc] tag#18 FAILED Result: hostbyte=DID_SOFT_ERROR driverbyte=DRIVER_OK
[97112.653340] sd 0:0:2:0: [sdc] tag#18 CDB: Read(16) 88 00 00 00 00 00 07 4a 28 00 00 00 04 00 00 00
[97112.653343] sd 0:0:2:0: [sdc] tag#4 Sense Key : Medium Error [current] 
[97112.653350] print_req_error: I/O error, dev sdc, sector 122300416
[97112.653353] sd 0:0:2:0: [sdc] tag#4 Add. Sense: Unrecovered read error
[97112.653363] sd 0:0:2:0: [sdc] tag#4 CDB: Read(16) 88 00 00 00 00 00 07 4a 5c 00 00 00 04 00 00 00
[97112.656056] print_req_error: critical medium error, dev sdc, sector 122313728
[97116.642055] mpt2sas_cm0: log_info(0x31080000): originator(PL), code(0x08), sub_code(0x0000)
[97116.642066] mpt2sas_cm0: log_info(0x31080000): originator(PL), code(0x08), sub_code(0x0000)
[97116.642072] mpt2sas_cm0: log_info(0x31080000): originator(PL), code(0x08), sub_code(0x0000)
[97116.642079] mpt2sas_cm0: log_info(0x31080000): originator(PL), code(0x08), sub_code(0x0000)
[97116.642085] mpt2sas_cm0: log_info(0x31080000): originator(PL), code(0x08), sub_code(0x0000)
[97116.642090] mpt2sas_cm0: log_info(0x31080000): originator(PL), code(0x08), sub_code(0x0000)
[97116.642096] mpt2sas_cm0: log_info(0x31080000): originator(PL), code(0x08), sub_code(0x0000)
[97116.642102] mpt2sas_cm0: log_info(0x31080000): originator(PL), code(0x08), sub_code(0x0000)
[97116.642108] mpt2sas_cm0: log_info(0x31080000): originator(PL), code(0x08), sub_code(0x0000)
[97116.642115] mpt2sas_cm0: log_info(0x31080000): originator(PL), code(0x08), sub_code(0x0000)
[97116.642121] mpt2sas_cm0: log_info(0x31080000): originator(PL), code(0x08), sub_code(0x0000)
[97116.642127] mpt2sas_cm0: log_info(0x31080000): originator(PL), code(0x08), sub_code(0x0000)
[97116.642133] mpt2sas_cm0: log_info(0x31080000): originator(PL), code(0x08), sub_code(0x0000)
[97116.642139] mpt2sas_cm0: log_info(0x31080000): originator(PL), code(0x08), sub_code(0x0000)
[97116.642145] mpt2sas_cm0: log_info(0x31080000): originator(PL), code(0x08), sub_code(0x0000)
[97116.642212] sd 0:0:2:0: [sdc] tag#3 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE
[97116.642230] sd 0:0:2:0: [sdc] tag#3 Sense Key : Medium Error [current] 
[97116.642240] sd 0:0:2:0: [sdc] tag#3 Add. Sense: Unrecovered read error
[97116.642252] sd 0:0:2:0: [sdc] tag#3 CDB: Read(16) 88 00 00 00 00 00 07 4a 34 00 00 00 04 00 00 00
[97116.642260] print_req_error: critical medium error, dev sdc, sector 122303488
[97120.653211] mpt2sas_cm0: log_info(0x31080000): originator(PL), code(0x08), sub_code(0x0000)
[97120.653221] mpt2sas_cm0: log_info(0x31080000): originator(PL), code(0x08), sub_code(0x0000)
[97120.653229] mpt2sas_cm0: log_info(0x31080000): originator(PL), code(0x08), sub_code(0x0000)
[97120.653236] mpt2sas_cm0: log_info(0x31080000): originator(PL), code(0x08), sub_code(0x0000)
[97120.653242] mpt2sas_cm0: log_info(0x31080000): originator(PL), code(0x08), sub_code(0x0000)
[97120.653248] mpt2sas_cm0: log_info(0x31080000): originator(PL), code(0x08), sub_code(0x0000)
[97120.653254] mpt2sas_cm0: log_info(0x31080000): originator(PL), code(0x08), sub_code(0x0000)
[97120.653260] mpt2sas_cm0: log_info(0x31080000): originator(PL), code(0x08), sub_code(0x0000)
[97120.653267] mpt2sas_cm0: log_info(0x31080000): originator(PL), code(0x08), sub_code(0x0000)
[97120.653272] mpt2sas_cm0: log_info(0x31080000): originator(PL), code(0x08), sub_code(0x0000)
[97120.653279] mpt2sas_cm0: log_info(0x31080000): originator(PL), code(0x08), sub_code(0x0000)
[97120.653285] mpt2sas_cm0: log_info(0x31080000): originator(PL), code(0x08), sub_code(0x0000)
[97120.653291] mpt2sas_cm0: log_info(0x31080000): originator(PL), code(0x08), sub_code(0x0000)
[97120.653297] mpt2sas_cm0: log_info(0x31080000): originator(PL), code(0x08), sub_code(0x0000)
[97120.653303] mpt2sas_cm0: log_info(0x31080000): originator(PL), code(0x08), sub_code(0x0000)
[97120.653367] sd 0:0:2:0: [sdc] tag#9 FAILED Result: hostbyte=DID_SOFT_ERROR driverbyte=DRIVER_OK
[97120.653388] sd 0:0:2:0: [sdc] tag#9 CDB: Read(16) 88 00 00 00 00 00 07 4a 7c 00 00 00 04 00 00 00
[97120.653397] print_req_error: I/O error, dev sdc, sector 122321920
[97120.653406] sd 0:0:2:0: [sdc] tag#0 FAILED Result: hostbyte=DID_SOFT_ERROR driverbyte=DRIVER_OK
[97120.656152] sd 0:0:2:0: [sdc] tag#0 CDB: Read(16) 88 00 00 00 00 00 07 4a 70 00 00 00 04 00 00 00
[97120.656161] print_req_error: I/O error, dev sdc, sector 122318848
[97120.656523] sd 0:0:2:0: [sdc] tag#10 FAILED Result: hostbyte=DID_SOFT_ERROR driverbyte=DRIVER_OK
[97120.658863] sd 0:0:2:0: [sdc] tag#10 CDB: Read(16) 88 00 00 00 00 00 07 4a 54 00 00 00 04 00 00 00
[97120.658870] print_req_error: I/O error, dev sdc, sector 122311680
[97120.659245] sd 0:0:2:0: [sdc] tag#2 FAILED Result: hostbyte=DID_SOFT_ERROR driverbyte=DRIVER_OK
[97120.661419] sd 0:0:2:0: [sdc] tag#2 CDB: Read(16) 88 00 00 00 00 00 07 4a 60 00 00 00 04 00 00 00
[97120.661427] print_req_error: I/O error, dev sdc, sector 122314752
[97120.661714] sd 0:0:2:0: [sdc] tag#11 FAILED Result: hostbyte=DID_SOFT_ERROR driverbyte=DRIVER_OK
[97120.663899] sd 0:0:2:0: [sdc] tag#11 CDB: Read(16) 88 00 00 00 00 00 07 4a 50 00 00 00 04 00 00 00
[97120.663907] print_req_error: I/O error, dev sdc, sector 122310656
[97120.664232] sd 0:0:2:0: [sdc] tag#1 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE
[97120.666326] sd 0:0:2:0: [sdc] tag#1 Sense Key : Medium Error [current] 
[97120.666331] sd 0:0:2:0: [sdc] tag#1 Add. Sense: Unrecovered read error
[97120.666336] sd 0:0:2:0: [sdc] tag#1 CDB: Read(16) 88 00 00 00 00 00 07 4a 6c 00 00 00 04 00 00 00
[97120.666340] print_req_error: critical medium error, dev sdc, sector 122317824
[97120.666475] sd 0:0:2:0: [sdc] tag#12 FAILED Result: hostbyte=DID_SOFT_ERROR driverbyte=DRIVER_OK
[97120.667298] sd 0:0:2:0: [sdc] tag#12 CDB: Read(16) 88 00 00 00 00 00 07 4a 40 00 00 00 04 00 00 00
[97120.667301] print_req_error: I/O error, dev sdc, sector 122306560
[97120.668433] sd 0:0:2:0: [sdc] tag#13 FAILED Result: hostbyte=DID_SOFT_ERROR driverbyte=DRIVER_OK
[97120.668439] sd 0:0:2:0: [sdc] tag#13 CDB: Read(16) 88 00 00 00 00 00 07 4a 3c 00 00 00 04 00 00 00
[97120.668442] print_req_error: I/O error, dev sdc, sector 122305536
[97120.669561] sd 0:0:2:0: [sdc] tag#14 FAILED Result: hostbyte=DID_SOFT_ERROR driverbyte=DRIVER_OK
[97120.669567] sd 0:0:2:0: [sdc] tag#14 CDB: Read(16) 88 00 00 00 00 00 07 4a 38 00 00 00 04 00 00 00
[97120.669570] print_req_error: I/O error, dev sdc, sector 122304512
[97124.686585] mpt2sas_cm0: log_info(0x31080000): originator(PL), code(0x08), sub_code(0x0000)
[97124.686596] mpt2sas_cm0: log_info(0x31080000): originator(PL), code(0x08), sub_code(0x0000)
[97124.686603] mpt2sas_cm0: log_info(0x31080000): originator(PL), code(0x08), sub_code(0x0000)
[97124.686610] mpt2sas_cm0: log_info(0x31080000): originator(PL), code(0x08), sub_code(0x0000)
[97124.686616] mpt2sas_cm0: log_info(0x31080000): originator(PL), code(0x08), sub_code(0x0000)
[97124.686685] sd 0:0:2:0: [sdc] tag#3 FAILED Result: hostbyte=DID_SOFT_ERROR driverbyte=DRIVER_OK
[97124.686705] sd 0:0:2:0: [sdc] tag#3 CDB: Read(16) 88 00 00 00 00 00 07 4a 78 00 00 00 04 00 00 00
[97124.686714] print_req_error: I/O error, dev sdc, sector 122320896
[97124.689721] sd 0:0:2:0: [sdc] tag#4 FAILED Result: hostbyte=DID_SOFT_ERROR driverbyte=DRIVER_OK
[97124.689737] sd 0:0:2:0: [sdc] tag#4 CDB: Read(16) 88 00 00 00 00 00 07 4a 74 00 00 00 04 00 00 00
[97124.689745] print_req_error: I/O error, dev sdc, sector 122319872
[97124.692635] sd 0:0:2:0: [sdc] tag#5 FAILED Result: hostbyte=DID_SOFT_ERROR driverbyte=DRIVER_OK
[97124.692650] sd 0:0:2:0: [sdc] tag#5 CDB: Read(16) 88 00 00 00 00 00 07 4a 64 00 00 00 04 00 00 00
[97124.692657] print_req_error: I/O error, dev sdc, sector 122315776
[97124.695561] sd 0:0:2:0: [sdc] tag#6 FAILED Result: hostbyte=DID_SOFT_ERROR driverbyte=DRIVER_OK
[97124.695576] sd 0:0:2:0: [sdc] tag#6 CDB: Read(16) 88 00 00 00 00 00 07 4a 58 00 00 00 04 00 00 00
[97124.695583] print_req_error: I/O error, dev sdc, sector 122312704
[97124.698121] sd 0:0:2:0: [sdc] tag#8 FAILED Result: hostbyte=DID_SOFT_ERROR driverbyte=DRIVER_OK
[97124.698127] sd 0:0:2:0: [sdc] tag#8 CDB: Read(16) 88 00 00 00 00 00 07 4a 44 00 00 00 04 00 00 00
[97124.698129] print_req_error: I/O error, dev sdc, sector 122307584
[97124.699461] sd 0:0:2:0: [sdc] tag#7 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE
[97124.699467] sd 0:0:2:0: [sdc] tag#7 Sense Key : Medium Error [current] 
[97124.699470] sd 0:0:2:0: [sdc] tag#7 Add. Sense: Unrecovered read error
[97124.699473] sd 0:0:2:0: [sdc] tag#7 CDB: Read(16) 88 00 00 00 00 00 07 4a 4c 00 00 00 04 00 00 00
[97124.699475] print_req_error: critical medium error, dev sdc, sector 122309632
[97128.653402] sd 0:0:2:0: [sdc] tag#0 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE
[97128.653439] sd 0:0:2:0: [sdc] tag#0 Sense Key : Medium Error [current] 
[97128.653450] sd 0:0:2:0: [sdc] tag#0 Add. Sense: Unrecovered read error
[97128.653463] sd 0:0:2:0: [sdc] tag#0 CDB: Read(16) 88 00 00 00 00 00 07 4a 2c 00 00 00 04 00 00 00
[97128.653472] print_req_error: critical medium error, dev sdc, sector 122301440
[97138.120049] mpt2sas_cm0: log_info(0x31080000): originator(PL), code(0x08), sub_code(0x0000)
[97138.120060] mpt2sas_cm0: log_info(0x31080000): originator(PL), code(0x08), sub_code(0x0000)
[97138.120066] mpt2sas_cm0: log_info(0x31080000): originator(PL), code(0x08), sub_code(0x0000)
[97138.120073] mpt2sas_cm0: log_info(0x31080000): originator(PL), code(0x08), sub_code(0x0000)
[97138.120078] mpt2sas_cm0: log_info(0x31080000): originator(PL), code(0x08), sub_code(0x0000)
[97138.120085] mpt2sas_cm0: log_info(0x31080000): originator(PL), code(0x08), sub_code(0x0000)
[97138.120092] mpt2sas_cm0: log_info(0x31080000): originator(PL), code(0x08), sub_code(0x0000)
[97138.120099] mpt2sas_cm0: log_info(0x31080000): originator(PL), code(0x08), sub_code(0x0000)
[97138.120105] mpt2sas_cm0: log_info(0x31080000): originator(PL), code(0x08), sub_code(0x0000)
[97138.120111] mpt2sas_cm0: log_info(0x31080000): originator(PL), code(0x08), sub_code(0x0000)
[97138.120135] mpt2sas_cm0: log_info(0x31080000): originator(PL), code(0x08), sub_code(0x0000)
[97138.120143] mpt2sas_cm0: log_info(0x31080000): originator(PL), code(0x08), sub_code(0x0000)
[97138.120150] mpt2sas_cm0: log_info(0x31080000): originator(PL), code(0x08), sub_code(0x0000)
[97138.120157] mpt2sas_cm0: log_info(0x31080000): originator(PL), code(0x08), sub_code(0x0000)
[97138.120164] mpt2sas_cm0: log_info(0x31080000): originator(PL), code(0x08), sub_code(0x0000)
[97138.120172] mpt2sas_cm0: log_info(0x31080000): originator(PL), code(0x08), sub_code(0x0000)
[97138.120180] mpt2sas_cm0: log_info(0x31080000): originator(PL), code(0x08), sub_code(0x0000)
[97138.120187] mpt2sas_cm0: log_info(0x31080000): originator(PL), code(0x08), sub_code(0x0000)
[97138.120195] mpt2sas_cm0: log_info(0x31080000): originator(PL), code(0x08), sub_code(0x0000)
[97138.120202] mpt2sas_cm0: log_info(0x31080000): originator(PL), code(0x08), sub_code(0x0000)
[97138.120210] mpt2sas_cm0: log_info(0x31080000): originator(PL), code(0x08), sub_code(0x0000)
[97138.120217] mpt2sas_cm0: log_info(0x31080000): originator(PL), code(0x08), sub_code(0x0000)
[97138.120305] sd 0:0:2:0: [sdc] tag#8 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE
[97138.120316] sd 0:0:2:0: [sdc] tag#8 Sense Key : Medium Error [current] 
[97138.120324] sd 0:0:2:0: [sdc] tag#8 Add. Sense: Unrecovered read error
[97138.120334] sd 0:0:2:0: [sdc] tag#8 CDB: Read(16) 88 00 00 00 00 00 07 4a 94 00 00 00 04 00 00 00
[97138.120341] print_req_error: critical medium error, dev sdc, sector 122328064
[97142.108977] mpt2sas_cm0: log_info(0x31080000): originator(PL), code(0x08), sub_code(0x0000)
[97142.108987] mpt2sas_cm0: log_info(0x31080000): originator(PL), code(0x08), sub_code(0x0000)
[97142.108993] mpt2sas_cm0: log_info(0x31080000): originator(PL), code(0x08), sub_code(0x0000)
[97142.108998] mpt2sas_cm0: log_info(0x31080000): originator(PL), code(0x08), sub_code(0x0000)
[97142.109004] mpt2sas_cm0: log_info(0x31080000): originator(PL), code(0x08), sub_code(0x0000)
[97142.109009] mpt2sas_cm0: log_info(0x31080000): originator(PL), code(0x08), sub_code(0x0000)
[97142.109016] mpt2sas_cm0: log_info(0x31080000): originator(PL), code(0x08), sub_code(0x0000)
[97142.109022] mpt2sas_cm0: log_info(0x31080000): originator(PL), code(0x08), sub_code(0x0000)
[97142.109028] mpt2sas_cm0: log_info(0x31080000): originator(PL), code(0x08), sub_code(0x0000)
[97142.109033] mpt2sas_cm0: log_info(0x31080000): originator(PL), code(0x08), sub_code(0x0000)
[97142.109038] mpt2sas_cm0: log_info(0x31080000): originator(PL), code(0x08), sub_code(0x0000)
[97142.109045] mpt2sas_cm0: log_info(0x31080000): originator(PL), code(0x08), sub_code(0x0000)
[97142.109051] mpt2sas_cm0: log_info(0x31080000): originator(PL), code(0x08), sub_code(0x0000)
[97142.109057] mpt2sas_cm0: log_info(0x31080000): originator(PL), code(0x08), sub_code(0x0000)
[97142.109063] mpt2sas_cm0: log_info(0x31080000): originator(PL), code(0x08), sub_code(0x0000)
[97142.109069] mpt2sas_cm0: log_info(0x31080000): originator(PL), code(0x08), sub_code(0x0000)
[97142.109075] mpt2sas_cm0: log_info(0x31080000): originator(PL), code(0x08), sub_code(0x0000)
[97142.109081] mpt2sas_cm0: log_info(0x31080000): originator(PL), code(0x08), sub_code(0x0000)
[97142.109087] mpt2sas_cm0: log_info(0x31080000): originator(PL), code(0x08), sub_code(0x0000)
[97142.109094] mpt2sas_cm0: log_info(0x31080000): originator(PL), code(0x08), sub_code(0x0000)
[97142.109100] mpt2sas_cm0: log_info(0x31080000): originator(PL), code(0x08), sub_code(0x0000)
[97142.109169] sd 0:0:2:0: [sdc] tag#18 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE
[97142.109179] sd 0:0:2:0: [sdc] tag#18 Sense Key : Medium Error [current] 
[97142.109185] sd 0:0:2:0: [sdc] tag#18 Add. Sense: Unrecovered read error
[97142.109192] sd 0:0:2:0: [sdc] tag#18 CDB: Read(16) 88 00 00 00 00 00 07 4a cc 00 00 00 04 00 00 00
[97142.109197] print_req_error: critical medium error, dev sdc, sector 122342400
[97146.109023] mpt2sas_cm0: log_info(0x31080000): originator(PL), code(0x08), sub_code(0x0000)
[97146.109033] mpt2sas_cm0: log_info(0x31080000): originator(PL), code(0x08), sub_code(0x0000)
[97146.109039] mpt2sas_cm0: log_info(0x31080000): originator(PL), code(0x08), sub_code(0x0000)
[97146.109044] mpt2sas_cm0: log_info(0x31080000): originator(PL), code(0x08), sub_code(0x0000)
[97146.109050] mpt2sas_cm0: log_info(0x31080000): originator(PL), code(0x08), sub_code(0x0000)
[97146.109056] mpt2sas_cm0: log_info(0x31080000): originator(PL), code(0x08), sub_code(0x0000)
[97146.109062] mpt2sas_cm0: log_info(0x31080000): originator(PL), code(0x08), sub_code(0x0000)
[97146.109068] mpt2sas_cm0: log_info(0x31080000): originator(PL), code(0x08), sub_code(0x0000)
[97146.109074] mpt2sas_cm0: log_info(0x31080000): originator(PL), code(0x08), sub_code(0x0000)
[97146.109080] mpt2sas_cm0: log_info(0x31080000): originator(PL), code(0x08), sub_code(0x0000)
[97146.109086] mpt2sas_cm0: log_info(0x31080000): originator(PL), code(0x08), sub_code(0x0000)
[97146.109092] mpt2sas_cm0: log_info(0x31080000): originator(PL), code(0x08), sub_code(0x0000)
[97146.109098] mpt2sas_cm0: log_info(0x31080000): originator(PL), code(0x08), sub_code(0x0000)
[97146.109104] mpt2sas_cm0: log_info(0x31080000): originator(PL), code(0x08), sub_code(0x0000)
[97146.109111] mpt2sas_cm0: log_info(0x31080000): originator(PL), code(0x08), sub_code(0x0000)
[97146.109116] mpt2sas_cm0: log_info(0x31080000): originator(PL), code(0x08), sub_code(0x0000)
[97146.109122] mpt2sas_cm0: log_info(0x31080000): originator(PL), code(0x08), sub_code(0x0000)
[97146.109129] mpt2sas_cm0: log_info(0x31080000): originator(PL), code(0x08), sub_code(0x0000)
[97146.109135] mpt2sas_cm0: log_info(0x31080000): originator(PL), code(0x08), sub_code(0x0000)
[97146.109140] mpt2sas_cm0: log_info(0x31080000): originator(PL), code(0x08), sub_code(0x0000)
[97146.109207] sd 0:0:2:0: [sdc] tag#4 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE
[97146.109217] sd 0:0:2:0: [sdc] tag#4 Sense Key : Medium Error [current] 
[97146.109223] sd 0:0:2:0: [sdc] tag#4 Add. Sense: Unrecovered read error
[97146.109231] sd 0:0:2:0: [sdc] tag#4 CDB: Read(16) 88 00 00 00 00 00 07 4a 98 00 00 00 04 00 00 00
[97146.109236] print_req_error: critical medium error, dev sdc, sector 122329088
[97150.120141] mpt2sas_cm0: log_info(0x31080000): originator(PL), code(0x08), sub_code(0x0000)
[97150.120150] mpt2sas_cm0: log_info(0x31080000): originator(PL), code(0x08), sub_code(0x0000)
[97150.120155] mpt2sas_cm0: log_info(0x31080000): originator(PL), code(0x08), sub_code(0x0000)
[97150.120160] mpt2sas_cm0: log_info(0x31080000): originator(PL), code(0x08), sub_code(0x0000)
[97150.120165] mpt2sas_cm0: log_info(0x31080000): originator(PL), code(0x08), sub_code(0x0000)
[97150.120169] mpt2sas_cm0: log_info(0x31080000): originator(PL), code(0x08), sub_code(0x0000)
[97150.120175] mpt2sas_cm0: log_info(0x31080000): originator(PL), code(0x08), sub_code(0x0000)
[97150.120180] mpt2sas_cm0: log_info(0x31080000): originator(PL), code(0x08), sub_code(0x0000)
[97150.120185] mpt2sas_cm0: log_info(0x31080000): originator(PL), code(0x08), sub_code(0x0000)
[97150.120189] mpt2sas_cm0: log_info(0x31080000): originator(PL), code(0x08), sub_code(0x0000)
[97150.120194] mpt2sas_cm0: log_info(0x31080000): originator(PL), code(0x08), sub_code(0x0000)
[97150.120200] mpt2sas_cm0: log_info(0x31080000): originator(PL), code(0x08), sub_code(0x0000)
[97150.120205] mpt2sas_cm0: log_info(0x31080000): originator(PL), code(0x08), sub_code(0x0000)
[97150.120211] mpt2sas_cm0: log_info(0x31080000): originator(PL), code(0x08), sub_code(0x0000)
[97150.120216] mpt2sas_cm0: log_info(0x31080000): originator(PL), code(0x08), sub_code(0x0000)
[97150.120221] mpt2sas_cm0: log_info(0x31080000): originator(PL), code(0x08), sub_code(0x0000)
[97150.120226] mpt2sas_cm0: log_info(0x31080000): originator(PL), code(0x08), sub_code(0x0000)
[97150.120231] mpt2sas_cm0: log_info(0x31080000): originator(PL), code(0x08), sub_code(0x0000)
[97150.120236] mpt2sas_cm0: log_info(0x31080000): originator(PL), code(0x08), sub_code(0x0000)
[97150.120304] sd 0:0:2:0: [sdc] tag#15 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE
[97150.120313] sd 0:0:2:0: [sdc] tag#15 Sense Key : Medium Error [current] 
[97150.120318] sd 0:0:2:0: [sdc] tag#15 Add. Sense: Unrecovered read error
[97150.120325] sd 0:0:2:0: [sdc] tag#15 CDB: Read(16) 88 00 00 00 00 00 07 4a 9c 00 00 00 04 00 00 00
[97150.120329] print_req_error: critical medium error, dev sdc, sector 122330112
[97154.109084] mpt2sas_cm0: log_info(0x31080000): originator(PL), code(0x08), sub_code(0x0000)
[97154.109093] mpt2sas_cm0: log_info(0x31080000): originator(PL), code(0x08), sub_code(0x0000)
[97154.109097] mpt2sas_cm0: log_info(0x31080000): originator(PL), code(0x08), sub_code(0x0000)
[97154.109102] mpt2sas_cm0: log_info(0x31080000): originator(PL), code(0x08), sub_code(0x0000)
[97154.109107] mpt2sas_cm0: log_info(0x31080000): originator(PL), code(0x08), sub_code(0x0000)
[97154.109111] mpt2sas_cm0: log_info(0x31080000): originator(PL), code(0x08), sub_code(0x0000)
[97154.109116] mpt2sas_cm0: log_info(0x31080000): originator(PL), code(0x08), sub_code(0x0000)
[97154.109121] mpt2sas_cm0: log_info(0x31080000): originator(PL), code(0x08), sub_code(0x0000)
[97154.109127] mpt2sas_cm0: log_info(0x31080000): originator(PL), code(0x08), sub_code(0x0000)
[97154.109132] mpt2sas_cm0: log_info(0x31080000): originator(PL), code(0x08), sub_code(0x0000)
[97154.109137] mpt2sas_cm0: log_info(0x31080000): originator(PL), code(0x08), sub_code(0x0000)
[97154.109143] mpt2sas_cm0: log_info(0x31080000): originator(PL), code(0x08), sub_code(0x0000)
[97154.109149] mpt2sas_cm0: log_info(0x31080000): originator(PL), code(0x08), sub_code(0x0000)
[97154.109154] mpt2sas_cm0: log_info(0x31080000): originator(PL), code(0x08), sub_code(0x0000)
[97154.109159] mpt2sas_cm0: log_info(0x31080000): originator(PL), code(0x08), sub_code(0x0000)
[97154.109164] mpt2sas_cm0: log_info(0x31080000): originator(PL), code(0x08), sub_code(0x0000)
[97154.109169] mpt2sas_cm0: log_info(0x31080000): originator(PL), code(0x08), sub_code(0x0000)
[97154.109174] mpt2sas_cm0: log_info(0x31080000): originator(PL), code(0x08), sub_code(0x0000)
[97154.109249] sd 0:0:2:0: [sdc] tag#4 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE
[97154.109258] sd 0:0:2:0: [sdc] tag#4 Sense Key : Medium Error [current] 
[97154.109263] sd 0:0:2:0: [sdc] tag#4 Add. Sense: Unrecovered read error
[97154.109272] sd 0:0:2:0: [sdc] tag#4 CDB: Read(16) 88 00 00 00 00 00 07 4a a0 00 00 00 04 00 00 00
[97154.109276] print_req_error: critical medium error, dev sdc, sector 122331136
[97158.086926] mpt2sas_cm0: log_info(0x31080000): originator(PL), code(0x08), sub_code(0x0000)
[97158.086937] mpt2sas_cm0: log_info(0x31080000): originator(PL), code(0x08), sub_code(0x0000)
[97158.086944] mpt2sas_cm0: log_info(0x31080000): originator(PL), code(0x08), sub_code(0x0000)
[97158.086950] mpt2sas_cm0: log_info(0x31080000): originator(PL), code(0x08), sub_code(0x0000)
[97158.086956] mpt2sas_cm0: log_info(0x31080000): originator(PL), code(0x08), sub_code(0x0000)
[97158.086963] mpt2sas_cm0: log_info(0x31080000): originator(PL), code(0x08), sub_code(0x0000)
[97158.086970] mpt2sas_cm0: log_info(0x31080000): originator(PL), code(0x08), sub_code(0x0000)
[97158.086976] mpt2sas_cm0: log_info(0x31080000): originator(PL), code(0x08), sub_code(0x0000)
[97158.086983] mpt2sas_cm0: log_info(0x31080000): originator(PL), code(0x08), sub_code(0x0000)
[97158.086988] mpt2sas_cm0: log_info(0x31080000): originator(PL), code(0x08), sub_code(0x0000)
[97158.086995] mpt2sas_cm0: log_info(0x31080000): originator(PL), code(0x08), sub_code(0x0000)
[97158.087002] mpt2sas_cm0: log_info(0x31080000): originator(PL), code(0x08), sub_code(0x0000)
[97158.087008] mpt2sas_cm0: log_info(0x31080000): originator(PL), code(0x08), sub_code(0x0000)
[97158.087014] sd 0:0:2:0: [sdc] tag#0 FAILED Result: hostbyte=DID_SOFT_ERROR driverbyte=DRIVER_OK
[97158.087018] mpt2sas_cm0: log_info(0x31080000): originator(PL), code(0x08), sub_code(0x0000)
[97158.087025] mpt2sas_cm0: log_info(0x31080000): originator(PL), code(0x08), sub_code(0x0000)
[97158.087031] mpt2sas_cm0: log_info(0x31080000): originator(PL), code(0x08), sub_code(0x0000)
[97158.087037] mpt2sas_cm0: log_info(0x31080000): originator(PL), code(0x08), sub_code(0x0000)
[97158.087060] sd 0:0:2:0: [sdc] tag#0 CDB: Read(16) 88 00 00 00 00 00 07 4a ec 00 00 00 01 00 00 00
[97158.087069] print_req_error: I/O error, dev sdc, sector 122350592
[97158.089156] sd 0:0:2:0: [sdc] tag#1 FAILED Result: hostbyte=DID_SOFT_ERROR driverbyte=DRIVER_OK
[97158.089169] sd 0:0:2:0: [sdc] tag#1 CDB: Read(16) 88 00 00 00 00 00 07 4a e8 00 00 00 04 00 00 00
[97158.089175] print_req_error: I/O error, dev sdc, sector 122349568
[97158.090591] sd 0:0:2:0: [sdc] tag#2 FAILED Result: hostbyte=DID_SOFT_ERROR driverbyte=DRIVER_OK
[97158.090602] sd 0:0:2:0: [sdc] tag#2 CDB: Read(16) 88 00 00 00 00 00 07 4a e4 00 00 00 04 00 00 00
[97158.090606] print_req_error: I/O error, dev sdc, sector 122348544
[97158.091662] sd 0:0:2:0: [sdc] tag#3 FAILED Result: hostbyte=DID_SOFT_ERROR driverbyte=DRIVER_OK
[97158.091672] sd 0:0:2:0: [sdc] tag#3 CDB: Read(16) 88 00 00 00 00 00 07 4a e0 00 00 00 04 00 00 00
[97158.091676] print_req_error: I/O error, dev sdc, sector 122347520
[97158.092720] sd 0:0:2:0: [sdc] tag#4 FAILED Result: hostbyte=DID_SOFT_ERROR driverbyte=DRIVER_OK
[97158.092733] sd 0:0:2:0: [sdc] tag#4 CDB: Read(16) 88 00 00 00 00 00 07 4a dc 00 00 00 04 00 00 00
[97158.092737] print_req_error: I/O error, dev sdc, sector 122346496
[97158.099294] sd 0:0:2:0: [sdc] tag#5 FAILED Result: hostbyte=DID_SOFT_ERROR driverbyte=DRIVER_OK
[97158.099302] sd 0:0:2:0: [sdc] tag#5 CDB: Read(16) 88 00 00 00 00 00 07 4a d8 00 00 00 04 00 00 00
[97158.099305] print_req_error: I/O error, dev sdc, sector 122345472
[97158.100394] sd 0:0:2:0: [sdc] tag#6 FAILED Result: hostbyte=DID_SOFT_ERROR driverbyte=DRIVER_OK
[97158.100399] sd 0:0:2:0: [sdc] tag#6 CDB: Read(16) 88 00 00 00 00 00 07 4a d4 00 00 00 04 00 00 00
[97158.100402] print_req_error: I/O error, dev sdc, sector 122344448
[97158.101577] sd 0:0:2:0: [sdc] tag#7 FAILED Result: hostbyte=DID_SOFT_ERROR driverbyte=DRIVER_OK
[97158.101583] sd 0:0:2:0: [sdc] tag#7 CDB: Read(16) 88 00 00 00 00 00 07 4a c4 00 00 00 04 00 00 00
[97158.101586] print_req_error: I/O error, dev sdc, sector 122340352
[97158.102597] sd 0:0:2:0: [sdc] tag#8 FAILED Result: hostbyte=DID_SOFT_ERROR driverbyte=DRIVER_OK
[97158.102603] sd 0:0:2:0: [sdc] tag#8 CDB: Read(16) 88 00 00 00 00 00 07 4a c0 00 00 00 04 00 00 00
[97158.102607] print_req_error: I/O error, dev sdc, sector 122339328
[97162.231458] scsi_io_completion: 9 callbacks suppressed
[97162.231498] sd 0:0:2:0: [sdc] tag#1 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE
[97162.231510] sd 0:0:2:0: [sdc] tag#1 Sense Key : Medium Error [current] 
[97162.231519] sd 0:0:2:0: [sdc] tag#1 Add. Sense: Unrecovered read error
[97162.231530] sd 0:0:2:0: [sdc] tag#1 CDB: Read(16) 88 00 00 00 00 00 07 4a ed 00 00 00 00 80 00 00
[97162.231537] print_req_error: 9 callbacks suppressed
[97162.231545] print_req_error: critical medium error, dev sdc, sector 122350848
[97166.331467] sd 0:0:2:0: [sdc] tag#1 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE
[97166.331494] sd 0:0:2:0: [sdc] tag#1 Sense Key : Medium Error [current] 
[97166.331503] sd 0:0:2:0: [sdc] tag#1 Add. Sense: Unrecovered read error
[97166.331513] sd 0:0:2:0: [sdc] tag#1 CDB: Read(16) 88 00 00 00 00 00 07 4a ed 80 00 00 00 80 00 00
[97166.331521] print_req_error: critical medium error, dev sdc, sector 122350976
[97170.320449] sd 0:0:2:0: [sdc] tag#1 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE
[97170.320487] sd 0:0:2:0: [sdc] tag#1 Sense Key : Medium Error [current] 
[97170.320498] sd 0:0:2:0: [sdc] tag#1 Add. Sense: Unrecovered read error
[97170.320511] sd 0:0:2:0: [sdc] tag#1 CDB: Read(16) 88 00 00 00 00 00 07 4a ee 00 00 00 00 80 00 00
[97170.320520] print_req_error: critical medium error, dev sdc, sector 122351104
[97174.387070] mpt2sas_cm0: log_info(0x31080000): originator(PL), code(0x08), sub_code(0x0000)
[97174.387080] mpt2sas_cm0: log_info(0x31080000): originator(PL), code(0x08), sub_code(0x0000)
[97174.387086] mpt2sas_cm0: log_info(0x31080000): originator(PL), code(0x08), sub_code(0x0000)
[97174.387092] mpt2sas_cm0: log_info(0x31080000): originator(PL), code(0x08), sub_code(0x0000)
[97174.387097] mpt2sas_cm0: log_info(0x31080000): originator(PL), code(0x08), sub_code(0x0000)
[97174.387102] mpt2sas_cm0: log_info(0x31080000): originator(PL), code(0x08), sub_code(0x0000)
[97174.387212] sd 0:0:2:0: [sdc] tag#2 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE
[97174.387222] sd 0:0:2:0: [sdc] tag#2 Sense Key : Medium Error [current] 
[97174.387228] sd 0:0:2:0: [sdc] tag#2 Add. Sense: Unrecovered read error
[97174.387236] sd 0:0:2:0: [sdc] tag#2 CDB: Read(16) 88 00 00 00 00 00 07 4a ee 80 00 00 04 00 00 00
[97174.387241] print_req_error: critical medium error, dev sdc, sector 122351232
[97178.364881] mpt2sas_cm0: log_info(0x31080000): originator(PL), code(0x08), sub_code(0x0000)
[97178.364890] mpt2sas_cm0: log_info(0x31080000): originator(PL), code(0x08), sub_code(0x0000)
[97178.364896] mpt2sas_cm0: log_info(0x31080000): originator(PL), code(0x08), sub_code(0x0000)
[97178.364901] mpt2sas_cm0: log_info(0x31080000): originator(PL), code(0x08), sub_code(0x0000)
[97178.364906] mpt2sas_cm0: log_info(0x31080000): originator(PL), code(0x08), sub_code(0x0000)
[97178.364992] sd 0:0:2:0: [sdc] tag#5 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE
[97178.365006] sd 0:0:2:0: [sdc] tag#5 Sense Key : Medium Error [current] 
[97178.365014] sd 0:0:2:0: [sdc] tag#5 Add. Sense: Unrecovered read error
[97178.365023] sd 0:0:2:0: [sdc] tag#5 CDB: Read(16) 88 00 00 00 00 00 07 4a f2 80 00 00 04 00 00 00
[97178.365030] print_req_error: critical medium error, dev sdc, sector 122352256
[97182.331607] mpt2sas_cm0: log_info(0x31080000): originator(PL), code(0x08), sub_code(0x0000)
[97182.331617] mpt2sas_cm0: log_info(0x31080000): originator(PL), code(0x08), sub_code(0x0000)
[97182.331624] mpt2sas_cm0: log_info(0x31080000): originator(PL), code(0x08), sub_code(0x0000)
[97182.331629] mpt2sas_cm0: log_info(0x31080000): originator(PL), code(0x08), sub_code(0x0000)
[97182.331723] sd 0:0:2:0: [sdc] tag#0 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE
[97182.331740] sd 0:0:2:0: [sdc] tag#0 Sense Key : Medium Error [current] 
[97182.331756] sd 0:0:2:0: [sdc] tag#0 Add. Sense: Unrecovered read error
[97182.331767] sd 0:0:2:0: [sdc] tag#0 CDB: Read(16) 88 00 00 00 00 00 07 4a f6 80 00 00 04 00 00 00
[97182.331774] print_req_error: critical medium error, dev sdc, sector 122353280
[97186.342745] mpt2sas_cm0: log_info(0x31080000): originator(PL), code(0x08), sub_code(0x0000)
[97186.342755] mpt2sas_cm0: log_info(0x31080000): originator(PL), code(0x08), sub_code(0x0000)
[97186.342761] mpt2sas_cm0: log_info(0x31080000): originator(PL), code(0x08), sub_code(0x0000)
[97186.342854] sd 0:0:2:0: [sdc] tag#3 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE
[97186.342870] sd 0:0:2:0: [sdc] tag#3 Sense Key : Medium Error [current] 
[97186.342878] sd 0:0:2:0: [sdc] tag#3 Add. Sense: Unrecovered read error
[97186.342888] sd 0:0:2:0: [sdc] tag#3 CDB: Read(16) 88 00 00 00 00 00 07 4a fa 80 00 00 04 00 00 00
[97186.342895] print_req_error: critical medium error, dev sdc, sector 122354304
[97190.742799] mpt2sas_cm0: log_info(0x31080000): originator(PL), code(0x08), sub_code(0x0000)
[97190.742809] mpt2sas_cm0: log_info(0x31080000): originator(PL), code(0x08), sub_code(0x0000)
[97190.742912] sd 0:0:2:0: [sdc] tag#0 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE
[97190.742929] sd 0:0:2:0: [sdc] tag#0 Sense Key : Medium Error [current] 
[97190.742939] sd 0:0:2:0: [sdc] tag#0 Add. Sense: Unrecovered read error
[97190.742950] sd 0:0:2:0: [sdc] tag#0 CDB: Read(16) 88 00 00 00 00 00 07 4a fe 80 00 00 04 00 00 00
[97190.742958] print_req_error: critical medium error, dev sdc, sector 122355328
[97194.731725] mpt2sas_cm0: log_info(0x31080000): originator(PL), code(0x08), sub_code(0x0000)
[97194.731816] sd 0:0:2:0: [sdc] tag#0 FAILED Result: hostbyte=DID_SOFT_ERROR driverbyte=DRIVER_OK
[97194.731837] sd 0:0:2:0: [sdc] tag#0 CDB: Read(16) 88 00 00 00 00 00 07 4b 06 80 00 00 02 80 00 00
[97194.731846] print_req_error: I/O error, dev sdc, sector 122357376
[97194.734371] sd 0:0:2:0: [sdc] tag#1 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE
[97194.734387] sd 0:0:2:0: [sdc] tag#1 Sense Key : Medium Error [current] 
[97194.734398] sd 0:0:2:0: [sdc] tag#1 Add. Sense: Unrecovered read error
[97194.734409] sd 0:0:2:0: [sdc] tag#1 CDB: Read(16) 88 00 00 00 00 00 07 4b 02 80 00 00 04 00 00 00
[97194.734416] print_req_error: critical medium error, dev sdc, sector 122356352
[97198.765146] sd 0:0:2:0: [sdc] tag#0 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE
[97198.765171] sd 0:0:2:0: [sdc] tag#0 Sense Key : Medium Error [current] 
[97198.765181] sd 0:0:2:0: [sdc] tag#0 Add. Sense: Unrecovered read error
[97198.765192] sd 0:0:2:0: [sdc] tag#0 CDB: Read(16) 88 00 00 00 00 00 07 4b 09 00 00 00 00 80 00 00
[97198.765200] print_req_error: critical medium error, dev sdc, sector 122358016
[97202.776330] sd 0:0:2:0: [sdc] tag#1 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE
[97202.776368] sd 0:0:2:0: [sdc] tag#1 Sense Key : Medium Error [current] 
[97202.776379] sd 0:0:2:0: [sdc] tag#1 Add. Sense: Unrecovered read error
[97202.776392] sd 0:0:2:0: [sdc] tag#1 CDB: Read(16) 88 00 00 00 00 00 07 4b 09 80 00 00 00 80 00 00
[97202.776402] print_req_error: critical medium error, dev sdc, sector 122358144
[97206.798565] sd 0:0:2:0: [sdc] tag#1 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE
[97206.798594] sd 0:0:2:0: [sdc] tag#1 Sense Key : Medium Error [current] 
[97206.798605] sd 0:0:2:0: [sdc] tag#1 Add. Sense: Unrecovered read error
[97206.798618] sd 0:0:2:0: [sdc] tag#1 CDB: Read(16) 88 00 00 00 00 00 07 4b 0a 00 00 00 00 80 00 00
[97206.798627] print_req_error: critical medium error, dev sdc, sector 122358272
[97210.820773] mpt2sas_cm0: log_info(0x31080000): originator(PL), code(0x08), sub_code(0x0000)
[97210.820783] mpt2sas_cm0: log_info(0x31080000): originator(PL), code(0x08), sub_code(0x0000)
[97210.820790] mpt2sas_cm0: log_info(0x31080000): originator(PL), code(0x08), sub_code(0x0000)
[97210.820796] mpt2sas_cm0: log_info(0x31080000): originator(PL), code(0x08), sub_code(0x0000)
[97210.820802] mpt2sas_cm0: log_info(0x31080000): originator(PL), code(0x08), sub_code(0x0000)
[97210.820808] mpt2sas_cm0: log_info(0x31080000): originator(PL), code(0x08), sub_code(0x0000)
[97210.820815] mpt2sas_cm0: log_info(0x31080000): originator(PL), code(0x08), sub_code(0x0000)
[97210.820821] mpt2sas_cm0: log_info(0x31080000): originator(PL), code(0x08), sub_code(0x0000)
[97210.820826] mpt2sas_cm0: log_info(0x31080000): originator(PL), code(0x08), sub_code(0x0000)
[97210.820832] mpt2sas_cm0: log_info(0x31080000): originator(PL), code(0x08), sub_code(0x0000)
[97210.820838] mpt2sas_cm0: log_info(0x31080000): originator(PL), code(0x08), sub_code(0x0000)
[97210.820844] mpt2sas_cm0: log_info(0x31080000): originator(PL), code(0x08), sub_code(0x0000)
[97210.820850] mpt2sas_cm0: log_info(0x31080000): originator(PL), code(0x08), sub_code(0x0000)
[97210.820858] mpt2sas_cm0: log_info(0x31080000): originator(PL), code(0x08), sub_code(0x0000)
[97210.820864] mpt2sas_cm0: log_info(0x31080000): originator(PL), code(0x08), sub_code(0x0000)
[97210.820869] mpt2sas_cm0: log_info(0x31080000): originator(PL), code(0x08), sub_code(0x0000)
[97210.820875] mpt2sas_cm0: log_info(0x31080000): originator(PL), code(0x08), sub_code(0x0000)
[97210.820882] mpt2sas_cm0: log_info(0x31080000): originator(PL), code(0x08), sub_code(0x0000)
[97210.820888] mpt2sas_cm0: log_info(0x31080000): originator(PL), code(0x08), sub_code(0x0000)
[97210.820894] mpt2sas_cm0: log_info(0x31080000): originator(PL), code(0x08), sub_code(0x0000)
[97210.820901] mpt2sas_cm0: log_info(0x31080000): originator(PL), code(0x08), sub_code(0x0000)
[97210.820908] mpt2sas_cm0: log_info(0x31080000): originator(PL), code(0x08), sub_code(0x0000)
[97210.820914] mpt2sas_cm0: log_info(0x31080000): originator(PL), code(0x08), sub_code(0x0000)
[97210.820920] mpt2sas_cm0: log_info(0x31080000): originator(PL), code(0x08), sub_code(0x0000)
[97210.820927] mpt2sas_cm0: log_info(0x31080000): originator(PL), code(0x08), sub_code(0x0000)
[97210.820934] mpt2sas_cm0: log_info(0x31080000): originator(PL), code(0x08), sub_code(0x0000)
[97210.820941] mpt2sas_cm0: log_info(0x31080000): originator(PL), code(0x08), sub_code(0x0000)
[97210.820948] mpt2sas_cm0: log_info(0x31080000): originator(PL), code(0x08), sub_code(0x0000)
[97210.820955] mpt2sas_cm0: log_info(0x31080000): originator(PL), code(0x08), sub_code(0x0000)
[97210.820961] mpt2sas_cm0: log_info(0x31080000): originator(PL), code(0x08), sub_code(0x0000)
[97210.820967] mpt2sas_cm0: log_info(0x31080000): originator(PL), code(0x08), sub_code(0x0000)
[97210.821081] sd 0:0:2:0: [sdc] tag#2 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE
[97210.821093] sd 0:0:2:0: [sdc] tag#2 Sense Key : Medium Error [current] 
[97210.821102] sd 0:0:2:0: [sdc] tag#2 Add. Sense: Unrecovered read error
[97210.821116] sd 0:0:2:0: [sdc] tag#2 CDB: Read(16) 88 00 00 00 00 00 07 4b 0a 80 00 00 04 00 00 00
[97210.821124] print_req_error: critical medium error, dev sdc, sector 122358400
[97214.787475] mpt2sas_cm0: log_info(0x31080000): originator(PL), code(0x08), sub_code(0x0000)
[97214.787486] mpt2sas_cm0: log_info(0x31080000): originator(PL), code(0x08), sub_code(0x0000)
[97214.787492] mpt2sas_cm0: log_info(0x31080000): originator(PL), code(0x08), sub_code(0x0000)
[97214.787498] mpt2sas_cm0: log_info(0x31080000): originator(PL), code(0x08), sub_code(0x0000)
[97214.787504] mpt2sas_cm0: log_info(0x31080000): originator(PL), code(0x08), sub_code(0x0000)
[97214.787510] mpt2sas_cm0: log_info(0x31080000): originator(PL), code(0x08), sub_code(0x0000)
[97214.787517] mpt2sas_cm0: log_info(0x31080000): originator(PL), code(0x08), sub_code(0x0000)
[97214.787523] mpt2sas_cm0: log_info(0x31080000): originator(PL), code(0x08), sub_code(0x0000)
[97214.787529] mpt2sas_cm0: log_info(0x31080000): originator(PL), code(0x08), sub_code(0x0000)
[97214.787535] mpt2sas_cm0: log_info(0x31080000): originator(PL), code(0x08), sub_code(0x0000)
[97214.787542] mpt2sas_cm0: log_info(0x31080000): originator(PL), code(0x08), sub_code(0x0000)
[97214.787547] mpt2sas_cm0: log_info(0x31080000): originator(PL), code(0x08), sub_code(0x0000)
[97214.787555] mpt2sas_cm0: log_info(0x31080000): originator(PL), code(0x08), sub_code(0x0000)
[97214.787562] mpt2sas_cm0: log_info(0x31080000): originator(PL), code(0x08), sub_code(0x0000)
[97214.787568] mpt2sas_cm0: log_info(0x31080000): originator(PL), code(0x08), sub_code(0x0000)
[97214.787574] mpt2sas_cm0: log_info(0x31080000): originator(PL), code(0x08), sub_code(0x0000)
[97214.787581] mpt2sas_cm0: log_info(0x31080000): originator(PL), code(0x08), sub_code(0x0000)
[97214.787587] mpt2sas_cm0: log_info(0x31080000): originator(PL), code(0x08), sub_code(0x0000)
[97214.787593] mpt2sas_cm0: log_info(0x31080000): originator(PL), code(0x08), sub_code(0x0000)
[97214.787600] mpt2sas_cm0: log_info(0x31080000): originator(PL), code(0x08), sub_code(0x0000)
[97214.787607] mpt2sas_cm0: log_info(0x31080000): originator(PL), code(0x08), sub_code(0x0000)
[97214.787613] mpt2sas_cm0: log_info(0x31080000): originator(PL), code(0x08), sub_code(0x0000)
[97214.787619] mpt2sas_cm0: log_info(0x31080000): originator(PL), code(0x08), sub_code(0x0000)
[97214.787626] mpt2sas_cm0: log_info(0x31080000): originator(PL), code(0x08), sub_code(0x0000)
[97214.787632] mpt2sas_cm0: log_info(0x31080000): originator(PL), code(0x08), sub_code(0x0000)
[97214.787639] mpt2sas_cm0: log_info(0x31080000): originator(PL), code(0x08), sub_code(0x0000)
[97214.787646] mpt2sas_cm0: log_info(0x31080000): originator(PL), code(0x08), sub_code(0x0000)
[97214.787652] mpt2sas_cm0: log_info(0x31080000): originator(PL), code(0x08), sub_code(0x0000)
[97214.787659] mpt2sas_cm0: log_info(0x31080000): originator(PL), code(0x08), sub_code(0x0000)
[97214.787665] mpt2sas_cm0: log_info(0x31080000): originator(PL), code(0x08), sub_code(0x0000)
[97214.787672] mpt2sas_cm0: log_info(0x31080000): originator(PL), code(0x08), sub_code(0x0000)
[97214.787790] sd 0:0:2:0: [sdc] tag#12 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE
[97214.787802] sd 0:0:2:0: [sdc] tag#12 Sense Key : Medium Error [current] 
[97214.787810] sd 0:0:2:0: [sdc] tag#12 Add. Sense: Unrecovered read error
[97214.787822] sd 0:0:2:0: [sdc] tag#12 CDB: Read(16) 88 00 00 00 00 00 07 4b 0e 80 00 00 04 00 00 00
[97214.787834] print_req_error: critical medium error, dev sdc, sector 122359424
[97218.776405] mpt2sas_cm0: log_info(0x31080000): originator(PL), code(0x08), sub_code(0x0000)
[97218.776415] mpt2sas_cm0: log_info(0x31080000): originator(PL), code(0x08), sub_code(0x0000)
[97218.776421] mpt2sas_cm0: log_info(0x31080000): originator(PL), code(0x08), sub_code(0x0000)
[97218.776426] mpt2sas_cm0: log_info(0x31080000): originator(PL), code(0x08), sub_code(0x0000)
[97218.776432] mpt2sas_cm0: log_info(0x31080000): originator(PL), code(0x08), sub_code(0x0000)
[97218.776438] mpt2sas_cm0: log_info(0x31080000): originator(PL), code(0x08), sub_code(0x0000)
[97218.776443] mpt2sas_cm0: log_info(0x31080000): originator(PL), code(0x08), sub_code(0x0000)
[97218.776449] mpt2sas_cm0: log_info(0x31080000): originator(PL), code(0x08), sub_code(0x0000)
[97218.776454] mpt2sas_cm0: log_info(0x31080000): originator(PL), code(0x08), sub_code(0x0000)
[97218.776459] mpt2sas_cm0: log_info(0x31080000): originator(PL), code(0x08), sub_code(0x0000)
[97218.776466] mpt2sas_cm0: log_info(0x31080000): originator(PL), code(0x08), sub_code(0x0000)
[97218.776473] mpt2sas_cm0: log_info(0x31080000): originator(PL), code(0x08), sub_code(0x0000)
[97218.776479] mpt2sas_cm0: log_info(0x31080000): originator(PL), code(0x08), sub_code(0x0000)
[97218.776484] mpt2sas_cm0: log_info(0x31080000): originator(PL), code(0x08), sub_code(0x0000)
[97218.776489] mpt2sas_cm0: log_info(0x31080000): originator(PL), code(0x08), sub_code(0x0000)
[97218.776495] mpt2sas_cm0: log_info(0x31080000): originator(PL), code(0x08), sub_code(0x0000)
[97218.776500] mpt2sas_cm0: log_info(0x31080000): originator(PL), code(0x08), sub_code(0x0000)
[97218.776506] mpt2sas_cm0: log_info(0x31080000): originator(PL), code(0x08), sub_code(0x0000)
[97218.776513] mpt2sas_cm0: log_info(0x31080000): originator(PL), code(0x08), sub_code(0x0000)
[97218.776518] mpt2sas_cm0: log_info(0x31080000): originator(PL), code(0x08), sub_code(0x0000)
[97218.776525] mpt2sas_cm0: log_info(0x31080000): originator(PL), code(0x08), sub_code(0x0000)
[97218.776531] mpt2sas_cm0: log_info(0x31080000): originator(PL), code(0x08), sub_code(0x0000)
[97218.776536] mpt2sas_cm0: log_info(0x31080000): originator(PL), code(0x08), sub_code(0x0000)
[97218.776542] mpt2sas_cm0: log_info(0x31080000): originator(PL), code(0x08), sub_code(0x0000)
[97218.776549] mpt2sas_cm0: log_info(0x31080000): originator(PL), code(0x08), sub_code(0x0000)
[97218.776555] mpt2sas_cm0: log_info(0x31080000): originator(PL), code(0x08), sub_code(0x0000)
[97218.776561] mpt2sas_cm0: log_info(0x31080000): originator(PL), code(0x08), sub_code(0x0000)
[97218.776607] mpt2sas_cm0: log_info(0x31080000): originator(PL), code(0x08), sub_code(0x0000)
[97218.776613] mpt2sas_cm0: log_info(0x31080000): originator(PL), code(0x08), sub_code(0x0000)
[97218.776619] mpt2sas_cm0: log_info(0x31080000): originator(PL), code(0x08), sub_code(0x0000)
[97218.776626] mpt2sas_cm0: log_info(0x31080000): originator(PL), code(0x08), sub_code(0x0000)
[97218.776710] sd 0:0:2:0: [sdc] tag#0 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE
[97218.776721] sd 0:0:2:0: [sdc] tag#0 Sense Key : Medium Error [current] 
[97218.776738] sd 0:0:2:0: [sdc] tag#0 Add. Sense: Unrecovered read error
[97218.776757] sd 0:0:2:0: [sdc] tag#0 CDB: Read(16) 88 00 00 00 00 00 07 4b 12 80 00 00 04 00 00 00
[97218.776765] print_req_error: critical medium error, dev sdc, sector 122360448
[97222.776431] mpt2sas_cm0: log_info(0x31080000): originator(PL), code(0x08), sub_code(0x0000)
[97222.776442] mpt2sas_cm0: log_info(0x31080000): originator(PL), code(0x08), sub_code(0x0000)
[97222.776449] mpt2sas_cm0: log_info(0x31080000): originator(PL), code(0x08), sub_code(0x0000)
[97222.776455] mpt2sas_cm0: log_info(0x31080000): originator(PL), code(0x08), sub_code(0x0000)
[97222.776460] mpt2sas_cm0: log_info(0x31080000): originator(PL), code(0x08), sub_code(0x0000)
[97222.776466] mpt2sas_cm0: log_info(0x31080000): originator(PL), code(0x08), sub_code(0x0000)
[97222.776473] mpt2sas_cm0: log_info(0x31080000): originator(PL), code(0x08), sub_code(0x0000)
[97222.776480] mpt2sas_cm0: log_info(0x31080000): originator(PL), code(0x08), sub_code(0x0000)
[97222.776486] mpt2sas_cm0: log_info(0x31080000): originator(PL), code(0x08), sub_code(0x0000)
[97222.776492] mpt2sas_cm0: log_info(0x31080000): originator(PL), code(0x08), sub_code(0x0000)
[97222.776499] mpt2sas_cm0: log_info(0x31080000): originator(PL), code(0x08), sub_code(0x0000)
[97222.776507] mpt2sas_cm0: log_info(0x31080000): originator(PL), code(0x08), sub_code(0x0000)
[97222.776513] mpt2sas_cm0: log_info(0x31080000): originator(PL), code(0x08), sub_code(0x0000)
[97222.776519] mpt2sas_cm0: log_info(0x31080000): originator(PL), code(0x08), sub_code(0x0000)
[97222.776526] mpt2sas_cm0: log_info(0x31080000): originator(PL), code(0x08), sub_code(0x0000)
[97222.776532] mpt2sas_cm0: log_info(0x31080000): originator(PL), code(0x08), sub_code(0x0000)
[97222.776539] mpt2sas_cm0: log_info(0x31080000): originator(PL), code(0x08), sub_code(0x0000)
[97222.776545] mpt2sas_cm0: log_info(0x31080000): originator(PL), code(0x08), sub_code(0x0000)
[97222.776552] mpt2sas_cm0: log_info(0x31080000): originator(PL), code(0x08), sub_code(0x0000)
[97222.776558] mpt2sas_cm0: log_info(0x31080000): originator(PL), code(0x08), sub_code(0x0000)
[97222.776565] mpt2sas_cm0: log_info(0x31080000): originator(PL), code(0x08), sub_code(0x0000)
[97222.776571] mpt2sas_cm0: log_info(0x31080000): originator(PL), code(0x08), sub_code(0x0000)
[97222.776578] mpt2sas_cm0: log_info(0x31080000): originator(PL), code(0x08), sub_code(0x0000)
[97222.776585] mpt2sas_cm0: log_info(0x31080000): originator(PL), code(0x08), sub_code(0x0000)
[97222.776591] mpt2sas_cm0: log_info(0x31080000): originator(PL), code(0x08), sub_code(0x0000)
[97222.776598] mpt2sas_cm0: log_info(0x31080000): originator(PL), code(0x08), sub_code(0x0000)
[97222.776604] mpt2sas_cm0: log_info(0x31080000): originator(PL), code(0x08), sub_code(0x0000)
[97222.776611] mpt2sas_cm0: log_info(0x31080000): originator(PL), code(0x08), sub_code(0x0000)
[97222.776617] mpt2sas_cm0: log_info(0x31080000): originator(PL), code(0x08), sub_code(0x0000)
[97222.776705] sd 0:0:2:0: [sdc] tag#14 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE
[97222.776716] sd 0:0:2:0: [sdc] tag#14 Sense Key : Medium Error [current] 
[97222.776723] sd 0:0:2:0: [sdc] tag#14 Add. Sense: Unrecovered read error
[97222.776752] sd 0:0:2:0: [sdc] tag#14 CDB: Read(16) 88 00 00 00 00 00 07 4b 7e 80 00 00 04 00 00 00
[97222.776757] print_req_error: critical medium error, dev sdc, sector 122388096
[97226.798698] mpt2sas_cm0: log_info(0x31080000): originator(PL), code(0x08), sub_code(0x0000)
[97226.798708] mpt2sas_cm0: log_info(0x31080000): originator(PL), code(0x08), sub_code(0x0000)
[97226.798714] mpt2sas_cm0: log_info(0x31080000): originator(PL), code(0x08), sub_code(0x0000)
[97226.798720] mpt2sas_cm0: log_info(0x31080000): originator(PL), code(0x08), sub_code(0x0000)
[97226.798725] mpt2sas_cm0: log_info(0x31080000): originator(PL), code(0x08), sub_code(0x0000)
[97226.798730] mpt2sas_cm0: log_info(0x31080000): originator(PL), code(0x08), sub_code(0x0000)
[97226.798735] mpt2sas_cm0: log_info(0x31080000): originator(PL), code(0x08), sub_code(0x0000)
[97226.798741] mpt2sas_cm0: log_info(0x31080000): originator(PL), code(0x08), sub_code(0x0000)
[97226.798747] mpt2sas_cm0: log_info(0x31080000): originator(PL), code(0x08), sub_code(0x0000)
[97226.798753] mpt2sas_cm0: log_info(0x31080000): originator(PL), code(0x08), sub_code(0x0000)
[97226.798760] mpt2sas_cm0: log_info(0x31080000): originator(PL), code(0x08), sub_code(0x0000)
[97226.798766] mpt2sas_cm0: log_info(0x31080000): originator(PL), code(0x08), sub_code(0x0000)
[97226.798771] mpt2sas_cm0: log_info(0x31080000): originator(PL), code(0x08), sub_code(0x0000)
[97226.798777] mpt2sas_cm0: log_info(0x31080000): originator(PL), code(0x08), sub_code(0x0000)
[97226.798782] mpt2sas_cm0: log_info(0x31080000): originator(PL), code(0x08), sub_code(0x0000)
[97226.798787] mpt2sas_cm0: log_info(0x31080000): originator(PL), code(0x08), sub_code(0x0000)
[97226.798793] mpt2sas_cm0: log_info(0x31080000): originator(PL), code(0x08), sub_code(0x0000)
[97226.798799] mpt2sas_cm0: log_info(0x31080000): originator(PL), code(0x08), sub_code(0x0000)
[97226.798805] mpt2sas_cm0: log_info(0x31080000): originator(PL), code(0x08), sub_code(0x0000)
[97226.798811] mpt2sas_cm0: log_info(0x31080000): originator(PL), code(0x08), sub_code(0x0000)
[97226.798817] mpt2sas_cm0: log_info(0x31080000): originator(PL), code(0x08), sub_code(0x0000)
[97226.798823] mpt2sas_cm0: log_info(0x31080000): originator(PL), code(0x08), sub_code(0x0000)
[97226.798828] mpt2sas_cm0: log_info(0x31080000): originator(PL), code(0x08), sub_code(0x0000)
[97226.798834] mpt2sas_cm0: log_info(0x31080000): originator(PL), code(0x08), sub_code(0x0000)
[97226.798841] mpt2sas_cm0: log_info(0x31080000): originator(PL), code(0x08), sub_code(0x0000)
[97226.798848] mpt2sas_cm0: log_info(0x31080000): originator(PL), code(0x08), sub_code(0x0000)
[97226.798853] mpt2sas_cm0: log_info(0x31080000): originator(PL), code(0x08), sub_code(0x0000)
[97226.798893] mpt2sas_cm0: log_info(0x31080000): originator(PL), code(0x08), sub_code(0x0000)
[97226.798978] sd 0:0:2:0: [sdc] tag#0 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE
[97226.798989] sd 0:0:2:0: [sdc] tag#0 Sense Key : Medium Error [current] 
[97226.798997] sd 0:0:2:0: [sdc] tag#0 Add. Sense: Unrecovered read error
[97226.799007] sd 0:0:2:0: [sdc] tag#0 CDB: Read(16) 88 00 00 00 00 00 07 4b 16 80 00 00 04 00 00 00
[97226.799014] print_req_error: critical medium error, dev sdc, sector 122361472
[97230.787640] mpt2sas_cm0: log_info(0x31080000): originator(PL), code(0x08), sub_code(0x0000)
[97230.787649] mpt2sas_cm0: log_info(0x31080000): originator(PL), code(0x08), sub_code(0x0000)
[97230.787654] mpt2sas_cm0: log_info(0x31080000): originator(PL), code(0x08), sub_code(0x0000)
[97230.787659] mpt2sas_cm0: log_info(0x31080000): originator(PL), code(0x08), sub_code(0x0000)
[97230.787663] mpt2sas_cm0: log_info(0x31080000): originator(PL), code(0x08), sub_code(0x0000)
[97230.787668] mpt2sas_cm0: log_info(0x31080000): originator(PL), code(0x08), sub_code(0x0000)
[97230.787673] mpt2sas_cm0: log_info(0x31080000): originator(PL), code(0x08), sub_code(0x0000)
[97230.787677] mpt2sas_cm0: log_info(0x31080000): originator(PL), code(0x08), sub_code(0x0000)
[97230.787682] mpt2sas_cm0: log_info(0x31080000): originator(PL), code(0x08), sub_code(0x0000)
[97230.787687] mpt2sas_cm0: log_info(0x31080000): originator(PL), code(0x08), sub_code(0x0000)
[97230.787693] mpt2sas_cm0: log_info(0x31080000): originator(PL), code(0x08), sub_code(0x0000)
[97230.787698] mpt2sas_cm0: log_info(0x31080000): originator(PL), code(0x08), sub_code(0x0000)
[97230.787704] mpt2sas_cm0: log_info(0x31080000): originator(PL), code(0x08), sub_code(0x0000)
[97230.787709] mpt2sas_cm0: log_info(0x31080000): originator(PL), code(0x08), sub_code(0x0000)
[97230.787713] mpt2sas_cm0: log_info(0x31080000): originator(PL), code(0x08), sub_code(0x0000)
[97230.787718] mpt2sas_cm0: log_info(0x31080000): originator(PL), code(0x08), sub_code(0x0000)
[97230.787723] mpt2sas_cm0: log_info(0x31080000): originator(PL), code(0x08), sub_code(0x0000)
[97230.787728] mpt2sas_cm0: log_info(0x31080000): originator(PL), code(0x08), sub_code(0x0000)
[97230.787733] mpt2sas_cm0: log_info(0x31080000): originator(PL), code(0x08), sub_code(0x0000)
[97230.787739] mpt2sas_cm0: log_info(0x31080000): originator(PL), code(0x08), sub_code(0x0000)
[97230.787744] mpt2sas_cm0: log_info(0x31080000): originator(PL), code(0x08), sub_code(0x0000)
[97230.787803] mpt2sas_cm0: log_info(0x31080000): originator(PL), code(0x08), sub_code(0x0000)
[97230.787809] mpt2sas_cm0: log_info(0x31080000): originator(PL), code(0x08), sub_code(0x0000)
[97230.787815] mpt2sas_cm0: log_info(0x31080000): originator(PL), code(0x08), sub_code(0x0000)
[97230.787821] mpt2sas_cm0: log_info(0x31080000): originator(PL), code(0x08), sub_code(0x0000)
[97230.787827] mpt2sas_cm0: log_info(0x31080000): originator(PL), code(0x08), sub_code(0x0000)
[97230.787833] mpt2sas_cm0: log_info(0x31080000): originator(PL), code(0x08), sub_code(0x0000)
[97230.787846] sd 0:0:2:0: [sdc] tag#11 FAILED Result: hostbyte=DID_SOFT_ERROR driverbyte=DRIVER_OK
[97230.787863] sd 0:0:2:0: [sdc] tag#11 CDB: Read(16) 88 00 00 00 00 00 07 4b 86 80 00 00 03 80 00 00
[97230.787871] print_req_error: I/O error, dev sdc, sector 122390144
[97230.787892] sd 0:0:2:0: [sdc] tag#0 FAILED Result: hostbyte=DID_SOFT_ERROR driverbyte=DRIVER_OK
[97230.789763] sd 0:0:2:0: [sdc] tag#0 CDB: Read(16) 88 00 00 00 00 00 07 4b 3e 80 00 00 04 00 00 00
[97230.789771] print_req_error: I/O error, dev sdc, sector 122371712
[97230.790066] sd 0:0:2:0: [sdc] tag#14 FAILED Result: hostbyte=DID_SOFT_ERROR driverbyte=DRIVER_OK
[97230.791610] sd 0:0:2:0: [sdc] tag#14 CDB: Read(16) 88 00 00 00 00 00 07 4b 6e 80 00 00 04 00 00 00
[97230.791616] print_req_error: I/O error, dev sdc, sector 122384000
[97230.791919] sd 0:0:2:0: [sdc] tag#1 FAILED Result: hostbyte=DID_SOFT_ERROR driverbyte=DRIVER_OK
[97230.793464] sd 0:0:2:0: [sdc] tag#1 CDB: Read(16) 88 00 00 00 00 00 07 4b 3a 80 00 00 04 00 00 00
[97230.793469] print_req_error: I/O error, dev sdc, sector 122370688
[97230.793728] sd 0:0:2:0: [sdc] tag#15 FAILED Result: hostbyte=DID_SOFT_ERROR driverbyte=DRIVER_OK
[97230.795297] sd 0:0:2:0: [sdc] tag#15 CDB: Read(16) 88 00 00 00 00 00 07 4b 6a 80 00 00 04 00 00 00
[97230.795303] print_req_error: I/O error, dev sdc, sector 122382976
[97230.795566] sd 0:0:2:0: [sdc] tag#2 FAILED Result: hostbyte=DID_SOFT_ERROR driverbyte=DRIVER_OK
[97230.797130] sd 0:0:2:0: [sdc] tag#2 CDB: Read(16) 88 00 00 00 00 00 07 4b 36 80 00 00 04 00 00 00
[97230.797136] print_req_error: I/O error, dev sdc, sector 122369664
[97230.797390] sd 0:0:2:0: [sdc] tag#16 FAILED Result: hostbyte=DID_SOFT_ERROR driverbyte=DRIVER_OK
[97230.798965] sd 0:0:2:0: [sdc] tag#16 CDB: Read(16) 88 00 00 00 00 00 07 4b 66 80 00 00 04 00 00 00
[97230.798971] print_req_error: I/O error, dev sdc, sector 122381952
[97230.799212] sd 0:0:2:0: [sdc] tag#3 FAILED Result: hostbyte=DID_SOFT_ERROR driverbyte=DRIVER_OK
[97230.800762] sd 0:0:2:0: [sdc] tag#3 CDB: Read(16) 88 00 00 00 00 00 07 4b 32 80 00 00 04 00 00 00
[97230.800768] print_req_error: I/O error, dev sdc, sector 122368640
[97230.801033] sd 0:0:2:0: [sdc] tag#17 FAILED Result: hostbyte=DID_SOFT_ERROR driverbyte=DRIVER_OK
[97230.802515] sd 0:0:2:0: [sdc] tag#17 CDB: Read(16) 88 00 00 00 00 00 07 4b 62 80 00 00 04 00 00 00
[97230.802520] print_req_error: I/O error, dev sdc, sector 122380928
[97234.732135] mpt2sas_cm0: log_info(0x31080000): originator(PL), code(0x08), sub_code(0x0000)
[97234.732146] mpt2sas_cm0: log_info(0x31080000): originator(PL), code(0x08), sub_code(0x0000)
[97234.732153] mpt2sas_cm0: log_info(0x31080000): originator(PL), code(0x08), sub_code(0x0000)
[97234.732222] scsi_io_completion: 15 callbacks suppressed
[97234.732248] sd 0:0:2:0: [sdc] tag#2 FAILED Result: hostbyte=DID_SOFT_ERROR driverbyte=DRIVER_OK
[97234.732260] sd 0:0:2:0: [sdc] tag#2 CDB: Read(16) 88 00 00 00 00 00 07 4b 8a 00 00 00 02 00 00 00
[97234.732272] print_req_error: 15 callbacks suppressed
[97234.732280] print_req_error: I/O error, dev sdc, sector 122391040
[97234.734523] sd 0:0:2:0: [sdc] tag#10 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE
[97234.734538] sd 0:0:2:0: [sdc] tag#10 Sense Key : Medium Error [current] 
[97234.734547] sd 0:0:2:0: [sdc] tag#10 Add. Sense: Unrecovered read error
[97234.734556] sd 0:0:2:0: [sdc] tag#10 CDB: Read(16) 88 00 00 00 00 00 07 4b 8c 00 00 00 04 00 00 00
[97234.734563] print_req_error: critical medium error, dev sdc, sector 122391552
[97241.727617] mpt2sas_cm0: log_info(0x31080000): originator(PL), code(0x08), sub_code(0x0000)
[97241.727734] sd 0:0:2:0: [sdc] tag#0 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE
[97241.727756] sd 0:0:2:0: [sdc] tag#0 Sense Key : Medium Error [current] 
[97241.727765] sd 0:0:2:0: [sdc] tag#0 Add. Sense: Unrecovered read error
[97241.727776] sd 0:0:2:0: [sdc] tag#0 CDB: Read(16) 88 00 00 00 00 00 07 4b 90 00 00 00 04 00 00 00
[97241.727784] print_req_error: critical medium error, dev sdc, sector 122392576
[97245.732322] sd 0:0:2:0: [sdc] tag#0 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE
[97245.732354] sd 0:0:2:0: [sdc] tag#0 Sense Key : Medium Error [current] 
[97245.732363] sd 0:0:2:0: [sdc] tag#0 Add. Sense: Unrecovered read error
[97245.732374] sd 0:0:2:0: [sdc] tag#0 CDB: Read(16) 88 00 00 00 00 00 07 4b 94 00 00 00 04 00 00 00
[97245.732381] print_req_error: critical medium error, dev sdc, sector 122393600
[97249.932356] sd 0:0:2:0: [sdc] tag#0 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE
[97249.932390] sd 0:0:2:0: [sdc] tag#0 Sense Key : Medium Error [current] 
[97249.932401] sd 0:0:2:0: [sdc] tag#0 Add. Sense: Unrecovered read error
[97249.932414] sd 0:0:2:0: [sdc] tag#0 CDB: Read(16) 88 00 00 00 00 00 07 4b 98 00 00 00 01 00 00 00
[97249.932423] print_req_error: critical medium error, dev sdc, sector 122394624
[97253.999021] sd 0:0:2:0: [sdc] tag#1 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE
[97253.999050] sd 0:0:2:0: [sdc] tag#1 Sense Key : Medium Error [current] 
[97253.999059] sd 0:0:2:0: [sdc] tag#1 Add. Sense: Unrecovered read error
[97253.999069] sd 0:0:2:0: [sdc] tag#1 CDB: Read(16) 88 00 00 00 00 00 07 4b 9a 00 00 00 01 00 00 00
[97253.999077] print_req_error: critical medium error, dev sdc, sector 122395136
[97258.032414] sd 0:0:2:0: [sdc] tag#1 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE
[97258.032443] sd 0:0:2:0: [sdc] tag#1 Sense Key : Medium Error [current] 
[97258.032454] sd 0:0:2:0: [sdc] tag#1 Add. Sense: Unrecovered read error
[97258.032466] sd 0:0:2:0: [sdc] tag#1 CDB: Read(16) 88 00 00 00 00 00 07 4b 9b 00 00 00 00 80 00 00
[97258.032476] print_req_error: critical medium error, dev sdc, sector 122395392
[97262.143571] sd 0:0:2:0: [sdc] tag#1 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE
[97262.143608] sd 0:0:2:0: [sdc] tag#1 Sense Key : Medium Error [current] 
[97262.143619] sd 0:0:2:0: [sdc] tag#1 Add. Sense: Unrecovered read error
[97262.143632] sd 0:0:2:0: [sdc] tag#1 CDB: Read(16) 88 00 00 00 00 00 07 4b 9b 80 00 00 00 80 00 00
[97262.143642] print_req_error: critical medium error, dev sdc, sector 122395520
[97266.210210] mpt2sas_cm0: log_info(0x31080000): originator(PL), code(0x08), sub_code(0x0000)
[97266.210312] sd 0:0:2:0: [sdc] tag#2 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE
[97266.210327] sd 0:0:2:0: [sdc] tag#2 Sense Key : Medium Error [current] 
[97266.210335] sd 0:0:2:0: [sdc] tag#2 Add. Sense: Unrecovered read error
[97266.210345] sd 0:0:2:0: [sdc] tag#2 CDB: Read(16) 88 00 00 00 00 00 07 4b 9c 00 00 00 04 00 00 00
[97266.210352] print_req_error: critical medium error, dev sdc, sector 122395648
[97270.232559] sd 0:0:2:0: [sdc] tag#0 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE
[97270.232589] sd 0:0:2:0: [sdc] tag#0 Sense Key : Medium Error [current] 
[97270.232597] sd 0:0:2:0: [sdc] tag#0 Add. Sense: Unrecovered read error
[97270.232607] sd 0:0:2:0: [sdc] tag#0 CDB: Read(16) 88 00 00 00 00 00 07 4b a0 00 00 00 04 00 00 00
[97270.232614] print_req_error: critical medium error, dev sdc, sector 122396672
[97274.321484] sd 0:0:2:0: [sdc] tag#1 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE
[97274.321517] sd 0:0:2:0: [sdc] tag#1 Sense Key : Medium Error [current] 
[97274.321528] sd 0:0:2:0: [sdc] tag#1 Add. Sense: Unrecovered read error
[97274.321541] sd 0:0:2:0: [sdc] tag#1 CDB: Read(16) 88 00 00 00 00 00 07 4b a4 80 00 00 00 80 00 00
[97274.321551] print_req_error: critical medium error, dev sdc, sector 122397824
[97278.343740] sd 0:0:2:0: [sdc] tag#0 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE
[97278.343769] sd 0:0:2:0: [sdc] tag#0 Sense Key : Medium Error [current] 
[97278.343779] sd 0:0:2:0: [sdc] tag#0 Add. Sense: Unrecovered read error
[97278.343790] sd 0:0:2:0: [sdc] tag#0 CDB: Read(16) 88 00 00 00 00 00 07 4b a5 00 00 00 00 80 00 00
[97278.343799] print_req_error: critical medium error, dev sdc, sector 122397952
[97282.332680] sd 0:0:2:0: [sdc] tag#1 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE
[97282.332713] sd 0:0:2:0: [sdc] tag#1 Sense Key : Medium Error [current] 
[97282.332740] sd 0:0:2:0: [sdc] tag#1 Add. Sense: Unrecovered read error
[97282.332753] sd 0:0:2:0: [sdc] tag#1 CDB: Read(16) 88 00 00 00 00 00 07 4b a5 80 00 00 00 80 00 00
[97282.332762] print_req_error: critical medium error, dev sdc, sector 122398080
[97289.378318] sd 0:0:2:0: [sdc] tag#1 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE
[97289.378342] sd 0:0:2:0: [sdc] tag#1 Sense Key : Medium Error [current] 
[97289.378350] sd 0:0:2:0: [sdc] tag#1 Add. Sense: Unrecovered read error
[97289.378360] sd 0:0:2:0: [sdc] tag#1 CDB: Read(16) 88 00 00 00 00 00 07 4b a6 00 00 00 00 80 00 00
[97289.378367] print_req_error: critical medium error, dev sdc, sector 122398208
[97293.399404] sd 0:0:2:0: [sdc] tag#1 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE
[97293.399430] sd 0:0:2:0: [sdc] tag#1 Sense Key : Medium Error [current] 
[97293.399439] sd 0:0:2:0: [sdc] tag#1 Add. Sense: Unrecovered read error
[97293.399450] sd 0:0:2:0: [sdc] tag#1 CDB: Read(16) 88 00 00 00 00 00 07 4b a6 80 00 00 00 80 00 00
[97293.399457] print_req_error: critical medium error, dev sdc, sector 122398336
[97297.410558] sd 0:0:2:0: [sdc] tag#1 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE
[97297.410579] sd 0:0:2:0: [sdc] tag#1 Sense Key : Medium Error [current] 
[97297.410588] sd 0:0:2:0: [sdc] tag#1 Add. Sense: Unrecovered read error
[97297.410598] sd 0:0:2:0: [sdc] tag#1 CDB: Read(16) 88 00 00 00 00 00 07 4b a7 00 00 00 00 80 00 00
[97297.410606] print_req_error: critical medium error, dev sdc, sector 122398464
[97301.399437] mpt2sas_cm0: log_info(0x31080000): originator(PL), code(0x08), sub_code(0x0000)
[97301.399447] mpt2sas_cm0: log_info(0x31080000): originator(PL), code(0x08), sub_code(0x0000)
[97301.399454] mpt2sas_cm0: log_info(0x31080000): originator(PL), code(0x08), sub_code(0x0000)
[97301.399460] mpt2sas_cm0: log_info(0x31080000): originator(PL), code(0x08), sub_code(0x0000)
[97301.399465] mpt2sas_cm0: log_info(0x31080000): originator(PL), code(0x08), sub_code(0x0000)
[97301.399471] mpt2sas_cm0: log_info(0x31080000): originator(PL), code(0x08), sub_code(0x0000)
[97301.399476] mpt2sas_cm0: log_info(0x31080000): originator(PL), code(0x08), sub_code(0x0000)
[97301.399482] mpt2sas_cm0: log_info(0x31080000): originator(PL), code(0x08), sub_code(0x0000)
[97301.399487] mpt2sas_cm0: log_info(0x31080000): originator(PL), code(0x08), sub_code(0x0000)
[97301.399493] mpt2sas_cm0: log_info(0x31080000): originator(PL), code(0x08), sub_code(0x0000)
[97301.399498] mpt2sas_cm0: log_info(0x31080000): originator(PL), code(0x08), sub_code(0x0000)
[97301.399504] mpt2sas_cm0: log_info(0x31080000): originator(PL), code(0x08), sub_code(0x0000)
[97301.399509] mpt2sas_cm0: log_info(0x31080000): originator(PL), code(0x08), sub_code(0x0000)
[97301.399514] mpt2sas_cm0: log_info(0x31080000): originator(PL), code(0x08), sub_code(0x0000)
[97301.399521] mpt2sas_cm0: log_info(0x31080000): originator(PL), code(0x08), sub_code(0x0000)
[97301.399527] mpt2sas_cm0: log_info(0x31080000): originator(PL), code(0x08), sub_code(0x0000)
[97301.399533] mpt2sas_cm0: log_info(0x31080000): originator(PL), code(0x08), sub_code(0x0000)
[97301.399539] mpt2sas_cm0: log_info(0x31080000): originator(PL), code(0x08), sub_code(0x0000)
[97301.399544] mpt2sas_cm0: log_info(0x31080000): originator(PL), code(0x08), sub_code(0x0000)
[97301.399550] mpt2sas_cm0: log_info(0x31080000): originator(PL), code(0x08), sub_code(0x0000)
[97301.399556] mpt2sas_cm0: log_info(0x31080000): originator(PL), code(0x08), sub_code(0x0000)
[97301.399563] mpt2sas_cm0: log_info(0x31080000): originator(PL), code(0x08), sub_code(0x0000)
[97301.399569] mpt2sas_cm0: log_info(0x31080000): originator(PL), code(0x08), sub_code(0x0000)
[97301.399576] mpt2sas_cm0: log_info(0x31080000): originator(PL), code(0x08), sub_code(0x0000)
[97301.399581] mpt2sas_cm0: log_info(0x31080000): originator(PL), code(0x08), sub_code(0x0000)
[97301.399587] mpt2sas_cm0: log_info(0x31080000): originator(PL), code(0x08), sub_code(0x0000)
[97301.399593] mpt2sas_cm0: log_info(0x31080000): originator(PL), code(0x08), sub_code(0x0000)
[97301.399628] mpt2sas_cm0: log_info(0x31080000): originator(PL), code(0x08), sub_code(0x0000)
[97301.399635] mpt2sas_cm0: log_info(0x31080000): originator(PL), code(0x08), sub_code(0x0000)
[97301.399641] mpt2sas_cm0: log_info(0x31080000): originator(PL), code(0x08), sub_code(0x0000)
[97301.399647] mpt2sas_cm0: log_info(0x31080000): originator(PL), code(0x08), sub_code(0x0000)
[97301.399732] sd 0:0:2:0: [sdc] tag#2 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE
[97301.399742] sd 0:0:2:0: [sdc] tag#2 Sense Key : Medium Error [current] 
[97301.399750] sd 0:0:2:0: [sdc] tag#2 Add. Sense: Unrecovered read error
[97301.399761] sd 0:0:2:0: [sdc] tag#2 CDB: Read(16) 88 00 00 00 00 00 07 4b a7 80 00 00 04 00 00 00
[97301.399768] print_req_error: critical medium error, dev sdc, sector 122398592
[121767.882927] ata2.00: exception Emask 0x10 SAct 0x4000000 SErr 0x2c0100 action 0x6 frozen
[121767.884033] ata2.00: irq_stat 0x08000000, interface fatal error
[121767.885129] ata2: SError: { UnrecovData CommWake 10B8B BadCRC }
[121767.886292] ata2.00: failed command: READ FPDMA QUEUED
[121767.887452] ata2.00: cmd 60/00:d0:00:0d:c0/01:00:5e:00:00/40 tag 26 ncq dma 131072 in
                         res 40/00:d4:00:0d:c0/00:00:5e:00:00/40 Emask 0x10 (ATA bus error)
[121767.889908] ata2.00: status: { DRDY }
[121767.891133] ata2: hard resetting link
[121768.362839] ata2: SATA link up 6.0 Gbps (SStatus 133 SControl 300)
[121768.364506] ata2.00: configured for UDMA/133
[121768.364639] ata2: EH complete
[142463.633090] ata2.00: exception Emask 0x10 SAct 0x8000000 SErr 0x2c0100 action 0x6 frozen
[142463.633996] ata2.00: irq_stat 0x08000000, interface fatal error
[142463.634885] ata2: SError: { UnrecovData CommWake 10B8B BadCRC }
[142463.635821] ata2.00: failed command: READ FPDMA QUEUED
[142463.636751] ata2.00: cmd 60/00:d8:00:84:5e/01:00:a6:00:00/40 tag 27 ncq dma 131072 in
                         res 40/00:dc:00:84:5e/00:00:a6:00:00/40 Emask 0x10 (ATA bus error)
[142463.638746] ata2.00: status: { DRDY }
[142463.639740] ata2: hard resetting link
[142464.112102] ata2: SATA link up 6.0 Gbps (SStatus 133 SControl 300)
[142464.124637] ata2.00: configured for UDMA/133
[142464.124726] ata2: EH complete
[161969.808989] ata2.00: exception Emask 0x10 SAct 0x800 SErr 0x2c0100 action 0x6 frozen
[161969.810006] ata2.00: irq_stat 0x08000000, interface fatal error
[161969.811015] ata2: SError: { UnrecovData CommWake 10B8B BadCRC }
[161969.812074] ata2.00: failed command: READ FPDMA QUEUED
[161969.813125] ata2.00: cmd 60/00:58:00:8c:f0/01:00:e4:00:00/40 tag 11 ncq dma 131072 in
                         res 40/00:5c:00:8c:f0/00:00:e4:00:00/40 Emask 0x10 (ATA bus error)
[161969.815318] ata2.00: status: { DRDY }
[161969.816437] ata2: hard resetting link
[161970.288295] ata2: SATA link up 6.0 Gbps (SStatus 133 SControl 300)
[161970.289769] ata2.00: configured for UDMA/133
[161970.289874] ata2: EH complete
[180086.968705] ata2.00: exception Emask 0x10 SAct 0x8 SErr 0x2c0100 action 0x6 frozen
[180086.969894] ata2.00: irq_stat 0x08000000, interface fatal error
[180086.971049] ata2: SError: { UnrecovData CommWake 10B8B BadCRC }
[180086.972244] ata2.00: failed command: READ FPDMA QUEUED
[180086.973438] ata2.00: cmd 60/00:18:00:2c:5e/01:00:1a:01:00/40 tag 3 ncq dma 131072 in
                         res 40/00:1c:00:2c:5e/00:00:1a:01:00/40 Emask 0x10 (ATA bus error)
[180086.975920] ata2.00: status: { DRDY }
[180086.977183] ata2: hard resetting link
[180087.451405] ata2: SATA link up 6.0 Gbps (SStatus 133 SControl 300)
[180087.452763] ata2.00: configured for UDMA/133
[180087.452854] ata2: EH complete
[182053.137581] EXT4-fs (dm-9): error count since last fsck: 36
[182053.137585] EXT4-fs (dm-9): initial error at time 1555459461: htree_dirblock_to_tree:1007: inode 207225819: block 828976633
[182053.137589] EXT4-fs (dm-9): last error at time 1555489211: htree_dirblock_to_tree:1007: inode 200803462: block 803370684
[184961.910473] ata2.00: exception Emask 0x10 SAct 0x400 SErr 0x2c0100 action 0x6 frozen
[184961.912261] ata2.00: irq_stat 0x08000000, interface fatal error
[184961.914074] ata2: SError: { UnrecovData CommWake 10B8B BadCRC }
[184961.915855] ata2.00: failed command: READ FPDMA QUEUED
[184961.917643] ata2.00: cmd 60/00:50:00:e1:b4/01:00:28:01:00/40 tag 10 ncq dma 131072 in
                         res 40/00:54:00:e1:b4/00:00:28:01:00/40 Emask 0x10 (ATA bus error)
[184961.921340] ata2.00: status: { DRDY }
[184961.923212] ata2: hard resetting link
[184962.397072] ata2: SATA link up 6.0 Gbps (SStatus 133 SControl 300)
[184962.398359] ata2.00: configured for UDMA/133
[184962.398434] ata2: EH complete
[203933.007037] md: md127: data-check done.
[223759.243215] ata2.00: exception Emask 0x10 SAct 0x8000000 SErr 0x2c0100 action 0x6 frozen
[223759.244668] ata2.00: irq_stat 0x08000000, interface fatal error
[223759.246140] ata2: SError: { UnrecovData CommWake 10B8B BadCRC }
[223759.247603] ata2.00: failed command: READ FPDMA QUEUED
[223759.249080] ata2.00: cmd 60/00:d8:00:60:f7/04:00:75:00:00/40 tag 27 ncq dma 524288 in
                         res 40/00:dc:00:60:f7/00:00:75:00:00/40 Emask 0x10 (ATA bus error)
[223759.252104] ata2.00: status: { DRDY }
[223759.253653] ata2: hard resetting link
[223759.726117] ata2: SATA link up 6.0 Gbps (SStatus 133 SControl 300)
[223759.727449] ata2.00: configured for UDMA/133
[223759.727533] ata2: EH complete
[236509.392565] md127: detected capacity change from 8999751647232 to 0
[236509.392581] md: md127 stopped.
[237069.168849] md/raid10:md127: active with 6 out of 6 devices
[237069.195214] md127: detected capacity change from 0 to 8999751647232
[237603.621049] EXT4-fs (dm-2): mounted filesystem with ordered data mode. Opts: (null)
[240970.885488] EXT4-fs (dm-3): mounted filesystem with ordered data mode. Opts: (null)
[241028.883617] mpt2sas_cm0: log_info(0x31080000): originator(PL), code(0x08), sub_code(0x0000)
[241028.883629] mpt2sas_cm0: log_info(0x31080000): originator(PL), code(0x08), sub_code(0x0000)
[241028.883637] mpt2sas_cm0: log_info(0x31080000): originator(PL), code(0x08), sub_code(0x0000)
[241028.883644] mpt2sas_cm0: log_info(0x31080000): originator(PL), code(0x08), sub_code(0x0000)
[241028.883652] mpt2sas_cm0: log_info(0x31080000): originator(PL), code(0x08), sub_code(0x0000)
[241028.883660] mpt2sas_cm0: log_info(0x31080000): originator(PL), code(0x08), sub_code(0x0000)
[241028.883668] mpt2sas_cm0: log_info(0x31080000): originator(PL), code(0x08), sub_code(0x0000)
[241028.883676] mpt2sas_cm0: log_info(0x31080000): originator(PL), code(0x08), sub_code(0x0000)
[241028.883684] mpt2sas_cm0: log_info(0x31080000): originator(PL), code(0x08), sub_code(0x0000)
[241028.883691] mpt2sas_cm0: log_info(0x31080000): originator(PL), code(0x08), sub_code(0x0000)
[241028.883698] mpt2sas_cm0: log_info(0x31080000): originator(PL), code(0x08), sub_code(0x0000)
[241028.883708] mpt2sas_cm0: log_info(0x31080000): originator(PL), code(0x08), sub_code(0x0000)
[241028.883714] mpt2sas_cm0: log_info(0x31080000): originator(PL), code(0x08), sub_code(0x0000)
[241028.883720] mpt2sas_cm0: log_info(0x31080000): originator(PL), code(0x08), sub_code(0x0000)
[241028.883727] mpt2sas_cm0: log_info(0x31080000): originator(PL), code(0x08), sub_code(0x0000)
[241028.883734] mpt2sas_cm0: log_info(0x31080000): originator(PL), code(0x08), sub_code(0x0000)
[241028.883741] mpt2sas_cm0: log_info(0x31080000): originator(PL), code(0x08), sub_code(0x0000)
[241028.883748] mpt2sas_cm0: log_info(0x31080000): originator(PL), code(0x08), sub_code(0x0000)
[241028.883755] mpt2sas_cm0: log_info(0x31080000): originator(PL), code(0x08), sub_code(0x0000)
[241028.883762] mpt2sas_cm0: log_info(0x31080000): originator(PL), code(0x08), sub_code(0x0000)
[241028.883768] mpt2sas_cm0: log_info(0x31080000): originator(PL), code(0x08), sub_code(0x0000)
[241028.883775] mpt2sas_cm0: log_info(0x31080000): originator(PL), code(0x08), sub_code(0x0000)
[241028.883782] mpt2sas_cm0: log_info(0x31080000): originator(PL), code(0x08), sub_code(0x0000)
[241028.883789] mpt2sas_cm0: log_info(0x31080000): originator(PL), code(0x08), sub_code(0x0000)
[241028.883796] mpt2sas_cm0: log_info(0x31080000): originator(PL), code(0x08), sub_code(0x0000)
[241028.883803] mpt2sas_cm0: log_info(0x31080000): originator(PL), code(0x08), sub_code(0x0000)
[241028.883867] sd 0:0:2:0: [sdc] tag#34 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE
[241028.883878] sd 0:0:2:0: [sdc] tag#34 Sense Key : Illegal Request [current] 
[241028.883885] sd 0:0:2:0: [sdc] tag#34 Add. Sense: Logical block address out of range
[241028.883894] sd 0:0:2:0: [sdc] tag#34 CDB: Write(16) 8a 00 00 00 00 00 07 4a 1c 00 00 00 04 00 00 00
[241028.883899] print_req_error: critical target error, dev sdc, sector 122297344
[241083.326055] sd 0:0:2:0: attempting task abort! scmd(000000001397c229)
[241083.326068] sd 0:0:2:0: [sdc] tag#19 CDB: Synchronize Cache(10) 35 00 00 00 00 00 00 00 00 00
[241083.326076] scsi target0:0:2: handle(0x000b), sas_address(0x4433221102000000), phy(2)
[241083.326081] scsi target0:0:2: enclosure logical id(0x5000000080000000), slot(1) 
[241087.313130] sd 0:0:2:0: task abort: SUCCESS scmd(000000001397c229)
[241087.326101] sd 0:0:2:0: [sdc] tag#0 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE
[241087.326111] sd 0:0:2:0: [sdc] tag#0 Sense Key : Not Ready [current] 
[241087.326115] sd 0:0:2:0: [sdc] tag#0 Add. Sense: Logical unit not ready, cause not reportable
[241087.326126] sd 0:0:2:0: [sdc] tag#0 CDB: Synchronize Cache(10) 35 00 00 00 00 00 00 00 00 00
[241087.326152] print_req_error: I/O error, dev sdc, sector 16
[241087.328798] md: super_written gets error=10
[241087.331440] md/raid10:md127: Disk failure on sdc, disabling device.
                md/raid10:md127: Operation continuing on 5 devices.
[241087.337077] sd 0:0:2:0: [sdc] tag#14 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE
[241087.337082] sd 0:0:2:0: [sdc] tag#14 Sense Key : Not Ready [current] 
[241087.337088] sd 0:0:2:0: [sdc] tag#14 Add. Sense: Logical unit not ready, cause not reportable
[241087.337092] sd 0:0:2:0: [sdc] tag#14 CDB: Write(16) 8a 00 00 00 00 00 07 4a 7c 00 00 00 04 00 00 00
[241087.337095] print_req_error: I/O error, dev sdc, sector 122321920
[241087.337097] sd 0:0:2:0: [sdc] tag#1 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE
[241087.338779] sd 0:0:2:0: [sdc] tag#1 Sense Key : Not Ready [current] 
[241087.338782] sd 0:0:2:0: [sdc] tag#21 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE
[241087.338783] sd 0:0:2:0: [sdc] tag#1 Add. Sense: Logical unit not ready, cause not reportable
[241087.338788] sd 0:0:2:0: [sdc] tag#1 CDB: Write(16) 8a 00 00 00 00 00 07 4a 54 00 00 00 03 f8 00 00
[241087.338790] sd 0:0:2:0: [sdc] tag#21 Sense Key : Not Ready [current] 
[241087.338792] print_req_error: I/O error, dev sdc, sector 122311680
[241087.338795] sd 0:0:2:0: [sdc] tag#15 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE
[241087.338815] sd 0:0:2:0: [sdc] tag#2 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE
[241087.338818] sd 0:0:2:0: [sdc] tag#2 Sense Key : Not Ready [current] 
[241087.338820] sd 0:0:2:0: [sdc] tag#2 Add. Sense: Logical unit not ready, cause not reportable
[241087.338823] sd 0:0:2:0: [sdc] tag#2 CDB: Write(16) 8a 00 00 00 00 00 07 4a 57 f8 00 00 00 08 00 00
[241087.338824] print_req_error: I/O error, dev sdc, sector 122312696
[241087.338835] sd 0:0:2:0: [sdc] tag#3 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE
[241087.338838] sd 0:0:2:0: [sdc] tag#3 Sense Key : Not Ready [current] 
[241087.338840] sd 0:0:2:0: [sdc] tag#3 Add. Sense: Logical unit not ready, cause not reportable
[241087.338842] sd 0:0:2:0: [sdc] tag#3 CDB: Write(16) 8a 00 00 00 00 00 07 4a 50 00 00 00 04 00 00 00
[241087.338844] print_req_error: I/O error, dev sdc, sector 122310656
[241087.338855] sd 0:0:2:0: [sdc] tag#4 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE
[241087.338858] sd 0:0:2:0: [sdc] tag#4 Sense Key : Not Ready [current] 
[241087.338860] sd 0:0:2:0: [sdc] tag#4 Add. Sense: Logical unit not ready, cause not reportable
[241087.338863] sd 0:0:2:0: [sdc] tag#4 CDB: Write(16) 8a 00 00 00 00 00 07 4a 4c 00 00 00 04 00 00 00
[241087.338864] print_req_error: I/O error, dev sdc, sector 122309632
[241087.338874] sd 0:0:2:0: [sdc] tag#5 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE
[241087.338877] sd 0:0:2:0: [sdc] tag#5 Sense Key : Not Ready [current] 
[241087.338879] sd 0:0:2:0: [sdc] tag#5 Add. Sense: Logical unit not ready, cause not reportable
[241087.338881] sd 0:0:2:0: [sdc] tag#5 CDB: Write(16) 8a 00 00 00 00 00 07 4a 48 00 00 00 04 00 00 00
[241087.338883] print_req_error: I/O error, dev sdc, sector 122308608
[241087.338896] sd 0:0:2:0: [sdc] tag#0 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE
[241087.338899] sd 0:0:2:0: [sdc] tag#0 Sense Key : Not Ready [current] 
[241087.338901] sd 0:0:2:0: [sdc] tag#0 Add. Sense: Logical unit not ready, cause not reportable
[241087.338904] sd 0:0:2:0: [sdc] tag#0 CDB: Write(16) 8a 00 00 00 00 00 07 4a 40 00 00 00 04 00 00 00
[241087.338905] print_req_error: I/O error, dev sdc, sector 122306560
[241087.338914] print_req_error: I/O error, dev sdc, sector 122305536
[241087.338924] print_req_error: I/O error, dev sdc, sector 122304512
[241087.340458] sd 0:0:2:0: [sdc] tag#21 Add. Sense: Logical unit not ready, cause not reportable
[241087.340461] sd 0:0:2:0: [sdc] tag#21 CDB: Write(16) 8a 00 00 00 00 00 07 4a 70 00 00 00 04 00 00 00
[241087.352343] sd 0:0:2:0: [sdc] tag#15 Sense Key : Not Ready [current] 
[241087.352346] sd 0:0:2:0: [sdc] tag#15 Add. Sense: Logical unit not ready, cause not reportable
[241087.352349] sd 0:0:2:0: [sdc] tag#15 CDB: Write(16) 8a 00 00 00 00 00 07 4a 78 00 00 00 04 00 00 00
[241092.423724] scsi_io_completion: 18449 callbacks suppressed
[241092.423745] sd 0:0:2:0: [sdc] tag#27 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE
[241092.423750] sd 0:0:2:0: [sdc] tag#27 Sense Key : Not Ready [current] 
[241092.423754] sd 0:0:2:0: [sdc] tag#27 Add. Sense: Logical unit not ready, cause not reportable
[241092.423760] sd 0:0:2:0: [sdc] tag#27 CDB: Write(16) 8a 00 00 00 00 00 07 4a 78 00 00 00 00 01 00 00
[241092.423763] print_req_error: 18449 callbacks suppressed
[241092.423766] print_req_error: I/O error, dev sdc, sector 122320896
[241092.425573] sd 0:0:2:0: [sdc] tag#10 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE
[241092.425579] sd 0:0:2:0: [sdc] tag#10 Sense Key : Not Ready [current] 
[241092.425583] sd 0:0:2:0: [sdc] tag#10 Add. Sense: Logical unit not ready, cause not reportable
[241092.425587] sd 0:0:2:0: [sdc] tag#10 CDB: Write(16) 8a 00 00 00 00 00 07 4a 78 01 00 00 00 01 00 00
[241092.425590] print_req_error: I/O error, dev sdc, sector 122320897
[241092.428263] sd 0:0:2:0: [sdc] tag#10 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE
[241092.428272] sd 0:0:2:0: [sdc] tag#10 Sense Key : Not Ready [current] 
[241092.428276] sd 0:0:2:0: [sdc] tag#10 Add. Sense: Logical unit not ready, cause not reportable
[241092.428281] sd 0:0:2:0: [sdc] tag#10 CDB: Write(16) 8a 00 00 00 00 00 07 4a 78 02 00 00 00 01 00 00
[241092.428286] print_req_error: I/O error, dev sdc, sector 122320898
[241092.430118] sd 0:0:2:0: [sdc] tag#11 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE
[241092.430123] sd 0:0:2:0: [sdc] tag#11 Sense Key : Not Ready [current] 
[241092.430127] sd 0:0:2:0: [sdc] tag#11 Add. Sense: Logical unit not ready, cause not reportable
[241092.430131] sd 0:0:2:0: [sdc] tag#11 CDB: Write(16) 8a 00 00 00 00 00 07 4a 78 03 00 00 00 01 00 00
[241092.430135] print_req_error: I/O error, dev sdc, sector 122320899
[241092.431914] sd 0:0:2:0: [sdc] tag#11 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE
[241092.431922] sd 0:0:2:0: [sdc] tag#11 Sense Key : Not Ready [current] 
[241092.431926] sd 0:0:2:0: [sdc] tag#11 Add. Sense: Logical unit not ready, cause not reportable
[241092.431931] sd 0:0:2:0: [sdc] tag#11 CDB: Write(16) 8a 00 00 00 00 00 07 4a 78 04 00 00 00 01 00 00
[241092.431934] print_req_error: I/O error, dev sdc, sector 122320900
[241092.433712] sd 0:0:2:0: [sdc] tag#11 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE
[241092.433715] sd 0:0:2:0: [sdc] tag#11 Sense Key : Not Ready [current] 
[241092.433717] sd 0:0:2:0: [sdc] tag#11 Add. Sense: Logical unit not ready, cause not reportable
[241092.433721] sd 0:0:2:0: [sdc] tag#11 CDB: Write(16) 8a 00 00 00 00 00 07 4a 78 05 00 00 00 01 00 00
[241092.433723] print_req_error: I/O error, dev sdc, sector 122320901
[241092.435506] sd 0:0:2:0: [sdc] tag#11 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE
[241092.435514] sd 0:0:2:0: [sdc] tag#11 Sense Key : Not Ready [current] 
[241092.435518] sd 0:0:2:0: [sdc] tag#11 Add. Sense: Logical unit not ready, cause not reportable
[241092.435522] sd 0:0:2:0: [sdc] tag#11 CDB: Write(16) 8a 00 00 00 00 00 07 4a 78 06 00 00 00 01 00 00
[241092.435525] print_req_error: I/O error, dev sdc, sector 122320902
[241092.437345] sd 0:0:2:0: [sdc] tag#11 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE
[241092.437352] sd 0:0:2:0: [sdc] tag#11 Sense Key : Not Ready [current] 
[241092.437355] sd 0:0:2:0: [sdc] tag#11 Add. Sense: Logical unit not ready, cause not reportable
[241092.437361] sd 0:0:2:0: [sdc] tag#11 CDB: Write(16) 8a 00 00 00 00 00 07 4a 78 07 00 00 00 01 00 00
[241092.437364] print_req_error: I/O error, dev sdc, sector 122320903
[241092.439144] sd 0:0:2:0: [sdc] tag#11 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE
[241092.439150] sd 0:0:2:0: [sdc] tag#11 Sense Key : Not Ready [current] 
[241092.439154] sd 0:0:2:0: [sdc] tag#11 Add. Sense: Logical unit not ready, cause not reportable
[241092.439159] sd 0:0:2:0: [sdc] tag#11 CDB: Write(16) 8a 00 00 00 00 00 07 4a 78 08 00 00 00 01 00 00
[241092.439162] print_req_error: I/O error, dev sdc, sector 122320904
[241092.440986] sd 0:0:2:0: [sdc] tag#11 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE
[241092.440992] sd 0:0:2:0: [sdc] tag#11 Sense Key : Not Ready [current] 
[241092.440995] sd 0:0:2:0: [sdc] tag#11 Add. Sense: Logical unit not ready, cause not reportable
[241092.441000] sd 0:0:2:0: [sdc] tag#11 CDB: Write(16) 8a 00 00 00 00 00 07 4a 78 09 00 00 00 01 00 00
[241092.441003] print_req_error: I/O error, dev sdc, sector 122320905

Do I understand it correctly that my sdc is fried? I know some errors can be fixed by reformatting which will trigger a remapping of bad sectors, but messages such as these

180086.968705] ata2.00: exception Emask 0x10 SAct 0x8 SErr 0x2c0100 action 0x6 frozen
[180086.969894] ata2.00: irq_stat 0x08000000, interface fatal error
[180086.971049] ata2: SError: { UnrecovData CommWake 10B8B BadCRC }
[180086.972244] ata2.00: failed command: READ FPDMA QUEUED
[180086.973438] ata2.00: cmd 60/00:18:00:2c:5e/01:00:1a:01:00/40 tag 3 ncq dma 131072 in
                         res 40/00:1c:00:2c:5e/00:00:1a:01:00/40 Emask 0x10 (ATA bus error)
[180086.975920] ata2.00: status: { DRDY }
[180086.977183] ata2: hard resetting link
[180087.451405] ata2: SATA link up 6.0 Gbps (SStatus 133 SControl 300)
[180087.452763] ata2.00: configured for UDMA/133
[180087.452854] ata2: EH complete

sound worse than just bad blocks existing on the device. Am I correct that there's a hardware error on the device? (It's still inside warranty, it's actually the most recent drive I got)

EDIT: Also, I just did a long SMART-test on all the drives. For sdc, this has the result "The previous self-test completed having the read element of the test failed." so indeed some sectors just cannot be read anymore. But why in the world, then, does SMART still return "SMART overall-health self-assessment test result: PASSED"?

smartctl 6.6 2017-11-05 r4594 [x86_64-linux-4.16.5-1-ARCH] (local build)
Copyright (C) 2002-17, Bruce Allen, Christian Franke, www.smartmontools.org

=== START OF INFORMATION SECTION ===
Model Family:     Western Digital Red
Device Model:     WDC WD30EFRX-68N32N0
Serial Number:    WD-WCC7K6FT392L
LU WWN Device Id: 5 0014ee 2644a9812
Firmware Version: 82.00A82
User Capacity:    3,000,592,982,016 bytes [3.00 TB]
Sector Sizes:     512 bytes logical, 4096 bytes physical
Rotation Rate:    5400 rpm
Form Factor:      3.5 inches
Device is:        In smartctl database [for details use: -P show]
ATA Version is:   ACS-3 T13/2161-D revision 5
SATA Version is:  SATA 3.1, 6.0 Gb/s (current: 6.0 Gb/s)
Local Time is:    Thu May  2 23:23:56 2019 UTC
SMART support is: Available - device has SMART capability.
SMART support is: Enabled

=== START OF READ SMART DATA SECTION ===
SMART overall-health self-assessment test result: PASSED

General SMART Values:
Offline data collection status:  (0x00)	Offline data collection activity
					was never started.
					Auto Offline Data Collection: Disabled.
Self-test execution status:      ( 121)	The previous self-test completed having
					the read element of the test failed.
Total time to complete Offline 
data collection: 		(31560) seconds.
Offline data collection
capabilities: 			 (0x7b) SMART execute Offline immediate.
					Auto Offline data collection on/off support.
					Suspend Offline collection upon new
					command.
					Offline surface scan supported.
					Self-test supported.
					Conveyance Self-test supported.
					Selective Self-test supported.
SMART capabilities:            (0x0003)	Saves SMART data before entering
					power-saving mode.
					Supports SMART auto save timer.
Error logging capability:        (0x01)	Error logging supported.
					General Purpose Logging supported.
Short self-test routine 
recommended polling time: 	 (   2) minutes.
Extended self-test routine
recommended polling time: 	 ( 337) minutes.
Conveyance self-test routine
recommended polling time: 	 (   5) minutes.
SCT capabilities: 	       (0x303d)	SCT Status supported.
					SCT Error Recovery Control supported.
					SCT Feature Control supported.
					SCT Data Table supported.

SMART Attributes Data Structure revision number: 16
Vendor Specific SMART Attributes with Thresholds:
ID# ATTRIBUTE_NAME          FLAG     VALUE WORST THRESH TYPE      UPDATED  WHEN_FAILED RAW_VALUE
  1 Raw_Read_Error_Rate     0x002f   200   198   051    Pre-fail  Always       -       0
  3 Spin_Up_Time            0x0027   215   184   021    Pre-fail  Always       -       4250
  4 Start_Stop_Count        0x0032   100   100   000    Old_age   Always       -       47
  5 Reallocated_Sector_Ct   0x0033   200   200   140    Pre-fail  Always       -       0
  7 Seek_Error_Rate         0x002e   200   200   000    Old_age   Always       -       0
  9 Power_On_Hours          0x0032   090   090   000    Old_age   Always       -       7960
 10 Spin_Retry_Count        0x0032   100   253   000    Old_age   Always       -       0
 11 Calibration_Retry_Count 0x0032   100   253   000    Old_age   Always       -       0
 12 Power_Cycle_Count       0x0032   100   100   000    Old_age   Always       -       42
192 Power-Off_Retract_Count 0x0032   200   200   000    Old_age   Always       -       38
193 Load_Cycle_Count        0x0032   200   200   000    Old_age   Always       -       71
194 Temperature_Celsius     0x0022   113   101   000    Old_age   Always       -       37
196 Reallocated_Event_Count 0x0032   200   200   000    Old_age   Always       -       0
197 Current_Pending_Sector  0x0032   200   200   000    Old_age   Always       -       6
198 Offline_Uncorrectable   0x0030   100   253   000    Old_age   Offline      -       0
199 UDMA_CRC_Error_Count    0x0032   200   200   000    Old_age   Always       -       0
200 Multi_Zone_Error_Rate   0x0008   100   253   000    Old_age   Offline      -       0

SMART Error Log Version: 1
No Errors Logged

SMART Self-test log structure revision number 1
Num  Test_Description    Status                  Remaining  LifeTime(hours)  LBA_of_first_error
# 1  Extended offline    Completed: read failure       90%      7958         122298368

SMART Selective self-test log data structure revision number 1
 SPAN  MIN_LBA  MAX_LBA  CURRENT_TEST_STATUS
    1        0        0  Not_testing
    2        0        0  Not_testing
    3        0        0  Not_testing
    4        0        0  Not_testing
    5        0        0  Not_testing
Selective self-test flags (0x0):
  After scanning selected spans, do NOT read-scan remainder of disk.
If Selective self-test is pending on power-up, resume after 0 minute delay.

Last edited by Natanji (2019-05-02 23:36:50)

Offline

#9 2019-05-03 09:12:18

Spider.007
Member
Registered: 2004-06-20
Posts: 1,175

Re: [solved] Server crash -> mdadm RAID superblocks missing from disks?!

Natanji wrote:

sound worse than just bad blocks existing on the device. Am I correct that there's a hardware error on the device? (It's still inside warranty, it's actually the most recent drive I got)

Just an FYI as you've been switching disks - a badly inserted SATA cable will have the same effect, so double check your connectors and possibly try a new cable before blaming the drive.

Offline

#10 2019-05-03 23:18:45

Natanji
Member
Registered: 2009-09-22
Posts: 133

Re: [solved] Server crash -> mdadm RAID superblocks missing from disks?!

Spider.007 wrote:
Natanji wrote:

sound worse than just bad blocks existing on the device. Am I correct that there's a hardware error on the device? (It's still inside warranty, it's actually the most recent drive I got)

Just an FYI as you've been switching disks - a badly inserted SATA cable will have the same effect, so double check your connectors and possibly try a new cable before blaming the drive.

Thanks for that idea, but I checked again with a new cable and the tests still fail, it's definitely a faulty drive.

As of today, my server is running again. Thank you all sooo much for the suggestions, especiall Frostschutz - I learned a lot about server admining from this crash, which might come in handy in the future! And it seems that I was indeed lucky and ultimately, no data was lost.

I'll mark the thread as [solved] now.

Last edited by Natanji (2019-05-03 23:20:27)

Offline

Board footer

Powered by FluxBB