Strange issues since kernel 3.2.5 update

Darkimmortal · 2012-02-15 17:50:29

Updated the kernel on my server last night, then suddenly today everything stopped working with errors such as

[root@weeaboo ~]# top
-bash: /usr/bin/top: Input/output error
[root@weeaboo ~]# htop
Segmentation fault
[root@weeaboo ~]# reboot
-bash: reboot: command not found
[root@weeaboo ~]# smartctl -a /dev/sda
smartctl 5.42 2011-10-20 r3458 [x86_64-linux-3.2.5-1-ARCH] (local build)
Copyright (C) 2002-11 by Bruce Allen, http://smartmontools.sourceforge.net

Bus error

Looking back in the logs I am seeing errors approx every 1ms:

Feb 15 17:26:04 weeaboo kernel: [58492.627664] end_request: I/O error, dev sda, sector 84562736
Feb 15 17:26:04 weeaboo kernel: [58492.628537] end_request: I/O error, dev sda, sector 84562736
Feb 15 17:26:04 weeaboo kernel: [58492.629121] end_request: I/O error, dev sda, sector 84562736

Feb 15 17:26:08 weeaboo kernel: [58495.758718] sd 0:0:0:0: [sda] Unhandled error code
Feb 15 17:26:08 weeaboo kernel: [58495.758719] sd 0:0:0:0: [sda]  Result: hostbyte=0x04 driverbyte=0x00
Feb 15 17:26:08 weeaboo kernel: [58495.758720] sd 0:0:0:0: [sda] CDB: cdb[0]=0x28: 28 00 02 6a 97 08 00 00 08 00
Feb 15 17:26:08 weeaboo kernel: [58495.758734] sd 0:0:0:0: [sda] Unhandled error code
Feb 15 17:26:08 weeaboo kernel: [58495.758735] sd 0:0:0:0: [sda]  Result: hostbyte=0x04 driverbyte=0x00
Feb 15 17:26:08 weeaboo kernel: [58495.758736] sd 0:0:0:0: [sda] CDB: cdb[0]=0x28: 28 00 05 0a 53 30 00 00 08 00

And also:

Feb 15 16:56:08 weeaboo kernel: [56702.247278] ata1.00: exception Emask 0x0 SAct 0x7fffffff SErr 0x0 action 0x6 frozen
Feb 15 16:56:08 weeaboo kernel: [56702.247509] ata1.00: failed command: READ FPDMA QUEUED
Feb 15 16:56:08 weeaboo kernel: [56702.247748] ata1.00: cmd 60/10:00:30:d8:14/00:00:02:00:00/40 tag 0 ncq 8192 in
Feb 15 16:56:08 weeaboo kernel: [56702.247749]          res 40/00:00:01:4f:c2/00:00:00:00:00/00 Emask 0x4 (timeout)
Feb 15 16:56:08 weeaboo kernel: [56702.248255] ata1.00: status: { DRDY }
Feb 15 16:56:08 weeaboo kernel: [56702.248523] ata1.00: failed command: READ FPDMA QUEUED
Feb 15 16:56:08 weeaboo kernel: [56702.248807] ata1.00: cmd 60/08:08:68:d8:14/00:00:02:00:00/40 tag 1 ncq 4096 in
Feb 15 16:56:08 weeaboo kernel: [56702.248808]          res 40/00:00:00:00:00/00:00:00:00:00/00 Emask 0x4 (timeout)
Feb 15 16:56:08 weeaboo kernel: [56702.249436] ata1.00: status: { DRDY }
Feb 15 16:56:08 weeaboo kernel: [56702.249773] ata1.00: failed command: READ FPDMA QUEUED
Feb 15 16:56:08 weeaboo kernel: [56702.250134] ata1.00: cmd 60/08:10:e0:d8:14/00:00:02:00:00/40 tag 2 ncq 4096 in
Feb 15 16:56:08 weeaboo kernel: [56702.250135]          res 40/00:00:00:00:00/00:00:00:00:00/00 Emask 0x4 (timeout)
Feb 15 16:56:08 weeaboo kernel: [56702.250949] ata1.00: status: { DRDY }
Feb 15 16:56:08 weeaboo kernel: [56702.251361] ata1.00: failed command: WRITE FPDMA QUEUED
Feb 15 16:56:08 weeaboo kernel: [56702.251795] ata1.00: cmd 61/08:18:a0:48:36/00:00:02:00:00/40 tag 3 ncq 4096 out
Feb 15 16:56:08 weeaboo kernel: [56702.251796]          res 40/00:00:00:00:00/00:00:00:00:00/00 Emask 0x4 (timeout)
Feb 15 16:56:08 weeaboo kernel: [56702.252683] ata1.00: status: { DRDY }
Feb 15 16:56:08 weeaboo kernel: [56702.253157] ata1.00: failed command: WRITE FPDMA QUEUED
Feb 15 16:56:08 weeaboo kernel: [56702.253592] ata1.00: cmd 61/10:20:08:5a:36/00:00:02:00:00/40 tag 4 ncq 8192 out
Feb 15 16:56:08 weeaboo kernel: [56702.253593]          res 40/00:00:00:00:00/00:00:00:00:00/00 Emask 0x4 (timeout)
Feb 15 16:56:08 weeaboo kernel: [56702.254315] ata1.00: status: { DRDY }
Feb 15 16:56:08 weeaboo kernel: [56702.254688] ata1.00: failed command: WRITE FPDMA QUEUED
Feb 15 16:56:08 weeaboo kernel: [56702.255079] ata1.00: cmd 61/08:28:68:a8:36/00:00:02:00:00/40 tag 5 ncq 4096 out
Feb 15 16:56:08 weeaboo kernel: [56702.255080]          res 40/00:00:00:00:00/00:00:00:00:00/00 Emask 0x4 (timeout)
Feb 15 16:56:08 weeaboo kernel: [56702.255901] ata1.00: status: { DRDY }

(repeats for a while)


Feb 15 16:56:18 weeaboo kernel: [56712.328400] ata1: softreset failed (device not ready)
Feb 15 16:56:28 weeaboo kernel: [56722.343068] ata1: softreset failed (device not ready)
Feb 15 16:57:03 weeaboo kernel: [56757.246493] ata1: softreset failed (device not ready)
Feb 15 16:57:09 weeaboo kernel: [56762.405059] ata1: softreset failed (device not ready)
Feb 15 16:57:09 weeaboo kernel: [56762.405920] ata1: reset failed, giving up
Feb 15 16:57:09 weeaboo kernel: [56762.406876] end_request: I/O error, dev sda, sector 32115224
Feb 15 16:57:09 weeaboo kernel: [56762.407736] Buffer I/O error on device sda2, logical block 3962947
Feb 15 16:57:09 weeaboo kernel: [56762.408624] Buffer I/O error on device sda2, logical block 3962948
Feb 15 16:57:09 weeaboo kernel: [56762.409520] end_request: I/O error, dev sda, sector 32115208
Feb 15 16:57:09 weeaboo kernel: [56762.410409] Buffer I/O error on device sda2, logical block 3962945
Feb 15 16:57:09 weeaboo kernel: [56762.411291] end_request: I/O error, dev sda, sector 32114960
Feb 15 16:57:09 weeaboo kernel: [56762.412144] Buffer I/O error on device sda2, logical block 3962914
Feb 15 16:57:09 weeaboo kernel: [56762.413037] end_request: I/O error, dev sda, sector 34930584
Feb 15 16:57:09 weeaboo kernel: [56762.413911] Buffer I/O error on device sda2, logical block 4314867
Feb 15 16:57:09 weeaboo kernel: [56762.414823] end_request: I/O error, dev sda, sector 34927112

smartctl output after having power cycled the box:

smartctl -a /dev/sda
smartctl 5.42 2011-10-20 r3458 [x86_64-linux-3.2.5-1-ARCH] (local build)
Copyright (C) 2002-11 by Bruce Allen, http://smartmontools.sourceforge.net

=== START OF INFORMATION SECTION ===
Model Family:     Crucial/Micron RealSSD C300/C400
Device Model:     M4-CT064M4SSD2
Serial Number:    00000000112103109D1E
LU WWN Device Id: 5 00a075 103109d1e
Firmware Version: 0002
User Capacity:    64,023,257,088 bytes [64.0 GB]
Sector Size:      512 bytes logical/physical
Device is:        In smartctl database [for details use: -P show]
ATA Version is:   8
ATA Standard is:  ATA-8-ACS revision 6
Local Time is:    Wed Feb 15 17:48:16 2012 GMT
SMART support is: Available - device has SMART capability.
SMART support is: Enabled

=== START OF READ SMART DATA SECTION ===
SMART overall-health self-assessment test result: PASSED

General SMART Values:
Offline data collection status:  (0x80) Offline data collection activity
                                        was never started.
                                        Auto Offline Data Collection: Enabled.
Self-test execution status:      (   0) The previous self-test routine completed
                                        without error or no self-test has ever 
                                        been run.
Total time to complete Offline 
data collection:                (  295) seconds.
Offline data collection
capabilities:                    (0x7b) SMART execute Offline immediate.
                                        Auto Offline data collection on/off support.
                                        Suspend Offline collection upon new
                                        command.
                                        Offline surface scan supported.
                                        Self-test supported.
                                        Conveyance Self-test supported.
                                        Selective Self-test supported.
SMART capabilities:            (0x0003) Saves SMART data before entering
                                        power-saving mode.
                                        Supports SMART auto save timer.
Error logging capability:        (0x01) Error logging supported.
                                        General Purpose Logging supported.
Short self-test routine 
recommended polling time:        (   2) minutes.
Extended self-test routine
recommended polling time:        (   4) minutes.
Conveyance self-test routine
recommended polling time:        (   3) minutes.
SCT capabilities:              (0x003d) SCT Status supported.
                                        SCT Error Recovery Control supported.
                                        SCT Feature Control supported.
                                        SCT Data Table supported.

SMART Attributes Data Structure revision number: 16
Vendor Specific SMART Attributes with Thresholds:
ID# ATTRIBUTE_NAME          FLAG     VALUE WORST THRESH TYPE      UPDATED  WHEN_FAILED RAW_VALUE
  1 Raw_Read_Error_Rate     0x002f   100   100   050    Pre-fail  Always       -       0
  5 Reallocated_Sector_Ct   0x0033   100   100   010    Pre-fail  Always       -       0
  9 Power_On_Hours          0x0032   100   100   001    Old_age   Always       -       5212
 12 Power_Cycle_Count       0x0032   100   100   001    Old_age   Always       -       175
170 Grown_Failing_Block_Ct  0x0033   100   100   010    Pre-fail  Always       -       0
171 Program_Fail_Count      0x0032   100   100   001    Old_age   Always       -       0
172 Erase_Fail_Count        0x0032   100   100   001    Old_age   Always       -       0
173 Wear_Levelling_Count    0x0033   099   099   010    Pre-fail  Always       -       34
174 Unexpect_Power_Loss_Ct  0x0032   100   100   001    Old_age   Always       -       0
181 Non4k_Aligned_Access    0x0022   100   100   001    Old_age   Always       -       20 3 16
183 SATA_Iface_Downshift    0x0032   100   100   001    Old_age   Always       -       0
184 End-to-End_Error        0x0033   100   100   050    Pre-fail  Always       -       0
187 Reported_Uncorrect      0x0032   100   100   001    Old_age   Always       -       0
188 Command_Timeout         0x0032   100   100   001    Old_age   Always       -       0
189 Factory_Bad_Block_Ct    0x000e   100   100   001    Old_age   Always       -       53
195 Hardware_ECC_Recovered  0x003c   100   100   001    Old_age   Offline      -       0
196 Reallocated_Event_Count 0x0032   100   100   001    Old_age   Always       -       0
197 Current_Pending_Sector  0x0032   100   100   001    Old_age   Always       -       0
198 Offline_Uncorrectable   0x0030   100   100   001    Old_age   Offline      -       0
199 UDMA_CRC_Error_Count    0x0032   100   100   001    Old_age   Always       -       0
202 Perc_Rated_Life_Used    0x0018   099   099   001    Old_age   Offline      -       1
206 Write_Error_Rate        0x000e   100   100   001    Old_age   Always       -       0

SMART Error Log Version: 1
No Errors Logged

SMART Self-test log structure revision number 1
Num  Test_Description    Status                  Remaining  LifeTime(hours)  LBA_of_first_error
# 1  Short offline       Completed without error       00%      5212         -
# 2  Vendor (0xff)       Completed without error       00%       109         -
# 3  Vendor (0xff)       Completed without error       00%        22         -

SMART Selective self-test log data structure revision number 1
 SPAN  MIN_LBA  MAX_LBA  CURRENT_TEST_STATUS
    1        0        0  Not_testing
    2        0        0  Not_testing
    3        0        0  Not_testing
    4        0        0  Not_testing
    5        0        0  Not_testing
Selective self-test flags (0x0):
  After scanning selected spans, do NOT read-scan remainder of disk.
If Selective self-test is pending on power-up, resume after 0 minute delay.

I don't have any reason to believe /dev/sda (a 64GB Crucial M4 SSD) is failing, so just wondering if anyone is aware of any issues in this new kernel?

Edit: Just happened again, some more errors:

[ 4413.797791] sd 0:0:0:0: [sda] Unhandled error code
[ 4413.797793] sd 0:0:0:0: [sda]  Result: hostbyte=0x04 driverbyte=0x00
[ 4413.797794] sd 0:0:0:0: [sda] CDB: cdb[0]=0x28: 28 00 01 de f1 78 00 00 08 00
[ 4413.797798] end_request: I/O error, dev sda, sector 31388024
[ 4413.797807] sd 0:0:0:0: [sda] Unhandled error code
[ 4413.797809] sd 0:0:0:0: [sda]  Result: hostbyte=0x04 driverbyte=0x00
[ 4413.797810] sd 0:0:0:0: [sda] CDB: cdb[0]=0x28: 28 00 01 de f1 78 00 00 08 00
[ 4413.797813] end_request: I/O error, dev sda, sector 31388024
[ 4413.797823] sd 0:0:0:0: [sda] Unhandled error code
[ 4413.797824] sd 0:0:0:0: [sda]  Result: hostbyte=0x04 driverbyte=0x00
[ 4413.797825] sd 0:0:0:0: [sda] CDB: cdb[0]=0x28: 28 00 01 de f1 78 00 00 08 00
[ 4413.797829] end_request: I/O error, dev sda, sector 31388024
[ 4413.799257] sd 0:0:0:0: [sda] Unhandled error code
[ 4413.799258] sd 0:0:0:0: [sda]  Result: hostbyte=0x04 driverbyte=0x00
[ 4413.799260] sd 0:0:0:0: [sda] CDB: cdb[0]=0x28: 28 00 01 e1 1b 70 00 00 08 00
[ 4413.799263] end_request: I/O error, dev sda, sector 31529840
[ 4413.799273] sd 0:0:0:0: [sda] Unhandled error code
[ 4413.799274] sd 0:0:0:0: [sda]  Result: hostbyte=0x04 driverbyte=0x00
[ 4413.799275] sd 0:0:0:0: [sda] CDB: cdb[0]=0x28: 28 00 01 e1 1b 70 00 00 08 00
[ 4413.799278] end_request: I/O error, dev sda, sector 31529840
[ 4413.799288] sd 0:0:0:0: [sda] Unhandled error code
[ 4413.799289] sd 0:0:0:0: [sda]  Result: hostbyte=0x04 driverbyte=0x00
[ 4413.799291] sd 0:0:0:0: [sda] CDB: cdb[0]=0x28: 28 00 01 e1 1b 70 00 00 08 00
[ 4413.799294] end_request: I/O error, dev sda, sector 31529840
[ 4413.799310] sd 0:0:0:0: [sda] Unhandled error code
[ 4413.799311] sd 0:0:0:0: [sda]  Result: hostbyte=0x04 driverbyte=0x00
[ 4413.799312] sd 0:0:0:0: [sda] CDB: cdb[0]=0x28: 28 00 01 de f1 80 00 00 20 00
[ 4413.799316] end_request: I/O error, dev sda, sector 31388032
[ 4413.799324] sd 0:0:0:0: [sda] Unhandled error code
[ 4413.799325] sd 0:0:0:0: [sda]  Result: hostbyte=0x04 driverbyte=0x00
[ 4413.799327] sd 0:0:0:0: [sda] CDB: cdb[0]=0x28: 28 00 01 de f1 80 00 00 08 00
[ 4413.799330] end_request: I/O error, dev sda, sector 31388032
[ 4413.799339] sd 0:0:0:0: [sda] Unhandled error code
[ 4413.799340] sd 0:0:0:0: [sda]  Result: hostbyte=0x04 driverbyte=0x00
[ 4413.799342] sd 0:0:0:0: [sda] CDB: cdb[0]=0x28: 28 00 01 de f1 80 00 00 08 00
[ 4413.799345] end_request: I/O error, dev sda, sector 31388032
[ 4413.799354] sd 0:0:0:0: [sda] Unhandled error code
[ 4413.799356] sd 0:0:0:0: [sda]  Result: hostbyte=0x04 driverbyte=0x00
[ 4413.799357] sd 0:0:0:0: [sda] CDB: cdb[0]=0x28: 28 00 01 de f1 88 00 00 08 00
[ 4413.799360] end_request: I/O error, dev sda, sector 31388040
[ 4413.799369] sd 0:0:0:0: [sda] Unhandled error code
[ 4413.799370] sd 0:0:0:0: [sda]  Result: hostbyte=0x04 driverbyte=0x00
[ 4413.799371] sd 0:0:0:0: [sda] CDB: cdb[0]=0x28: 28 00 01 de f1 88 00 00 08 00
[ 4413.799375] end_request: I/O error, dev sda, sector 31388040
[ 4414.251769] INFO: rcu_sched detected stall on CPU 2 (t=218685 jiffies)
[ 4414.251982] sending NMI to all CPUs:
[ 4414.251987] NMI backtrace for cpu 3
[ 4414.251989] CPU 3 
[ 4414.251990] Modules linked in: fuse ip6table_filter ip6_tables it87 hwmon_vid coretemp iptable_nat nf_nat xt_CHECKSUM iptable_mangle nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack xt_tcpudp iptable_filter ip_tables x_tables tcp_illinois ext2 dm_mod snd_hda_codec_hdmi snd_hda_codec_realtek raid1 md_mod snd_hda_intel snd_hda_codec snd_hwdep snd_pcm tun snd_page_alloc snd_timer bridge mei(C) snd stp r8168(O) i2c_i801 iTCO_wdt llc serio_raw pcspkr iTCO_vendor_support soundcore ipv6 kvm_intel kvm processor evdev vboxnetflt(O) vboxdrv(O) ext4 crc16 jbd2 mbcache sd_mod ehci_hcd ahci libahci libata xhci_hcd scsi_mod usbcore usb_common i915 video button i2c_algo_bit intel_agp intel_gtt drm_kms_helper drm i2c_core
[ 4414.252032] 
[ 4414.252034] Pid: 667, comm: ntpd Tainted: G         C O 3.2.5-1-ARCH #1 Gigabyte Technology Co., Ltd. H67MA-UD2H-B3/H67MA-UD2H-B3
[ 4414.252038] RIP: 0033:[<00007ffff39ff7b8>]  [<00007ffff39ff7b8>] 0x7ffff39ff7b7
[ 4414.252043] RSP: 002b:00007ffff38eef80  EFLAGS: 00000296
[ 4414.252044] RAX: bc4db82cd525bd7c RBX: 000000000021cef0 RCX: 000000004f3bfc5d
[ 4414.252046] RDX: 0000000022c0fb84 RSI: 00007ffff38eefb0 RDI: 00007ffff38eefb0
[ 4414.252048] RBP: 00007ffff38eef90 R08: 0000000000000001 R09: 00007ffff38eecec
[ 4414.252049] R10: 0000000000000000 R11: 0000000000000246 R12: 0000000000405cec
[ 4414.252051] R13: 00007ffff38ef208 R14: 0000000000000004 R15: 0000000000000004
[ 4414.252053] FS:  00007f2fc79d2700(0000) GS:ffff88041fac0000(0000) knlGS:0000000000000000
[ 4414.252055] CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b
[ 4414.252057] CR2: 00000000f6e941b8 CR3: 0000000404098000 CR4: 00000000000426e0
[ 4414.252058] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[ 4414.252060] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
[ 4414.252062] Process ntpd (pid: 667, threadinfo ffff880404762000, task ffff880402cfdca0)
[ 4414.252063] 
[ 4414.252064] Call Trace:
[ 4414.252066] NMI backtrace for cpu 7
[ 4414.252067] CPU 7 
[ 4414.252068] Modules linked in: fuse ip6table_filter ip6_tables it87 hwmon_vid coretemp iptable_nat nf_nat xt_CHECKSUM iptable_mangle nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack xt_tcpudp iptable_filter ip_tables x_tables tcp_illinois ext2 dm_mod snd_hda_codec_hdmi snd_hda_codec_realtek raid1 md_mod snd_hda_intel snd_hda_codec snd_hwdep snd_pcm tun snd_page_alloc snd_timer bridge mei(C) snd stp r8168(O) i2c_i801 iTCO_wdt llc serio_raw pcspkr iTCO_vendor_support soundcore ipv6 kvm_intel kvm processor evdev vboxnetflt(O) vboxdrv(O) ext4 crc16 jbd2 mbcache sd_mod ehci_hcd ahci libahci libata xhci_hcd scsi_mod usbcore usb_common i915 video button i2c_algo_bit intel_agp intel_gtt drm_kms_helper drm i2c_core
[ 4414.252097] 
[ 4414.252099] Pid: 2553, comm: deluge-web Tainted: G         C O 3.2.5-1-ARCH #1 Gigabyte Technology Co., Ltd. H67MA-UD2H-B3/H67MA-UD2H-B3
[ 4414.252102] RIP: 0033:[<00007fffbcc927b8>]  [<00007fffbcc927b8>] 0x7fffbcc927b7
[ 4414.252105] RSP: 002b:00007fffbcc37d10  EFLAGS: 00000292
[ 4414.252106] RAX: 86380e775a146228 RBX: 000000000021cef0 RCX: 000000004f3bfc5d
[ 4414.252108] RDX: 000000000b0ba6e5 RSI: 0000000000000000 RDI: 00007fffbcc37d50
[ 4414.252109] RBP: 00007fffbcc37d20 R08: 00007fa1f0f3dd28 R09: 0000000000b870a0
[ 4414.252111] R10: 0000000000000000 R11: 00007fa1efcc8420 R12: 0000000000000000
[ 4414.252112] R13: 0000000001190d98 R14: 0000000000000000 R15: 00007fa1f10246c8
[ 4414.252114] FS:  00007fa1f1137700(0000) GS:ffff88041fbc0000(0000) knlGS:0000000000000000
[ 4414.252116] CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b
[ 4414.252117] CR2: 00007f4fd5dff000 CR3: 00000003fbc3b000 CR4: 00000000000426e0
[ 4414.252119] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[ 4414.252120] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
[ 4414.252122] Process deluge-web (pid: 2553, threadinfo ffff8803fb570000, task ffff880402143900)
[ 4414.252124] 
[ 4414.252124] Call Trace:
[ 4414.252126] NMI backtrace for cpu 5
[ 4414.252128] CPU 5 
[ 4414.252129] Modules linked in: fuse ip6table_filter ip6_tables it87 hwmon_vid coretemp iptable_nat nf_nat xt_CHECKSUM iptable_mangle nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack xt_tcpudp iptable_filter ip_tables x_tables tcp_illinois ext2 dm_mod snd_hda_codec_hdmi snd_hda_codec_realtek raid1 md_mod snd_hda_intel snd_hda_codec snd_hwdep snd_pcm tun snd_page_alloc snd_timer bridge mei(C) snd stp r8168(O) i2c_i801 iTCO_wdt llc serio_raw pcspkr iTCO_vendor_support soundcore ipv6 kvm_intel kvm processor evdev vboxnetflt(O) vboxdrv(O) ext4 crc16 jbd2 mbcache sd_mod ehci_hcd ahci libahci libata xhci_hcd scsi_mod usbcore usb_common i915 video button i2c_algo_bit intel_agp intel_gtt drm_kms_helper drm i2c_core
[ 4414.252164] 
[ 4414.252166] Pid: 3032, comm: VBoxHeadless Tainted: G         C O 3.2.5-1-ARCH #1 Gigabyte Technology Co., Ltd. H67MA-UD2H-B3/H67MA-UD2H-B3
[ 4414.252170] RIP: 0033:[<00007fffbc7ff7b8>]  [<00007fffbc7ff7b8>] 0x7fffbc7ff7b7
[ 4414.252173] RSP: 002b:00007f015cc96840  EFLAGS: 00000216
[ 4414.252175] RAX: 3e4450cd077fae7c RBX: 000000000021cef0 RCX: 000000004f3bfc5d
[ 4414.252177] RDX: 000000004013d585 RSI: 0000000000000000 RDI: 00007f015cc96880
[ 4414.252178] RBP: 00007f015cc96850 R08: 000000000162d458 R09: 0000000000000bd8
[ 4414.252180] R10: 00007f0159cccab0 R11: 0000000000000000 R12: 0000000000000000
[ 4414.252182] R13: 00007f0158130066 R14: 00007f015cc9692c R15: 00007f015cc96b30
[ 4414.252184] FS:  00007f015cc97700(0000) GS:ffff88041fb40000(0000) knlGS:0000000000000000
[ 4414.252186] CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b
[ 4414.252187] CR2: 00000000f0997000 CR3: 00000003f9365000 CR4: 00000000000426e0
[ 4414.252189] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[ 4414.252191] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
[ 4414.252193] Process VBoxHeadless (pid: 3032, threadinfo ffff8803fc13e000, task ffff8803fc64e3c0)
[ 4414.252194] 
[ 4414.252195] Call Trace:
[ 4414.252196] NMI backtrace for cpu 1
[ 4414.252197] CPU 1 
[ 4414.252198] Modules linked in: fuse ip6table_filter ip6_tables it87 hwmon_vid coretemp iptable_nat nf_nat xt_CHECKSUM iptable_mangle nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack xt_tcpudp iptable_filter ip_tables x_tables tcp_illinois ext2 dm_mod snd_hda_codec_hdmi snd_hda_codec_realtek raid1 md_mod snd_hda_intel snd_hda_codec snd_hwdep snd_pcm tun snd_page_alloc snd_timer bridge mei(C) snd stp r8168(O) i2c_i801 iTCO_wdt llc serio_raw pcspkr iTCO_vendor_support soundcore ipv6 kvm_intel kvm processor evdev vboxnetflt(O) vboxdrv(O) ext4 crc16 jbd2 mbcache sd_mod ehci_hcd ahci libahci libata xhci_hcd scsi_mod usbcore usb_common i915 video button i2c_algo_bit intel_agp intel_gtt drm_kms_helper drm i2c_core
[ 4414.252228] 
[ 4414.252229] Pid: 19012, comm: xfce4-panel Tainted: G         C O 3.2.5-1-ARCH #1 Gigabyte Technology Co., Ltd. H67MA-UD2H-B3/H67MA-UD2H-B3
[ 4414.252232] RIP: 0033:[<00007fffb079d888>]  [<00007fffb079d888>] 0x7fffb079d887
[ 4414.252235] RSP: 002b:00007fffb0694370  EFLAGS: 00000216
[ 4414.252236] RAX: ffffff80c1df91c4 RBX: 000000000021cef0 RCX: 0000000000000018
[ 4414.252238] RDX: 00000000004b771e RSI: 00007fffb06943b0 RDI: 00007fffb06943b0
[ 4414.252240] RBP: 00007fffb0694390 R08: 0000000001e842b8 R09: 0000000000004a44
[ 4414.252241] R10: 0000000000000001 R11: 0000000000000001 R12: 000000042bab2ff4
[ 4414.252243] R13: 0769ff608960efec R14: 00007fb1c8c2f020 R15: 00007fb1c8981110
[ 4414.252245] FS:  00007fb1cb3608c0(0000) GS:ffff88041fa40000(0000) knlGS:0000000000000000
[ 4414.252246] CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b
[ 4414.252248] CR2: 00000000083796a0 CR3: 00000000c6c1d000 CR4: 00000000000426e0
[ 4414.252249] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[ 4414.252251] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
[ 4414.252253] Process xfce4-panel (pid: 19012, threadinfo ffff88009f11c000, task ffff88018187c740)
[ 4414.252254] 
[ 4414.252255] Call Trace:
[ 4414.252256] NMI backtrace for cpu 6
[ 4414.252257] CPU 6 
[ 4414.252258] Modules linked in: fuse ip6table_filter ip6_tables it87 hwmon_vid coretemp iptable_nat nf_nat xt_CHECKSUM iptable_mangle nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack xt_tcpudp iptable_filter ip_tables x_tables tcp_illinois ext2 dm_mod snd_hda_codec_hdmi snd_hda_codec_realtek raid1 md_mod snd_hda_intel snd_hda_codec snd_hwdep snd_pcm tun snd_page_alloc snd_timer bridge mei(C) snd stp r8168(O) i2c_i801 iTCO_wdt llc serio_raw pcspkr iTCO_vendor_support soundcore ipv6 kvm_intel kvm processor evdev vboxnetflt(O) vboxdrv(O) ext4 crc16 jbd2 mbcache sd_mod ehci_hcd ahci libahci libata xhci_hcd scsi_mod usbcore usb_common i915 video button i2c_algo_bit intel_agp intel_gtt drm_kms_helper drm i2c_core
[ 4414.252292] 
[ 4414.252294] Pid: 2842, comm: imap Tainted: G         C O 3.2.5-1-ARCH #1 Gigabyte Technology Co., Ltd. H67MA-UD2H-B3/H67MA-UD2H-B3
[ 4414.252297] RIP: 0033:[<00007ffff0aed7b8>]  [<00007ffff0aed7b8>] 0x7ffff0aed7b7
[ 4414.252301] RSP: 002b:00007ffff0a15dd0  EFLAGS: 00000216
[ 4414.252303] RAX: 610e1cee8dac77d1 RBX: 000000000021cef0 RCX: 000000004f3bfc5d
[ 4414.252305] RDX: 00000000aaa9ba63 RSI: 0000000000000000 RDI: 00007ffff0a15e40
[ 4414.252306] RBP: 00007ffff0a15de0 R08: 00000000022d80c0 R09: 0101010101010101
[ 4414.252308] R10: 0000000000000000 R11: 00007f5c2a837520 R12: 0000000000000000
[ 4414.252310] R13: 00007ffff0a16060 R14: 0000000000000000 R15: 0000000000000000
[ 4414.252312] FS:  00007f5c2b1c3700(0000) GS:ffff88041fb80000(0000) knlGS:0000000000000000
[ 4414.252314] CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b
[ 4414.252315] CR2: 00000000e6c91b84 CR3: 00000003fc623000 CR4: 00000000000426e0
[ 4414.252317] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[ 4414.252319] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
[ 4414.252320] Process imap (pid: 2842, threadinfo ffff8803fb934000, task ffff880403cf6ae0)
[ 4414.252322] 
[ 4414.252323] Call Trace:
[ 4414.252324] NMI backtrace for cpu 4
[ 4414.252326] CPU 4 
[ 4414.252327] Modules linked in: fuse ip6table_filter ip6_tables it87 hwmon_vid coretemp iptable_nat nf_nat xt_CHECKSUM iptable_mangle nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack xt_tcpudp iptable_filter ip_tables x_tables tcp_illinois ext2 dm_mod snd_hda_codec_hdmi snd_hda_codec_realtek raid1 md_mod snd_hda_intel snd_hda_codec snd_hwdep snd_pcm tun snd_page_alloc snd_timer bridge mei(C) snd stp r8168(O) i2c_i801 iTCO_wdt llc serio_raw pcspkr iTCO_vendor_support soundcore ipv6 kvm_intel kvm processor evdev vboxnetflt(O) vboxdrv(O) ext4 crc16 jbd2 mbcache sd_mod ehci_hcd ahci libahci libata xhci_hcd scsi_mod usbcore usb_common i915 video button i2c_algo_bit intel_agp intel_gtt drm_kms_helper drm i2c_core
[ 4414.252361] 
[ 4414.252363] Pid: 18981, comm: autocutsel Tainted: G         C O 3.2.5-1-ARCH #1 Gigabyte Technology Co., Ltd. H67MA-UD2H-B3/H67MA-UD2H-B3
[ 4414.252367] RIP: 0033:[<00007fff02dff7b8>]  [<00007fff02dff7b8>] 0x7fff02dff7b7
[ 4414.252370] RSP: 002b:00007fff02dc8300  EFLAGS: 00000296
[ 4414.252372] RAX: d6ed70714c5e0d10 RBX: 000000000021cef0 RCX: 000000004f3bfc5d
[ 4414.252374] RDX: 00000000b067b05c RSI: 0000000000000000 RDI: 00007fff02dc84e0
[ 4414.252375] RBP: 00007fff02dc8310 R08: 0000000000000000 R09: 0000000000000001
[ 4414.252377] R10: 0000000000000000 R11: 0000000000000246 R12: 0000000000000000
[ 4414.252379] R13: 0000000000000002 R14: 0000000000000000 R15: 0000000000000001
[ 4414.252381] FS:  00007f0e7f50c740(0000) GS:ffff88041fb00000(0000) knlGS:0000000000000000
[ 4414.252383] CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b
[ 4414.252384] CR2: 00000000f30741b8 CR3: 00000000c6e75000 CR4: 00000000000426e0
[ 4414.252386] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[ 4414.252388] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
[ 4414.252390] Process autocutsel (pid: 18981, threadinfo ffff880037a9e000, task ffff880287a74740)
[ 4414.252391] 
[ 4414.252392] Call Trace:
[ 4414.252393] NMI backtrace for cpu 0
[ 4414.252395] CPU 0 
[ 4414.252395] Modules linked in: fuse ip6table_filter ip6_tables it87 hwmon_vid coretemp iptable_nat nf_nat xt_CHECKSUM iptable_mangle nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack xt_tcpudp iptable_filter ip_tables x_tables tcp_illinois ext2 dm_mod snd_hda_codec_hdmi snd_hda_codec_realtek raid1 md_mod snd_hda_intel snd_hda_codec snd_hwdep snd_pcm tun snd_page_alloc snd_timer bridge mei(C) snd stp r8168(O) i2c_i801 iTCO_wdt llc serio_raw pcspkr iTCO_vendor_support soundcore ipv6 kvm_intel kvm processor evdev vboxnetflt(O) vboxdrv(O) ext4 crc16 jbd2 mbcache sd_mod ehci_hcd ahci libahci libata xhci_hcd scsi_mod usbcore usb_common i915 video button i2c_algo_bit intel_agp intel_gtt drm_kms_helper drm i2c_core
[ 4414.252425] 
[ 4414.252426] Pid: 19001, comm: gpg-agent Tainted: G         C O 3.2.5-1-ARCH #1 Gigabyte Technology Co., Ltd. H67MA-UD2H-B3/H67MA-UD2H-B3
[ 4414.252429] RIP: 0033:[<00007fffa6f4e7b8>]  [<00007fffa6f4e7b8>] 0x7fffa6f4e7b7
[ 4414.252432] RSP: 002b:0000000001c3d420  EFLAGS: 00000212
[ 4414.252433] RAX: 099d39afc2285e49 RBX: 000000000021cef0 RCX: 000000004f3bfc5d
[ 4414.252435] RDX: 0000000022381691 RSI: 0000000000000000 RDI: 0000000001c3d778
[ 4414.252437] RBP: 0000000001c3d430 R08: 0000000000000000 R09: 0000000000000000
[ 4414.252438] R10: 0000000000000006 R11: 0000000000000000 R12: 0000000000000000
[ 4414.252440] R13: 00007f2571309fc0 R14: 00007f2571309f90 R15: 0000000000000000
[ 4414.252441] FS:  00007f2571b8e700(0000) GS:ffff88041fa00000(0000) knlGS:0000000000000000
[ 4414.252443] CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b
[ 4414.252445] CR2: 00007f19a00d2000 CR3: 00000000c6dc9000 CR4: 00000000000426f0
[ 4414.252446] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[ 4414.252448] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
[ 4414.252449] Process gpg-agent (pid: 19001, threadinfo ffff8800d68e2000, task ffff880254c45ca0)
[ 4414.252451] 
[ 4414.252451] Call Trace:
[ 4414.252453] NMI backtrace for cpu 2
[ 4414.252454] CPU 2 
[ 4414.252454] Modules linked in: fuse ip6table_filter ip6_tables it87 hwmon_vid coretemp iptable_nat nf_nat xt_CHECKSUM iptable_mangle nf_conntrack_ipv4 nf_defrag_ipv4 xt_state nf_conntrack xt_tcpudp iptable_filter ip_tables x_tables tcp_illinois ext2 dm_mod snd_hda_codec_hdmi snd_hda_codec_realtek raid1 md_mod snd_hda_intel snd_hda_codec snd_hwdep snd_pcm tun snd_page_alloc snd_timer bridge mei(C) snd stp r8168(O) i2c_i801 iTCO_wdt llc serio_raw pcspkr iTCO_vendor_support soundcore ipv6 kvm_intel kvm processor evdev vboxnetflt(O) vboxdrv(O) ext4 crc16 jbd2 mbcache sd_mod ehci_hcd ahci libahci libata xhci_hcd scsi_mod usbcore usb_common i915 video button i2c_algo_bit intel_agp intel_gtt drm_kms_helper drm i2c_core
[ 4414.252484] 
[ 4414.252485] Pid: 15, comm: ksoftirqd/2 Tainted: G         C O 3.2.5-1-ARCH #1 Gigabyte Technology Co., Ltd. H67MA-UD2H-B3/H67MA-UD2H-B3
[ 4414.252488] RIP: 0010:[<ffffffff81241122>]  [<ffffffff81241122>] __const_udelay+0x12/0x40
[ 4414.252493] RSP: 0018:ffff88041fa83da8  EFLAGS: 00000006
[ 4414.252495] RAX: 0000000000000c00 RBX: 0000000000002710 RCX: 0000000000000082
[ 4414.252496] RDX: 0000000000ac8a7d RSI: 0000000000000002 RDI: 0000000000418958
[ 4414.252498] RBP: ffff88041fa83da8 R08: 0000000000000000 R09: 0000000000000000
[ 4414.252499] R10: 0000000000000000 R11: 0000000000000001 R12: ffffffff8182ef80
[ 4414.252501] R13: ffffffff8182f080 R14: ffff88041fa8e580 R15: 0000000000000000
[ 4414.252503] FS:  0000000000000000(0000) GS:ffff88041fa80000(0000) knlGS:0000000000000000
[ 4414.252504] CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b
[ 4414.252506] CR2: 00007f9b581bed28 CR3: 00000000c6e75000 CR4: 00000000000426e0
[ 4414.252507] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[ 4414.252509] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
[ 4414.252511] Process ksoftirqd/2 (pid: 15, threadinfo ffff88040558e000, task ffff8804054ceae0)
[ 4414.252512] Stack:
[ 4414.252726]  ffff88041fa83dc8 ffffffff810351aa 0000000000010000 ffff88041fa8eb00
[ 4414.252729]  ffff88041fa83e18 ffffffff810d10c9 0000000000000000 00000406a921cb0b
[ 4414.252732]  ffff88041fa83e28 0000000000000002 0000000000000000 0000000000000002
[ 4414.252735] Call Trace:
[ 4414.252952]  <IRQ> 
[ 4414.253174]  [<ffffffff810351aa>] arch_trigger_all_cpu_backtrace+0x6a/0x90
[ 4414.253178]  [<ffffffff810d10c9>] __rcu_pending+0x169/0x400
[ 4414.253181]  [<ffffffff810d2957>] rcu_check_callbacks+0xb7/0x2d0
[ 4414.253184]  [<ffffffff81076d48>] update_process_times+0x48/0x90
[ 4414.253188]  [<ffffffff8109a0c4>] tick_sched_timer+0x64/0xc0
[ 4414.253191]  [<ffffffff8108c704>] __run_hrtimer+0x74/0x2c0
[ 4414.253194]  [<ffffffff8109a060>] ? tick_nohz_handler+0x100/0x100
[ 4414.253196]  [<ffffffff8101d639>] ? read_tsc+0x9/0x20
[ 4414.253199]  [<ffffffff8108d3c3>] hrtimer_interrupt+0xf3/0x210
[ 4414.253202]  [<ffffffff8142b9ac>] ? call_softirq+0x1c/0x30
[ 4414.253204]  [<ffffffff8142c349>] smp_apic_timer_interrupt+0x69/0x99
[ 4414.253208]  [<ffffffff8142a21e>] apic_timer_interrupt+0x6e/0x80
[ 4414.253209]  <EOI> 
[ 4414.253433]  [<ffffffff81066d32>] ? vprintk+0x362/0x510
[ 4414.253435]  [<ffffffff8108e4c2>] ? up+0x32/0x50
[ 4414.253439]  [<ffffffff8141ef52>] printk+0x41/0x43
[ 4414.253444]  [<ffffffff8121d758>] blk_update_request+0x288/0x520
[ 4414.253447]  [<ffffffff8121da21>] blk_update_bidi_request+0x31/0x90
[ 4414.253450]  [<ffffffff8121e91c>] blk_end_bidi_request+0x2c/0x80
[ 4414.253452]  [<ffffffff8121e980>] blk_end_request+0x10/0x20
[ 4414.253455]  [<ffffffff8121e9c4>] blk_end_request_err+0x34/0x60
[ 4414.253461]  [<ffffffffa018509a>] scsi_io_completion+0x3aa/0x5e0 [scsi_mod]
[ 4414.253465]  [<ffffffffa017c940>] scsi_finish_command+0xb0/0xe0 [scsi_mod]
[ 4414.253470]  [<ffffffffa0184b75>] scsi_softirq_done+0xa5/0x140 [scsi_mod]
[ 4414.253473]  [<ffffffff812241d3>] blk_done_softirq+0x83/0xa0
[ 4414.253475]  [<ffffffff8106d780>] __do_softirq+0xb0/0x270
[ 4414.253477]  [<ffffffff8106da65>] run_ksoftirqd+0x125/0x230
[ 4414.253479]  [<ffffffff8106d940>] ? __do_softirq+0x270/0x270
[ 4414.253482]  [<ffffffff8108842c>] kthread+0x8c/0xa0
[ 4414.253484]  [<ffffffff8142b8b4>] kernel_thread_helper+0x4/0x10
[ 4414.253486]  [<ffffffff810883a0>] ? kthread_worker_fn+0x190/0x190
[ 4414.253488]  [<ffffffff8142b8b0>] ? gs_change+0x13/0x13
[ 4414.253490] Code: 66 90 ff 15 d9 28 61 00 5d c3 66 66 66 66 66 66 2e 0f 1f 84 00 00 00 00 00 55 48 89 e5 66 66 66 66 90 65 48 8b 14 25 58 33 01 00 <48> 8d 0c 92 48 8d 04 bd 00 00 00 00 48 89 ca 48 c1 e2 04 48 29 
[ 4414.254091] INFO: rcu_sched detected stalls on CPUs/tasks: { 2} (detected by 4, t=218685 jiffies)
[ 4414.254095] INFO: rcu_bh detected stalls on CPUs/tasks: { 2} (detected by 4, t=218685 jiffies)
[ 4414.254097] INFO: rcu_preempt detected stalls on CPUs/tasks: { 2} (detected by 4, t=218685 jiffies)
[ 4414.254191] Clocksource tsc unstable (delta = 599932202606 ns)
[ 4414.255279] Call Trace:
[ 4414.255281]  <IRQ>  [<ffffffff810351aa>] arch_trigger_all_cpu_backtrace+0x6a/0x90
[ 4414.255289]  [<ffffffff810d10c9>] __rcu_pending+0x169/0x400
[ 4414.255291]  [<ffffffff810d2957>] rcu_check_callbacks+0xb7/0x2d0
[ 4414.255295]  [<ffffffff81076d48>] update_process_times+0x48/0x90
[ 4414.255298]  [<ffffffff8109a0c4>] tick_sched_timer+0x64/0xc0
[ 4414.255301]  [<ffffffff8108c704>] __run_hrtimer+0x74/0x2c0
[ 4414.255304]  [<ffffffff8109a060>] ? tick_nohz_handler+0x100/0x100
[ 4414.255307]  [<ffffffff8101d639>] ? read_tsc+0x9/0x20
[ 4414.255309]  [<ffffffff8108d3c3>] hrtimer_interrupt+0xf3/0x210
[ 4414.255312]  [<ffffffff8142b9ac>] ? call_softirq+0x1c/0x30
[ 4414.255314]  [<ffffffff8142c349>] smp_apic_timer_interrupt+0x69/0x99
[ 4414.255318]  [<ffffffff8142a21e>] apic_timer_interrupt+0x6e/0x80
[ 4414.255319]  <EOI>  [<ffffffff81066d32>] ? vprintk+0x362/0x510
[ 4414.255324]  [<ffffffff8108e4c2>] ? up+0x32/0x50
[ 4414.255332]  [<ffffffff8141ef52>] printk+0x41/0x43
[ 4414.255336]  [<ffffffff8121d758>] blk_update_request+0x288/0x520
[ 4414.255340]  [<ffffffff8121da21>] blk_update_bidi_request+0x31/0x90
[ 4414.255343]  [<ffffffff8121e91c>] blk_end_bidi_request+0x2c/0x80
[ 4414.255345]  [<ffffffff8121e980>] blk_end_request+0x10/0x20
[ 4414.255348]  [<ffffffff8121e9c4>] blk_end_request_err+0x34/0x60
[ 4414.255353]  [<ffffffffa018509a>] scsi_io_completion+0x3aa/0x5e0 [scsi_mod]
[ 4414.255358]  [<ffffffffa017c940>] scsi_finish_command+0xb0/0xe0 [scsi_mod]
[ 4414.255362]  [<ffffffffa0184b75>] scsi_softirq_done+0xa5/0x140 [scsi_mod]
[ 4414.255365]  [<ffffffff812241d3>] blk_done_softirq+0x83/0xa0
[ 4414.255367]  [<ffffffff8106d780>] __do_softirq+0xb0/0x270
[ 4414.255369]  [<ffffffff8106da65>] run_ksoftirqd+0x125/0x230
[ 4414.255371]  [<ffffffff8106d940>] ? __do_softirq+0x270/0x270
[ 4414.255374]  [<ffffffff8108842c>] kthread+0x8c/0xa0
[ 4414.255376]  [<ffffffff8142b8b4>] kernel_thread_helper+0x4/0x10
[ 4414.255378]  [<ffffffff810883a0>] ? kthread_worker_fn+0x190/0x190
[ 4414.255380]  [<ffffffff8142b8b0>] ? gs_change+0x13/0x13
[ 4414.256423] sd 0:0:0:0: [sda] Unhandled error code
[ 4414.256425] sd 0:0:0:0: [sda]  Result: hostbyte=0x04 driverbyte=0x00
[ 4414.256428] sd 0:0:0:0: [sda] CDB: cdb[0]=0x28: 28 00 03 74 0e 38 00 00 28 00
[ 4414.256434] end_request: I/O error, dev sda, sector 57937464
[ 4414.256858] sd 0:0:0:0: [sda] Unhandled error code
[ 4414.256859] sd 0:0:0:0: [sda]  Result: hostbyte=0x04 driverbyte=0x00
[ 4414.256861] sd 0:0:0:0: [sda] CDB: cdb[0]=0x28: 28 00 03 74 0e 00 00 00 30 00
[ 4414.256866] end_request: I/O error, dev sda, sector 57937408
[ 4414.257286] sd 0:0:0:0: [sda] Unhandled error code
[ 4414.257287] sd 0:0:0:0: [sda]  Result: hostbyte=0x04 driverbyte=0x00
[ 4414.257289] sd 0:0:0:0: [sda] CDB: cdb[0]=0x28: 28 00 03 74 0d e0 00 00 10 00
[ 4414.257294] end_request: I/O error, dev sda, sector 57937376
[ 4414.257714] sd 0:0:0:0: [sda] Unhandled error code
[ 4414.257715] sd 0:0:0:0: [sda]  Result: hostbyte=0x04 driverbyte=0x00
[ 4414.257717] sd 0:0:0:0: [sda] CDB: cdb[0]=0x2a: 2a 00 00 99 38 c0 00 00 20 00
[ 4414.257722] end_request: I/O error, dev sda, sector 10041536
[ 4414.258155] Buffer I/O error on device sda2, logical block 1203736
[ 4414.258608] Buffer I/O error on device sda2, logical block 1203737
[ 4414.259065] Buffer I/O error on device sda2, logical block 1203738
[ 4414.259518] Buffer I/O error on device sda2, logical block 1203739
[ 4414.259956] EXT4-fs warning (device sda2): ext4_end_bio:244: I/O error writing to inode 3423353 (offset 509706240 size 16384 starting block 1255196)
[ 4414.259969] sd 0:0:0:0: [sda] Unhandled error code
[ 4414.259970] sd 0:0:0:0: [sda]  Result: hostbyte=0x04 driverbyte=0x00
[ 4414.259972] sd 0:0:0:0: [sda] CDB: cdb[0]=0x2a: 2a 00 00 98 dc 80 00 00 20 00

2012 Feb 15 18:53:43 weeaboo [ 4414.252512] Stack:
2012 Feb 15 18:53:43 weeaboo [ 4414.252735] Call Trace:
2012 Feb 15 18:53:43 weeaboo [ 4414.252952]  <IRQ> 
2012 Feb 15 18:53:43 weeaboo [ 4414.253209]  <EOI> 
2012 Feb 15 18:53:43 weeaboo [ 4414.253490] Code: 66 90 ff 15 d9 28 61 00 5d c3 66 66 66 66 66 66 2e 0f 1f 84 00 00 00 00 00 55 48 89 e5 66 66 66 66 90 65 48 8b 14 25 58 33 01 00 <48> 8d 0c 92 48 8d 04 bd 00 00 00 00 48 89 ca 48 c1 e2 04 48 29 
htop

Edit 2: Couldn't stop laughing when I found this firmware update for the SSD :V

Release Date: 01/13/2012
Change Log:
Changes made in version 0009 (m4 can be updated to revision 0309 directly from either revision 0001, 0002, or 0009)
Correct a condition where an incorrect response to a SMART counter will cause the m4 drive to become unresponsive after 5184 hours of Power-on time. The drive will recover after a power cycle, however, this failure will repeat once per hour after reaching this point. The condition will allow the end user to successfully update firmware, and poses no risk to user or system data stored on the drive.
This firmware update is STRONGLY RECOMMENDED for drives in the field. Although the failure mode due to the SMART Power On Hours counter poses no risk to saved user data, the failure mode can become repetitive, and pose a nuisance to the end user. If the end user has not yet observed this failure mode, this update is required to prevent it from happening.

Last edited by Darkimmortal (2012-02-16 02:15:49)

monzer4819 · 2012-02-20 15:23:44

Thank you for this post. Same problem, same drive. Assuming you leave your server on 24/7, I'm guessing you bought your drive 5 days before me

Did the firmware correct the issue?

Darkimmortal · 2012-02-20 16:45:23

monzer4819 wrote:

Thank you for this post. Same problem, same drive. Assuming you leave your server on 24/7, I'm guessing you bought your drive 5 days before me
Did the firmware correct the issue?

Hehe cool and yes all sorted now

Only ill effect was having to run a full recovery on the separate RAID in the system which lost its superblocks or something due to IO completely dying so many times during rebuilds.

Arch Linux

#1 2012-02-15 17:50:29

Strange issues since kernel 3.2.5 update

#2 2012-02-20 15:23:44

Re: Strange issues since kernel 3.2.5 update

#3 2012-02-20 16:45:23

Re: Strange issues since kernel 3.2.5 update

Board footer