You are not logged in.

Pages: 1

#1 2022-12-30 12:55:38

djmwj1
Member
Registered: 2013-10-04
Posts: 32

Hard Drive Diagnostic Help // AHCI AMD-Vi Issue?

Happy Holidays! I recently purchased some seagate EXOS16 drives and I am having troubles identifying which drive I am having problems with.

Two Drives:
/dev/sdc
/dev/sdd

Both have dm-crypt luks volumes on sdc1 and sdd1 respectively.

Originally I put both of them into a btrfs RAID1 volume and experienced an issue when ryncing data to the new drives. The BTRFS volume went offline so i started troubleshooting.

I ran on both drives:
SMART Tests:
SeaChest_SMART --shortDST 
-No issues found
SeaChest_SMART --longDST
-No issues found

Generic Tests:
SeaChest_GenericTests --twoMinuteGeneric
-No issues found
SeaChest_GenericTests --twoMinuteGeneric
-No issues found

SeaChest_GenericTests --longGeneric --genericMode write
-No issues found

With all of these tests not turning up any issues, I decided to stand up one drive ((/dev/sdc) and redo the btrfs filesystem over a new dm-crypt partition (named crypt_media1) (sdc1 again) and rsync all the data.
--It performed flawlessly.

So then I re-added /dev/sdd with its own encrypted volume (crypt_media2) and ran the balance operation below. It resulted in an error and the drives went offline.
--This leads me to believe the issue is with /dev/sdd but looking at the logs to my untrained eyes, it looks like /dev/sdc popped the first error? Looking for help on deciphering which drive is bad so i can get a replacement. Thanks!


Command used for balance:

sudo btrfs balance start -dconvert=raid1 -mconvert=raid1 /var/lib/btrfs_media

Logs of failure during BTRFS balance operation:

[ 3481.835349] BTRFS info (device dm-10): found 8 extents, stage: update data pointers
[ 3481.993914] BTRFS info (device dm-10): relocating block group 1520448831488 flags data|raid0
[ 3491.580041] BTRFS info (device dm-10): found 8 extents, stage: move data extents
[ 3491.837502] BTRFS info (device dm-10): found 8 extents, stage: update data pointers
[ 3492.011510] BTRFS info (device dm-10): relocating block group 1519375089664 flags data|raid0
[ 3501.678745] BTRFS info (device dm-10): found 8 extents, stage: move data extents
[ 3501.993379] BTRFS info (device dm-10): found 8 extents, stage: update data pointers
[ 3502.166789] BTRFS info (device dm-10): relocating block group 1518301347840 flags data|raid0
[ 3503.151770] ahci 0000:48:00.0: AMD-Vi: Event logged [IO_PAGE_FAULT domain=0x0044 address=0x7ffffe00000 flags=0x0020]
[ 3503.151786] ahci 0000:48:00.0: AMD-Vi: Event logged [IO_PAGE_FAULT domain=0x0044 address=0x7ffffe00180 flags=0x0020]
[ 3503.151796] ahci 0000:48:00.0: AMD-Vi: Event logged [IO_PAGE_FAULT domain=0x0044 address=0x7ffffe00200 flags=0x0020]
[ 3503.151805] ahci 0000:48:00.0: AMD-Vi: Event logged [IO_PAGE_FAULT domain=0x0044 address=0x7ffffe00300 flags=0x0020]
[ 3503.151814] ahci 0000:48:00.0: AMD-Vi: Event logged [IO_PAGE_FAULT domain=0x0044 address=0x7ffffe00400 flags=0x0020]
[ 3503.151823] ahci 0000:48:00.0: AMD-Vi: Event logged [IO_PAGE_FAULT domain=0x0044 address=0x7ffffe00500 flags=0x0020]
[ 3503.151832] ahci 0000:48:00.0: AMD-Vi: Event logged [IO_PAGE_FAULT domain=0x0044 address=0x7ffffe00600 flags=0x0020]
[ 3503.151841] ahci 0000:48:00.0: AMD-Vi: Event logged [IO_PAGE_FAULT domain=0x0044 address=0x7ffffe00680 flags=0x0020]
[ 3503.151854] ahci 0000:48:00.0: AMD-Vi: Event logged [IO_PAGE_FAULT domain=0x0044 address=0x7ffffe00780 flags=0x0020]
[ 3503.151867] ahci 0000:48:00.0: AMD-Vi: Event logged [IO_PAGE_FAULT domain=0x0044 address=0x7ffffe00880 flags=0x0020]
[ 3503.155603] AMD-Vi: IOMMU event log overflow
[ 3503.156519] BTRFS warning (device dm-10): csum failed root -9 ino 535 off 245891072 csum 0x976b4e2c expected csum 0xc5f8675c mirror 1
[ 3503.156531] BTRFS error (device dm-10): bdev /dev/mapper/crypt_media1 errs: wr 0, rd 0, flush 0, corrupt 1, gen 0
[ 3503.156534] BTRFS warning (device dm-10): csum failed root -9 ino 535 off 245956608 csum 0x065dcd93 expected csum 0x2b5bfc2a mirror 1
[ 3503.156537] BTRFS warning (device dm-10): csum failed root -9 ino 535 off 246022144 csum 0x9cb04056 expected csum 0xd3a133c4 mirror 1
[ 3503.156538] BTRFS warning (device dm-10): csum failed root -9 ino 535 off 246087680 csum 0x57e5e6e1 expected csum 0x1c24c176 mirror 1
[ 3503.156538] BTRFS warning (device dm-10): csum failed root -9 ino 535 off 246153216 csum 0x41f4d368 expected csum 0x8a4d5288 mirror 1
[ 3503.156542] BTRFS error (device dm-10): bdev /dev/mapper/crypt_media1 errs: wr 0, rd 0, flush 0, corrupt 2, gen 0
[ 3503.156544] BTRFS error (device dm-10): bdev /dev/mapper/crypt_media1 errs: wr 0, rd 0, flush 0, corrupt 3, gen 0
[ 3503.156543] BTRFS warning (device dm-10): csum failed root -9 ino 535 off 246218752 csum 0xb920132f expected csum 0x465bc074 mirror 1
[ 3503.156547] BTRFS warning (device dm-10): csum failed root -9 ino 535 off 245895168 csum 0x31f1b25e expected csum 0xcad09771 mirror 1
[ 3503.156548] BTRFS error (device dm-10): bdev /dev/mapper/crypt_media1 errs: wr 0, rd 0, flush 0, corrupt 5, gen 0
[ 3503.156548] BTRFS error (device dm-10): bdev /dev/mapper/crypt_media1 errs: wr 0, rd 0, flush 0, corrupt 5, gen 0
[ 3503.156549] BTRFS error (device dm-10): bdev /dev/mapper/crypt_media1 errs: wr 0, rd 0, flush 0, corrupt 6, gen 0
[ 3503.156548] BTRFS warning (device dm-10): csum failed root -9 ino 535 off 246349824 csum 0xb7678f7c expected csum 0x9366328a mirror 1
[ 3503.156552] BTRFS error (device dm-10): bdev /dev/mapper/crypt_media1 errs: wr 0, rd 0, flush 0, corrupt 7, gen 0
[ 3503.156556] BTRFS error (device dm-10): bdev /dev/mapper/crypt_media1 errs: wr 0, rd 0, flush 0, corrupt 8, gen 0
[ 3503.156562] BTRFS warning (device dm-10): csum failed root -9 ino 535 off 246288384 csum 0x46ec4655 expected csum 0xd51aaa18 mirror 1
[ 3503.156563] BTRFS warning (device dm-10): csum failed root -9 ino 535 off 245899264 csum 0x8e609c9f expected csum 0x7e31a862 mirror 1
[ 3503.156563] BTRFS error (device dm-10): bdev /dev/mapper/crypt_media1 errs: wr 0, rd 0, flush 0, corrupt 9, gen 0
[ 3503.156567] BTRFS error (device dm-10): bdev /dev/mapper/crypt_media1 errs: wr 0, rd 0, flush 0, corrupt 10, gen 0
[ 3503.159274] AMD-Vi: IOMMU event log overflow
[ 3503.298664] AMD-Vi: IOMMU event log overflow
[ 3503.302509] AMD-Vi: IOMMU event log overflow
[ 3504.309706] AMD-Vi: IOMMU event log overflow
[ 3504.313478] AMD-Vi: IOMMU event log overflow
[ 3504.480924] AMD-Vi: IOMMU event log overflow
[ 3504.484463] AMD-Vi: IOMMU event log overflow
[ 3504.590742] AMD-Vi: IOMMU event log overflow
[ 3504.594682] AMD-Vi: IOMMU event log overflow
[ 3508.282563] amd_iommu_report_page_fault: 58620 callbacks suppressed
[ 3508.282569] ahci 0000:48:00.0: AMD-Vi: Event logged [IO_PAGE_FAULT domain=0x0044 address=0x7ffffe00000 flags=0x0020]
[ 3508.282583] ahci 0000:48:00.0: AMD-Vi: Event logged [IO_PAGE_FAULT domain=0x0044 address=0x7ffffe00180 flags=0x0020]
[ 3508.282592] ahci 0000:48:00.0: AMD-Vi: Event logged [IO_PAGE_FAULT domain=0x0044 address=0x7ffffe00200 flags=0x0020]
[ 3508.282601] ahci 0000:48:00.0: AMD-Vi: Event logged [IO_PAGE_FAULT domain=0x0044 address=0x7ffffe00300 flags=0x0020]
[ 3508.282610] ahci 0000:48:00.0: AMD-Vi: Event logged [IO_PAGE_FAULT domain=0x0044 address=0x7ffffe00400 flags=0x0020]
[ 3508.282619] ahci 0000:48:00.0: AMD-Vi: Event logged [IO_PAGE_FAULT domain=0x0044 address=0x7ffffe00500 flags=0x0020]
[ 3508.282628] ahci 0000:48:00.0: AMD-Vi: Event logged [IO_PAGE_FAULT domain=0x0044 address=0x7ffffe00600 flags=0x0020]
[ 3508.282637] ahci 0000:48:00.0: AMD-Vi: Event logged [IO_PAGE_FAULT domain=0x0044 address=0x7ffffe00700 flags=0x0020]
[ 3508.282646] ahci 0000:48:00.0: AMD-Vi: Event logged [IO_PAGE_FAULT domain=0x0044 address=0x7ffffe00800 flags=0x0020]
[ 3508.282655] ahci 0000:48:00.0: AMD-Vi: Event logged [IO_PAGE_FAULT domain=0x0044 address=0x7ffffe00900 flags=0x0020]
[ 3508.286147] amd_iommu_int_thread: 98 callbacks suppressed
[ 3508.286148] AMD-Vi: IOMMU event log overflow
[ 3508.287789] btrfs_print_data_csum_error: 20456 callbacks suppressed
[ 3508.287793] btrfs_dev_stat_inc_and_print: 20453 callbacks suppressed
[ 3508.287795] BTRFS warning (device dm-10): csum failed root -9 ino 535 off 977534976 csum 0xfa962bc7 expected csum 0x3285215e mirror 1
[ 3508.287797] BTRFS error (device dm-10): bdev /dev/mapper/crypt_media1 errs: wr 0, rd 0, flush 0, corrupt 20481, gen 0
[ 3508.287796] BTRFS warning (device dm-10): csum failed root -9 ino 535 off 977862656 csum 0x856615ab expected csum 0x811f09ba mirror 1
[ 3508.287796] BTRFS warning (device dm-10): csum failed root -9 ino 535 off 977666048 csum 0xbed87a8a expected csum 0x3ffb5546 mirror 1
[ 3508.287798] BTRFS warning (device dm-10): csum failed root -9 ino 535 off 977797120 csum 0xc5aacf58 expected csum 0xa4442f95 mirror 1
[ 3508.287804] BTRFS error (device dm-10): bdev /dev/mapper/crypt_media1 errs: wr 0, rd 0, flush 0, corrupt 20482, gen 0
[ 3508.287807] BTRFS warning (device dm-10): csum failed root -9 ino 535 off 977604608 csum 0x56d29ae1 expected csum 0xa3019da3 mirror 1
[ 3508.287807] BTRFS error (device dm-10): bdev /dev/mapper/crypt_media1 errs: wr 0, rd 0, flush 0, corrupt 20483, gen 0
[ 3508.287806] BTRFS warning (device dm-10): csum failed root -9 ino 535 off 977731584 csum 0x887dc44a expected csum 0xd0266353 mirror 1
[ 3508.287807] BTRFS error (device dm-10): bdev /dev/mapper/crypt_media1 errs: wr 0, rd 0, flush 0, corrupt 20484, gen 0
[ 3508.287812] BTRFS error (device dm-10): bdev /dev/mapper/crypt_media1 errs: wr 0, rd 0, flush 0, corrupt 20485, gen 0
[ 3508.287814] BTRFS error (device dm-10): bdev /dev/mapper/crypt_media1 errs: wr 0, rd 0, flush 0, corrupt 20486, gen 0
[ 3508.287813] BTRFS warning (device dm-10): csum failed root -9 ino 535 off 977993728 csum 0xf027f21c expected csum 0xe489b804 mirror 1
[ 3508.287815] BTRFS warning (device dm-10): csum failed root -9 ino 535 off 977866752 csum 0x714b948c expected csum 0xb1614fca mirror 1
[ 3508.287816] BTRFS warning (device dm-10): csum failed root -9 ino 535 off 977670144 csum 0x4cd0cd73 expected csum 0x0557d928 mirror 1
[ 3508.287818] BTRFS error (device dm-10): bdev /dev/mapper/crypt_media1 errs: wr 0, rd 0, flush 0, corrupt 20487, gen 0
[ 3508.287818] BTRFS error (device dm-10): bdev /dev/mapper/crypt_media1 errs: wr 0, rd 0, flush 0, corrupt 20488, gen 0
[ 3508.287816] BTRFS warning (device dm-10): csum failed root -9 ino 535 off 978124800 csum 0xd4ac2027 expected csum 0xcddd1178 mirror 1
[ 3508.287819] BTRFS error (device dm-10): bdev /dev/mapper/crypt_media1 errs: wr 0, rd 0, flush 0, corrupt 20489, gen 0
[ 3508.287821] BTRFS error (device dm-10): bdev /dev/mapper/crypt_media1 errs: wr 0, rd 0, flush 0, corrupt 20490, gen 0
[ 3508.290052] AMD-Vi: IOMMU event log overflow
[ 3508.293639] AMD-Vi: IOMMU event log overflow
[ 3508.297161] AMD-Vi: IOMMU event log overflow
[ 3508.407219] AMD-Vi: IOMMU event log overflow
[ 3508.411053] AMD-Vi: IOMMU event log overflow
[ 3508.415015] AMD-Vi: IOMMU event log overflow
[ 3508.418585] AMD-Vi: IOMMU event log overflow
[ 3508.538137] AMD-Vi: IOMMU event log overflow
[ 3508.541939] AMD-Vi: IOMMU event log overflow
[ 3514.365523] BTRFS info (device dm-10): found 8 extents, stage: move data extents
[ 3514.648207] BTRFS info (device dm-10): found 8 extents, stage: update data pointers
[ 3514.821181] BTRFS info (device dm-10): relocating block group 1517227606016 flags data|raid0
[ 3515.710497] amd_iommu_report_page_fault: 12914 callbacks suppressed
[ 3515.710504] ahci 0000:48:00.0: AMD-Vi: Event logged [IO_PAGE_FAULT domain=0x0044 address=0x7ffffe00000 flags=0x0020]
[ 3515.710518] ahci 0000:48:00.0: AMD-Vi: Event logged [IO_PAGE_FAULT domain=0x0044 address=0x7ffffe00180 flags=0x0020]
[ 3515.710528] ahci 0000:48:00.0: AMD-Vi: Event logged [IO_PAGE_FAULT domain=0x0044 address=0x7ffffe00200 flags=0x0020]
[ 3515.710537] ahci 0000:48:00.0: AMD-Vi: Event logged [IO_PAGE_FAULT domain=0x0044 address=0x7ffffe00300 flags=0x0020]
[ 3515.710546] ahci 0000:48:00.0: AMD-Vi: Event logged [IO_PAGE_FAULT domain=0x0044 address=0x7ffffe00400 flags=0x0020]
[ 3515.710555] ahci 0000:48:00.0: AMD-Vi: Event logged [IO_PAGE_FAULT domain=0x0044 address=0x7ffffe00500 flags=0x0020]
[ 3515.710564] ahci 0000:48:00.0: AMD-Vi: Event logged [IO_PAGE_FAULT domain=0x0044 address=0x7ffffe00600 flags=0x0020]
[ 3515.710573] ahci 0000:48:00.0: AMD-Vi: Event logged [IO_PAGE_FAULT domain=0x0044 address=0x7ffffe00700 flags=0x0020]
[ 3515.710582] ahci 0000:48:00.0: AMD-Vi: Event logged [IO_PAGE_FAULT domain=0x0044 address=0x7ffffe00800 flags=0x0020]
[ 3515.710591] ahci 0000:48:00.0: AMD-Vi: Event logged [IO_PAGE_FAULT domain=0x0044 address=0x7ffffe00900 flags=0x0020]
[ 3515.714435] amd_iommu_int_thread: 13 callbacks suppressed
[ 3515.714437] AMD-Vi: IOMMU event log overflow
[ 3515.715668] btrfs_print_data_csum_error: 5428 callbacks suppressed
[ 3515.715674] BTRFS warning (device dm-10): csum failed root -9 ino 536 off 220725248 csum 0x9408324a expected csum 0xaf0a073e mirror 1
[ 3515.715682] btrfs_dev_stat_inc_and_print: 5430 callbacks suppressed
[ 3515.715683] BTRFS error (device dm-10): bdev /dev/mapper/crypt_media1 errs: wr 0, rd 0, flush 0, corrupt 25921, gen 0
[ 3515.715687] BTRFS warning (device dm-10): csum failed root -9 ino 536 off 220790784 csum 0xbfce5258 expected csum 0xbbf133cc mirror 1
[ 3515.715690] BTRFS warning (device dm-10): csum failed root -9 ino 536 off 220856320 csum 0xeaa1a306 expected csum 0x046ec329 mirror 1
[ 3515.715697] BTRFS error (device dm-10): bdev /dev/mapper/crypt_media1 errs: wr 0, rd 0, flush 0, corrupt 25922, gen 0
[ 3515.715699] BTRFS error (device dm-10): bdev /dev/mapper/crypt_media1 errs: wr 0, rd 0, flush 0, corrupt 25923, gen 0
[ 3515.715699] BTRFS warning (device dm-10): csum failed root -9 ino 536 off 220729344 csum 0x91baf3b7 expected csum 0x24f010cf mirror 1
[ 3515.715698] BTRFS warning (device dm-10): csum failed root -9 ino 536 off 220921856 csum 0xbf9bc082 expected csum 0xc7234c51 mirror 1
[ 3515.715700] BTRFS warning (device dm-10): csum failed root -9 ino 536 off 220987392 csum 0x51550e05 expected csum 0x58c5478b mirror 1
[ 3515.715701] BTRFS warning (device dm-10): csum failed root -9 ino 536 off 221052928 csum 0xf8f1e7a7 expected csum 0xc5c55fcf mirror 1
[ 3515.715707] BTRFS error (device dm-10): bdev /dev/mapper/crypt_media1 errs: wr 0, rd 0, flush 0, corrupt 25924, gen 0
[ 3515.715708] BTRFS error (device dm-10): bdev /dev/mapper/crypt_media1 errs: wr 0, rd 0, flush 0, corrupt 25925, gen 0
[ 3515.715711] BTRFS warning (device dm-10): csum failed root -9 ino 536 off 220860416 csum 0x601450c9 expected csum 0x8a8ffc19 mirror 1
[ 3515.715711] BTRFS error (device dm-10): bdev /dev/mapper/crypt_media1 errs: wr 0, rd 0, flush 0, corrupt 25926, gen 0
[ 3515.715713] BTRFS warning (device dm-10): csum failed root -9 ino 536 off 220794880 csum 0xff896d50 expected csum 0x2be7a306 mirror 1
[ 3515.715715] BTRFS error (device dm-10): bdev /dev/mapper/crypt_media1 errs: wr 0, rd 0, flush 0, corrupt 25927, gen 0
[ 3515.715715] BTRFS error (device dm-10): bdev /dev/mapper/crypt_media1 errs: wr 0, rd 0, flush 0, corrupt 25928, gen 0
[ 3515.715714] BTRFS warning (device dm-10): csum failed root -9 ino 536 off 221118464 csum 0xc32599d1 expected csum 0x85c411e2 mirror 1
[ 3515.715716] BTRFS error (device dm-10): bdev /dev/mapper/crypt_media1 errs: wr 0, rd 0, flush 0, corrupt 25929, gen 0
[ 3515.715718] BTRFS error (device dm-10): bdev /dev/mapper/crypt_media1 errs: wr 0, rd 0, flush 0, corrupt 25931, gen 0
[ 3515.718028] AMD-Vi: IOMMU event log overflow
[ 3515.729840] AMD-Vi: IOMMU event log overflow
[ 3515.733400] AMD-Vi: IOMMU event log overflow
[ 3516.937855] AMD-Vi: IOMMU event log overflow
[ 3516.941671] AMD-Vi: IOMMU event log overflow
[ 3516.945675] AMD-Vi: IOMMU event log overflow
[ 3516.949469] AMD-Vi: IOMMU event log overflow
[ 3517.069693] AMD-Vi: IOMMU event log overflow
[ 3517.073659] AMD-Vi: IOMMU event log overflow
[ 3520.801101] amd_iommu_report_page_fault: 85590 callbacks suppressed
[ 3520.801106] ahci 0000:48:00.0: AMD-Vi: Event logged [IO_PAGE_FAULT domain=0x0044 address=0x7ffffe00000 flags=0x0020]
[ 3520.801121] ahci 0000:48:00.0: AMD-Vi: Event logged [IO_PAGE_FAULT domain=0x0044 address=0x7ffffe00180 flags=0x0020]
[ 3520.801130] ahci 0000:48:00.0: AMD-Vi: Event logged [IO_PAGE_FAULT domain=0x0044 address=0x7ffffe00200 flags=0x0020]
[ 3520.801139] ahci 0000:48:00.0: AMD-Vi: Event logged [IO_PAGE_FAULT domain=0x0044 address=0x7ffffe00300 flags=0x0020]
[ 3520.801148] ahci 0000:48:00.0: AMD-Vi: Event logged [IO_PAGE_FAULT domain=0x0044 address=0x7ffffe00380 flags=0x0020]
[ 3520.801157] ahci 0000:48:00.0: AMD-Vi: Event logged [IO_PAGE_FAULT domain=0x0044 address=0x7ffffe00480 flags=0x0020]
[ 3520.801166] ahci 0000:48:00.0: AMD-Vi: Event logged [IO_PAGE_FAULT domain=0x0044 address=0x7ffffe00580 flags=0x0020]
[ 3520.801175] ahci 0000:48:00.0: AMD-Vi: Event logged [IO_PAGE_FAULT domain=0x0044 address=0x7ffffe00600 flags=0x0020]
[ 3520.801184] ahci 0000:48:00.0: AMD-Vi: Event logged [IO_PAGE_FAULT domain=0x0044 address=0x7ffffe00700 flags=0x0020]
[ 3520.801193] ahci 0000:48:00.0: AMD-Vi: Event logged [IO_PAGE_FAULT domain=0x0044 address=0x7ffffe00780 flags=0x0020]
[ 3520.804406] btrfs_print_data_csum_error: 32292 callbacks suppressed
[ 3520.804408] btrfs_dev_stat_inc_and_print: 32294 callbacks suppressed
[ 3520.804410] BTRFS warning (device dm-10): csum failed root -9 ino 536 off 788529152 csum 0xb00b207c expected csum 0xa717e125 mirror 1
[ 3520.804412] BTRFS error (device dm-10): bdev /dev/mapper/crypt_media1 errs: wr 0, rd 0, flush 0, corrupt 58241, gen 0
[ 3520.804411] BTRFS warning (device dm-10): csum failed root -9 ino 536 off 788660224 csum 0xcf55fba7 expected csum 0x809448cc mirror 1
[ 3520.804418] BTRFS error (device dm-10): bdev /dev/mapper/crypt_media1 errs: wr 0, rd 0, flush 0, corrupt 58242, gen 0
[ 3520.804418] BTRFS warning (device dm-10): csum failed root -9 ino 536 off 788725760 csum 0x24ac14c7 expected csum 0xccef8dfa mirror 1
[ 3520.804421] BTRFS error (device dm-10): bdev /dev/mapper/crypt_media1 errs: wr 0, rd 0, flush 0, corrupt 58243, gen 0
[ 3520.804422] BTRFS error (device dm-10): bdev /dev/mapper/crypt_media1 errs: wr 0, rd 0, flush 0, corrupt 58244, gen 0
[ 3520.804420] BTRFS warning (device dm-10): csum failed root -9 ino 536 off 788922368 csum 0x637d036f expected csum 0x6a975163 mirror 1
[ 3520.804421] BTRFS warning (device dm-10): csum failed root -9 ino 536 off 789053440 csum 0x8273d9ea expected csum 0xa9b92a42 mirror 1
[ 3520.804423] BTRFS warning (device dm-10): csum failed root -9 ino 536 off 789118976 csum 0x8626b041 expected csum 0x735a53cc mirror 1
[ 3520.804423] BTRFS warning (device dm-10): csum failed root -9 ino 536 off 788598784 csum 0x0d2303c1 expected csum 0x9e0b7e81 mirror 1
[ 3520.804428] BTRFS error (device dm-10): bdev /dev/mapper/crypt_media1 errs: wr 0, rd 0, flush 0, corrupt 58245, gen 0
[ 3520.804430] BTRFS error (device dm-10): bdev /dev/mapper/crypt_media1 errs: wr 0, rd 0, flush 0, corrupt 58247, gen 0
[ 3520.804430] BTRFS error (device dm-10): bdev /dev/mapper/crypt_media1 errs: wr 0, rd 0, flush 0, corrupt 58246, gen 0
[ 3520.804430] BTRFS error (device dm-10): bdev /dev/mapper/crypt_media1 errs: wr 0, rd 0, flush 0, corrupt 58248, gen 0
[ 3520.804431] BTRFS error (device dm-10): bdev /dev/mapper/crypt_media1 errs: wr 0, rd 0, flush 0, corrupt 58249, gen 0
[ 3520.804431] BTRFS warning (device dm-10): csum failed root -9 ino 536 off 788664320 csum 0x0e80019c expected csum 0x1cf8bab9 mirror 1
[ 3520.804431] BTRFS warning (device dm-10): csum failed root -9 ino 536 off 788860928 csum 0x211c3c0f expected csum 0x4d8e8f43 mirror 1
[ 3520.804436] BTRFS error (device dm-10): bdev /dev/mapper/crypt_media1 errs: wr 0, rd 0, flush 0, corrupt 58250, gen 0
[ 3520.804444] BTRFS warning (device dm-10): csum failed root -9 ino 536 off 788729856 csum 0x0a0a7c10 expected csum 0xab0aca5f mirror 1
[ 3520.804669] amd_iommu_int_thread: 149 callbacks suppressed
[ 3520.804671] AMD-Vi: IOMMU event log overflow
[ 3520.808155] AMD-Vi: IOMMU event log overflow
[ 3520.812017] AMD-Vi: IOMMU event log overflow
[ 3520.815534] AMD-Vi: IOMMU event log overflow
[ 3520.874004] AMD-Vi: IOMMU event log overflow
[ 3520.883072] AMD-Vi: IOMMU event log overflow
[ 3520.886846] AMD-Vi: IOMMU event log overflow
[ 3520.890488] AMD-Vi: IOMMU event log overflow
[ 3520.965877] AMD-Vi: IOMMU event log overflow
[ 3520.973896] AMD-Vi: IOMMU event log overflow
[ 3525.842381] amd_iommu_report_page_fault: 110728 callbacks suppressed
[ 3525.842387] ahci 0000:48:00.0: AMD-Vi: Event logged [IO_PAGE_FAULT domain=0x0044 address=0x7ffffe00000 flags=0x0020]
[ 3525.842401] ahci 0000:48:00.0: AMD-Vi: Event logged [IO_PAGE_FAULT domain=0x0044 address=0x7ffffe00180 flags=0x0020]
[ 3525.842411] ahci 0000:48:00.0: AMD-Vi: Event logged [IO_PAGE_FAULT domain=0x0044 address=0x7ffffe00200 flags=0x0020]
[ 3525.842420] ahci 0000:48:00.0: AMD-Vi: Event logged [IO_PAGE_FAULT domain=0x0044 address=0x7ffffe00300 flags=0x0020]
[ 3525.842429] ahci 0000:48:00.0: AMD-Vi: Event logged [IO_PAGE_FAULT domain=0x0044 address=0x7ffffe00400 flags=0x0020]
[ 3525.842437] ahci 0000:48:00.0: AMD-Vi: Event logged [IO_PAGE_FAULT domain=0x0044 address=0x7ffffe00580 flags=0x0020]
[ 3525.842446] ahci 0000:48:00.0: AMD-Vi: Event logged [IO_PAGE_FAULT domain=0x0044 address=0x7ffffe00600 flags=0x0020]
[ 3525.842455] ahci 0000:48:00.0: AMD-Vi: Event logged [IO_PAGE_FAULT domain=0x0044 address=0x7ffffe00700 flags=0x0020]
[ 3525.842464] ahci 0000:48:00.0: AMD-Vi: Event logged [IO_PAGE_FAULT domain=0x0044 address=0x7ffffe00800 flags=0x0020]
[ 3525.842473] ahci 0000:48:00.0: AMD-Vi: Event logged [IO_PAGE_FAULT domain=0x0044 address=0x7ffffe00900 flags=0x0020]
[ 3525.845653] btrfs_print_data_csum_error: 48901 callbacks suppressed
[ 3525.845657] BTRFS warning (device dm-10): csum failed root -9 ino 536 off 1006632960 csum 0x51cdf7a4 expected csum 0x150a17ea mirror 1
[ 3525.845658] BTRFS warning (device dm-10): csum failed root -9 ino 536 off 1006698496 csum 0x7863d4ad expected csum 0xfdc6eb35 mirror 1
[ 3525.845659] BTRFS warning (device dm-10): csum failed root -9 ino 536 off 1006764032 csum 0xd51e70e1 expected csum 0x1a40906e mirror 1
[ 3525.845665] btrfs_dev_stat_inc_and_print: 48928 callbacks suppressed
[ 3525.845668] BTRFS error (device dm-10): bdev /dev/mapper/crypt_media1 errs: wr 0, rd 0, flush 0, corrupt 107201, gen 0
[ 3525.845668] BTRFS error (device dm-10): bdev /dev/mapper/crypt_media1 errs: wr 0, rd 0, flush 0, corrupt 107202, gen 0
[ 3525.845670] BTRFS error (device dm-10): bdev /dev/mapper/crypt_media1 errs: wr 0, rd 0, flush 0, corrupt 107203, gen 0
[ 3525.845671] BTRFS warning (device dm-10): csum failed root -9 ino 536 off 1006829568 csum 0x00c91843 expected csum 0xba914b02 mirror 1
[ 3525.845673] BTRFS warning (device dm-10): csum failed root -9 ino 536 off 1006895104 csum 0x3c9460f1 expected csum 0x1bf72459 mirror 1
[ 3525.845674] BTRFS warning (device dm-10): csum failed root -9 ino 536 off 1006960640 csum 0xbc575c3f expected csum 0xe210bd69 mirror 1
[ 3525.845676] BTRFS warning (device dm-10): csum failed root -9 ino 536 off 1006702592 csum 0xaa563dea expected csum 0xea6d7e0e mirror 1
[ 3525.845679] BTRFS warning (device dm-10): csum failed root -9 ino 536 off 1006637056 csum 0xb7ee8b8c expected csum 0x0d7d01e1 mirror 1
[ 3525.845681] BTRFS error (device dm-10): bdev /dev/mapper/crypt_media1 errs: wr 0, rd 0, flush 0, corrupt 107205, gen 0
[ 3525.845682] BTRFS error (device dm-10): bdev /dev/mapper/crypt_media1 errs: wr 0, rd 0, flush 0, corrupt 107206, gen 0
[ 3525.845682] BTRFS warning (device dm-10): csum failed root -9 ino 536 off 1007026176 csum 0xa594a743 expected csum 0x2bd4eda1 mirror 1
[ 3525.845684] BTRFS error (device dm-10): bdev /dev/mapper/crypt_media1 errs: wr 0, rd 0, flush 0, corrupt 107209, gen 0
[ 3525.845682] BTRFS warning (device dm-10): csum failed root -9 ino 536 off 1007091712 csum 0x2c92ba5a expected csum 0xcde3323c mirror 1
[ 3525.845684] BTRFS error (device dm-10): bdev /dev/mapper/crypt_media1 errs: wr 0, rd 0, flush 0, corrupt 107209, gen 0
[ 3525.845686] BTRFS error (device dm-10): bdev /dev/mapper/crypt_media1 errs: wr 0, rd 0, flush 0, corrupt 107211, gen 0
[ 3525.845686] BTRFS error (device dm-10): bdev /dev/mapper/crypt_media1 errs: wr 0, rd 0, flush 0, corrupt 107210, gen 0
[ 3525.845924] amd_iommu_int_thread: 194 callbacks suppressed
[ 3525.845925] AMD-Vi: IOMMU event log overflow
[ 3525.851977] AMD-Vi: IOMMU event log overflow
[ 3525.855694] AMD-Vi: IOMMU event log overflow
[ 3525.859175] AMD-Vi: IOMMU event log overflow
[ 3525.942614] AMD-Vi: IOMMU event log overflow
[ 3525.951176] AMD-Vi: IOMMU event log overflow
[ 3525.954923] AMD-Vi: IOMMU event log overflow
[ 3525.958408] AMD-Vi: IOMMU event log overflow
[ 3526.076250] AMD-Vi: IOMMU event log overflow
[ 3526.084009] AMD-Vi: IOMMU event log overflow
[ 3531.090971] amd_iommu_report_page_fault: 33561 callbacks suppressed
[ 3531.090976] ahci 0000:48:00.0: AMD-Vi: Event logged [IO_PAGE_FAULT domain=0x0044 address=0x7ffffe00000 flags=0x0000]
[ 3531.090990] ahci 0000:48:00.0: AMD-Vi: Event logged [IO_PAGE_FAULT domain=0x0044 address=0x7ffffe00380 flags=0x0000]
[ 3531.090999] ahci 0000:48:00.0: AMD-Vi: Event logged [IO_PAGE_FAULT domain=0x0044 address=0x7ffffe00480 flags=0x0000]
[ 3531.091008] ahci 0000:48:00.0: AMD-Vi: Event logged [IO_PAGE_FAULT domain=0x0044 address=0x7ffffe00780 flags=0x0000]
[ 3561.245510] ata4.00: exception Emask 0x0 SAct 0x7ff00 SErr 0x0 action 0x6 frozen
[ 3561.245521] ata4.00: failed command: WRITE FPDMA QUEUED
[ 3561.245524] ata4.00: cmd 61/05:40:cb:d3:62/01:00:04:00:00/40 tag 8 ncq dma 1069056 ou
                        res 40/00:01:00:00:00/00:00:00:00:00/00 Emask 0x4 (timeout)
[ 3561.245532] ata4.00: status: { DRDY }
[ 3561.245534] ata4.00: failed command: WRITE FPDMA QUEUED
[ 3561.245536] ata4.00: cmd 61/00:48:d0:d4:62/01:00:04:00:00/40 tag 9 ncq dma 1048576 ou
                        res 40/00:81:82:00:00/00:00:00:00:00/40 Emask 0x4 (timeout)
[ 3561.245542] ata4.00: status: { DRDY }
[ 3561.245544] ata4.00: failed command: WRITE FPDMA QUEUED
[ 3561.245545] ata4.00: cmd 61/00:50:d0:d5:62/01:00:04:00:00/40 tag 10 ncq dma 1048576 ou
                        res 40/00:01:00:4f:c2/00:00:00:00:00/00 Emask 0x4 (timeout)
[ 3561.245551] ata4.00: status: { DRDY }
[ 3561.245553] ata4.00: failed command: WRITE FPDMA QUEUED
[ 3561.245555] ata4.00: cmd 61/00:58:d0:d6:62/01:00:04:00:00/40 tag 11 ncq dma 1048576 ou
                        res 40/00:01:00:4f:c2/00:00:00:00:00/00 Emask 0x4 (timeout)
[ 3561.245560] ata4.00: status: { DRDY }
[ 3561.245562] ata4.00: failed command: WRITE FPDMA QUEUED
[ 3561.245564] ata4.00: cmd 61/00:60:d0:d7:62/01:00:04:00:00/40 tag 12 ncq dma 1048576 ou
                        res 40/00:00:00:00:00/00:00:00:00:00/00 Emask 0x4 (timeout)
[ 3561.245569] ata4.00: status: { DRDY }
[ 3561.245571] ata4.00: failed command: WRITE FPDMA QUEUED
[ 3561.245572] ata4.00: cmd 61/00:68:d0:d8:62/01:00:04:00:00/40 tag 13 ncq dma 1048576 ou
                        res 40/00:00:00:00:00/00:00:00:00:00/00 Emask 0x4 (timeout)
[ 3561.245578] ata4.00: status: { DRDY }
[ 3561.245580] ata4.00: failed command: WRITE FPDMA QUEUED
[ 3561.245581] ata4.00: cmd 61/00:70:d0:d9:62/01:00:04:00:00/40 tag 14 ncq dma 1048576 ou
                        res 40/00:00:00:00:00/00:00:00:00:00/00 Emask 0x4 (timeout)
[ 3561.245587] ata4.00: status: { DRDY }
[ 3561.245588] ata4.00: failed command: WRITE FPDMA QUEUED
[ 3561.245590] ata4.00: cmd 61/00:78:d0:da:62/01:00:04:00:00/40 tag 15 ncq dma 1048576 ou
                        res 40/00:01:00:00:00/00:00:00:00:00/00 Emask 0x4 (timeout)
[ 3561.245595] ata4.00: status: { DRDY }
[ 3561.245597] ata4.00: failed command: WRITE FPDMA QUEUED
[ 3561.245599] ata4.00: cmd 61/00:80:d0:db:62/01:00:04:00:00/40 tag 16 ncq dma 1048576 ou
                        res 40/00:00:00:00:00/00:00:00:00:00/00 Emask 0x4 (timeout)
[ 3561.245604] ata4.00: status: { DRDY }
[ 3561.245606] ata4.00: failed command: WRITE FPDMA QUEUED
[ 3561.245607] ata4.00: cmd 61/00:88:d0:dc:62/01:00:04:00:00/40 tag 17 ncq dma 1048576 ou
                        res 40/00:00:00:00:00/00:00:00:00:00/00 Emask 0x4 (timeout)
[ 3561.245613] ata4.00: status: { DRDY }
[ 3561.245614] ata4.00: failed command: WRITE FPDMA QUEUED
[ 3561.245616] ata4.00: cmd 61/00:90:d0:dd:62/01:00:04:00:00/40 tag 18 ncq dma 1048576 ou
                        res 40/00:00:00:00:00/00:00:00:00:00/00 Emask 0x4 (timeout)
[ 3561.245621] ata4.00: status: { DRDY }
[ 3561.245625] ata4: hard resetting link
[ 3561.245643] ata3.00: exception Emask 0x0 SAct 0x80fe1fff SErr 0x0 action 0x6 frozen
[ 3561.245654] ata3.00: failed command: WRITE FPDMA QUEUED
[ 3561.245656] ata3.00: cmd 61/00:00:d0:1b:6b/01:00:1e:00:00/40 tag 0 ncq dma 1048576 ou
                        res 40/00:01:00:00:00/00:00:00:00:00/a0 Emask 0x4 (timeout)
[ 3561.245665] ata3.00: status: { DRDY }
[ 3561.245668] ata3.00: failed command: WRITE FPDMA QUEUED
[ 3561.245670] ata3.00: cmd 61/00:08:d0:1c:6b/01:00:1e:00:00/40 tag 1 ncq dma 1048576 ou
                        res 40/00:01:00:4f:c2/00:00:00:00:00/00 Emask 0x4 (timeout)
[ 3561.245676] ata3.00: status: { DRDY }
[ 3561.245678] ata3.00: failed command: WRITE FPDMA QUEUED
[ 3561.245680] ata3.00: cmd 61/00:10:d0:1d:6b/01:00:1e:00:00/40 tag 2 ncq dma 1048576 ou
                        res 40/00:01:00:4f:c2/00:00:00:00:00/00 Emask 0x4 (timeout)
[ 3561.245686] ata3.00: status: { DRDY }
[ 3561.245688] ata3.00: failed command: WRITE FPDMA QUEUED
[ 3561.245690] ata3.00: cmd 61/00:18:d0:1e:6b/01:00:1e:00:00/40 tag 3 ncq dma 1048576 ou
                        res 40/00:01:00:00:00/00:00:00:00:00/00 Emask 0x4 (timeout)
[ 3561.245696] ata3.00: status: { DRDY }
[ 3561.245697] ata3.00: failed command: WRITE FPDMA QUEUED
[ 3561.245699] ata3.00: cmd 61/00:20:d0:1f:6b/01:00:1e:00:00/40 tag 4 ncq dma 1048576 ou
                        res 40/00:ff:ff:00:00/00:00:00:00:00/40 Emask 0x4 (timeout)
[ 3561.245705] ata3.00: status: { DRDY }
[ 3561.245707] ata3.00: failed command: WRITE FPDMA QUEUED
[ 3561.245708] ata3.00: cmd 61/d4:28:d0:20:6b/00:00:1e:00:00/40 tag 5 ncq dma 868352 out
                        res 40/00:00:00:00:00/00:00:00:00:00/00 Emask 0x4 (timeout)
[ 3561.245714] ata3.00: status: { DRDY }
[ 3561.245716] ata3.00: failed command: WRITE FPDMA QUEUED
[ 3561.245718] ata3.00: cmd 61/00:30:a4:21:6b/01:00:1e:00:00/40 tag 6 ncq dma 1048576 ou
                        res 40/00:00:00:4f:c2/00:00:00:00:00/40 Emask 0x4 (timeout)
[ 3561.245724] ata3.00: status: { DRDY }
[ 3561.245726] ata3.00: failed command: WRITE FPDMA QUEUED
[ 3561.245727] ata3.00: cmd 61/00:38:a4:22:6b/01:00:1e:00:00/40 tag 7 ncq dma 1048576 ou
                        res 40/00:00:00:4f:c2/00:00:00:00:00/40 Emask 0x4 (timeout)
[ 3561.245733] ata3.00: status: { DRDY }
[ 3561.245735] ata3.00: failed command: WRITE FPDMA QUEUED
[ 3561.245736] ata3.00: cmd 61/00:40:a4:23:6b/01:00:1e:00:00/40 tag 8 ncq dma 1048576 ou
                        res 40/00:01:00:4f:c2/00:00:00:00:00/00 Emask 0x4 (timeout)
[ 3561.245742] ata3.00: status: { DRDY }
[ 3561.245744] ata3.00: failed command: WRITE FPDMA QUEUED
[ 3561.245745] ata3.00: cmd 61/d6:48:a4:24:6b/00:00:1e:00:00/40 tag 9 ncq dma 876544 out
                        res 40/00:00:00:00:00/00:00:00:00:00/00 Emask 0x4 (timeout)
[ 3561.245751] ata3.00: status: { DRDY }
[ 3561.245753] ata3.00: failed command: WRITE FPDMA QUEUED
[ 3561.245754] ata3.00: cmd 61/27:50:7a:25:6b/01:00:1e:00:00/40 tag 10 ncq dma 1208320 ou
                        res 40/00:00:00:00:00/00:00:00:00:00/00 Emask 0x4 (timeout)
[ 3561.245760] ata3.00: status: { DRDY }
[ 3561.245762] ata3.00: failed command: WRITE FPDMA QUEUED
[ 3561.245764] ata3.00: cmd 61/00:58:a1:26:6b/01:00:1e:00:00/40 tag 11 ncq dma 1048576 ou
                        res 40/00:01:00:4f:c2/00:00:00:00:00/00 Emask 0x4 (timeout)
[ 3561.245770] ata3.00: status: { DRDY }
[ 3561.245771] ata3.00: failed command: WRITE FPDMA QUEUED
[ 3561.245773] ata3.00: cmd 61/00:60:a1:27:6b/01:00:1e:00:00/40 tag 12 ncq dma 1048576 ou
                        res 40/00:00:00:00:00/00:00:00:00:00/00 Emask 0x4 (timeout)
[ 3561.245779] ata3.00: status: { DRDY }
[ 3561.245781] ata3.00: failed command: WRITE FPDMA QUEUED
[ 3561.245782] ata3.00: cmd 61/3f:88:91:14:6b/01:00:1e:00:00/40 tag 17 ncq dma 1306624 ou
                        res 40/00:ff:ff:00:00/00:00:00:00:00/40 Emask 0x4 (timeout)
[ 3561.245788] ata3.00: status: { DRDY }
[ 3561.245790] ata3.00: failed command: WRITE FPDMA QUEUED
[ 3561.245791] ata3.00: cmd 61/00:90:d0:15:6b/01:00:1e:00:00/40 tag 18 ncq dma 1048576 ou
                        res 40/00:00:00:00:00/00:00:00:00:00/00 Emask 0x4 (timeout)
[ 3561.245797] ata3.00: status: { DRDY }
[ 3561.245799] ata3.00: failed command: WRITE FPDMA QUEUED
[ 3561.245800] ata3.00: cmd 61/00:98:d0:16:6b/01:00:1e:00:00/40 tag 19 ncq dma 1048576 ou
                        res 40/00:00:00:00:00/00:00:00:00:00/00 Emask 0x4 (timeout)
[ 3561.245806] ata3.00: status: { DRDY }
[ 3561.245808] ata3.00: failed command: WRITE FPDMA QUEUED
[ 3561.245809] ata3.00: cmd 61/b1:a0:d0:17:6b/00:00:1e:00:00/40 tag 20 ncq dma 724992 out
                        res 40/00:ff:ff:00:00/00:00:00:00:00/40 Emask 0x4 (timeout)
[ 3561.245815] ata3.00: status: { DRDY }
[ 3561.245817] ata3.00: failed command: WRITE FPDMA QUEUED
[ 3561.245818] ata3.00: cmd 61/4f:a8:81:18:6b/00:00:1e:00:00/40 tag 21 ncq dma 323584 out
                        res 40/00:01:00:4f:c2/00:00:00:00:00/00 Emask 0x4 (timeout)
[ 3561.245824] ata3.00: status: { DRDY }
[ 3561.245826] ata3.00: failed command: WRITE FPDMA QUEUED
[ 3561.245827] ata3.00: cmd 61/00:b0:d0:18:6b/01:00:1e:00:00/40 tag 22 ncq dma 1048576 ou
                        res 40/00:00:00:00:00/00:00:00:00:00/00 Emask 0x4 (timeout)
[ 3561.245833] ata3.00: status: { DRDY }
[ 3561.245835] ata3.00: failed command: WRITE FPDMA QUEUED
[ 3561.245836] ata3.00: cmd 61/00:b8:d0:19:6b/01:00:1e:00:00/40 tag 23 ncq dma 1048576 ou
                        res 40/00:01:00:00:00/00:00:00:00:00/00 Emask 0x4 (timeout)
[ 3561.245842] ata3.00: status: { DRDY }
[ 3561.245844] ata3.00: failed command: WRITE FPDMA QUEUED
[ 3561.245845] ata3.00: cmd 61/00:f8:d0:1a:6b/01:00:1e:00:00/40 tag 31 ncq dma 1048576 ou
                        res 40/00:00:00:00:00/00:00:00:00:00/00 Emask 0x4 (timeout)
[ 3561.245851] ata3.00: status: { DRDY }
[ 3561.245854] ata3: hard resetting link
[ 3571.248763] ata3: softreset failed (1st FIS failed)
[ 3571.248771] ata3: hard resetting link
[ 3571.248933] ata4: softreset failed (1st FIS failed)
[ 3571.248941] ata4: hard resetting link
[ 3581.252030] ata4: softreset failed (1st FIS failed)
[ 3581.252037] ata4: hard resetting link
[ 3581.252365] ata3: softreset failed (1st FIS failed)
[ 3581.252372] ata3: hard resetting link
[ 3616.254364] ata4: softreset failed (1st FIS failed)
[ 3616.254374] ata4: limiting SATA link speed to 3.0 Gbps
[ 3616.254376] ata4: hard resetting link
[ 3616.254872] ata3: softreset failed (1st FIS failed)
[ 3616.254882] ata3: limiting SATA link speed to 3.0 Gbps
[ 3616.254884] ata3: hard resetting link
[ 3621.257746] ata3: softreset failed (1st FIS failed)
[ 3621.257753] ata3: reset failed, giving up
[ 3621.257756] ata3.00: disable device
[ 3621.258058] ata3: EH complete
[ 3621.258085] ata4: softreset failed (1st FIS failed)
[ 3621.258093] ata4: reset failed, giving up
[ 3621.258096] ata4.00: disable device
[ 3621.258096] sd 2:0:0:0: [sdc] tag#29 FAILED Result: hostbyte=DID_BAD_TARGET driverbyte=DRIVER_OK cmd_age=90s
[ 3621.258109] sd 2:0:0:0: [sdc] tag#29 CDB: Write(10) 2a 00 1e 6b 1b d0 00 01 00 00
[ 3621.258112] I/O error, dev sdc, sector 4082687616 op 0x1:(WRITE) flags 0x4800 phys_seg 97 prio class 2
[ 3621.258159] sd 2:0:0:0: [sdc] tag#30 FAILED Result: hostbyte=DID_BAD_TARGET driverbyte=DRIVER_OK cmd_age=90s
[ 3621.258163] sd 2:0:0:0: [sdc] tag#30 CDB: Write(10) 2a 00 1e 6b 1c d0 00 01 00 00
[ 3621.258165] I/O error, dev sdc, sector 4082689664 op 0x1:(WRITE) flags 0x4800 phys_seg 65 prio class 2
[ 3621.258264] ata4: EH complete
[ 3621.258283] sd 3:0:0:0: [sdd] tag#1 FAILED Result: hostbyte=DID_BAD_TARGET driverbyte=DRIVER_OK cmd_age=90s
[ 3621.258289] sd 3:0:0:0: [sdd] tag#1 CDB: Write(10) 2a 00 04 62 d3 cb 00 01 05 00
[ 3621.258290] I/O error, dev sdd, sector 588684888 op 0x1:(WRITE) flags 0x4800 phys_seg 140 prio class 2
[ 3621.258306] sd 2:0:0:0: [sdc] tag#0 FAILED Result: hostbyte=DID_BAD_TARGET driverbyte=DRIVER_OK cmd_age=90s
[ 3621.258310] sd 2:0:0:0: [sdc] tag#0 CDB: Write(10) 2a 00 1e 6b 1d d0 00 01 00 00
[ 3621.258313] I/O error, dev sdc, sector 4082691712 op 0x1:(WRITE) flags 0x4800 phys_seg 87 prio class 2
[ 3621.258347] sd 2:0:0:0: [sdc] tag#1 FAILED Result: hostbyte=DID_BAD_TARGET driverbyte=DRIVER_OK cmd_age=90s
[ 3621.258350] sd 2:0:0:0: [sdc] tag#1 CDB: Write(10) 2a 00 1e 6b 1e d0 00 01 00 00
[ 3621.258352] I/O error, dev sdc, sector 4082693760 op 0x1:(WRITE) flags 0x4800 phys_seg 81 prio class 2
[ 3621.258383] sd 2:0:0:0: [sdc] tag#2 FAILED Result: hostbyte=DID_BAD_TARGET driverbyte=DRIVER_OK cmd_age=90s
[ 3621.258386] sd 2:0:0:0: [sdc] tag#2 CDB: Write(10) 2a 00 1e 6b 1f d0 00 01 00 00
[ 3621.258388] I/O error, dev sdc, sector 4082695808 op 0x1:(WRITE) flags 0x4800 phys_seg 56 prio class 2
[ 3621.258394] sd 3:0:0:0: [sdd] tag#2 FAILED Result: hostbyte=DID_BAD_TARGET driverbyte=DRIVER_OK cmd_age=90s
[ 3621.258397] sd 3:0:0:0: [sdd] tag#2 CDB: Write(10) 2a 00 04 62 d4 d0 00 01 00 00
[ 3621.258398] I/O error, dev sdd, sector 588686976 op 0x1:(WRITE) flags 0x4800 phys_seg 110 prio class 2
[ 3621.258432] btrfs_dev_stat_inc_and_print: 15012 callbacks suppressed
[ 3621.258435] BTRFS error (device dm-10): bdev /dev/mapper/crypt_media2 errs: wr 1, rd 0, flush 0, corrupt 0, gen 0
[ 3621.258442] sd 2:0:0:0: [sdc] tag#3 FAILED Result: hostbyte=DID_BAD_TARGET driverbyte=DRIVER_OK cmd_age=90s
[ 3621.258445] sd 2:0:0:0: [sdc] tag#3 CDB: Write(10) 2a 00 1e 6b 20 d0 00 00 d4 00
[ 3621.258447] I/O error, dev sdc, sector 4082697856 op 0x1:(WRITE) flags 0x4800 phys_seg 130 prio class 2
[ 3621.258492] BTRFS error (device dm-10): bdev /dev/mapper/crypt_media1 errs: wr 1, rd 0, flush 0, corrupt 122240, gen 0
[ 3621.258501] sd 2:0:0:0: [sdc] tag#4 FAILED Result: hostbyte=DID_BAD_TARGET driverbyte=DRIVER_OK cmd_age=90s
[ 3621.258504] sd 2:0:0:0: [sdc] tag#4 CDB: Write(10) 2a 00 1e 6b 21 a4 00 01 00 00
[ 3621.258507] I/O error, dev sdc, sector 4082699552 op 0x1:(WRITE) flags 0x4800 phys_seg 39 prio class 2
[ 3621.258568] sd 2:0:0:0: [sdc] tag#5 FAILED Result: hostbyte=DID_BAD_TARGET driverbyte=DRIVER_OK cmd_age=90s
[ 3621.258571] sd 2:0:0:0: [sdc] tag#5 CDB: Write(10) 2a 00 1e 6b 22 a4 00 01 00 00
[ 3621.258573] I/O error, dev sdc, sector 4082701600 op 0x1:(WRITE) flags 0x4800 phys_seg 38 prio class 2
[ 3621.258699] BTRFS error (device dm-10): bdev /dev/mapper/crypt_media1 errs: wr 2, rd 0, flush 0, corrupt 122240, gen 0
[ 3621.258928] BTRFS error (device dm-10): bdev /dev/mapper/crypt_media1 errs: wr 3, rd 0, flush 0, corrupt 122240, gen 0
[ 3621.259101] BTRFS error (device dm-10): bdev /dev/mapper/crypt_media1 errs: wr 4, rd 0, flush 0, corrupt 122240, gen 0
[ 3621.259123] BTRFS error (device dm-10): bdev /dev/mapper/crypt_media2 errs: wr 2, rd 0, flush 0, corrupt 0, gen 0
[ 3621.259214] BTRFS error (device dm-10): bdev /dev/mapper/crypt_media1 errs: wr 5, rd 0, flush 0, corrupt 122240, gen 0
[ 3621.259494] BTRFS error (device dm-10): bdev /dev/mapper/crypt_media1 errs: wr 6, rd 0, flush 0, corrupt 122240, gen 0
[ 3621.259701] BTRFS error (device dm-10): bdev /dev/mapper/crypt_media1 errs: wr 7, rd 0, flush 0, corrupt 122240, gen 0
[ 3621.259792] BTRFS error (device dm-10): bdev /dev/mapper/crypt_media2 errs: wr 3, rd 0, flush 0, corrupt 0, gen 0
[ 3621.300069] BTRFS: error (device dm-10) in btrfs_commit_transaction:2436: errno=-5 IO failure (Error while writing out transaction)
[ 3621.300080] BTRFS info (device dm-10: state E): forced readonly
[ 3621.300083] BTRFS warning (device dm-10: state E): Skipping commit of aborted transaction.
[ 3621.300085] BTRFS: error (device dm-10: state EA) in cleanup_transaction:1983: errno=-5 IO failure
[ 3621.493715] BTRFS info (device dm-10: state EA): balance: ended with status: -5
[ 3621.494388] audit: type=1106 audit(1672401228.685:478): pid=11399 uid=1000 auid=1000 ses=4 msg='op=PAM:session_close grantors=pam_systemd_home,pam_limits,pam_unix,pam_permit acct="root" exe="/usr/bin/sudo" hostname=? addr=? terminal=/dev/pts/0 res=success'
[ 3621.494435] audit: type=1104 audit(1672401228.685:479): pid=11399 uid=1000 auid=1000 ses=4 msg='op=PAM:setcred grantors=pam_faillock,pam_permit,pam_env,pam_faillock acct="root" exe="/usr/bin/sudo" hostname=? addr=? terminal=/dev/pts/0 res=success'
[ 3623.637713] audit: type=1130 audit(1672401230.828:480): pid=1 uid=0 auid=4294967295 ses=4294967295 msg='unit=mysql_dump comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=failed'
[ 3768.857896] scsi_io_completion_action: 845 callbacks suppressed
[ 3768.857905] sd 3:0:0:0: [sdd] tag#29 FAILED Result: hostbyte=DID_BAD_TARGET driverbyte=DRIVER_OK cmd_age=0s
[ 3768.857910] sd 3:0:0:0: [sdd] tag#29 CDB: ATA command pass through(16) 85 06 20 00 00 00 00 00 00 00 00 00 00 00 e5 00
[ 3768.907562] sd 2:0:0:0: [sdc] tag#24 FAILED Result: hostbyte=DID_BAD_TARGET driverbyte=DRIVER_OK cmd_age=0s
[ 3768.907566] sd 2:0:0:0: [sdc] tag#24 CDB: ATA command pass through(16) 85 06 20 00 00 00 00 00 00 00 00 00 00 00 e5 00
[ 3882.303721] audit: type=1130 audit(1672401489.498:481): pid=1 uid=0 auid=4294967295 ses=4294967295 msg='unit=flatpak-system-helper comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=success'
[ 4016.638490] audit: type=1130 audit(1672401623.837:482): pid=1 uid=0 auid=4294967295 ses=4294967295 msg='unit=snapper-timeline comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=success'
[ 4016.655863] audit: type=1130 audit(1672401623.854:483): pid=1 uid=0 auid=4294967295 ses=4294967295 msg='unit=snapperd comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=success'
[ 4017.091483] audit: type=1131 audit(1672401624.291:484): pid=1 uid=0 auid=4294967295 ses=4294967295 msg='unit=snapper-timeline comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=success'
[ 4077.116724] audit: type=1131 audit(1672401684.319:485): pid=1 uid=0 auid=4294967295 ses=4294967295 msg='unit=snapperd comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=success'
[ 4368.845536] sd 3:0:0:0: [sdd] tag#10 FAILED Result: hostbyte=DID_BAD_TARGET driverbyte=DRIVER_OK cmd_age=0s
[ 4368.845542] sd 3:0:0:0: [sdd] tag#10 CDB: ATA command pass through(16) 85 06 20 00 00 00 00 00 00 00 00 00 00 00 e5 00
[ 4368.895132] sd 2:0:0:0: [sdc] tag#11 FAILED Result: hostbyte=DID_BAD_TARGET driverbyte=DRIVER_OK cmd_age=0s
[ 4368.895136] sd 2:0:0:0: [sdc] tag#11 CDB: ATA command pass through(16) 85 06 20 00 00 00 00 00 00 00 00 00 00 00 e5 00
[ 4499.879321] audit: type=1131 audit(1672402107.086:486): pid=1 uid=0 auid=4294967295 ses=4294967295 msg='unit=flatpak-system-helper comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=success'
[ 4968.835261] sd 3:0:0:0: [sdd] tag#11 FAILED Result: hostbyte=DID_BAD_TARGET driverbyte=DRIVER_OK cmd_age=0s
[ 4968.835268] sd 3:0:0:0: [sdd] tag#11 CDB: ATA command pass through(16) 85 06 20 00 00 00 00 00 00 00 00 00 00 00 e5 00
[ 4968.885080] sd 2:0:0:0: [sdc] tag#12 FAILED Result: hostbyte=DID_BAD_TARGET driverbyte=DRIVER_OK cmd_age=0s
[ 4968.885084] sd 2:0:0:0: [sdc] tag#12 CDB: ATA command pass through(16) 85 06 20 00 00 00 00 00 00 00 00 00 00 00 e5 00
[ 5568.748355] sd 3:0:0:0: [sdd] tag#18 FAILED Result: hostbyte=DID_BAD_TARGET driverbyte=DRIVER_OK cmd_age=0s
[ 5568.748361] sd 3:0:0:0: [sdd] tag#18 CDB: ATA command pass through(16) 85 06 20 00 00 00 00 00 00 00 00 00 00 00 e5 00
[ 5568.797652] sd 2:0:0:0: [sdc] tag#15 FAILED Result: hostbyte=DID_BAD_TARGET driverbyte=DRIVER_OK cmd_age=0s
[ 5568.797657] sd 2:0:0:0: [sdc] tag#15 CDB: ATA command pass through(16) 85 06 20 00 00 00 00 00 00 00 00 00 00 00 e5 00
[ 6168.809566] sd 3:0:0:0: [sdd] tag#13 FAILED Result: hostbyte=DID_BAD_TARGET driverbyte=DRIVER_OK cmd_age=0s
[ 6168.809573] sd 3:0:0:0: [sdd] tag#13 CDB: ATA command pass through(16) 85 06 20 00 00 00 00 00 00 00 00 00 00 00 e5 00
[ 6168.859365] sd 2:0:0:0: [sdc] tag#6 FAILED Result: hostbyte=DID_BAD_TARGET driverbyte=DRIVER_OK cmd_age=0s
[ 6168.859369] sd 2:0:0:0: [sdc] tag#6 CDB: ATA command pass through(16) 85 06 20 00 00 00 00 00 00 00 00 00 00 00 e5 00

Kernel Version: 6.0.12-arch1-1

Last edited by djmwj1 (2023-01-04 21:30:28)

Offline

#2 2023-01-02 10:51:15

d.ALT
Member
Registered: 2019-05-10
Posts: 920

Re: Hard Drive Diagnostic Help // AHCI AMD-Vi Issue?

ata4.00: exception Emask 0x0 SAct 0x7ff00 SErr 0x0 action 0x6 frozen
ata4.00: failed command: WRITE FPDMA QUEUED
ata4.00: cmd 61/00:48:d0:d4:62/01:00:04:00:00/40 tag 9 ncq dma 1048576 ou
                        res 40/00:81:82:00:00/00:00:00:00:00/40 Emask 0x4 (timeout)

Other than that, please check both SATA and power cables are good and well connected on both ends.
Contact manufacturer for RMA.

If you can LOSE ALL OF YOUR DATA:

--captive --idd long

and also

--captive --longDST
--longDST
		Execute a long diagnostic drive self test. This test takes
		hours to complete.  A 2TB drive may take six (6) hours to
		complete. Use with the --poll argument to let SeaChest check
		for progress and print it to the screen until complete.
		Use the --progress dst command to check on the completion
		percentage(%) and test result.
		This test stops on the first error. Use --abortDST
		to manually stop the test. SAS drives give status in 1%
		increments. SATA drives give status in 10% increments which
		means more than an hour may elapse between updates on a SATA
		drive > 2TB.

A 2TB drive may take six (6) hours to complete

https://github.com/Seagate/ToolBin/blob … _SMART.txt

Last edited by d.ALT (2023-01-02 10:51:39)

<49,17,III,I>    Fama di loro il mondo esser non lassa;
<50,17,III,I>    misericordia e giustizia li sdegna:
<51,17,III,I>    non ragioniam di lor, ma guarda e passa.

Offline

#3 2023-01-04 12:16:46

djmwj1
Member
Registered: 2013-10-04
Posts: 32

Re: Hard Drive Diagnostic Help // AHCI AMD-Vi Issue?

Thanks for your reply,
So the plot thickens, I have been trying different things to try to get this working, what I have found is the error does not follow the drive. If I swap the drives around the same error seems to occur on the sata port not the drive. I have swapped power and SATA cables but the issue persists. The fact that it seems to be based on the sata ports leads me to believe its not the drives, but rather something else.  Below is a snip of the latest logs after moving to new cables and swapping the drives around. 

Also the error does not crop up until about 1.3-1.5TB of data has been transferred, and this seems very consistent.... not sure if its relevant but again just seems odd.

I also changed my boot cmd to add

 iommu=soft

to see if that helped, but it does not seem to change this behavior. It does appear that this change stopped the repeated "AMD-Vi: Event logged [IO_PAGE_FAULT domain=0x0027 address=0x7ffffe00000 flags=0x0000]" and instead just shows one error followed by the rest. 

Jan 03 22:28:36 desktop kernel: ahci 0000:47:00.0: AMD-Vi: Event logged [IO_PAGE_FAULT domain=0x0027 address=0x7ffffe00000 flags=0x0000]
Jan 03 22:29:08 desktop kernel: ata1.00: exception Emask 0x0 SAct 0x3e00000 SErr 0x0 action 0x6 frozen
Jan 03 22:29:08 desktop kernel: ata1.00: failed command: WRITE FPDMA QUEUED
Jan 03 22:29:08 desktop kernel: ata1.00: cmd 61/3b:a8:f8:0f:fa/01:00:13:00:00/40 tag 21 ncq dma 1290240 ou
                                               res 40/00:01:00:4f:c2/00:00:00:00:00/00 Emask 0x4 (timeout)
Jan 03 22:29:08 desktop kernel: ata1.00: status: { DRDY }
Jan 03 22:29:08 desktop kernel: ata1.00: failed command: WRITE FPDMA QUEUED
Jan 03 22:29:08 desktop kernel: ata1.00: cmd 61/00:b0:33:11:fa/01:00:13:00:00/40 tag 22 ncq dma 1048576 ou
                                               res 40/00:00:00:4f:c2/00:00:00:00:00/40 Emask 0x4 (timeout)
Jan 03 22:29:08 desktop kernel: ata1.00: status: { DRDY }
Jan 03 22:29:08 desktop kernel: ata1.00: failed command: WRITE FPDMA QUEUED
Jan 03 22:29:08 desktop kernel: ata1.00: cmd 61/00:b8:33:12:fa/01:00:13:00:00/40 tag 23 ncq dma 1048576 ou
                                               res 40/00:ff:ff:00:00/00:00:00:00:00/40 Emask 0x4 (timeout)
Jan 03 22:29:08 desktop kernel: ata1.00: status: { DRDY }
Jan 03 22:29:08 desktop kernel: ata1.00: failed command: WRITE FPDMA QUEUED
Jan 03 22:29:08 desktop kernel: ata1.00: cmd 61/5b:c0:33:13:fa/00:00:13:00:00/40 tag 24 ncq dma 372736 out
                                               res 40/00:00:00:00:00/00:00:00:00:00/00 Emask 0x4 (timeout)
Jan 03 22:29:08 desktop kernel: ata1.00: status: { DRDY }
Jan 03 22:29:08 desktop kernel: ata1.00: failed command: WRITE FPDMA QUEUED
Jan 03 22:29:08 desktop kernel: ata1.00: cmd 61/00:c8:8e:13:fa/01:00:13:00:00/40 tag 25 ncq dma 1048576 ou
                                               res 40/00:01:00:4f:c2/00:00:00:00:00/00 Emask 0x4 (timeout)
Jan 03 22:29:08 desktop kernel: ata1.00: status: { DRDY }
Jan 03 22:29:08 desktop kernel: ata1: hard resetting link
Jan 03 22:29:08 desktop kernel: ata1: SATA link up 6.0 Gbps (SStatus 133 SControl 300)
Jan 03 22:29:25 desktop kernel: ata1.00: qc timeout after 15000 msecs (cmd 0x47)
Jan 03 22:29:25 desktop kernel: ata1.00: Read log 0x00 page 0x00 failed, Emask 0x40
Jan 03 22:29:25 desktop kernel: ata1.00: NCQ Send/Recv Log not supported
Jan 03 22:29:25 desktop kernel: ata1.00: Read log 0x00 page 0x00 failed, Emask 0x40
Jan 03 22:29:25 desktop kernel: ata1.00: configured for UDMA/133
Jan 03 22:29:25 desktop kernel: ata1.00: device reported invalid CHS sector 0
Jan 03 22:29:35 desktop kernel: ata1: softreset failed (1st FIS failed)
Jan 03 22:29:45 desktop kernel: ata1: softreset failed (1st FIS failed)
Jan 03 22:30:20 desktop kernel: ata1: softreset failed (1st FIS failed)
Jan 03 22:30:20 desktop kernel: ata1: limiting SATA link speed to 3.0 Gbps
Jan 03 22:30:25 desktop kernel: ata1: softreset failed (1st FIS failed)
Jan 03 22:30:25 desktop kernel: ata1: reset failed, giving up
Jan 03 22:30:25 desktop kernel: ata1.00: disable device
Jan 03 22:30:25 desktop kernel: ata1: EH complete
Jan 03 22:30:25 desktop kernel: sd 0:0:0:0: [sda] tag#13 FAILED Result: hostbyte=DID_BAD_TARGET driverbyte=DRIVER_OK cmd_age=108s
Jan 03 22:30:25 desktop kernel: sd 0:0:0:0: [sda] tag#13 CDB: Write(10) 2a 00 13 fa 0f f8 00 01 3b 00
Jan 03 22:30:25 desktop kernel: I/O error, dev sda, sector 2681241536 op 0x1:(WRITE) flags 0x104000 phys_seg 160 prio class 2
Jan 03 22:30:25 desktop kernel: sd 0:0:0:0: [sda] tag#8 FAILED Result: hostbyte=DID_BAD_TARGET driverbyte=DRIVER_OK cmd_age=0s
Jan 03 22:30:25 desktop kernel: sd 0:0:0:0: [sda] Read Capacity(16) failed: Result: hostbyte=DID_BAD_TARGET driverbyte=DRIVER_OK
Jan 03 22:30:25 desktop kernel: sd 0:0:0:0: [sda] tag#8 CDB: ATA command pass through(16) 85 06 20 00 00 00 00 00 00 00 00 00 00 00 e5 00
Jan 03 22:30:25 desktop kernel: sd 0:0:0:0: [sda] Sense not available.
Jan 03 22:30:25 desktop kernel: sd 0:0:0:0: [sda] Read Capacity(10) failed: Result: hostbyte=DID_BAD_TARGET driverbyte=DRIVER_OK
Jan 03 22:30:25 desktop kernel: sd 0:0:0:0: [sda] Sense not available.
Jan 03 22:30:25 desktop kernel: BTRFS error (device dm-5): bdev /dev/mapper/crypt_media1 errs: wr 1, rd 0, flush 0, corrupt 0, gen 0
Jan 03 22:30:25 desktop kernel: BTRFS error (device dm-5): bdev /dev/mapper/crypt_media1 errs: wr 1, rd 1, flush 0, corrupt 0, gen 0
Jan 03 22:30:25 desktop kernel: sd 0:0:0:0: [sda] tag#14 FAILED Result: hostbyte=DID_BAD_TARGET driverbyte=DRIVER_OK cmd_age=108s
Jan 03 22:30:25 desktop kernel: sd 0:0:0:0: [sda] tag#14 CDB: Write(10) 2a 00 13 fa 11 33 00 01 00 00
Jan 03 22:30:25 desktop kernel: I/O error, dev sda, sector 2681244056 op 0x1:(WRITE) flags 0x104000 phys_seg 79 prio class 2
Jan 03 22:30:25 desktop kernel: sd 0:0:0:0: [sda] tag#15 FAILED Result: hostbyte=DID_BAD_TARGET driverbyte=DRIVER_OK cmd_age=108s
Jan 03 22:30:25 desktop kernel: sd 0:0:0:0: [sda] tag#15 CDB: Write(10) 2a 00 13 fa 12 33 00 01 00 00
Jan 03 22:30:25 desktop kernel: I/O error, dev sda, sector 2681246104 op 0x1:(WRITE) flags 0x104000 phys_seg 83 prio class 2
Jan 03 22:30:25 desktop kernel: sd 0:0:0:0: [sda] tag#16 FAILED Result: hostbyte=DID_BAD_TARGET driverbyte=DRIVER_OK cmd_age=108s
Jan 03 22:30:25 desktop kernel: sd 0:0:0:0: [sda] tag#16 CDB: Write(10) 2a 00 13 fa 13 33 00 00 5b 00
Jan 03 22:30:25 desktop kernel: I/O error, dev sda, sector 2681248152 op 0x1:(WRITE) flags 0x104000 phys_seg 31 prio class 2
Jan 03 22:30:25 desktop kernel: BTRFS error (device dm-5): bdev /dev/mapper/crypt_media1 errs: wr 2, rd 1, flush 0, corrupt 0, gen 0
Jan 03 22:30:25 desktop kernel: BTRFS error (device dm-5): bdev /dev/mapper/crypt_media1 errs: wr 2, rd 2, flush 0, corrupt 0, gen 0
Jan 03 22:30:25 desktop kernel: sd 0:0:0:0: [sda] tag#17 FAILED Result: hostbyte=DID_BAD_TARGET driverbyte=DRIVER_OK cmd_age=108s
Jan 03 22:30:25 desktop kernel: sd 0:0:0:0: [sda] tag#17 CDB: Write(10) 2a 00 13 fa 13 8e 00 01 00 00
Jan 03 22:30:25 desktop kernel: I/O error, dev sda, sector 2681248880 op 0x1:(WRITE) flags 0x104000 phys_seg 68 prio class 2
Jan 03 22:30:25 desktop kernel: sd 0:0:0:0: [sda] tag#31 FAILED Result: hostbyte=DID_BAD_TARGET driverbyte=DRIVER_OK cmd_age=0s
Jan 03 22:30:25 desktop kernel: sd 0:0:0:0: [sda] tag#31 CDB: Write(10) 2a 00 13 fa 14 8e 00 01 00 00
Jan 03 22:30:25 desktop kernel: I/O error, dev sda, sector 2681250928 op 0x1:(WRITE) flags 0x100000 phys_seg 70 prio class 2
Jan 03 22:30:25 desktop kernel: sd 0:0:0:0: [sda] 0 4096-byte logical blocks: (0 B/0 B)
Jan 03 22:30:25 desktop kernel: sda: detected capacity change from 27344764928 to 0
Jan 03 22:30:25 desktop kernel: sd 0:0:0:0: [sda] tag#13 access beyond end of device
Jan 03 22:30:25 desktop kernel: I/O error, dev sda, sector 2681259216 op 0x1:(WRITE) flags 0x104000 phys_seg 68 prio class 2
Jan 03 22:30:25 desktop kernel: sd 0:0:0:0: [sda] tag#14 access beyond end of device
Jan 03 22:30:25 desktop kernel: I/O error, dev sda, sector 2681261264 op 0x1:(WRITE) flags 0x104000 phys_seg 67 prio class 2
Jan 03 22:30:25 desktop kernel: sd 0:0:0:0: [sda] tag#15 access beyond end of device
Jan 03 22:30:25 desktop kernel: I/O error, dev sda, sector 2681263312 op 0x1:(WRITE) flags 0x104000 phys_seg 82 prio class 2
Jan 03 22:30:25 desktop kernel: BTRFS error (device dm-5): bdev /dev/mapper/crypt_media1 errs: wr 3, rd 2, flush 0, corrupt 0, gen 0
Jan 03 22:30:25 desktop kernel: BTRFS error (device dm-5): bdev /dev/mapper/crypt_media1 errs: wr 3, rd 3, flush 0, corrupt 0, gen 0
Jan 03 22:30:25 desktop kernel: sd 0:0:0:0: [sda] tag#18 access beyond end of device
Jan 03 22:30:25 desktop kernel: I/O error, dev sda, sector 2681265552 op 0x1:(WRITE) flags 0x104000 phys_seg 68 prio class 2
Jan 03 22:30:25 desktop kernel: sd 0:0:0:0: [sda] tag#19 access beyond end of device
Jan 03 22:30:25 desktop kernel: sd 0:0:0:0: [sda] tag#20 access beyond end of device
Jan 03 22:30:25 desktop kernel: BTRFS error (device dm-5): bdev /dev/mapper/crypt_media1 errs: wr 4, rd 3, flush 0, corrupt 0, gen 0
Jan 03 22:30:25 desktop kernel: BTRFS error (device dm-5): bdev /dev/mapper/crypt_media1 errs: wr 4, rd 4, flush 0, corrupt 0, gen 0
Jan 03 22:30:25 desktop kernel: sd 0:0:0:0: [sda] tag#21 access beyond end of device
Jan 03 22:30:25 desktop kernel: sd 0:0:0:0: [sda] tag#22 access beyond end of device
Jan 03 22:30:25 desktop kernel: sd 0:0:0:0: [sda] tag#23 access beyond end of device
Jan 03 22:30:25 desktop kernel: sd 0:0:0:0: [sda] tag#24 access beyond end of device
Jan 03 22:30:25 desktop kernel: sd 0:0:0:0: [sda] tag#25 access beyond end of device
Jan 03 22:30:25 desktop kernel: BTRFS error (device dm-5): bdev /dev/mapper/crypt_media1 errs: wr 5, rd 4, flush 0, corrupt 0, gen 0
Jan 03 22:30:25 desktop kernel: BTRFS error (device dm-5): bdev /dev/mapper/crypt_media1 errs: wr 5, rd 5, flush 0, corrupt 0, gen 0
Jan 03 22:30:25 desktop kernel: sd 0:0:0:0: [sda] tag#26 access beyond end of device
Jan 03 22:30:25 desktop kernel: sd 0:0:0:0: [sda] tag#27 access beyond end of device
Jan 03 22:30:25 desktop kernel: sd 0:0:0:0: [sda] tag#28 access beyond end of device
Jan 03 22:30:25 desktop kernel: sd 0:0:0:0: [sda] tag#29 access beyond end of device
Jan 03 22:30:25 desktop kernel: sd 0:0:0:0: [sda] tag#30 access beyond end of device
Jan 03 22:30:25 desktop kernel: sd 0:0:0:0: [sda] tag#0 access beyond end of device
Jan 03 22:30:25 desktop kernel: sd 0:0:0:0: [sda] tag#1 access beyond end of device
Jan 03 22:30:25 desktop kernel: sd 0:0:0:0: [sda] tag#2 access beyond end of device
Jan 03 22:30:25 desktop kernel: sd 0:0:0:0: [sda] tag#3 access beyond end of device
Jan 03 22:30:25 desktop kernel: sd 0:0:0:0: [sda] tag#4 access beyond end of device
Jan 03 22:30:25 desktop kernel: sd 0:0:0:0: [sda] tag#8 access beyond end of device
Jan 03 22:30:25 desktop kernel: sd 0:0:0:0: [sda] tag#9 access beyond end of device
Jan 03 22:30:25 desktop kernel: sd 0:0:0:0: [sda] tag#10 access beyond end of device
Jan 03 22:30:25 desktop kernel: sd 0:0:0:0: [sda] tag#12 access beyond end of device
Jan 03 22:30:25 desktop kernel: sd 0:0:0:0: [sda] tag#13 access beyond end of device
Jan 03 22:30:25 desktop kernel: sd 0:0:0:0: [sda] tag#14 access beyond end of device
Jan 03 22:30:25 desktop kernel: sd 0:0:0:0: [sda] tag#15 access beyond end of device
Jan 03 22:30:25 desktop kernel: sd 0:0:0:0: [sda] tag#16 access beyond end of device
Jan 03 22:30:25 desktop kernel: sd 0:0:0:0: [sda] tag#17 access beyond end of device
Jan 03 22:30:25 desktop kernel: sd 0:0:0:0: [sda] tag#18 access beyond end of device
Jan 03 22:30:25 desktop kernel: sd 0:0:0:0: [sda] tag#19 access beyond end of device
Jan 03 22:30:25 desktop kernel: sd 0:0:0:0: [sda] tag#20 access beyond end of device
Jan 03 22:30:25 desktop kernel: sd 0:0:0:0: [sda] tag#21 access beyond end of device
Jan 03 22:30:25 desktop kernel: sd 0:0:0:0: [sda] tag#22 access beyond end of device
Jan 03 22:30:25 desktop kernel: sd 0:0:0:0: [sda] tag#23 access beyond end of device
Jan 03 22:30:25 desktop kernel: sd 0:0:0:0: [sda] tag#31 access beyond end of device
Jan 03 22:30:25 desktop kernel: sd 0:0:0:0: [sda] tag#0 access beyond end of device
Jan 03 22:30:25 desktop kernel: sd 0:0:0:0: [sda] tag#1 access beyond end of device
Jan 03 22:30:25 desktop kernel: sd 0:0:0:0: [sda] tag#2 access beyond end of device
Jan 03 22:30:25 desktop kernel: sd 0:0:0:0: [sda] tag#3 access beyond end of device
Jan 03 22:30:25 desktop kernel: sd 0:0:0:0: [sda] tag#4 access beyond end of device
Jan 03 22:30:25 desktop kernel: sd 0:0:0:0: [sda] tag#8 access beyond end of device
Jan 03 22:30:25 desktop kernel: sd 0:0:0:0: [sda] tag#9 access beyond end of device
Jan 03 22:30:25 desktop kernel: sd 0:0:0:0: [sda] tag#10 access beyond end of device
Jan 03 22:30:25 desktop kernel: sd 0:0:0:0: [sda] tag#12 access beyond end of device
Jan 03 22:30:25 desktop kernel: sd 0:0:0:0: [sda] tag#13 access beyond end of device
Jan 03 22:30:25 desktop kernel: sd 0:0:0:0: [sda] tag#14 access beyond end of device
Jan 03 22:30:25 desktop kernel: sd 0:0:0:0: [sda] tag#15 access beyond end of device
Jan 03 22:30:25 desktop kernel: sd 0:0:0:0: [sda] tag#16 access beyond end of device
Jan 03 22:30:25 desktop kernel: sd 0:0:0:0: [sda] tag#17 access beyond end of device
Jan 03 22:30:25 desktop kernel: sd 0:0:0:0: [sda] tag#18 access beyond end of device
Jan 03 22:30:25 desktop kernel: sd 0:0:0:0: [sda] tag#19 access beyond end of device
Jan 03 22:30:25 desktop kernel: sd 0:0:0:0: [sda] tag#20 access beyond end of device
Jan 03 22:30:25 desktop kernel: sd 0:0:0:0: [sda] tag#21 access beyond end of device
Jan 03 22:30:25 desktop kernel: sd 0:0:0:0: [sda] tag#22 access beyond end of device
Jan 03 22:30:25 desktop kernel: sd 0:0:0:0: [sda] tag#23 access beyond end of device
Jan 03 22:30:25 desktop kernel: sd 0:0:0:0: [sda] tag#24 access beyond end of device
Jan 03 22:30:25 desktop kernel: sd 0:0:0:0: [sda] tag#25 access beyond end of device
Jan 03 22:30:25 desktop kernel: sd 0:0:0:0: [sda] tag#26 access beyond end of device
Jan 03 22:30:25 desktop kernel: sd 0:0:0:0: [sda] tag#27 access beyond end of device
Jan 03 22:30:25 desktop kernel: sd 0:0:0:0: [sda] tag#28 access beyond end of device
Jan 03 22:30:25 desktop kernel: sd 0:0:0:0: [sda] tag#29 access beyond end of device
Jan 03 22:30:25 desktop kernel: sd 0:0:0:0: [sda] tag#30 access beyond end of device
Jan 03 22:30:25 desktop kernel: sd 0:0:0:0: [sda] tag#5 FAILED Result: hostbyte=DID_BAD_TARGET driverbyte=DRIVER_OK cmd_age=0s
Jan 03 22:30:25 desktop kernel: sd 0:0:0:0: [sda] tag#5 CDB: Write(10) 2a 00 13 fa 15 8e 00 00 c3 00
Jan 03 22:30:25 desktop kernel: sd 0:0:0:0: [sda] tag#9 access beyond end of device
Jan 03 22:30:25 desktop kernel: sd 0:0:0:0: [sda] tag#10 access beyond end of device
Jan 03 22:30:25 desktop kernel: sd 0:0:0:0: [sda] tag#6 FAILED Result: hostbyte=DID_BAD_TARGET driverbyte=DRIVER_OK cmd_age=0s
Jan 03 22:30:25 desktop kernel: sd 0:0:0:0: [sda] tag#6 CDB: Write(10) 2a 00 13 fa 16 51 00 01 00 00
Jan 03 22:30:25 desktop kernel: sd 0:0:0:0: [sda] tag#7 FAILED Result: hostbyte=DID_BAD_TARGET driverbyte=DRIVER_OK cmd_age=0s
Jan 03 22:30:25 desktop kernel: sd 0:0:0:0: [sda] tag#7 CDB: Write(10) 2a 00 13 fa 17 51 00 01 00 00
Jan 03 22:30:25 desktop kernel: sd 0:0:0:0: [sda] tag#12 access beyond end of device
Jan 03 22:30:25 desktop kernel: sd 0:0:0:0: [sda] tag#13 access beyond end of device
Jan 03 22:30:25 desktop kernel: sd 0:0:0:0: [sda] tag#14 access beyond end of device
Jan 03 22:30:25 desktop kernel: sd 0:0:0:0: [sda] tag#15 access beyond end of device
Jan 03 22:30:25 desktop kernel: sd 0:0:0:0: [sda] tag#16 access beyond end of device

And goes on like that
d.ALT wrote:

If you can LOSE ALL OF YOUR DATA:

--captive --idd long

and also

--captive --longDST

do these apply to the SeaChest_genericTest command like:

SeaChest_genericTest --captive --longDST

Last edited by djmwj1 (2023-01-04 12:20:51)

Offline

#4 2023-01-04 21:17:00

djmwj1
Member
Registered: 2013-10-04
Posts: 32

Re: Hard Drive Diagnostic Help // AHCI AMD-Vi Issue?

Well its been scrubbing the volumes all day and has obviously been working hard to correct the errors. It is intersting because it is still giving AMD-Vi ahci errors but they don't seem like they are as fatal during the scrub, the drive has remained online and it looks like the scrub is still fixing up errors. 

33749.226339] BTRFS error (device dm-2): bdev /dev/mapper/crypt_media1 errs: wr 2332322, rd 1054654, flush 0, corrupt 12425879, gen 4027
[33752.901566] scrub_handle_errored_block: 1506 callbacks suppressed
[33752.901569] BTRFS error (device dm-2): fixed up error at logical 1417591443456 on dev /dev/mapper/crypt_media1
[33752.901681] BTRFS error (device dm-2): fixed up error at logical 1417591500800 on dev /dev/mapper/crypt_media1
[33752.959697] BTRFS error (device dm-2): fixed up error at logical 1417591447552 on dev /dev/mapper/crypt_media1
[33752.959813] BTRFS error (device dm-2): fixed up error at logical 1417591504896 on dev /dev/mapper/crypt_media1
[33752.961145] BTRFS error (device dm-2): fixed up error at logical 1417591934976 on dev /dev/mapper/crypt_media1
[33752.967537] BTRFS error (device dm-2): fixed up error at logical 1417591451648 on dev /dev/mapper/crypt_media1
[33752.984716] BTRFS error (device dm-2): fixed up error at logical 1417591508992 on dev /dev/mapper/crypt_media1
[33752.986251] BTRFS error (device dm-2): fixed up error at logical 1417591939072 on dev /dev/mapper/crypt_media1
[33752.992693] BTRFS error (device dm-2): fixed up error at logical 1417591513088 on dev /dev/mapper/crypt_media1
[33753.001010] BTRFS error (device dm-2): fixed up error at logical 1417591517184 on dev /dev/mapper/crypt_media1
[33754.142820] amd_iommu_report_page_fault: 1090 callbacks suppressed
[33754.142824] ahci 0000:47:00.0: AMD-Vi: Event logged [IO_PAGE_FAULT domain=0x0027 address=0x7ffffe00000 flags=0x0020]
[33754.142837] ahci 0000:47:00.0: AMD-Vi: Event logged [IO_PAGE_FAULT domain=0x0027 address=0x7ffffe00180 flags=0x0020]
[33754.142847] ahci 0000:47:00.0: AMD-Vi: Event logged [IO_PAGE_FAULT domain=0x0027 address=0x7ffffe00280 flags=0x0020]
[33754.142856] ahci 0000:47:00.0: AMD-Vi: Event logged [IO_PAGE_FAULT domain=0x0027 address=0x7ffffe00380 flags=0x0020]
[33754.142865] ahci 0000:47:00.0: AMD-Vi: Event logged [IO_PAGE_FAULT domain=0x0027 address=0x7ffffe00400 flags=0x0020]
[33754.142874] ahci 0000:47:00.0: AMD-Vi: Event logged [IO_PAGE_FAULT domain=0x0027 address=0x7ffffe00500 flags=0x0020]
[33754.142884] ahci 0000:47:00.0: AMD-Vi: Event logged [IO_PAGE_FAULT domain=0x0027 address=0x7ffffe00600 flags=0x0020]
[33754.142893] ahci 0000:47:00.0: AMD-Vi: Event logged [IO_PAGE_FAULT domain=0x0027 address=0x7ffffe00700 flags=0x0020]
[33754.142902] ahci 0000:47:00.0: AMD-Vi: Event logged [IO_PAGE_FAULT domain=0x0027 address=0x7ffffe00800 flags=0x0020]
[33754.142911] ahci 0000:47:00.0: AMD-Vi: Event logged [IO_PAGE_FAULT domain=0x0027 address=0x7ffffe00900 flags=0x0020]
[33754.146315] AMD-Vi: IOMMU event log overflow
[33754.149894] AMD-Vi: IOMMU event log overflow
...
[33769.277497] BTRFS error (device dm-2): bdev /dev/mapper/crypt_media1 errs: wr 2332322, rd 1054654, flush 0, corrupt 12434152, gen 4027
[33772.947552] scrub_handle_errored_block: 1625 callbacks suppressed
[33772.947557] BTRFS error (device dm-2): fixed up error at logical 1417625935872 on dev /dev/mapper/crypt_media1
[33772.956719] BTRFS error (device dm-2): fixed up error at logical 1417626066944 on dev /dev/mapper/crypt_media1
[33773.009987] BTRFS error (device dm-2): fixed up error at logical 1417624952832 on dev /dev/mapper/crypt_media1
[33773.011598] BTRFS error (device dm-2): fixed up error at logical 1417625354240 on dev /dev/mapper/crypt_media1
[33773.012962] BTRFS error (device dm-2): fixed up error at logical 1417625190400 on dev /dev/mapper/crypt_media1
[33773.020737] BTRFS error (device dm-2): fixed up error at logical 1417625427968 on dev /dev/mapper/crypt_media1
[33773.029932] BTRFS error (device dm-2): fixed up error at logical 1417625559040 on dev /dev/mapper/crypt_media1
[33773.038740] BTRFS error (device dm-2): fixed up error at logical 1417625690112 on dev /dev/mapper/crypt_media1
[33773.047535] BTRFS error (device dm-2): fixed up error at logical 1417625939968 on dev /dev/mapper/crypt_media1
[33773.056708] BTRFS error (device dm-2): fixed up error at logical 1417626071040 on dev /dev/mapper/crypt_media1
[33774.228483] scrub_handle_errored_block: 1155 callbacks suppressed
....

btrfs scrub status:

Scrub started:    Wed Jan  4 06:52:56 2023
Status:           running
Duration:         9:24:15
Time left:        1:53:06
ETA:              Wed Jan  4 18:10:19 2023
Total to scrub:   3.41TiB
Bytes scrubbed:   2.84TiB  (83.30%)
Rate:             88.00MiB/s
Error summary:    super=3 verify=4027 csum=12554042
  Corrected:      12558065
  Uncorrectable:  0
  Unverified:     0

I don't understand the ahci errors.... its only effecting this volume... I have 3 other RAID1 btrfs volumes that have not shown any issues.... the only differences i can think of are:
1) this btrfs volume was built on a newer kernel
2) this is the only volume that is on 4k sector format hard drives

not sure if the 4k sector thing is the hidden issue here, but i have verified both dm-crypt shows it as 4k sectors:

cryptsetup luksDump 

Data segments:
  0: crypt
	offset: 16777216 [bytes]
	length: (whole device)
	cipher: aes-xts-plain64
	sector: 4096 [bytes]

Last edited by djmwj1 (2023-01-04 21:28:42)

Offline

#5 2023-01-05 08:11:32

d.ALT
Member
Registered: 2019-05-10
Posts: 920

Re: Hard Drive Diagnostic Help // AHCI AMD-Vi Issue?

djmwj1 wrote:

the error does not follow the drive. If I swap the drives around the same error seems to occur on the sata port not the drive. I have swapped power and SATA cables but the issue persists. The fact that it seems to be based on the sata ports leads me to believe its not the drives, but rather something else.

Mmm... What's your motherboard exact model? Is you PSU (what model?) "strong" enough to handle all of your HDD(s) wattages?

Speaking about IOMMU thing... Worth giving it a try (although it is quite (out)dated): https://bbs.archlinux.org/viewtopic.php … 7#p1321877

Can you post here a full dmesg?

<49,17,III,I>    Fama di loro il mondo esser non lassa;
<50,17,III,I>    misericordia e giustizia li sdegna:
<51,17,III,I>    non ragioniam di lor, ma guarda e passa.

Offline

#6 2023-01-05 14:30:06

djmwj1
Member
Registered: 2013-10-04
Posts: 32

Re: Hard Drive Diagnostic Help // AHCI AMD-Vi Issue?

Thanks, I have changed over to IOMMU=SOFT for kernel parameters as suggested in the post you linked, but it has not changed the behavior.
I believe the kernel parameter is being honored as seen in my dmesg logs (using SWIOTLB):

Jan 04 06:51:18 david-desktop kernel: PCI-DMA: Using software bounce buffering for IO (SWIOTLB)
Jan 04 06:51:18 david-desktop kernel: software IO TLB: mapped [mem 0x000000007bc1b000-0x000000007fc1b000] (64MB)

It is interesting that it still is using some amd-vi even with SWIOTLB, but I certainly cannot speak intelligently to the inner working of IOMMU/AHCI.

My Hardware:

Motherboard: ASUS ROG ZENITH II EXTREME ALPHA
Processor: AMD Ryzen Threadripper 3970X 32-Core Processor
PSU: 1000W  EVGA Supernova G+ https://www.evga.com/products/product.a … GP-1000-X1

I calculate out my full load at 867 watts (but these transactions are not being done while my graphics cards are heavily loaded so I suspect I am well below that.

I could post the full dmesg, but because of the error spamming its about 104MB. I am happy to excise parts that you think are relevent. Like the bootup?

Thanks again for the help.

Offline

#7 2023-01-05 22:35:50

djmwj1
Member
Registered: 2013-10-04
Posts: 32

Re: Hard Drive Diagnostic Help // AHCI AMD-Vi Issue?

Well, I am more flabbergasted now. I simply swapped SATA cables (not ports) between the two drives, remade luks partitions and remade the BTRFS RAID1. I transferred all of the data without a single error, AHCI or BTRFS! I have no idea why..... maybe the SATA cable was not seated right?!?!?!?! but it seems it worked..... not sure if i should trust it, not sure how else to test it.....

btrfs dev stats:

[/dev/mapper/crypt_media1].write_io_errs    0
[/dev/mapper/crypt_media1].read_io_errs     0
[/dev/mapper/crypt_media1].flush_io_errs    0
[/dev/mapper/crypt_media1].corruption_errs  0
[/dev/mapper/crypt_media1].generation_errs  0
[/dev/mapper/crypt_media2].write_io_errs    0
[/dev/mapper/crypt_media2].read_io_errs     0
[/dev/mapper/crypt_media2].flush_io_errs    0
[/dev/mapper/crypt_media2].corruption_errs  0
[/dev/mapper/crypt_media2].generation_errs  0

Offline

Pages: 1

Board footer

Powered by FluxBB