You are not logged in.
To Long / didn't read: This documents my journey to to setup and test a btrfs filesystem.
Goal:
I want to setup a btrfs filesystem spanning 3 drives, capable of loosing 2 drives without loosing data.
Scenario:
I have a PC with 4 hard drives. All data on these drives is disposable. One drive is currently in use as a boot drive. The other 3 are 1TB drives with partitions with various sizes. This is intentional, as I want to simulate an array where some drives were replaced with bigger drives.
/dev/sdb6 930.6G
/dev/sdc1 223.6G
/dev/sdd6 930.6G
What's working sofar:
I've formatted each drive with a btrfs filesystem using these commands:
/dev/sdb6
# mkfs.btrfs /dev/sdb6
btrfs-progs v5.9
See http://btrfs.wiki.kernel.org for more information.
/dev/sdb6 appears to contain an existing filesystem (btrfs).
ERROR: use the -f option to force overwrite of /dev/sdb6
[root@cedric-nas cedric]# mkfs.btrfs /dev/sdb6 -f
btrfs-progs v5.9
See http://btrfs.wiki.kernel.org for more information.
Label: (null)
UUID: a3740759-5f92-499a-8788-4bba4aa67dfe
Node size: 16384
Sector size: 4096
Filesystem size: 930.62GiB
Block group profiles:
Data: single 8.00MiB
Metadata: DUP 1.00GiB
System: DUP 8.00MiB
SSD detected: no
Incompat features: extref, skinny-metadata
Runtime features:
Checksum: crc32c
Number of devices: 1
Devices:
ID SIZE PATH
1 930.62GiB /dev/sdb6/dev/sdc1
# mkfs.btrfs /dev/sdc1
btrfs-progs v5.9
See http://btrfs.wiki.kernel.org for more information.
/dev/sdc1 appears to contain an existing filesystem (btrfs).
ERROR: use the -f option to force overwrite of /dev/sdc1
[root@cedric-nas cedric]# mkfs.btrfs /dev/sdc1 -f
btrfs-progs v5.9
See http://btrfs.wiki.kernel.org for more information.
Label: (null)
UUID: 75f5afd6-601a-4434-bc92-c8ca57e7ff9c
Node size: 16384
Sector size: 4096
Filesystem size: 223.57GiB
Block group profiles:
Data: single 8.00MiB
Metadata: DUP 1.00GiB
System: DUP 8.00MiB
SSD detected: no
Incompat features: extref, skinny-metadata
Runtime features:
Checksum: crc32c
Number of devices: 1
Devices:
ID SIZE PATH
1 223.57GiB /dev/sdc1/dev/sdd6
# mkfs.btrfs /dev/sdd6
btrfs-progs v5.9
See http://btrfs.wiki.kernel.org for more information.
/dev/sdd6 appears to contain an existing filesystem (btrfs).
ERROR: use the -f option to force overwrite of /dev/sdd6
[root@cedric-nas cedric]# mkfs.btrfs /dev/sdd6 -f
btrfs-progs v5.9
See http://btrfs.wiki.kernel.org for more information.
Label: (null)
UUID: 986b9ff2-349f-4edd-96bb-8bdc53486b5d
Node size: 16384
Sector size: 4096
Filesystem size: 930.62GiB
Block group profiles:
Data: single 8.00MiB
Metadata: DUP 1.00GiB
System: DUP 8.00MiB
SSD detected: no
Incompat features: extref, skinny-metadata
Runtime features:
Checksum: crc32c
Number of devices: 1
Devices:
ID SIZE PATH
1 930.62GiB /dev/sdd6Now I want to combine these filesystems into one mountpoint. So I mount the first partition:
# mount /dev/sdb6 /storage-btrfs/
# df -h
/dev/sdb6 931G 3.8M 929G 1% /storage-btrfsAs an aside, btrfs device scan does not find any btrfs volumes, even though I have 3 of them. I don't understand.
# btrfs device scan
Scanning for Btrfs filesystems
# btrfs device scan /dev/sdb
Scanning for btrfs filesystems on '/dev/sdb'
ERROR: device scan failed on '/dev/sdb': Device or resource busyNow add the second partition to the pool:
# btrfs device add /dev/sdc1 /storage-btrfs
/dev/sdc1 appears to contain an existing filesystem (btrfs).
ERROR: use the -f option to force overwrite of /dev/sdc1
# btrfs device add /dev/sdc1 /storage-btrfs -f
[root@cedric-nas cedric]# df -h
/dev/sdb6 1.2T 3.8M 1.2T 1% /storage-btrfs
# btrfs filesystem show
Label: none uuid: a3740759-5f92-499a-8788-4bba4aa67dfe
Total devices 2 FS bytes used 384.00KiB
devid 1 size 930.62GiB used 2.02GiB path /dev/sdb6
devid 2 size 223.57GiB used 0.00B path /dev/sdc1
Label: none uuid: 986b9ff2-349f-4edd-96bb-8bdc53486b5d
Total devices 1 FS bytes used 128.00KiB
devid 1 size 930.62GiB used 2.02GiB path /dev/sdd6Conclusion: this will destroy all data on the to be added drive.
Is there a way to add a drive that is partially filled to the array without loosing that data?
Why is there 2 gigabyte used on /dev/sdb6? Is that metadata?
Now convert to raid1 (2 copies)
# # btrfs balance start -dconvert=raid1 -mconvert=raid1 /storage-btrfs
Done, had to relocate 3 out of 3 chunks
[root@cedric-nas cedric]# DF -H
bash: DF: command not found
[root@cedric-nas cedric]# df -h
/dev/sdb6 578G 3.9M 223G 1% /storage-btrfs
[root@cedric-nas cedric]# btrfs filesystem show
Label: none uuid: a3740759-5f92-499a-8788-4bba4aa67dfe
Total devices 2 FS bytes used 640.00KiB
devid 1 size 930.62GiB used 3.06GiB path /dev/sdb6
devid 2 size 223.57GiB used 3.06GiB path /dev/sdc1
Label: none uuid: 986b9ff2-349f-4edd-96bb-8bdc53486b5d
Total devices 1 FS bytes used 128.00KiB
devid 1 size 930.62GiB used 2.02GiB path /dev/sdd6As expected, the usable size of the array shrinks to the size of the smallest drive.
Why does "btrfs filesystem show" show two separate filesystems?
Now add the third partition to the pool:
# btrfs device add /dev/sdd6 /storage-btrfs
/dev/sdd6 appears to contain an existing filesystem (btrfs).
ERROR: use the -f option to force overwrite of /dev/sdd6
[root@cedric-nas cedric]# btrfs device add /dev/sdd6 /storage-btrfs -f
[root@cedric-nas cedric]# df -h
/dev/sdb6 1.1T 3.9M 930G 1% /storage-btrfs
[root@cedric-nas cedric]# btrfs filesystem show
Label: none uuid: a3740759-5f92-499a-8788-4bba4aa67dfe
Total devices 3 FS bytes used 640.00KiB
devid 1 size 930.62GiB used 3.06GiB path /dev/sdb6
devid 2 size 223.57GiB used 3.06GiB path /dev/sdc1
devid 3 size 930.62GiB used 0.00B path /dev/sdd6Next balance again:
# btrfs balance start /storage-btrfs/
WARNING:
Full balance without filters requested. This operation is very
intense and takes potentially very long. It is recommended to
use the balance filters to narrow down the scope of balance.
Use 'btrfs balance start --full-balance' option to skip this
warning. The operation will start in 10 seconds.
Use Ctrl-C to stop it.
10 9 8 7 6 5 4 3 2 1
Starting balance without any filters.
Done, had to relocate 5 out of 5 chunks
# btrfs filesystem show
Label: none uuid: a3740759-5f92-499a-8788-4bba4aa67dfe
Total devices 3 FS bytes used 912.00KiB
devid 1 size 930.62GiB used 2.03GiB path /dev/sdb6
devid 2 size 223.57GiB used 0.00B path /dev/sdc1
devid 3 size 930.62GiB used 2.03GiB path /dev/sdd6
[root@cedric-nas cedric]# df -h
Filesystem Size Used Avail Use% Mounted on
/dev/sdb6 1.1T 3.9M 930G 1% /storage-btrfsNow convert it into a raid1c3 (3 copies)
# btrfs balance start -dconvert=raid1c3 -mconvert=raid1c3 /storage-btrfs/
Done, had to relocate 3 out of 3 chunks
[root@cedric-nas cedric]# btrfs filesystem show
Label: none uuid: a3740759-5f92-499a-8788-4bba4aa67dfe
Total devices 3 FS bytes used 656.00KiB
devid 1 size 930.62GiB used 3.06GiB path /dev/sdb6
devid 2 size 223.57GiB used 3.06GiB path /dev/sdc1
devid 3 size 930.62GiB used 3.06GiB path /dev/sdd6
[root@cedric-nas cedric]# df -h
/dev/sdb6 695G 3.9M 223G 1% /storage-btrfsLast edited by cdwijs (2020-12-11 13:42:18)
Offline
Start a scrub of the files in use:
# btrfs scrub start /storage-btrfs/
# btrfs scrub status /storage-btrfs/
UUID: a3740759-5f92-499a-8788-4bba4aa67dfe
Scrub started: Fri Dec 11 13:36:45 2020
Status: running
Duration: 0:01:35
Time left: 0:01:03
ETA: Fri Dec 11 13:39:25 2020
Total to scrub: 46.02GiB
Bytes scrubbed: 27.56GiB
Rate: 297.10MiB/s
Error summary: no errors foundNow let's corrupt some data on one of the drives. Warning: running this command will write 100MB of random data to /dev/sdb6, and will lead to data loss:
dd if=/dev/urandom of=/dev/sdb6 bs=1M count=100
100+0 records in
100+0 records out
104857600 bytes (105 MB, 100 MiB) copied, 0.651539 s, 161 MB/sWhile reading some of the data, lot's of error messages pop up in the syslog
# md5sum *
Dec 11 13:45:57 cedric-nas kernel: BTRFS warning (device sdb6): csum failed root 5 ino 276 off 0 csum 0xe3d70a02 expected csum 0x6596037e mirror 2
Dec 11 13:45:57 cedric-nas kernel: BTRFS warning (device sdb6): csum failed root 5 ino 276 off 65536 csum 0xc914e938 expected csum 0x690d874b mirror 2
Dec 11 13:45:57 cedric-nas kernel: BTRFS error (device sdb6): bdev /dev/sdb6 errs: wr 0, rd 0, flush 0, corrupt 2, gen 0
Dec 11 13:45:57 cedric-nas kernel: BTRFS warning (device sdb6): csum failed root 5 ino 276 off 69632 csum 0x77ab1109 expected csum 0x027da353 mirror 2
Dec 11 13:45:57 cedric-nas kernel: BTRFS error (device sdb6): bdev /dev/sdb6 errs: wr 0, rd 0, flush 0, corrupt 3, gen 0
Dec 11 13:45:57 cedric-nas kernel: BTRFS warning (device sdb6): csum failed root 5 ino 276 off 73728 csum 0xfdbc7c20 expected csum 0xeb389a6d mirror 2
Dec 11 13:45:57 cedric-nas kernel: BTRFS error (device sdb6): bdev /dev/sdb6 errs: wr 0, rd 0, flush 0, corrupt 4, gen 0
Dec 11 13:45:57 cedric-nas kernel: BTRFS warning (device sdb6): csum failed root 5 ino 276 off 77824 csum 0x9a309df8 expected csum 0xc015cd50 mirror 2
Dec 11 13:45:57 cedric-nas kernel: BTRFS error (device sdb6): bdev /dev/sdb6 errs: wr 0, rd 0, flush 0, corrupt 5, gen 0
Dec 11 13:45:57 cedric-nas kernel: BTRFS warning (device sdb6): csum failed root 5 ino 276 off 81920 csum 0x387eb64e expected csum 0x35aa9487 mirror 2
Dec 11 13:45:57 cedric-nas kernel: BTRFS error (device sdb6): bdev /dev/sdb6 errs: wr 0, rd 0, flush 0, corrupt 6, gen 0
Dec 11 13:45:57 cedric-nas kernel: BTRFS warning (device sdb6): csum failed root 5 ino 276 off 4096 csum 0x5e7c8ef0 expected csum 0x55e90252 mirror 2
Dec 11 13:45:57 cedric-nas kernel: BTRFS warning (device sdb6): csum failed root 5 ino 276 off 86016 csum 0x95bd42c8 expected csum 0x60a1fc2f mirror 2
Dec 11 13:45:57 cedric-nas kernel: BTRFS error (device sdb6): bdev /dev/sdb6 errs: wr 0, rd 0, flush 0, corrupt 7, gen 0
Dec 11 13:45:57 cedric-nas kernel: BTRFS error (device sdb6): bdev /dev/sdb6 errs: wr 0, rd 0, flush 0, corrupt 8, gen 0
Dec 11 13:45:57 cedric-nas kernel: BTRFS warning (device sdb6): csum failed root 5 ino 276 off 90112 csum 0xe537100b expected csum 0x5ca14c47 mirror 2
Dec 11 13:45:57 cedric-nas kernel: BTRFS error (device sdb6): bdev /dev/sdb6 errs: wr 0, rd 0, flush 0, corrupt 9, gen 0
Dec 11 13:45:57 cedric-nas kernel: BTRFS warning (device sdb6): csum failed root 5 ino 276 off 94208 csum 0xc995bb36 expected csum 0x30d87089 mirror 2
Dec 11 13:45:57 cedric-nas kernel: BTRFS error (device sdb6): bdev /dev/sdb6 errs: wr 0, rd 0, flush 0, corrupt 10, gen 0
Dec 11 13:45:57 cedric-nas kernel: BTRFS info (device sdb6): read error corrected: ino 276 off 0 (dev /dev/sdb6 sector 2048)
Dec 11 13:45:57 cedric-nas kernel: BTRFS info (device sdb6): read error corrected: ino 276 off 73728 (dev /dev/sdb6 sector 2192)
Dec 11 13:45:57 cedric-nas kernel: BTRFS info (device sdb6): read error corrected: ino 276 off 69632 (dev /dev/sdb6 sector 2184)
Dec 11 13:45:57 cedric-nas kernel: BTRFS info (device sdb6): read error corrected: ino 276 off 77824 (dev /dev/sdb6 sector 2200)
Dec 11 13:45:57 cedric-nas kernel: BTRFS info (device sdb6): read error corrected: ino 276 off 65536 (dev /dev/sdb6 sector 2176)
Dec 11 13:45:57 cedric-nas kernel: BTRFS info (device sdb6): read error corrected: ino 276 off 81920 (dev /dev/sdb6 sector 2208)
Dec 11 13:45:57 cedric-nas kernel: BTRFS info (device sdb6): read error corrected: ino 276 off 94208 (dev /dev/sdb6 sector 2232)
Dec 11 13:45:57 cedric-nas kernel: BTRFS info (device sdb6): read error corrected: ino 276 off 90112 (dev /dev/sdb6 sector 2224)
Dec 11 13:45:57 cedric-nas kernel: BTRFS info (device sdb6): read error corrected: ino 276 off 98304 (dev /dev/sdb6 sector 2240)
Dec 11 13:45:57 cedric-nas kernel: BTRFS info (device sdb6): read error corrected: ino 276 off 86016 (dev /dev/sdb6 sector 2216)
Dec 11 13:46:17 cedric-nas kernel: btrfs_print_data_csum_error: 10102 callbacks suppressed
Dec 11 13:46:17 cedric-nas kernel: btrfs_dev_stat_print_on_error: 10101 callbacks suppressed
Dec 11 13:46:17 cedric-nas kernel: BTRFS warning (device sdb6): csum failed root 5 ino 290 off 0 csum 0x84d94a6c expected csum 0x6a506320 mirror 2
Dec 11 13:46:17 cedric-nas kernel: BTRFS error (device sdb6): bdev /dev/sdb6 errs: wr 0, rd 0, flush 0, corrupt 10113, gen 0
Dec 11 13:46:17 cedric-nas kernel: BTRFS error (device sdb6): bdev /dev/sdb6 errs: wr 0, rd 0, flush 0, corrupt 10114, gen 0
Dec 11 13:46:17 cedric-nas kernel: BTRFS warning (device sdb6): csum failed root 5 ino 290 off 69632 csum 0x54e68b0b expected csum 0xf5bf81ba mirror 2
Dec 11 13:46:17 cedric-nas kernel: BTRFS error (device sdb6): bdev /dev/sdb6 errs: wr 0, rd 0, flush 0, corrupt 10115, gen 0
Dec 11 13:46:17 cedric-nas kernel: BTRFS warning (device sdb6): csum failed root 5 ino 290 off 4096 csum 0x52a7877f expected csum 0xac098a77 mirror 2
Dec 11 13:46:17 cedric-nas kernel: BTRFS error (device sdb6): bdev /dev/sdb6 errs: wr 0, rd 0, flush 0, corrupt 10116, gen 0
Dec 11 13:46:17 cedric-nas kernel: BTRFS warning (device sdb6): csum failed root 5 ino 290 off 8192 csum 0xd5ca89e5 expected csum 0xaf21c625 mirror 2
Dec 11 13:46:17 cedric-nas kernel: BTRFS error (device sdb6): bdev /dev/sdb6 errs: wr 0, rd 0, flush 0, corrupt 10117, gen 0
Dec 11 13:46:17 cedric-nas kernel: BTRFS warning (device sdb6): csum failed root 5 ino 290 off 12288 csum 0xb5cf4785 expected csum 0xf41c777b mirror 2
Dec 11 13:46:17 cedric-nas kernel: BTRFS error (device sdb6): bdev /dev/sdb6 errs: wr 0, rd 0, flush 0, corrupt 10118, gen 0
Dec 11 13:46:17 cedric-nas kernel: BTRFS warning (device sdb6): csum failed root 5 ino 290 off 16384 csum 0x9fb6f34d expected csum 0xf49dff73 mirror 2
Dec 11 13:46:17 cedric-nas kernel: BTRFS error (device sdb6): bdev /dev/sdb6 errs: wr 0, rd 0, flush 0, corrupt 10119, gen 0
Dec 11 13:46:17 cedric-nas kernel: BTRFS warning (device sdb6): csum failed root 5 ino 290 off 20480 csum 0xad956ab2 expected csum 0x612b1841 mirror 2
Dec 11 13:46:17 cedric-nas kernel: BTRFS error (device sdb6): bdev /dev/sdb6 errs: wr 0, rd 0, flush 0, corrupt 10120, gen 0
Dec 11 13:46:17 cedric-nas kernel: BTRFS warning (device sdb6): csum failed root 5 ino 290 off 24576 csum 0x67373762 expected csum 0x80ae299e mirror 2
Dec 11 13:46:17 cedric-nas kernel: BTRFS error (device sdb6): bdev /dev/sdb6 errs: wr 0, rd 0, flush 0, corrupt 10121, gen 0
Dec 11 13:46:17 cedric-nas kernel: BTRFS warning (device sdb6): csum failed root 5 ino 290 off 28672 csum 0x3172186d expected csum 0x9b1ff623 mirror 2
Dec 11 13:46:17 cedric-nas kernel: BTRFS error (device sdb6): bdev /dev/sdb6 errs: wr 0, rd 0, flush 0, corrupt 10122, gen 0
Dec 11 13:46:17 cedric-nas kernel: BTRFS warning (device sdb6): csum failed root 5 ino 290 off 32768 csum 0xfb48990e expected csum 0x9491be79 mirror 2
Dec 11 13:46:17 cedric-nas kernel: repair_io_failure: 10091 callbacks suppressed
Dec 11 13:46:17 cedric-nas kernel: BTRFS info (device sdb6): read error corrected: ino 290 off 65536 (dev /dev/sdb6 sector 83072)
Dec 11 13:46:17 cedric-nas kernel: BTRFS info (device sdb6): read error corrected: ino 290 off 8192 (dev /dev/sdb6 sector 82960)
Dec 11 13:46:17 cedric-nas kernel: BTRFS info (device sdb6): read error corrected: ino 290 off 4096 (dev /dev/sdb6 sector 82952)
Dec 11 13:46:17 cedric-nas kernel: BTRFS info (device sdb6): read error corrected: ino 290 off 0 (dev /dev/sdb6 sector 82944)
Dec 11 13:46:17 cedric-nas kernel: BTRFS info (device sdb6): read error corrected: ino 290 off 12288 (dev /dev/sdb6 sector 82968)
Dec 11 13:46:17 cedric-nas kernel: BTRFS info (device sdb6): read error corrected: ino 290 off 20480 (dev /dev/sdb6 sector 82984)
Dec 11 13:46:17 cedric-nas kernel: BTRFS info (device sdb6): read error corrected: ino 290 off 16384 (dev /dev/sdb6 sector 82976)
Dec 11 13:46:17 cedric-nas kernel: BTRFS info (device sdb6): read error corrected: ino 290 off 28672 (dev /dev/sdb6 sector 83000)
Dec 11 13:46:17 cedric-nas kernel: BTRFS info (device sdb6): read error corrected: ino 290 off 24576 (dev /dev/sdb6 sector 82992)
Dec 11 13:46:17 cedric-nas kernel: BTRFS info (device sdb6): read error corrected: ino 290 off 36864 (dev /dev/sdb6 sector 83016)let's scrub the data again:
# btrfs scrub start /storage-btrfs/
# WARNING: errors detected during scrubbing, corrected
# btrfs scrub status /storage-btrfs/
UUID: a3740759-5f92-499a-8788-4bba4aa67dfe
Scrub started: Fri Dec 11 13:49:45 2020
Status: finished
Duration: 0:04:57
Total to scrub: 46.02GiB
Rate: 158.69MiB/s
Error summary: super=2 csum=9200
Corrected: 9200
Uncorrectable: 0
Unverified: 0The filesystem seems to be OK, and the md5 sums of the data are OK.
Now let's pull the sata cable from one of the drives, while the command md5sum * is runnning, and plug it back in after a minute. This gives an impressive amount of errors, and the md5 command runs much slower than usual
Dec 11 14:04:28 cedric-nas kernel: ata4.00: exception Emask 0x0 SAct 0x0 SErr 0x0 action 0x6 frozen
Dec 11 14:04:28 cedric-nas kernel: ata4.00: failed command: READ DMA EXT
Dec 11 14:04:28 cedric-nas kernel: ata4.00: cmd 25/00:00:00:6d:e2/00:0a:00:00:00/e0 tag 0 dma 1310720 in
res 40/00:01:00:00:00/00:00:00:00:00/00 Emask 0x4 (timeout)
Dec 11 14:04:28 cedric-nas kernel: ata4.00: status: { DRDY }
Dec 11 14:04:33 cedric-nas kernel: ata4: link is slow to respond, please be patient (ready=0)
Dec 11 14:04:38 cedric-nas kernel: ata4: device not ready (errno=-16), forcing hardreset
Dec 11 14:04:38 cedric-nas kernel: ata4: soft resetting link
Dec 11 14:04:43 cedric-nas kernel: ata4: link is slow to respond, please be patient (ready=0)
Dec 11 14:04:48 cedric-nas kernel: ata4: SRST failed (errno=-16)
Dec 11 14:04:48 cedric-nas kernel: ata4: soft resetting link
Dec 11 14:04:53 cedric-nas kernel: ata4: link is slow to respond, please be patient (ready=0)
Dec 11 14:04:58 cedric-nas kernel: ata4: SRST failed (errno=-16)
Dec 11 14:04:58 cedric-nas kernel: ata4: soft resetting link
Dec 11 14:05:03 cedric-nas kernel: ata4: link is slow to respond, please be patient (ready=0)
Dec 11 14:05:33 cedric-nas kernel: ata4: SRST failed (errno=-16)
Dec 11 14:05:33 cedric-nas kernel: ata4: soft resetting link
Dec 11 14:05:38 cedric-nas kernel: ata4: SRST failed (errno=-16)
Dec 11 14:05:38 cedric-nas kernel: ata4: reset failed, giving up
Dec 11 14:05:38 cedric-nas kernel: ata4.00: disabled
Dec 11 14:05:38 cedric-nas kernel: ata4.01: disabled
Dec 11 14:05:38 cedric-nas kernel: sd 3:0:0:0: [sdc] tag#0 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE cmd_age=101s
Dec 11 14:05:38 cedric-nas kernel: sd 3:0:0:0: [sdc] tag#0 Sense Key : Not Ready [current]
Dec 11 14:05:38 cedric-nas kernel: sd 3:0:0:0: [sdc] tag#0 Add. Sense: Logical unit not ready, hard reset required
Dec 11 14:05:38 cedric-nas kernel: sd 3:0:0:0: [sdc] tag#0 CDB: Read(10) 28 00 00 e2 6d 00 00 0a 00 00
Dec 11 14:05:38 cedric-nas kernel: blk_update_request: I/O error, dev sdc, sector 14839040 op 0x0:(READ) flags 0x84700 phys_seg 60 prio class 0
Dec 11 14:05:38 cedric-nas kernel: ata4: EH complete
Dec 11 14:05:38 cedric-nas kernel: sd 3:0:0:0: [sdc] tag#0 FAILED Result: hostbyte=DID_BAD_TARGET driverbyte=DRIVER_OK cmd_age=101s
Dec 11 14:05:38 cedric-nas kernel: sd 3:0:0:0: [sdc] tag#0 CDB: Read(10) 28 00 00 e2 6d 80 00 09 80 00
Dec 11 14:05:38 cedric-nas kernel: blk_update_request: I/O error, dev sdc, sector 14839168 op 0x0:(READ) flags 0x84000 phys_seg 57 prio class 0
Dec 11 14:05:38 cedric-nas kernel: sd 3:0:0:0: [sdc] tag#0 FAILED Result: hostbyte=DID_BAD_TARGET driverbyte=DRIVER_OK cmd_age=0s
Dec 11 14:05:38 cedric-nas kernel: sd 3:0:0:0: [sdc] tag#0 CDB: Read(10) 28 00 00 e2 77 00 00 0a 00 00
Dec 11 14:05:38 cedric-nas kernel: blk_update_request: I/O error, dev sdc, sector 14841600 op 0x0:(READ) flags 0x84700 phys_seg 60 prio class 0
Dec 11 14:05:38 cedric-nas kernel: sd 3:0:0:0: [sdc] tag#0 FAILED Result: hostbyte=DID_BAD_TARGET driverbyte=DRIVER_OK cmd_age=0s
Dec 11 14:05:38 cedric-nas kernel: sd 3:0:0:0: [sdc] tag#0 CDB: Read(10) 28 00 00 e2 77 80 00 09 80 00
Dec 11 14:05:38 cedric-nas kernel: blk_update_request: I/O error, dev sdc, sector 14841728 op 0x0:(READ) flags 0x84000 phys_seg 57 prio class 0
Dec 11 14:05:38 cedric-nas kernel: sd 3:0:0:0: [sdc] tag#0 FAILED Result: hostbyte=DID_BAD_TARGET driverbyte=DRIVER_OK cmd_age=0s
Dec 11 14:05:38 cedric-nas kernel: sd 3:0:0:0: [sdc] tag#0 CDB: Read(10) 28 00 00 e2 6d 00 00 00 08 00
Dec 11 14:05:38 cedric-nas kernel: blk_update_request: I/O error, dev sdc, sector 14839040 op 0x0:(READ) flags 0x100 phys_seg 1 prio class 0
Dec 11 14:05:38 cedric-nas kernel: btrfs_dev_stat_print_on_error: 30 callbacks suppressed
Dec 11 14:05:38 cedric-nas kernel: BTRFS error (device sdb6): bdev /dev/sdc1 errs: wr 0, rd 1, flush 0, corrupt 0, gen 0
Dec 11 14:05:38 cedric-nas kernel: sd 3:0:0:0: [sdc] tag#0 FAILED Result: hostbyte=DID_BAD_TARGET driverbyte=DRIVER_OK cmd_age=0s
Dec 11 14:05:38 cedric-nas kernel: sd 3:0:0:0: [sdc] tag#0 CDB: Read(10) 28 00 00 e2 6d 08 00 00 08 00
Dec 11 14:05:38 cedric-nas kernel: blk_update_request: I/O error, dev sdc, sector 14839048 op 0x0:(READ) flags 0x100 phys_seg 1 prio class 0
Dec 11 14:05:38 cedric-nas kernel: BTRFS error (device sdb6): bdev /dev/sdc1 errs: wr 0, rd 2, flush 0, corrupt 0, gen 0
Dec 11 14:05:38 cedric-nas kernel: sd 3:0:0:0: [sdc] tag#0 FAILED Result: hostbyte=DID_BAD_TARGET driverbyte=DRIVER_OK cmd_age=0s
Dec 11 14:05:38 cedric-nas kernel: sd 3:0:0:0: [sdc] tag#0 CDB: Read(10) 28 00 00 e2 6d 10 00 00 08 00
Dec 11 14:05:38 cedric-nas kernel: blk_update_request: I/O error, dev sdc, sector 14839056 op 0x0:(READ) flags 0x100 phys_seg 1 prio class 0
Dec 11 14:05:38 cedric-nas kernel: BTRFS error (device sdb6): bdev /dev/sdc1 errs: wr 0, rd 3, flush 0, corrupt 0, gen 0
Dec 11 14:05:38 cedric-nas kernel: sd 3:0:0:0: [sdc] tag#0 FAILED Result: hostbyte=DID_BAD_TARGET driverbyte=DRIVER_OK cmd_age=0s
Dec 11 14:05:38 cedric-nas kernel: sd 3:0:0:0: [sdc] tag#0 CDB: Read(10) 28 00 00 e2 6d 18 00 00 08 00
Dec 11 14:05:38 cedric-nas kernel: blk_update_request: I/O error, dev sdc, sector 14839064 op 0x0:(READ) flags 0x100 phys_seg 1 prio class 0
Dec 11 14:05:38 cedric-nas kernel: BTRFS error (device sdb6): bdev /dev/sdc1 errs: wr 0, rd 4, flush 0, corrupt 0, gen 0
Dec 11 14:05:38 cedric-nas kernel: sd 3:0:0:0: [sdc] tag#0 FAILED Result: hostbyte=DID_BAD_TARGET driverbyte=DRIVER_OK cmd_age=0s
Dec 11 14:05:38 cedric-nas kernel: sd 3:0:0:0: [sdc] tag#0 CDB: Read(10) 28 00 00 e2 6d 20 00 00 08 00
Dec 11 14:05:38 cedric-nas kernel: blk_update_request: I/O error, dev sdc, sector 14839072 op 0x0:(READ) flags 0x100 phys_seg 1 prio class 0
Dec 11 14:05:38 cedric-nas kernel: BTRFS error (device sdb6): bdev /dev/sdc1 errs: wr 0, rd 5, flush 0, corrupt 0, gen 0
Dec 11 14:05:38 cedric-nas kernel: sd 3:0:0:0: [sdc] tag#0 FAILED Result: hostbyte=DID_BAD_TARGET driverbyte=DRIVER_OK cmd_age=0s
Dec 11 14:05:38 cedric-nas kernel: sd 3:0:0:0: [sdc] tag#0 CDB: Read(10) 28 00 00 e2 6d 28 00 00 08 00
Dec 11 14:05:38 cedric-nas kernel: blk_update_request: I/O error, dev sdc, sector 14839080 op 0x0:(READ) flags 0x100 phys_seg 1 prio class 0
Dec 11 14:05:38 cedric-nas kernel: BTRFS error (device sdb6): bdev /dev/sdc1 errs: wr 0, rd 6, flush 0, corrupt 0, gen 0
Dec 11 14:05:38 cedric-nas kernel: BTRFS error (device sdb6): bdev /dev/sdc1 errs: wr 0, rd 7, flush 0, corrupt 0, gen 0
Dec 11 14:05:38 cedric-nas kernel: BTRFS error (device sdb6): bdev /dev/sdc1 errs: wr 0, rd 8, flush 0, corrupt 0, gen 0
Dec 11 14:05:38 cedric-nas kernel: BTRFS error (device sdb6): bdev /dev/sdc1 errs: wr 0, rd 9, flush 0, corrupt 0, gen 0
Dec 11 14:05:38 cedric-nas kernel: BTRFS error (device sdb6): bdev /dev/sdc1 errs: wr 0, rd 10, flush 0, corrupt 0, gen 0
Dec 11 14:05:43 cedric-nas kernel: scsi_io_completion_action: 73098 callbacks suppressed
Dec 11 14:05:43 cedric-nas kernel: sd 3:0:1:0: [sdd] tag#0 FAILED Result: hostbyte=DID_BAD_TARGET driverbyte=DRIVER_OK cmd_age=0s
Dec 11 14:05:43 cedric-nas kernel: sd 3:0:1:0: [sdd] tag#0 CDB: Write(10) 2a 00 01 02 fd 00 00 00 08 00
Dec 11 14:05:43 cedric-nas kernel: print_req_error: 73098 callbacks suppressed
Dec 11 14:05:43 cedric-nas kernel: blk_update_request: I/O error, dev sdd, sector 16973056 op 0x1:(WRITE) flags 0x800 phys_seg 1 prio class 0
Dec 11 14:05:43 cedric-nas kernel: btrfs_dev_stat_print_on_error: 80876 callbacks suppressed
Dec 11 14:05:43 cedric-nas kernel: BTRFS error (device sdb6): bdev /dev/sdd6 errs: wr 25601, rd 27648, flush 0, corrupt 0, gen 0
Dec 11 14:05:43 cedric-nas kernel: sd 3:0:1:0: [sdd] tag#0 FAILED Result: hostbyte=DID_BAD_TARGET driverbyte=DRIVER_OK cmd_age=0s
Dec 11 14:05:43 cedric-nas kernel: sd 3:0:1:0: [sdd] tag#0 CDB: Write(10) 2a 00 01 02 fd 08 00 00 08 00
Dec 11 14:05:43 cedric-nas kernel: blk_update_request: I/O error, dev sdd, sector 16973064 op 0x1:(WRITE) flags 0x800 phys_seg 1 prio class 0
Dec 11 14:05:43 cedric-nas kernel: BTRFS error (device sdb6): bdev /dev/sdd6 errs: wr 25602, rd 27648, flush 0, corrupt 0, gen 0
Dec 11 14:05:43 cedric-nas kernel: sd 3:0:1:0: [sdd] tag#0 FAILED Result: hostbyte=DID_BAD_TARGET driverbyte=DRIVER_OK cmd_age=0s
Dec 11 14:05:43 cedric-nas kernel: sd 3:0:1:0: [sdd] tag#0 CDB: Write(10) 2a 00 01 02 fd 10 00 00 08 00
Dec 11 14:05:43 cedric-nas kernel: blk_update_request: I/O error, dev sdd, sector 16973072 op 0x1:(WRITE) flags 0x800 phys_seg 1 prio class 0
Dec 11 14:05:43 cedric-nas kernel: BTRFS error (device sdb6): bdev /dev/sdd6 errs: wr 25603, rd 27648, flush 0, corrupt 0, gen 0
Dec 11 14:05:43 cedric-nas kernel: sd 3:0:1:0: [sdd] tag#0 FAILED Result: hostbyte=DID_BAD_TARGET driverbyte=DRIVER_OK cmd_age=0s
Dec 11 14:05:43 cedric-nas kernel: sd 3:0:1:0: [sdd] tag#0 CDB: Write(10) 2a 00 01 02 fd 18 00 00 08 00
Dec 11 14:05:43 cedric-nas kernel: blk_update_request: I/O error, dev sdd, sector 16973080 op 0x1:(WRITE) flags 0x800 phys_seg 1 prio class 0interestingly, disconnecting the dive took out both /dev/sdc and /dev/sdd. Replugging the drive did not automatically solve that problem:
# fdisk /dev/sda
Welcome to fdisk (util-linux 2.36).
Changes will remain in memory only, until you decide to write them.
Be careful before using the write command.
Command (m for help): q
[root@cedric-nas ~]# fdisk /dev/sdb
Welcome to fdisk (util-linux 2.36).
Changes will remain in memory only, until you decide to write them.
Be careful before using the write command.
Command (m for help): q
[root@cedric-nas ~]# fdisk /dev/sdc
Welcome to fdisk (util-linux 2.36).
Changes will remain in memory only, until you decide to write them.
Be careful before using the write command.
fdisk: cannot open /dev/sdc: Input/output error
[root@cedric-nas ~]# fdisk /dev/sdd
Welcome to fdisk (util-linux 2.36).
Changes will remain in memory only, until you decide to write them.
Be careful before using the write command.
fdisk: cannot open /dev/sdd: Input/output errorAfter rebooting the computer, all the drives show up again.
Offline
Now umount the array, fake a failed drive, and fix it again:
# fdisk /dev/sdd
Welcome to fdisk (util-linux 2.36).
Changes will remain in memory only, until you decide to write them.
Be careful before using the write command.
Command (m for help): p
Disk /dev/sdd: 931.51 GiB, 1000204886016 bytes, 1953525168 sectors
Disk model: Hitachi HDT72101
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disklabel type: dos
Disk identifier: 0x00000000
Device Boot Start End Sectors Size Id Type
/dev/sdd1 63 594404 594342 290.2M 83 Linux
/dev/sdd2 594405 1590434 996030 486.3M 83 Linux
/dev/sdd4 1590435 1953520064 1951929630 930.8G 5 Extended
/dev/sdd5 1590498 1863539 273042 133.3M 82 Linux swap / Solaris
Command (m for help): n
Partition type
p primary (2 primary, 1 extended, 1 free)
l logical (numbered from 5)
Select (default p): l
Adding logical partition 6
First sector (1865588-1953520064, default 1865728):
Last sector, +/-sectors or +/-size{K,M,G,T,P} (1865728-1953520064, default 1953520064):
Created a new partition 6 of type 'Linux' and of size 930.6 GiB.
Partition #6 contains a btrfs signature.
Do you want to remove the signature? [Y]es/[N]o: y
The signature will be removed by a write command.
Command (m for help): wq
The partition table has been altered.
Syncing disks.
[root@cedric-nas cedric]# umount /storage-btrfs/
[root@cedric-nas cedric]# mount /dev/sdb6 /storage-btrfs/
mount: /storage-btrfs: wrong fs type, bad option, bad superblock on /dev/sdb6, missing codepage or helper program, or other error.
[root@cedric-nas cedric]# mount /dev/sdb6 /storage-btrfs/ -o degraded
[root@cedric-nas cedric]# btrfs filesystem show
Label: none uuid: a3740759-5f92-499a-8788-4bba4aa67dfe
Total devices 3 FS bytes used 15.34GiB
devid 1 size 930.62GiB used 17.03GiB path /dev/sdb6
devid 2 size 223.57GiB used 0.00B path /dev/sdc1
*** Some devices missing
[root@cedric-nas cedric]# btrfs replace start 3 /dev/sdd6 /storage-btrfs/
[root@cedric-nas cedric]# btrfs filesystem show
Label: none uuid: a3740759-5f92-499a-8788-4bba4aa67dfe
Total devices 4 FS bytes used 15.34GiB
devid 0 size 930.62GiB used 17.03GiB path /dev/sdd6
devid 1 size 930.62GiB used 17.03GiB path /dev/sdb6
devid 2 size 223.57GiB used 0.00B path /dev/sdc1
*** Some devices missing
[root@cedric-nas cedric]# btrfs replace status
btrfs replace status: exactly 1 argument expected, 0 given
[root@cedric-nas cedric]# btrfs replace status /storage-btrfs
Started on 11.Dec 16:28:12, finished on 11.Dec 16:30:42, 0 write errs, 0 uncorr. read errs
[root@cedric-nas cedric]# btrfs filesystem show
Label: none uuid: a3740759-5f92-499a-8788-4bba4aa67dfe
Total devices 3 FS bytes used 15.34GiB
devid 1 size 930.62GiB used 17.03GiB path /dev/sdb6
devid 2 size 223.57GiB used 32.00MiB path /dev/sdc1
devid 3 size 930.62GiB used 17.00GiB path /dev/sdd6Offline