You are not logged in.

#1 2015-10-14 18:17:03

akawoz
Member
Registered: 2015-02-22
Posts: 6

BTRFS device delete taking too long

Hi guys, I started getting errors on a HDD that was obviously failing so I've issued a btrfs device delete /dev/sdc /mnt/media

After 12 hours it has only drained about 120GB of the 2.50TB - I'm assuming because of retry delays caused by the below critical medium errors.

This pool is RAID1 for both data and metadata. Since there appears to be no way to cancel a device delete I'm considering forcibly removing the failing drive and letting a btrfs balance sort it out.

BTW: I'm running a kernel RC because I'm using a couple 8TB SMR drives and ran into this bug: https://bugzilla.kernel.org/show_bug.cgi?id=93581

Thoughts, any other ideas?

Linux cloud.warrenhughes.net 4.3.0-rc5-mainline #1 SMP PREEMPT Tue Oct 13 15:03:09 NZDT 2015 x86_64 GNU/Linux

btrfs-progs v4.2.1

Label: none  uuid: 643c3145-8371-4011-8c34-20240e1bbaff
        Total devices 12 FS bytes used 13.77TiB
        devid    8 size 2.73TiB used 2.55TiB path /dev/sdh
        devid    9 size 2.73TiB used 2.42TiB path /dev/sdc
        devid   10 size 2.73TiB used 2.55TiB path /dev/sdf
        devid   11 size 1.82TiB used 1.64TiB path /dev/sdn
        devid   12 size 2.73TiB used 2.55TiB path /dev/sdg
        devid   14 size 2.73TiB used 2.55TiB path /dev/sda
        devid   15 size 2.73TiB used 2.55TiB path /dev/sdd
        devid   16 size 2.73TiB used 2.54TiB path /dev/sdk
        devid   17 size 2.73TiB used 2.55TiB path /dev/sdl
        devid   18 size 3.64TiB used 3.46TiB path /dev/sdm
        devid   19 size 7.28TiB used 2.12TiB path /dev/sdb
        devid   20 size 7.28TiB used 133.00GiB path /dev/sde


[61802.090145] blk_update_request: critical medium error, dev sdc, sector 5200523024
[61805.069882] sd 0:0:2:0: [sdc] tag#8 UNKNOWN(0x2003) Result: hostbyte=0x00 driverbyte=0x08
[61805.069887] sd 0:0:2:0: [sdc] tag#8 Sense Key : 0x3 [current] [descriptor]
[61805.069889] sd 0:0:2:0: [sdc] tag#8 ASC=0x11 ASCQ=0x0
[61805.069892] sd 0:0:2:0: [sdc] tag#8 CDB: opcode=0x88 88 00 00 00 00 01 35 f9 b5 60 00 00 00 08 00 00
[61805.069894] blk_update_request: critical medium error, dev sdc, sector 5200524640
[61807.949538] sd 0:0:2:0: [sdc] tag#4 UNKNOWN(0x2003) Result: hostbyte=0x00 driverbyte=0x08
[61807.949543] sd 0:0:2:0: [sdc] tag#4 Sense Key : 0x3 [current] [descriptor]
[61807.949545] sd 0:0:2:0: [sdc] tag#4 ASC=0x11 ASCQ=0x0
[61807.949547] sd 0:0:2:0: [sdc] tag#4 CDB: opcode=0x88 88 00 00 00 00 01 35 f9 b4 80 00 00 00 08 00 00
[61807.949549] blk_update_request: critical medium error, dev sdc, sector 5200524416
[61810.851475] sd 0:0:2:0: [sdc] tag#0 UNKNOWN(0x2003) Result: hostbyte=0x00 driverbyte=0x08
[61810.851482] sd 0:0:2:0: [sdc] tag#0 Sense Key : 0x3 [current] [descriptor]
[61810.851485] sd 0:0:2:0: [sdc] tag#0 ASC=0x11 ASCQ=0x0
[61810.851488] sd 0:0:2:0: [sdc] tag#0 CDB: opcode=0x88 88 00 00 00 00 01 35 f9 b2 c0 00 00 00 08 00 00
[61810.851491] blk_update_request: critical medium error, dev sdc, sector 5200523968
[61813.731107] sd 0:0:2:0: [sdc] tag#6 UNKNOWN(0x2003) Result: hostbyte=0x00 driverbyte=0x08
[61813.731113] sd 0:0:2:0: [sdc] tag#6 Sense Key : 0x3 [current] [descriptor]
[61813.731115] sd 0:0:2:0: [sdc] tag#6 ASC=0x11 ASCQ=0x0
[61813.731118] sd 0:0:2:0: [sdc] tag#6 CDB: opcode=0x88 88 00 00 00 00 01 35 f9 a6 08 00 00 00 f8 00 00
[61813.731121] blk_update_request: critical medium error, dev sdc, sector 5200520712

[wsh@cloud ~]$ sudo btrfs de st /mnt/media
[/dev/sdh].write_io_errs   0
[/dev/sdh].read_io_errs    0
[/dev/sdh].flush_io_errs   0
[/dev/sdh].corruption_errs 0
[/dev/sdh].generation_errs 0
[/dev/sdc].write_io_errs   0
[/dev/sdc].read_io_errs    198
[/dev/sdc].flush_io_errs   0
[/dev/sdc].corruption_errs 0
[/dev/sdc].generation_errs 0
[/dev/sdf].write_io_errs   0
[/dev/sdf].read_io_errs    0
[/dev/sdf].flush_io_errs   0
[/dev/sdf].corruption_errs 0
[/dev/sdf].generation_errs 0
[/dev/sdn].write_io_errs   0
[/dev/sdn].read_io_errs    0
[/dev/sdn].flush_io_errs   0
[/dev/sdn].corruption_errs 0
[/dev/sdn].generation_errs 0
[/dev/sdg].write_io_errs   0
[/dev/sdg].read_io_errs    0
[/dev/sdg].flush_io_errs   0
[/dev/sdg].corruption_errs 0
[/dev/sdg].generation_errs 0
[/dev/sda].write_io_errs   0
[/dev/sda].read_io_errs    0
[/dev/sda].flush_io_errs   0
[/dev/sda].corruption_errs 0
[/dev/sda].generation_errs 0
[/dev/sdd].write_io_errs   0
[/dev/sdd].read_io_errs    0
[/dev/sdd].flush_io_errs   0
[/dev/sdd].corruption_errs 0
[/dev/sdd].generation_errs 0
[/dev/sdk].write_io_errs   0
[/dev/sdk].read_io_errs    0
[/dev/sdk].flush_io_errs   0
[/dev/sdk].corruption_errs 0
[/dev/sdk].generation_errs 0
[/dev/sdl].write_io_errs   0
[/dev/sdl].read_io_errs    0
[/dev/sdl].flush_io_errs   0
[/dev/sdl].corruption_errs 0
[/dev/sdl].generation_errs 0
[/dev/sdm].write_io_errs   0
[/dev/sdm].read_io_errs    0
[/dev/sdm].flush_io_errs   0
[/dev/sdm].corruption_errs 0
[/dev/sdm].generation_errs 0
[/dev/sdb].write_io_errs   0
[/dev/sdb].read_io_errs    0
[/dev/sdb].flush_io_errs   0
[/dev/sdb].corruption_errs 0
[/dev/sdb].generation_errs 0
[/dev/sde].write_io_errs   0
[/dev/sde].read_io_errs    0
[/dev/sde].flush_io_errs   0
[/dev/sde].corruption_errs 0
[/dev/sde].generation_errs 0

[wsh@cloud ~]$ sudo btrfs de us /mnt/media
/dev/sda, ID: 14
   Device size:             2.73TiB
   Data,RAID1:              2.55TiB
   Metadata,RAID1:          2.00GiB
   Unallocated:           181.52GiB

/dev/sdb, ID: 19
   Device size:             7.28TiB
   Data,RAID1:              2.12TiB
   Metadata,RAID1:          2.00GiB
   System,RAID1:           32.00MiB
   Unallocated:             5.16TiB

/dev/sdc, ID: 9
   Device size:             2.73TiB
   Data,RAID1:              2.42TiB
   Metadata,RAID1:          2.00GiB
   Unallocated:           314.52GiB

/dev/sdd, ID: 15
   Device size:             2.73TiB
   Data,RAID1:              2.55TiB
   Metadata,RAID1:          4.00GiB
   Unallocated:           181.52GiB

/dev/sde, ID: 20
   Device size:             7.28TiB
   Data,RAID1:            133.00GiB
   Unallocated:             7.15TiB

/dev/sdf, ID: 10
   Device size:             2.73TiB
   Data,RAID1:              2.55TiB
   Metadata,RAID1:          2.00GiB
   Unallocated:           181.52GiB

/dev/sdg, ID: 12
   Device size:             2.73TiB
   Data,RAID1:              2.55TiB
   Metadata,RAID1:          2.00GiB
   Unallocated:           181.52GiB

/dev/sdh, ID: 8
   Device size:             2.73TiB
   Data,RAID1:              2.55TiB
   Metadata,RAID1:          2.00GiB
   Unallocated:           181.52GiB

/dev/sdk, ID: 16
   Device size:             2.73TiB
   Data,RAID1:              2.53TiB
   Metadata,RAID1:          7.00GiB
   Unallocated:           194.52GiB

/dev/sdl, ID: 17
   Device size:             2.73TiB
   Data,RAID1:              2.55TiB
   Metadata,RAID1:          5.00GiB
   System,RAID1:           32.00MiB
   Unallocated:           181.49GiB

/dev/sdm, ID: 18
   Device size:             3.64TiB
   Data,RAID1:              3.46TiB
   Metadata,RAID1:          5.00GiB
   Unallocated:           182.02GiB

/dev/sdn, ID: 11
   Device size:             1.82TiB
   Data,RAID1:              1.64TiB
   Metadata,RAID1:          1.00GiB
   Unallocated:           182.02GiB

Offline

#2 2015-10-14 19:29:20

zamzing
Member
From: Washington, DC
Registered: 2015-10-08
Posts: 24

Re: BTRFS device delete taking too long

I'd advice you to just wait and let it complete. btrfs device delete will force all data from the deleted drive to be rebalanced and the command completes when the drive has been freed up.  That can take many hours and even days!

See this blog.

Offline

#3 2015-10-14 19:51:44

akawoz
Member
Registered: 2015-02-22
Posts: 6

Re: BTRFS device delete taking too long

Thanks for the link - yes I'll let it run. Cheers

Offline

Board footer

Powered by FluxBB