Arch Linux

masterkoppa

Member

Registered: 2009-04-14

Posts: 92

[Solved] BRTFS No space left errors, force into read-only mode

It looks like I've run into an (apparently) infamous issue with BTRFS configurations, where the metadata is full but there's unallocated space on drives.

Attempting to delete any files, regardless of size, results in a forced read-only filesystem. Attempts to scrub, balance (with various filters, for data, metadata, and specific devids) or add new devices also results in a read-only filesystem.

This is the current state, as reported by btrfs filesystem usage

Overall:
    Device size:                  31.84TiB
    Device allocated:             29.89TiB
    Device unallocated:            1.95TiB
    Device missing:                  0.00B
    Used:                         29.72TiB
    Free (estimated):              1.67TiB      (min: 633.74GiB)
    Free (statfs, df):             1.11TiB
    Data ratio:                       1.27
    Metadata ratio:                   4.00
    Global reserve:              512.00MiB      (used: 0.00B)
    Multiple profiles:                  no

Data,RAID5: Size:23.50TiB, Used:23.37TiB (99.44%)
   /dev/sdb1       3.64TiB
   /dev/sda1       3.64TiB
   /dev/sdf1       5.43TiB
   /dev/sde1       4.52TiB
   /dev/sdc        6.27TiB
   /dev/sdg        6.27TiB

Metadata,RAID1C4: Size:30.00GiB, Used:29.96GiB (99.87%)
   /dev/sdf1      30.00GiB
   /dev/sde1      30.00GiB
   /dev/sdc       30.00GiB
   /dev/sdg       30.00GiB

System,RAID1C4: Size:32.00MiB, Used:1.47MiB (4.59%)
   /dev/sdf1      32.00MiB
   /dev/sde1      32.00MiB
   /dev/sdc       32.00MiB
   /dev/sdg       32.00MiB

Unallocated:
   /dev/sdb1       1.00MiB
   /dev/sda1       1.00MiB
   /dev/sdf1       1.00MiB
   /dev/sde1       1.00MiB
   /dev/sdc      998.01GiB
   /dev/sdg      998.01GiB

This is the output from btrfs filesystem usage for good measure:

/dev/sdb1, ID: 1
   Device size:             3.64TiB
   Device slack:            3.50KiB
   Data,RAID5/6:            3.64TiB
   Unallocated:             1.00MiB

/dev/sda1, ID: 2
   Device size:             3.64TiB
   Device slack:            3.50KiB
   Data,RAID5/6:            3.64TiB
   Unallocated:             1.00MiB

/dev/sdf1, ID: 3
   Device size:             5.46TiB
   Device slack:            3.50KiB
   Data,RAID5/6:            3.64TiB
   Data,RAID5/4:          901.47GiB
   Data,RAID5/3:          931.50GiB
   Metadata,RAID1C4:       30.00GiB
   System,RAID1C4:         32.00MiB
   Unallocated:             1.00MiB

/dev/sde1, ID: 4
   Device size:             4.55TiB
   Device slack:            3.50KiB
   Data,RAID5/6:            3.64TiB
   Data,RAID5/4:          901.47GiB
   Metadata,RAID1C4:       30.00GiB
   System,RAID1C4:         32.00MiB
   Unallocated:             1.00MiB

/dev/sdc, ID: 5
   Device size:             7.28TiB
   Device slack:              0.00B
   Data,RAID5/6:            3.64TiB
   Data,RAID5/4:          901.47GiB
   Data,RAID5/3:          931.50GiB
   Data,RAID5/2:          866.00GiB
   Metadata,RAID1C4:       30.00GiB
   System,RAID1C4:         32.00MiB
   Unallocated:           997.01GiB

/dev/sdg, ID: 6
   Device size:             7.28TiB
   Device slack:              0.00B
   Data,RAID5/6:            3.64TiB
   Data,RAID5/4:          901.47GiB
   Data,RAID5/3:          931.50GiB
   Data,RAID5/2:          866.00GiB
   Metadata,RAID1C4:       30.00GiB
   System,RAID1C4:         32.00MiB
   Unallocated:           997.01GiB

I'm aware of the various pitfalls and issues with BTRFS and RAID5. The setup is obviously non-ideal but it works for the particular use case (a media server). All data is backed up and can be restored if needed, would just like to avoid that since it's a non-trivial task to retrieve and restore from backups

As mentioned above, I've attempted to follow all the guides on the official BTRFS wiki for dealing with this issue. The filesystem mounts in rw mode, but  for example, when running balance with slowly increasing % filters on data or metadata, once it finds blocks to reallocate it will fail and runs into the no space left. Attempting to add a new 3TB device to the pool, also fails. I've also found this (apparently) popular resource, to no avail.

Currently my best idea is to attempt to convert the metadata to a RAID1 setup, so it could be re-balanced on the 2 drives with remaining un-allocated space, however I do not know if such an operation would succeed or leave me with a force recovery scenario.

My question to folks here is, is there other things that I should try before that? Would the RAID1 conversion for the metadata (btrfs balance -mconvert=raid1) succeed, or just lead to more headaches / broken filesystem? Any thoughts or ideas beyond just starting from scratch avoiding this setup / scenario?

Last edited by masterkoppa (2022-01-22 02:47:25)

masterkoppa

Member

Registered: 2009-04-14

Posts: 92

Re: [Solved] BRTFS No space left errors, force into read-only mode

After triple checking that backups were in place, up to date and valid I took the plunge and converted the metadata to raid1 and sure enough, it succeeded.

It used the unallocated space on the last 2 devid's to create a new raid1 for both system and metadata, and the filesystem is now operational. Hope this helps anyone running into anything like this in the future.