[SOLVED] BTRFS balance returns enospc error with 80% free space

zrianc · 2015-08-11 08:34:49

I have a recently installed a x86_64 system on a 50GB btrfs partition and every time I try btrfs balance start it gives me an enospc error even though I have less than 20% of the available space full.

I have tried the recommended method and with -dusage I can go up to -dusage=100 with no problems but with -musage it works until 34 and then at musage=35 it fails with the enospc error.

I tested if that free space is "real" by mounting the system without compression and filling the free space with a file written by dd from /dev/zero and the maximum size is exactly the size of the free space that is reported.
I have tried deleting all my snapshots and deleting the pkg cache until I was only using 6 GB out of 50 GB and the exact same errors.
I have tried adding more files until I reached 11 GB to see if get write errors when I add lots of small files and no problems and also still the same error.

Here is more detailed information about my setup and output of several commands:

lsblk -o NAME,SIZE,FSTYPE,UUID,PARTLABEL
NAME SIZE FSTYPE UUID PARTLABEL
sda 50G
├─sda1 2M BIOS boot partition
└─sda2 50G btrfs 6816726f-71ed-4b64-9071-60684a445e71 Linux x86-64 root (/)
sr0 1024M

cat /etc/fstab
# /dev/sda2 LABEL=ArchLinux
UUID=6816726f-71ed-4b64-9071-60684a445e71 / btrfs    rw,noatime,compress-force=lzo,space_cache,autodefrag 0 0
UUID=6816726f-71ed-4b64-9071-60684a445e71 /var/abs    btrfs    rw,noatime,compress-force=lzo,space_cache,autodefrag,subvol=subvolume_abs 0 0
UUID=6816726f-71ed-4b64-9071-60684a445e71 /var/cache/pacman/pkg btrfs    rw,noatime,compress-force=lzo,space_cache,autodefrag,subvol=subvolume_pkg 0 0
UUID=6816726f-71ed-4b64-9071-60684a445e71 /.snapshots btrfs    rw,noatime,compress-force=lzo,space_cache,autodefrag,noauto,subvol=subvolume_snapshots 0 0
UUID=6816726f-71ed-4b64-9071-60684a445e71 /mnt/btrfs_zero btrfs    defaults,noatime,noauto,subvolid=0 0 0

(I know I should not use compress-force=lzo on /var/cache/pacman/pkg but the compression is forced by the fact that the parent directory has the compression flag so since I can't find any solution to disable it I enabled it explicitly so at least it's clear how things are)

btrfs subvolume list -a -t --sort=-gen /
ID gen top level path
-- --- --------- ----
257 71683 5 <FS_TREE>/subvolume_root
258 71679 5 <FS_TREE>/subvolume_snapshots
326 71679 5 <FS_TREE>/subvolume_abs
327 71679 5 <FS_TREE>/subvolume_pkg
678 71677 258 <FS_TREE>/subvolume_snapshots/snapshot_root_1
677 71653 258 <FS_TREE>/subvolume_snapshots/snapshot_login
660 71600 258 <FS_TREE>/subvolume_snapshots/snapshot_root_3
661 71600 258 <FS_TREE>/subvolume_snapshots/snapshot_root_2
663 71600 258 <FS_TREE>/subvolume_snapshots/snapshot_daily
262 39 257 subvolume_root/var/lib/machines

(like I said I have also tried with all the snapshots deleted)

btrfs filesystem usage /
Overall:
Device size:    50.00GiB
Device allocated:    12.31GiB
Device unallocated:    37.68GiB
Device missing:    0.00B
Used:    10.21GiB
Free (estimated):    38.17GiB (min: 19.33GiB)
Data ratio:    1.00
Metadata ratio:    2.00
Global reserve:    128.00MiB (used: 0.00B)
Data,single: Size:10.00GiB, Used:9.52GiB
/dev/sda2    10.00GiB
Metadata,DUP: Size:1.12GiB, Used:354.16MiB
/dev/sda2    2.25GiB
System,DUP: Size:32.00MiB, Used:16.00KiB
/dev/sda2    64.00MiB
Unallocated:
/dev/sda2    37.68GiB
sudo btrfs filesystem df /
Data, single: total=10.00GiB, used=9.52GiB
System, DUP: total=32.00MiB, used=16.00KiB
Metadata, DUP: total=1.12GiB, used=354.31MiB
GlobalReserve, single: total=128.00MiB, used=0.00B

-This is the maximum size I have, I have also tested with only 6 GiB used
-I used bold for the metadata field because I suspected that there is where the problem is so I added in fstab metadata_ratio=20 to all subvolumes rebooted and nothing changed.

sudo btrfs scrub start -B /
scrub done for 6816726f-71ed-4b64-9071-60684a445e71
scrub started at Tue Aug 11 11:07:36 2015 and finished after 00:01:42
total bytes scrubbed: 10.21GiB with 0 errors

btrfs check output when run from the install cd with the partition unmounted:
Checking filesystem on /dev/sda2
UUID: 6816726f-71ed-4b64-9071-60684a445e71
found 10541920267 bytes used err is 0
total csum bytes: 9906264
total tree bytes: 370245632
total fs tree bytes: 337903616
total extent tree bytes: 20758528
btree space waste bytes: 63326339
file data blocks allocated: 10473455616
referenced 14596616192
btrfs-progs v4.1.2

btrfs balance start output with the following options:
-dusage 100:
Dumping filters: flags 0x1, state 0x0, force is off
DATA (flags 0x2): balancing, usage=100
Done, had to relocate 2 out of 13 chunks
-dusage 100, second, third, ... run:
Dumping filters: flags 0x1, state 0x0, force is off
DATA (flags 0x2): balancing, usage=100
Done, had to relocate 1 out of 13 chunks
-musage 33, first run:
Dumping filters: flags 0x6, state 0x0, force is off
METADATA (flags 0x2): balancing, usage=33
SYSTEM (flags 0x2): balancing, usage=33
Done, had to relocate 2 out of 13 chunks
-musage 33, second, third,.... run:
Dumping filters: flags 0x6, state 0x0, force is off
METADATA (flags 0x2): balancing, usage=33
SYSTEM (flags 0x2): balancing, usage=33
Done, had to relocate 1 out of 12 chunks
-musage 35 always gives an error:
Dumping filters: flags 0x6, state 0x0, force is off
METADATA (flags 0x2): balancing, usage=35
SYSTEM (flags 0x2): balancing, usage=35
ERROR: error during balancing '/' - No space left on device
There may be more info in syslog - try dmesg | tail
output of dmesg | tail (after repeated trying):
[ 2481.262199] BTRFS info (device sda2): found 1 extents
[ 2487.331921] BTRFS info (device sda2): relocating block group 683432476672 flags 34
[ 2498.583018] BTRFS info (device sda2): relocating block group 683466031104 flags 34
[ 2503.843304] BTRFS info (device sda2): relocating block group 683499585536 flags 34
[ 2511.407124] BTRFS info (device sda2): relocating block group 683533139968 flags 34
[ 2511.444678] BTRFS info (device sda2): 1 enospc errors during balance
[ 2528.820892] BTRFS info (device sda2): relocating block group 683600248832 flags 36
[ 2528.888670] BTRFS info (device sda2): relocating block group 683566694400 flags 34
[ 2528.915590] BTRFS info (device sda2): found 1 extents
[ 2528.936589] BTRFS info (device sda2): 1 enospc errors during balance

uname -a
Linux ArchLinux 4.1.4-1-ARCH #1 SMP PREEMPT Mon Aug 3 21:30:37 UTC 2015 x86_64 GNU/Linux
btrfs version
btrfs-progs v4.1.2

I have tried running the command both from inside the system and mounted from the install cd with different combinations of mount options like enabling and disabling space-cache / nospace_cache , clear_cache, enospc_debug, enable and disable compression or autodefrag.
I have tried defragmenting everything, filling all the space, adding files, deleting files, making snapshots, deleting snapshots still the same problem.
I have run the balance command on both the root subvolume and on subvolid=0.
I have tried putting the balance commands with options that work inside a for to run 1000 times hoping that maybe that one relocated chunk it says about might actually solve something in time but it doesn't.

Everything else works fine, the system is very fast, good compression, no other errors and I have no other problems but the fact that I have this error means something is wrong and I don't know what is the problem and how to solve it.

Last edited by zrianc (2015-08-12 18:39:25)

nstgc · 2015-08-11 11:35:55

Not that I'm confident that it would help, but by "defrag'ed everything" are you sure that includes directories? The command "btrfs fi defrag -r" doesn't defragment folders.

"find / -xdev -type d -exec btrfs file defrag '{}' \;" for example would find every directory in / and defragment it without crossing into other volumes.

As for your issue, I had this as well with my RAID 5 volume. Defragmenting everything (files and directories) didn't help, nor anything else I could think of. I eventually cleared out a ton of space and tried again.

Not really helpful, I know, but it's something.

zrianc · 2015-08-11 11:40:33

nstgc wrote:

Not that I'm confident that it would help, but by "defrag'ed everything" are you sure that includes directories? The command "btrfs fi defrag -r" doesn't defragment folders.
"find / -xdev -type d -exec btrfs file defrag '{}' \;" for example would find every directory in / and defragment it without crossing into other volumes.

Yes, I defragmented both files and folders by using both commands.

nstgc wrote:

As for your issue, I had this as well with my RAID 5 volume. Defragmenting everything (files and directories) didn't help, nor anything else I could think of. I eventually cleared out a ton of space and tried again.
Not really helpful, I know, but it's something.

I cleared space until I had 6GB used and 44 GB free and no snapshots...

nstgc · 2015-08-12 11:17:12

zrianc wrote:

I cleared space until I had 6GB used and 44 GB free and no snapshots...

Wow, okay, yeah. My thoughts than go two ways. If you want to fix it, jump on the btrfs IRC channel and ask for help there, or wait for someone more competent than myself to help out here. However, with only 6GB in use, I'd rsync that data to someplace else, reformat, and than rsync it back.

Slithery · 2015-08-12 18:37:15

I know that there have been issues with certain recent kernel versions and the btrfs balance command as I ran across problems myself (I can't remember if the error was the the same or not though).

I worked round it by booting to a different kernel version to run the balance (linux instead of my usual linux-ck if I remember correctly), but this was a while ago now.
It may be something worth investigating further though.

zrianc · 2015-08-12 18:37:17

Today I tried to force all installed packages to reinstall but before that I deleted once again all the caches (pkg+abs+pkgfile) and existing snapshots and BEFORE trying to reinstall I run again btrfs balance and this time it worked even though I have more space occupied now than I had when the error first appeared!

So theoretically my problem is SOLVED but I still think there is either something wrong with balance since it should have been able to do it before or there is something wrong with scrub or check because they should have reported an error if there was one.

I have also reported this issue on the btrfs mailing lists.

nstgc · 2015-08-14 18:31:52

Could you provide the link to message? I'd be interested in following it.

zrianc · 2015-08-15 03:43:58

nstgc wrote:

Could you provide the link to message? I'd be interested in following it.

Ok - http://www.spinics.net/lists/linux-btrfs/msg46009.html
But no solution or explanation was received, probably the mailing list is for more serious problems.

nstgc · 2015-08-15 11:49:33

If there is something wrong is balance, that sounds serious enough. Do write off your issue as a minor one. For us desktop users (I assume you are a desktop user), this might not be all that bad, but for a large server, this could be quite troublesome.

Arch Linux

#1 2015-08-11 08:34:49

[SOLVED] BTRFS balance returns enospc error with 80% free space

#2 2015-08-11 11:35:55

Re: [SOLVED] BTRFS balance returns enospc error with 80% free space

#3 2015-08-11 11:40:33

Re: [SOLVED] BTRFS balance returns enospc error with 80% free space

#4 2015-08-12 11:17:12

Re: [SOLVED] BTRFS balance returns enospc error with 80% free space

#5 2015-08-12 18:37:15

Re: [SOLVED] BTRFS balance returns enospc error with 80% free space

#6 2015-08-12 18:37:17

Re: [SOLVED] BTRFS balance returns enospc error with 80% free space

#7 2015-08-14 18:31:52

Re: [SOLVED] BTRFS balance returns enospc error with 80% free space

#8 2015-08-15 03:43:58

Re: [SOLVED] BTRFS balance returns enospc error with 80% free space

#9 2015-08-15 11:49:33

Re: [SOLVED] BTRFS balance returns enospc error with 80% free space

Board footer