“no space left on device” with 54% du

flying sheep · 2014-09-26 15:29:41

Hi, I use btrfs and am aware of the metadata thing (i.e. metadata not being accounted when using df), but this is another level:

i started to have a really unstable system (core dumping all around), and journald, which died itself frequently told me

Failed to create coredump file /var/lib/systemd/coredump/[long-ass filename]: No space left on device

a df -h / during that time told me:

Filesystem      Size  Used Avail Use% Mounted on
/dev/sdc5       126G   67G   59G  54% /

and one after running pacman -Scc, my system became stable (currently typing from firefox). df -h / gave:

Filesystem      Size  Used Avail Use% Mounted on
/dev/sdc5       126G   62G   63G  50% /

so this can’t be the metadata’s fault, as btrfs filesystem df tells me there’s 3.45G reserved for metadata and 122G for files/data.

so why does my system become unstable once it reaches about 55% (67/122) fill level?

or am i misinterpreting some things?

PS: we really need tags for inline code, my misused ins tags make my skin crawl. maybe [c][/c] which translates to <code></code>?

Last edited by flying sheep (2014-09-26 15:32:07)

Trilby · 2014-09-26 16:54:13

I know next to nothing about btrfs, but is this a single partition, or do you have a separate var partition?

As for the inline code tag suggestion, that'd have to go upstream to the fluxBB devs.

anatolik · 2014-09-26 17:02:17

I see the same thing with btrfs. Its free space calculation algorithm is weird and I still did not figure out what it means saying "free space". But you can recover some of it by defragmenting and rebalancing the filesystem.

alphaniner · 2014-09-26 17:18:02

flying sheep wrote:

PS: we really need tags for inline code, my misused ins tags make my skin crawl. maybe [c][/c] which translates to <code></code>?

If you were using the Cobalt theme, they'd make your eyes bleed. IMO bold is suitable for inline code in most cases.

flying sheep · 2014-09-26 18:14:01

anatolik wrote:

I see the same thing with btrfs. Its free space calculation algorithm is weird and I still did not figure out what it means saying "free space". But you can recover some of it by defragmenting and rebalancing the filesystem.

aah, here we go. you’re right, there’s another way in which btrfs isn’t filled as much as it tells us, i.e. unbalancedness.

$ sudo btrfs fi show /
Label: 'Arch'  uuid: ad537972-5f03-4f3b-96d4-cf45f8dcf581
        Total devices 1 FS bytes used 61.93GiB
        devid    1 size 125.46GiB used 125.46GiB path /dev/sdc5

it was news to me that an unbalanced btrfs can be twice as big as the real used code. after sudo btrfs balance start /:

$ sudo btrfs fi show /
Label: 'Arch'  uuid: ad537972-5f03-4f3b-96d4-cf45f8dcf581
        Total devices 1 FS bytes used 61.91GiB
        devid    1 size 125.46GiB used 65.47GiB path /dev/sdc5

but i wonder what that did to my poor SSD…

Last edited by flying sheep (2014-09-26 18:16:29)

WorMzy · 2014-09-26 19:29:24

btrfs output is quite confusing at times. Here "used" has two different meanings.

        Total devices 1 FS bytes used 61.93GiB

Here it means how much space is actually taken up by data.

        devid    1 size 125.46GiB used 125.46GiB path /dev/sdc5

And here it means "allocated".

Balancing reclaims any unused allocated space, so that it can be redistributed as required.

What I suspect was happening here, was that you couldn't create files because your metadata pool was full, but there wasn't any unallocated space left to add to it (because it was all allocated to the data pool). Balancing reclaimed the space from the data pool, so that space could be added to the metadata poll, and now you can create files again.

At least that's my theory. The evidence doesn't seem to support it however -- I can't imagine you managed to fill all 3.45G of the allocated metadata space you mentioned in your first post. The output of

# btrfs fi usage /

may shed some light on the matter though.

nstgc · 2014-09-27 14:33:02

What version of btrfs-progs are you using? In 3.16 they made some improvements to df.

ewaller · 2014-09-27 16:46:40

I know nothing of btrfs, but is it possible you are out of inodes? Some file systems have trouble with lots and lots of little files.

nstgc · 2014-09-27 17:10:56

Oh, another way to find the data used is by running "btrfs scrub start -B {mount point or device}". It will return how much data was actually read, which as far as I know is how much data is on the volume or drive.

Arch Linux

#1 2014-09-26 15:29:41

“no space left on device” with 54% du

#2 2014-09-26 16:54:13

Re: “no space left on device” with 54% du

#3 2014-09-26 17:02:17

Re: “no space left on device” with 54% du

#4 2014-09-26 17:18:02

Re: “no space left on device” with 54% du

#5 2014-09-26 18:14:01

Re: “no space left on device” with 54% du

#6 2014-09-26 19:29:24

Re: “no space left on device” with 54% du

#7 2014-09-27 14:33:02

Re: “no space left on device” with 54% du

#8 2014-09-27 16:46:40

Re: “no space left on device” with 54% du

#9 2014-09-27 17:10:56

Re: “no space left on device” with 54% du

Board footer