Out of disk space errors but df says there's plenty left

agnomen · 2024-04-16 12:42:42

Not sure where this should go, but this forum seemed like the most appropriate.

I have recently twice encountered an issue where the system became borderline unusable and multiple applications complained that the disk was full. However df said that there was "plenty" diskspace left (around 13% or so left). I started deleting things but I kept getting the error until I went under around 75% disk utilization. I have not set up any quotas or anything similar that I can remember. As far as I can recall I have previously not had any issue with having around 13% free on the disk.

Current output of df -hT:

➜  ~ df -hT
Filesystem          Type      Size  Used Avail Use% Mounted on
dev                 devtmpfs   16G     0   16G   0% /dev
run                 tmpfs      16G  2.0M   16G   1% /run
efivarfs            efivarfs  128K   34K   90K  28% /sys/firmware/efi/efivars
/dev/mapper/luksdev btrfs     931G  661G  269G  72% /
tmpfs               tmpfs      16G  541M   16G   4% /dev/shm
/dev/mapper/luksdev btrfs     931G  661G  269G  72% /.snapshots
/dev/mapper/luksdev btrfs     931G  661G  269G  72% /home
/dev/mapper/luksdev btrfs     931G  661G  269G  72% /var/cache/pacman/pkg
/dev/mapper/luksdev btrfs     931G  661G  269G  72% /var/log
tmpfs               tmpfs      16G   13M   16G   1% /tmp
/dev/nvme0n1p1      vfat      510M  144M  367M  29% /boot
tmpfs               tmpfs     3.2G  112K  3.2G   1% /run/user/1000

I still saw these issues even with 85% utilization. What can cause applications to behave like there is no diskspace available while df says there is diskspace available? Are there some btrfs considerations I don't know about? Any ideas about what to look into the next time this occurs would be helpful.

agnomen · 2024-04-16 12:43:48

Googling this mostly leads to posts where df actually agrees that the disk is full

schard · 2024-04-16 12:52:34

What does

# btrfs fi us /

yield?
Also read: https://wiki.tnonline.net/w/Btrfs/Balance

Scimmia · 2024-04-16 12:55:45

agnomen wrote:

and multiple applications complained that the disk was full.

We need specifics. What applications and what errors? Wild guess is that /tmp or RAM was full.

agnomen · 2024-04-16 13:31:35

schard wrote:

What does
# btrfs fi us /
yield?

Overall:
    Device size:                 931.00GiB
    Device allocated:            923.00GiB
    Device unallocated:            8.00GiB
    Device missing:                  0.00B
    Device slack:                    0.00B
    Used:                        659.74GiB
    Free (estimated):            268.60GiB      (min: 264.60GiB)
    Free (statfs, df):           268.60GiB
    Data ratio:                       1.00
    Metadata ratio:                   2.00
    Global reserve:              512.00MiB      (used: 0.00B)
    Multiple profiles:                  no

Data,single: Size:886.02GiB, Used:625.42GiB (70.59%)
   /dev/mapper/luksdev   886.02GiB

Metadata,DUP: Size:18.48GiB, Used:17.16GiB (92.86%)
   /dev/mapper/luksdev    36.96GiB

System,DUP: Size:8.00MiB, Used:128.00KiB (1.56%)
   /dev/mapper/luksdev    16.00MiB

Unallocated:
   /dev/mapper/luksdev     8.00GiB

schard wrote:

Also read: https://wiki.tnonline.net/w/Btrfs/Balance

This looks like what I needed. It specifically calls out df as a liar "don't trust 'df'". While the output of 'btrfs fi us /' looks good to me at the moment, this sounds like it was the issue. Neither have I performed any 'btrfs balance' previously either.

Running the suggested balance does seem to yield a better situation:

➜  ~ sudo btrfs balance start -dusage=5 /
Done, had to relocate 110 out of 910 chunks
➜  ~ sudo btrfs fi us /                  
Overall:
    Device size:                 931.00GiB
    Device allocated:            813.00GiB
    Device unallocated:          118.00GiB
    Device missing:                  0.00B
    Device slack:                    0.00B
    Used:                        659.75GiB
    Free (estimated):            268.59GiB      (min: 209.59GiB)
    Free (statfs, df):           268.59GiB
    Data ratio:                       1.00
    Metadata ratio:                   2.00
    Global reserve:              512.00MiB      (used: 0.00B)
    Multiple profiles:                  no

Data,single: Size:776.02GiB, Used:625.43GiB (80.59%)
   /dev/mapper/luksdev   776.02GiB

Metadata,DUP: Size:18.48GiB, Used:17.16GiB (92.87%)
   /dev/mapper/luksdev    36.96GiB

System,DUP: Size:8.00MiB, Used:128.00KiB (1.56%)
   /dev/mapper/luksdev    16.00MiB

Unallocated:
   /dev/mapper/luksdev   118.00GiB

agnomen · 2024-04-16 13:35:47

Scimmia wrote:

agnomen wrote:
and multiple applications complained that the disk was full.
We need specifics. What applications and what errors? Wild guess is that /tmp or RAM was full.

I think schard nailed the issue already. /tmp usage is correctly displayed by df AFAIK so that was not the issue. While I didn't make an exhaustive list of the errors, they did not appear connected to /tmp in any way. One example that's left in the logs:

[ 1623.979120] systemd-journald[2708]: Failed to rotate /var/log/journal/a1ff8a23704c4d239c7442729358af9a/system.journal: No space left on device
[ 1624.017248] systemd-journald[2708]: /var/log/journal/a1ff8a23704c4d239c7442729358af9a/user-1000.journal: IO error, rotating.

ayekat · 2024-04-24 20:59:47

Check the inodes count:

df -i

Arch Linux

#1 2024-04-16 12:42:42

Out of disk space errors but df says there's plenty left

#2 2024-04-16 12:43:48

Re: Out of disk space errors but df says there's plenty left

#3 2024-04-16 12:52:34

Re: Out of disk space errors but df says there's plenty left

#4 2024-04-16 12:55:45

Re: Out of disk space errors but df says there's plenty left

#5 2024-04-16 13:31:35

Re: Out of disk space errors but df says there's plenty left

#6 2024-04-16 13:35:47

Re: Out of disk space errors but df says there's plenty left

#7 2024-04-24 20:59:47

Re: Out of disk space errors but df says there's plenty left

Board footer