You are not logged in.

#1 2024-04-16 12:42:42

agnomen
Member
Registered: 2024-04-16
Posts: 4

Out of disk space errors but df says there's plenty left

Not sure where this should go, but this forum seemed like the most appropriate.

I have recently twice encountered an issue where the system became borderline unusable and multiple applications complained that the disk was full. However df said that there was "plenty" diskspace left (around 13% or so left). I started deleting things but I kept getting the error until I went under around 75% disk utilization. I have not set up any quotas or anything similar that I can remember. As far as I can recall I have previously not had any issue with having around 13% free on the disk.

Current output of df -hT:

➜  ~ df -hT
Filesystem          Type      Size  Used Avail Use% Mounted on
dev                 devtmpfs   16G     0   16G   0% /dev
run                 tmpfs      16G  2.0M   16G   1% /run
efivarfs            efivarfs  128K   34K   90K  28% /sys/firmware/efi/efivars
/dev/mapper/luksdev btrfs     931G  661G  269G  72% /
tmpfs               tmpfs      16G  541M   16G   4% /dev/shm
/dev/mapper/luksdev btrfs     931G  661G  269G  72% /.snapshots
/dev/mapper/luksdev btrfs     931G  661G  269G  72% /home
/dev/mapper/luksdev btrfs     931G  661G  269G  72% /var/cache/pacman/pkg
/dev/mapper/luksdev btrfs     931G  661G  269G  72% /var/log
tmpfs               tmpfs      16G   13M   16G   1% /tmp
/dev/nvme0n1p1      vfat      510M  144M  367M  29% /boot
tmpfs               tmpfs     3.2G  112K  3.2G   1% /run/user/1000

I still saw these issues even with 85% utilization. What can cause applications to behave like there is no diskspace available while df says there is diskspace available? Are there some btrfs considerations I don't know about? Any ideas about what to look into the next time this occurs would be helpful.

Offline

#2 2024-04-16 12:43:48

agnomen
Member
Registered: 2024-04-16
Posts: 4

Re: Out of disk space errors but df says there's plenty left

Googling this mostly leads to posts where df actually agrees that the disk is full

Offline

#3 2024-04-16 12:52:34

schard
Forum Moderator
From: Hannover
Registered: 2016-05-06
Posts: 2,117
Website

Re: Out of disk space errors but df says there's plenty left

What does

# btrfs fi us /

yield?
Also read: https://wiki.tnonline.net/w/Btrfs/Balance


Inofficial first vice president of the Rust Evangelism Strike Force

Offline

#4 2024-04-16 12:55:45

Scimmia
Fellow
Registered: 2012-09-01
Posts: 12,131

Re: Out of disk space errors but df says there's plenty left

agnomen wrote:

and multiple applications complained that the disk was full.

We need specifics. What applications and what errors? Wild guess is that /tmp or RAM was full.

Offline

#5 2024-04-16 13:31:35

agnomen
Member
Registered: 2024-04-16
Posts: 4

Re: Out of disk space errors but df says there's plenty left

schard wrote:

What does

# btrfs fi us /

yield?

Overall:
    Device size:                 931.00GiB
    Device allocated:            923.00GiB
    Device unallocated:            8.00GiB
    Device missing:                  0.00B
    Device slack:                    0.00B
    Used:                        659.74GiB
    Free (estimated):            268.60GiB      (min: 264.60GiB)
    Free (statfs, df):           268.60GiB
    Data ratio:                       1.00
    Metadata ratio:                   2.00
    Global reserve:              512.00MiB      (used: 0.00B)
    Multiple profiles:                  no

Data,single: Size:886.02GiB, Used:625.42GiB (70.59%)
   /dev/mapper/luksdev   886.02GiB

Metadata,DUP: Size:18.48GiB, Used:17.16GiB (92.86%)
   /dev/mapper/luksdev    36.96GiB

System,DUP: Size:8.00MiB, Used:128.00KiB (1.56%)
   /dev/mapper/luksdev    16.00MiB

Unallocated:
   /dev/mapper/luksdev     8.00GiB

This looks like what I needed. It specifically calls out df as a liar "don't trust 'df'". While the output of 'btrfs fi us /' looks good to me at the moment, this sounds like it was the issue. Neither have I performed any 'btrfs balance' previously either.

Running the suggested balance does seem to yield a better situation:

➜  ~ sudo btrfs balance start -dusage=5 /
Done, had to relocate 110 out of 910 chunks
➜  ~ sudo btrfs fi us /                  
Overall:
    Device size:                 931.00GiB
    Device allocated:            813.00GiB
    Device unallocated:          118.00GiB
    Device missing:                  0.00B
    Device slack:                    0.00B
    Used:                        659.75GiB
    Free (estimated):            268.59GiB      (min: 209.59GiB)
    Free (statfs, df):           268.59GiB
    Data ratio:                       1.00
    Metadata ratio:                   2.00
    Global reserve:              512.00MiB      (used: 0.00B)
    Multiple profiles:                  no

Data,single: Size:776.02GiB, Used:625.43GiB (80.59%)
   /dev/mapper/luksdev   776.02GiB

Metadata,DUP: Size:18.48GiB, Used:17.16GiB (92.87%)
   /dev/mapper/luksdev    36.96GiB

System,DUP: Size:8.00MiB, Used:128.00KiB (1.56%)
   /dev/mapper/luksdev    16.00MiB

Unallocated:
   /dev/mapper/luksdev   118.00GiB

Offline

#6 2024-04-16 13:35:47

agnomen
Member
Registered: 2024-04-16
Posts: 4

Re: Out of disk space errors but df says there's plenty left

Scimmia wrote:
agnomen wrote:

and multiple applications complained that the disk was full.

We need specifics. What applications and what errors? Wild guess is that /tmp or RAM was full.

I think schard nailed the issue already. /tmp usage is correctly displayed by df AFAIK so that was not the issue. While I didn't make an exhaustive list of the errors,  they did not appear connected to /tmp in any way. One example that's left in the logs:

[ 1623.979120] systemd-journald[2708]: Failed to rotate /var/log/journal/a1ff8a23704c4d239c7442729358af9a/system.journal: No space left on device
[ 1624.017248] systemd-journald[2708]: /var/log/journal/a1ff8a23704c4d239c7442729358af9a/user-1000.journal: IO error, rotating.

Offline

#7 2024-04-24 20:59:47

ayekat
Member
Registered: 2011-01-17
Posts: 1,611

Re: Out of disk space errors but df says there's plenty left

Check the inodes count:

df -i

pkgshackscfgblag

Offline

Board footer

Powered by FluxBB