[SOLVED] Disk thinks it is full but it is not. (Edit: it was)

benm · 2022-03-31 02:38:51

Dolphin believes I have 821MiB free of 915.8GiB on my ext4 root partition, /dev/sda1. This is an unrealistic number as I had nowhere near this loaded on the disk intentionally unless there is some hidden thing I am unaware of.

This installation is new (about two weeks old) with a new drive (about the same age).

I would occasionally delete a file or two just to temporarily make space and find that it later was full again. At one point it reflected 0 free space, and attempting to install via pacman a < 1MiB command line application failed due to no disk space.

df -h gives

Filesystem      Size  Used Avail Use% Mounted on
dev             7.8G     0  7.8G   0% /dev
run             7.8G  1.8M  7.8G   1% /run
/dev/sda1       916G  869G  822M 100% /
tmpfs           7.8G   57M  7.8G   1% /dev/shm
tmpfs           7.8G  8.8M  7.8G   1% /tmp
/dev/sdb1       1.8T  1.2T  563G  68% /run/media/ben/WD-Blue
tmpfs           1.6G  148K  1.6G   1% /run/user/1000

Filelight at launch reflects the 821 MiB out of 915.8 GiB, but after running a scan of root, says that only 312 GiB are consumed. 312 GiB is a realistic number based on actual installs and files I intentionally loaded on the drive.

I am looking for help understanding what is driving the system to think that it is full, whether it actually is full or not and if it is full how to figure out what is consuming the space since normal scans do not seem to reveal it.

Possibly relevant: on reboot, it spends a long time (sometimes > 60s) doing something to the journal which seems abnormal.

Last edited by benm (2022-03-31 12:47:41)

Trilby · 2022-03-31 02:44:01

"doing something"? What exactly? Even before that line my suspicion would have been that some process is spamming the journal which is in turn growing out of control. You can edit journald.conf to cap the out of control growth to make sure the system does freeze up due to running out of space, but then look at what's filling the journal and solve the underlying problem.

benm · 2022-03-31 02:57:20

The message it gives is

Starting version 250.4-2-arch
/dev/sda1: clean, 1729670/61054976 files, 231763236/244190390 blocks

Do you have a recommendation on where to cap the journal? That being said, it does not look like the culprit:

$ ls /var/log/journal/* -lah
/var/log/journal/acb59ce79f0f4e6382c7a53e5ba187f1:
total 525M
drwxr-sr-x+ 2 root systemd-journal 4.0K Mar 30 22:51 .
drwxr-sr-x+ 4 root systemd-journal 4.0K Mar 15 16:36 ..
-rw-r-----+ 1 root systemd-journal  24M Mar 16 07:26 system@0005da5437d7177d-a7d1b43eb838cada.journal~
-rw-r-----+ 1 root systemd-journal  16M Mar 27 13:10 system@0005db364aa7c9f2-769dfb9e8b05069d.journal~
-rw-r-----+ 1 root systemd-journal  69M Mar 29 17:55 system@6dc92fd797914df6b976cbdb05b4a252-0000000000000001-0005db364aa7345a.journal
-rw-r-----+ 1 root systemd-journal  69M Mar 30 22:50 system@6dc92fd797914df6b976cbdb05b4a252-0000000000012e2c-0005db627d54ea0a.journal
-rw-r-----+ 1 root systemd-journal  43M Mar 24 13:28 system@93fc2b9d3d7c4962b763ff0f5f7a01d8-0000000000000001-0005da5437d68829.journal
-rw-r-----+ 1 root systemd-journal 8.2M Mar 24 20:27 system@93fc2b9d3d7c4962b763ff0f5f7a01d8-00000000000296f0-0005dafa2c9cfdb5.journal
-rw-r-----+ 1 root systemd-journal 8.0M Mar 30 22:56 system.journal
-rw-r-----+ 1 root systemd-journal  16M Mar 16 07:27 user-1000@0005da5439df42f2-7e4b868a451536a5.journal~
-rw-r-----+ 1 root systemd-journal  16M Mar 27 13:10 user-1000@0005db364b503498-2a9d105a0ca6f8b3.journal~
-rw-r-----+ 1 root systemd-journal  17M Mar 29 17:55 user-1000@1bf0a775bf1c4700b07e16b4af9611ba-0000000000000589-0005db364b502fc2.journal
-rw-r-----+ 1 root systemd-journal  18M Mar 30 22:50 user-1000@1bf0a775bf1c4700b07e16b4af9611ba-0000000000012e46-0005db627e905935.journal
-rw-r-----+ 1 root systemd-journal 128M Mar 24 13:28 user-1000@99c083650ed84c168c835c39eb2e13d0-0000000000000583-0005da5439df3f22.journal
-rw-r-----+ 1 root systemd-journal 128M Mar 24 20:27 user-1000@99c083650ed84c168c835c39eb2e13d0-0000000000028550-0005dafa2c1b3a81.journal
-rw-r-----+ 1 root systemd-journal 8.0M Mar 30 22:56 user-1000.journal

/var/log/journal/remote:
total 16K
drwxr-sr-x+ 2 root systemd-journal-remote 4.0K Mar 15 16:23 .
drwxr-sr-x+ 4 root systemd-journal        4.0K Mar 15 16:36 ..

That being said, if it were, why would it not show up in the disk usage tools?

mpan · 2022-03-31 03:40:55

Use community/ncdu to see what uses most of the disk:

sudo ncdu -x /

Last edited by mpan (2022-03-31 03:44:50)

seth · 2022-03-31 07:11:08

but after running a scan of root, says that only 312 GiB are consumed. 312 GiB is a realistic number

Does a reboot initially return the lost space?
See eg. https://bbs.archlinux.org/viewtopic.php?id=273590 - esp. if you do similar stuff w/ your xsession-errors

benm · 2022-03-31 11:25:34

Ok, this was a stupid error on my part. I had run a variety of disk usage checkers but apparently did not run ncdu with sudo or with the -x option. I found 400GB+ of data that I had intended to be on an alternate drive. It was also a blockchain that was intended to be trimmed to 2GB upon completion but apparently never was.

The solution was to run ncdu the right way.

Arch Linux

#1 2022-03-31 02:38:51

[SOLVED] Disk thinks it is full but it is not. (Edit: it was)

#2 2022-03-31 02:44:01

Re: [SOLVED] Disk thinks it is full but it is not. (Edit: it was)

#3 2022-03-31 02:57:20

Re: [SOLVED] Disk thinks it is full but it is not. (Edit: it was)

#4 2022-03-31 03:40:55

Re: [SOLVED] Disk thinks it is full but it is not. (Edit: it was)

#5 2022-03-31 07:11:08

Re: [SOLVED] Disk thinks it is full but it is not. (Edit: it was)

#6 2022-03-31 11:25:34

Re: [SOLVED] Disk thinks it is full but it is not. (Edit: it was)

Board footer