Disk Space running full

stevek-pro · 2022-01-27 16:09:25

I have a problem of my disk running full, disk size is 432G and by no means I am using this intentionally.

This is the troubleshooting I conducted so far:

$ sudo du -hs /mnt/root/* | sort -rh
30G	/mnt/root/var
23G	/mnt/root/home
9,1G	/mnt/root/usr
971M	/mnt/root/opt

$ ncdu
. 114,5 GiB [#####################] /run
.  34,5 GiB [######               ] /mnt
.  21,7 GiB [###                  ] /home
.   9,1 GiB [#                    ] /usr
.   2,9 GiB [                     ] /var
. 970,2 MiB [                     ] /opt

$ df -h
Dateisystem    Größe Benutzt Verf. Verw% Eingehängt auf
dev              16G       0   16G    0% /dev
run              16G    9,9M   16G    1% /run
/dev/nvme0n1p2  432G    406G  3,9G  100% /
tmpfs            16G     74M   16G    1% /dev/shm
tmpfs            16G     71M   16G    1% /tmp
/dev/loop1       44M     44M     0  100% /var/lib/snapd/snap/snapd/14295
/dev/loop0       56M     56M     0  100% /var/lib/snapd/snap/core18/2253
/dev/nvme0n1p1  510M    296K  510M    1% /boot/efi
tmpfs           3,2G     84K  3,2G    1% /run/user/1000
/dev/nvme0n1p5  514G    116G  399G   23% /run/media/username/500G

Disk always runs full completely. I also deleted

/var/log

and

.xsessions

is piped to

/dev/null

in my bashrc. Yet, disk keeps on running full to 100% used rate again. Any ideas on how to troubleshoot this any further?

Last edited by stevek-pro (2022-01-27 16:17:58)

seth · 2022-01-27 16:43:04

The first box looks like an offline inspection and there're no offending paths? (/var is perhaps the pacman cache)
Does "df" still claim the disk to be full in that offline context?

stevek-pro · 2022-01-27 17:25:20

Hi @seth, thanks for your reply. First box precursor:

    sudo mkdir -p /mnt/root
    sudo mount --bind / /mnt/root
    shopt -s dotglob

I get offending paths if using du without privilege escalation, or what did you refer to with offending paths. df -h gives the same after above procedure and without.

Trilby · 2022-01-27 17:31:06

115G in /run?? Normally that's a tmpfs, do you have a disk-mounted /run? Or do you have an enormous swap file limit that is being consumed by /run overrunning your RAM? Regardles, 115G in /run is absolutely ridiculous. Some process is spamming it's /run directory. For reference, my /run is currently just over 1MB.

EDIT: *headdesk* never mind this ... you have mounted another disk under /run.

Last edited by Trilby (2022-01-27 17:32:20)

stevek-pro · 2022-01-27 17:48:53

Hi Trilby, yes thats the other disk. Btw, with 32gi of memory I thought I give it a shot w/o Swap file. I have this in ~/.bashrc, I hope I did this right:

# If the .xsession-errors file is not a symbolic link, delete it and create it as such
if [ ! -h $HOME/.xsession-errors ]; then
 /bin/rm $HOME/.xsession-errors
 ln -s /dev/null $HOME/.xsession-errors
fi

This is strange:

$ sudo du -cks * | sort -rn | head
120010542	insgesamt
120000406	media
8192	log
1580	udev
192	systemd
76	user
32	NetworkManager
8	docker
4	utmp
4	udisks2

Last edited by stevek-pro (2022-01-27 17:54:36)

Trilby · 2022-01-27 17:54:17

.xsession-errors shouldn't take up much (if any) space at all. If that was ever an issue (which I'd gather it was if you felt the need to send it all to nowhere) then you should address whatever your system is yelling at you about not just ignore the yelling. Though that in itself is not directly related to your concern in this thread ... but I am curious if it could be indirectly related. Could what would be filling up .xsession-errors give us a hint about problematic processes that may also be filling up other logs or otherwise eating disk space? The only way to know is to *not* send xsession-errors to /dev/null but rather look at what it says.

stevek-pro · 2022-01-27 17:56:10

I agree, I deleted the system's ear plugs which you mentioned and rebooted. Do now I see

$ df -h
dev 16G 0 16G 0% /dev
run 16G 9,8M 16G 1% /run
/dev/nvme0n1p2 432G 62G 348G 16% /
tmpfs 16G 17M 16G 1% /dev/shm
tmpfs 16G 116K 16G 1% /tmp
/dev/loop0 56M 56M 0 100% /var/lib/snapd/snap/core18/2253
/dev/loop1 44M 44M 0 100% /var/lib/snapd/snap/snapd/14295
/dev/nvme0n1p1 510M 296K 510M 1% /boot/efi
tmpfs 3,2G 80K 3,2G 1% /run/user/1000

So, that's a relief. I will be watching my .xsessions-errors now, I guess somehow the /dev/null redirection did not work, whatever the root cause was, I hope to find out, I will update here.

So far I'm getting:

sort ~/.xsession-errors | uniq -c | sort -nr | head -10
     26
      9 xdg-settings: default-url-scheme-handler not implemented for xfce

This is the preliminary content of above file:

https://gist.github.com/stevek-pro/667c … 8c264681a9

Last edited by stevek-pro (2022-01-27 18:35:54)

seth · 2022-01-27 20:57:43

Not sure what the systems ear plugs were, but what I meant by the offending paths are those which take a lot of space.
Your original post has

30G	/mnt/root/var
23G	/mnt/root/home
9,1G	/mnt/root/usr
971M	/mnt/root/opt

what kinda doesn't add up to > 400GB at all.
So my concern is/was that the space taking data doesn't exis if the system isn't running and inspected from the outside (what then means that you'll have to look at the space distribution from the running system)

Trilby · 2022-01-27 22:41:32

seth wrote:

Not sure what the systems ear plugs were...

It took me a few rereadings before I realized this just meant not ignoring xsession-errors (i.e., plugging one's ears to block out noise).

There's a fair bit in that .xsession-errors - much of it is likely benign, but I'd look into all the ERROR / Critical stuff. Much of it is for software I'm not familiar with, and a lot also in a language I'm also not so familiar with (German?). Perhaps a good approach would be to limit what software you run for a bit to see if there is a pattern: do you only see the disk use issue when a certain program / utility is running.

seth · 2022-01-28 08:28:35

Is german - already running gpg-agent, "mailspring2 executable could not be found, neither could "set_once.sh" and xfce4-screensaver runs into a concurrent screensaver
Might be signs of a bogus session, but even if we assume that the redirection is too late and the session writes into the unlinked FD, we're still looking for 300+ unaccounted GBs…

stevek-pro · 2022-01-28 20:35:05

Trilby wrote:

.. should address whatever your system is yelling at you about not just ignore the yelling.

Indeed I referred to looking into logs instead of dev nulling them, when using the term "ear plugs". Now that "they" are out if I look into xsessions-errors and see it full of this line.

[00007f8e140ac0f0] vdpau_chroma filter error: video mixer attributes failure: An invalid handle value was provided.

There's the perpetrator

349G -rw------- 1 user user 349G 28. Jan 21:38 /home/user/.xsession-errors

A quick search already revealed it's about nvidia.

Last edited by stevek-pro (2022-01-28 20:42:31)

seth · 2022-01-28 20:42:31

Does the ./xsessions-errors file grow fast enough to explain your space shortage?

stevek-pro · 2022-01-28 20:43:44

It went from 0 to 349G in about 1 day.

seth · 2022-01-28 20:47:01

Your script in #5 hits too late - by this time the session has already created the filehandle and stuff can write there.
You're just hiding that by removing the file from the filesystem (what explains the stealth space loss)

stevek-pro · 2022-01-28 20:55:32

seth wrote:

Your script in #5 hits too late - by this time the session has already created the filehandle and stuff can write there. You're just hiding that by removing the file from the filesystem (what explains the stealth space loss)

Last edited by stevek-pro (2022-01-28 20:57:36)

Trilby · 2022-01-28 20:57:00

Wow. I had an outside hope that looking at what was in that file might point to where the problem might be - I wasn't even thinking that file itself would be the problem. Several hundred GB of log spam is impressive.

Last edited by Trilby (2022-01-28 20:57:26)

stevek-pro · 2022-01-28 21:00:43

While doing research on similar issues I read of people experiencing Terrabytes of lost space. In anycase, thanks a lot for sharing your wisdom, much appreciated!

Root cause likely https://bugs.launchpad.net/ubuntu/+sour … ug/1911354

Last edited by stevek-pro (2022-01-28 21:13:42)

Arch Linux

#1 2022-01-27 16:09:25

Disk Space running full

#2 2022-01-27 16:43:04

Re: Disk Space running full

#3 2022-01-27 17:25:20

Re: Disk Space running full

#4 2022-01-27 17:31:06

Re: Disk Space running full

#5 2022-01-27 17:48:53

Re: Disk Space running full

#6 2022-01-27 17:54:17

Re: Disk Space running full

#7 2022-01-27 17:56:10

Re: Disk Space running full

#8 2022-01-27 20:57:43

Re: Disk Space running full

#9 2022-01-27 22:41:32

Re: Disk Space running full

#10 2022-01-28 08:28:35

Re: Disk Space running full

#11 2022-01-28 20:35:05

Re: Disk Space running full

#12 2022-01-28 20:42:31

Re: Disk Space running full

#13 2022-01-28 20:43:44

Re: Disk Space running full

#14 2022-01-28 20:47:01

Re: Disk Space running full

#15 2022-01-28 20:55:32

Re: Disk Space running full

#16 2022-01-28 20:57:00

Re: Disk Space running full

#17 2022-01-28 21:00:43

Re: Disk Space running full

Board footer