Arch Linux

alocacoc

Member

From: Germany

Registered: 2013-06-27

Posts: 104

[Solved] Root space problem

I have a 200GB root and a 750GB home partition.

The system sais, the root partition is full, but when I count all together, it only contains 17,5 GB. /usr is 11,6GB and /var is 3,7GB. Nothing close to 200GB.

The harddisk is checked every boot, sais 'clean'.

df -h
Filesystem      Size  Used Avail Use% Mounted on
dev             2.0G     0  2.0G   0% /dev
run             2.0G  1.1M  2.0G   1% /run
/dev/sda4       193G  184G     0 100% /
tmpfs           2.0G  172K  2.0G   1% /dev/shm
tmpfs           2.0G     0  2.0G   0% /sys/fs/cgroup
tmpfs           2.0G  2.5M  2.0G   1% /tmp
/dev/sdb1       2.7T  2.7T   21G 100% /mnt/hdd1
/dev/sda2       969M   38M  866M   5% /boot
/dev/sda5       705G  662G   36G  95% /home
tmpfs           396M   16K  396M   1% /run/user/1000


df -h /
Filesystem      Size  Used Avail Use% Mounted on
/dev/sda4       193G  184G     0 100% /

sda4 is root.

The system works fine and fast, nothing changed, but a few programs doesn't work, which want to write somewhere to root, like cups.

Any clue how to find out the files which consume like 175GB?

Thanks

Last edited by alocacoc (2015-04-30 20:07:30)

ewaller

Administrator

From: Pasadena, CA

Registered: 2009-07-13

Posts: 19,804

Re: [Solved] Root space problem

A good, first order approximation could be learned from: du -hs /* 2>/dev/null

Last edited by ewaller (2015-04-29 15:17:10)

---

Nothing is too wonderful to be true, if it be consistent with the laws of nature -- Michael Faraday
Sometimes it is the people no one can imagine anything of who do the things no one can imagine. -- Alan Turing
---
How to Ask Questions the Smart Way

Scimmia

Fellow

Registered: 2012-09-01

Posts: 11,599

Re: [Solved] Root space problem

ncdu is a great program for finding where your space went. Unfortunately, you won't be able to install it since / is full.

gmtch

Member

Registered: 2015-03-29

Posts: 7

Re: [Solved] Root space problem

How do you know that / is only using 17.5 GB?

If that is coming from du but df is telling you 184GB then you might have a deleted file which is using space because a process still has the file open.

Simplest way to confirm and resolve at the same time is to reboot. If afterwards you get  the space back then that was it.

What type of filesystem is / ? If it  is ext4 there might be 5% of the filesystem being reserved.

This will confirm:

tune2fs -l /dev/sda4  | grep "Reserved block count"

You can set it to a 1% reserved with:

tune2fs -m1 /dev/sda4

That might give you some space to work with.

To find any files that are only on / and are bigger than say 50M try:

find / -xdev -type f -size +50M -ls

If +50M finds too many files then up it to +100M and so on.

alocacoc

Member

From: Germany

Registered: 2013-06-27

Posts: 104

Re: [Solved] Root space problem

du -hs /* 2>/dev/null
0	/46F3511B365F
4.0K	/680db508
0	/bin
36M	/boot
168K	/dev
21M	/etc
662G	/home
0	/lib
0	/lib64
16K	/lost+found
4.0K	/media
2.9T	/mnt
763M	/opt
0	/proc
56M	/root
1.2M	/run
0	/sbin
12K	/srv
0	/sys
2.5M	/tmp
13G	/usr
4.1G	/var
4.0K	/yum-cron.service

ncdu /

ncdu 1.11 ~ Use the arrow keys to navigate, press ? for help                            
--- / ----------------------------------------------------------------------------------
    2.8 TiB [##########] /mnt                                                           
  661.2 GiB [##        ] /home
   12.2 GiB [          ] /usr
    4.0 GiB [          ] /var
  762.2 MiB [          ] /opt
   55.8 MiB [          ] /root
   35.8 MiB [          ] /boot
   20.5 MiB [          ] /etc
    2.4 MiB [          ] /tmp
.   1.1 MiB [          ] /run
  172.0 KiB [          ] /dev
e  16.0 KiB [          ] /lost+found
   12.0 KiB [          ] /srv
   12.0 KiB [          ] /.Trash-1000
    8.0 KiB [          ] /.config
e   4.0 KiB [          ] /media
    4.0 KiB [          ]  680db508
    4.0 KiB [          ]  yum-cron.service
.   0.0   B [          ] /proc
    0.0   B [          ] /sys
@   0.0   B [          ]  sbin
@   0.0   B [          ]  lib64
@   0.0   B [          ]  lib
@   0.0   B [          ]  bin
    0.0   B [          ]  46F3511B365F

tune2fs -m1 /dev/sda4
tune2fs 1.42.12 (29-Aug-2014)
Setting reserved blocks percentage to 1% (512000 blocks)


tune2fs -l /dev/sda4  | grep "Reserved block count"
Reserved block count:     512000

Setting the reserve space to less % gave me 6GB, but even before the system worked well. I could even install programs with pacman and update the system. Still it said the / is full. Once I free some space, with cleaning log files for example, gave me almost 1 GB, but itquickly filled up again to give me 0MB free after a few days.

17,5Gb were estimated by me by checking and combining the single folder sizes.

The PC is shutdown and used daily.

The problem appears since several weeks, took me a while to find out the cause. The two programs which doen't work anymore are cups and wine, where wine became glitchy. Glitches are gone after I free up space from root. Cups work again after I free space from root and stops working again once root is full again.

Root partition file system is ext4.

"find / -xdev -type f -size +200M -ls" listed only files on /mnt, /mnt contains only hdds which are not the root partition.

Thank you for your help.

Last edited by alocacoc (2015-04-30 18:25:58)

alphaniner

Member

From: Ancapistan

Registered: 2010-07-12

Posts: 2,810

Re: [Solved] Root space problem

"find / -xdev -type f -size +200M -ls" listed only files on /mnt, /mnt contains only hdds which are not the root partition.

With -xdev find doesn't show anything at all on other volumes. You must be writing to a directory on /mnt where you think something is mounted but actually isn't.

Last edited by alphaniner (2015-04-30 19:06:50)

---

But whether the Constitution really be one thing, or another, this much is certain - that it has either authorized such a government as we have had, or has been powerless to prevent it. In either case, it is unfit to exist.
-Lysander Spooner

lucke

Member

From: Poland

Registered: 2004-11-30

Posts: 4,018

Re: [Solved] Root space problem

/dev/sdb1       2.7T  2.7T   21G 100% /mnt/hdd1

2.9T	/mnt

Do you see it?

Spider.007

Member

Registered: 2004-06-20

Posts: 1,175

Re: [Solved] Root space problem

Maybe you wrote files to /mnt/hdd1 when /dev/sdb1 wasn't mounted; unmount that and have a look whats in /mnt then...

alocacoc

Member

From: Germany

Registered: 2013-06-27

Posts: 104

Re: [Solved] Root space problem

Problem found.

I had an external 1GB HDD mounted to /mnt/hdd2. This device was used as a backup harddisk and gave me trouble, so I cut it out of fstab and unplugged it after the next shutdown. And also forgot about the rsync backup cron jobs.

I tried to umount /mnt/hdd1, my 3TB disc, like Spider.007 suggested. Umount worked fine, folder was empty.

So I looked into /mnt/hdd2, sure the folder was still there, but inside was not empty, it contained files from the disconnected harddisc. Appearently, the files were written to the root partition somewhere instead to the disconnected harddisk. I deleted the whole /mnt/hdd2 folder as root and 173GB are free on root.

The rsync cron tried to do the backup, it couldn't find the disconnected disc, so it appearently made the backup to the root partition.

This explains why the space was always quickly filled up again, because it tried to complete the backup.

I removed the cron job now, deleted the files, this should do it.

Thank you all for your help.

WorMzy

Forum Moderator

From: Scotland

Registered: 2010-06-16

Posts: 11,901

Website

Re: [Solved] Root space problem

You might want to add some extra safety checks in your script. Checking that <path> is a mountpoint is easy with.. well, "mountpoint".

---

Sakura:-
Mobo: MSI MAG X570S TORPEDO MAX // Processor: AMD Ryzen 9 5950X @4.9GHz // GFX: AMD Radeon RX 5700 XT // RAM: 32GB (4x 8GB) Corsair DDR4 (@ 3000MHz) // Storage: 1x 3TB HDD, 6x 1TB SSD, 2x 120GB SSD, 1x 275GB M2 SSD

Making lemonade from lemons since 2015.

alocacoc

Member

From: Germany

Registered: 2013-06-27

Posts: 104

Re: [Solved] Root space problem

I will more carefully set up the sync next time I got a replacement harddisk. Thanks for the tips.