Arch Linux

ChatNoir

Member

Registered: 2011-01-13

Posts: 10

[SOLVED] Root fs appears full but nothing big shows up under du

Hi,

I was trying to install a package with pacman and ran into an issue where it told me the disk was full.

I have a 20GB root partition and 100GB home partition, df says that rootfs is full (20GB) however sudo du -hsx run in / shows only 3.3GB.

Here are the outputs of the commands:

bitdivision at ArchBox in / 
$ sudo du -hsx
3.3G	.

and here is df showing 100% usage:

$ df -hT                                       
Filesystem     Type      Size  Used Avail Use% Mounted on
rootfs         rootfs     20G   20G     0 100% /
dev            devtmpfs  3.8G     0  3.8G   0% /dev
run            tmpfs     3.8G  380K  3.8G   1% /run
/dev/sda3      ext4       20G   20G     0 100% /
shm            tmpfs     3.8G  1.4M  3.8G   1% /dev/shm
tmpfs          tmpfs     3.8G  180K  3.8G   1% /tmp
/dev/sda4      ext4      100G   18G   77G  19% /home
/dev/sda1      ext2       98M   19M   74M  20% /boot
/dev/sdb1      ext4      466G  124G  319G  28% /media/BackupDrive
/dev/sdc1      fuseblk   932G  818G  114G  88% /media/Obsidian

and finally here is findmnt to show my partition layout if it's needed:

$ findmnt
TARGET               SOURCE    FSTYPE OPTIONS
/                    /dev/sda3 ext4   rw,relatime,data=ordere
├─/proc              proc      proc   rw,nosuid,nodev,noexec,
│ └─/proc/sys/fs/binfmt_misc
                     binfmt    binfmt rw,relatime
├─/sys               sys       sysfs  rw,nosuid,nodev,noexec,
├─/dev               dev       devtmp rw,nosuid,relatime,size
│ ├─/dev/pts         devpts    devpts rw,nosuid,noexec,relati
│ └─/dev/shm         shm       tmpfs  rw,nosuid,nodev,relatim
├─/run               run       tmpfs  rw,nosuid,nodev,relatim
├─/tmp               tmpfs     tmpfs  rw,nosuid,nodev,relatim
├─/home              /dev/sda4 ext4   rw,noatime,discard,data
│ └─/home/bitdivision/.gvfs
                     gvfs-fuse-daemon
                               fuse.g rw,nosuid,nodev,relatim
├─/boot              /dev/sda1 ext2   rw,relatime
├─/media/BackupDrive /dev/sdb1 ext4   rw,nosuid,nodev,noexec,
└─/media/Obsidian    /dev/sdc1 fusebl rw,nosuid,nodev,noexec,

Thanks

Last edited by ChatNoir (2012-07-20 11:49:17)

DSpider

Member

From: Romania

Registered: 2009-08-23

Posts: 2,273

Re: [SOLVED] Root fs appears full but nothing big shows up under du

20 GB for just "/" is quite a lot. Check your /var/log folder. That's usually a good indicator.

And maybe clear your pacman cache.

# pacman -Scc

---

"How to Succeed with Linux"

I have made a personal commitment not to reply in topics that start with a lowercase letter. Proper grammar and punctuation is a sign of respect, and if you do not show any, you will NOT receive any help (at least not from me).

tigrezno

Member

Registered: 2004-05-10

Posts: 64

Re: [SOLVED] Root fs appears full but nothing big shows up under du

try this:

du -hx --max-depth=1 /

Last edited by tigrezno (2012-07-20 07:31:21)

ChatNoir

Member

Registered: 2011-01-13

Posts: 10

Re: [SOLVED] Root fs appears full but nothing big shows up under du

tigrenzo:

$ sudo du -hx --max-depth=1 /
Password: 
49M	/lib
4.0K	/media
4.0K	/mnt
4.2M	/bin
10M	/sbin
888M	/var
12M	/opt
132K	/root
2.4G	/usr
16K	/lost+found
6.4M	/etc
12K	/srv
3.3G	/

Still doesn't help much unfortunately...

DSpider:

Clearing the cached helped a little but I'm still at 19G on df and 3.3G on du. Really I'm wondering how it's possible for there to be a discrepancy there. It's almost like df is including my home partition in the rootfs size.

fatboy

Member

From: India

Registered: 2012-03-17

Posts: 73

Re: [SOLVED] Root fs appears full but nothing big shows up under du

Googling for the du vs df gives a lot of links. Here is some info from ext3 FAQ (I guess it applies to ext4 too):

The standard cause for this is some user process keeping a deleted file open. When this happens, the space is not visible via 'du', since the file is no longer visible in the directory tree. However, the space is still used by the file until it is deallocated, and that can only happen once the last process which has the file open either closes its file descriptor to the file, or the process exits. You can use the lsof program to try to find which process is keeping an open file. Usually it's some log file, or some large data base file which gets rotated out, but some older process are still keeping the log file open.

This is a longer article giving some more reasons, and says that du is more accurate than df.

EDIT: But the difference is seriously huge in your case!

Last edited by fatboy (2012-07-20 10:39:11)

ChatNoir

Member

Registered: 2011-01-13

Posts: 10

Re: [SOLVED] Root fs appears full but nothing big shows up under du

fatboy:

Thanks for the info, looks like that might have something to do with it.

The only thing is that I have restarted the box multiple times with no change. I would have thought that this would clear any deleted files no?

fatboy

Member

From: India

Registered: 2012-03-17

Posts: 73

Re: [SOLVED] Root fs appears full but nothing big shows up under du

Maybe try running fsck?

Found another weird case, where the guy had mounted an external drive on a non-empty /mnt/Backup, so du didn't count it.

EDIT: I guess it's not weird anymore

Last edited by fatboy (2012-07-20 13:26:32)

ChatNoir

Member

Registered: 2011-01-13

Posts: 10

Re: [SOLVED] Root fs appears full but nothing big shows up under du

fatboy:

You solved it! Thank you.

The issue was that I have a cron job to run backups onto BackupDrive. A little while ago BackupDrive didn't mount on startup before I switched to UUIDs and the cron job must have run while it was unmounted. This created a 17G directory on the rootfs which appeared invisible to du -x because the drive was mounted at the time.

I believe these are called overlay mounts.

Thanks very much for all your help.