[SOLVED] Full root file system and cannot pinpoint the culprit

ideogon · 2010-04-10 23:33:55

Apparently my root file system is full, which is causing write-errors while daemons attempt to start on boot.

[root@bigbird /]# df -h
Filesystem    Size    Used    Avail   Use%    Mounted on
/dev/sdb3     7.3G    7.3G    0       100%    /
udev          10M     120K    9.9M    2%      /dev
none          4.0G    0       4.0G    0%      /dev/shm
/dev/sda1     138G    61G     71G     47%     /mnt/backups
/dev/sdc1     459G    111G    325G    26%     /mnt/media
/dev/sdb1     38M     31M     5.4M    86%     /boot
/dev/sdb4     131G    5.4G    119G    5%      /home

I think I had accidentally mounted /dev/sdb1 as my /mnt/backups file system (which is supposed to be my root file system). For lack of understanding of what is going on, I think that may have caused my root file system to inappropriately fill up whenever I boot into Arch (possibly what was going on in this thread: http://bbs.archlinux.org/viewtopic.php?id=63201). I have since edited /etc/fstab to mount the backups directory from /dev/sda1, as the df -h output shows above.

Anyway, my root file system is still completely full, and I have yet to find a permutation of a df or du command to show me which file(s) are causing this.

--
David

Last edited by ideogon (2010-04-11 01:29:49)

jowilkin · 2010-04-10 23:35:02

Check your pacman cache directory.

Also to narrow it down you can go to root and run 'du --max-depth=1 -h'

This will show you how much each directory is using, then go to the culprit directory and do it again until you find the issue.

If you're running an X server, use the gnome disk usage utility and it will show you.

Last edited by jowilkin (2010-04-10 23:37:46)

ideogon · 2010-04-10 23:46:50

[root@bigbird /]# du --max-depth=1 -h /var/cache/
...
1.1G    /var/cache/pacman
...

[root@bigbird /]# du --max-depth=1 -h
du: cannot access `/proc/32281/task/32281/fd/4`: No such file or directory
du: cannot access `/proc/32281/task/32281/fdinfo/4`: No such file or directory
du: cannot access `/proc/32281/fd/4`: No such file or directory
du: cannot access `/proc/32281/fdinfo/4`: No such file or directory
0       /proc
3.9G    /usr
14M     /sbin
16K     /lost+found
12K     /tmp
31M     /boot
16K     /media
1.3G    /var
120K    /dev
127M    /opt
5.2G    /home
4.1M    /root
171G    /mnt
146M    /lib
5.3M    /bin
0       /sys
12K     /srv
4.0K    /lib64
14M     /etc
181G    /

karol · 2010-04-10 23:47:18

ncdu is a small app perfectly suited for finding what ate your free drivespace.

fsckd · 2010-04-10 23:49:57

Umm, no, du has a -x option, ignore the --max-depth and -h options (do not use them) and pipe the output to a certain useful sorting command which has a -g option.

Edit: forgot to mention, use su or sudo, this root after all

Last edited by fsckd (2010-04-10 23:52:09)

ideogon · 2010-04-10 23:54:24

[root@bigbird /]# du -x / | sort -g | tail
468240  /usr/bin
485816  /usr/share/locale
1065732 /var/cache/pacman/pkg
1065736 /var/cache/pacman
1154892 /var/cache
1296752 /usr/lib
1337972 /var
1893528 /usr/share
4016784 /usr
5670849 /

karol · 2010-04-10 23:55:22

@fsckd
Umm, 'du -x | sort -rg | head' makes more sense, but IMHO ncdu is still a better option. Aaaand '--max-depth=1' makes du less messy, so ignoring it may be a bad idea.

fsckd · 2010-04-11 00:02:54

ideogon wrote:

[root@bigbird /]# du -x / | sort -g | tail
468240  /usr/bin
485816  /usr/share/locale
1065732 /var/cache/pacman/pkg
1065736 /var/cache/pacman
1154892 /var/cache
1296752 /usr/lib
1337972 /var
1893528 /usr/share
4016784 /usr
5670849 /

There you go. Clear your pacman cache as jowilkin suggested and plan to move your root fs to a larger partition. You've installed far too much for ~7GiB.

ideogon · 2010-04-11 00:04:37

I have previously cleared my pacman cache with pacman -Sc, but not the more destructive pacman -Scc. I just ran it again, and df -h still reports a full root file system.

Should I run the more destructive pacman -Scc, or should I look for other culprits?

ideogon · 2010-04-11 00:05:40

fsckd wrote:

ideogon wrote:
[root@bigbird /]# du -x / | sort -g | tail
468240  /usr/bin
485816  /usr/share/locale
1065732 /var/cache/pacman/pkg
1065736 /var/cache/pacman
1154892 /var/cache
1296752 /usr/lib
1337972 /var
1893528 /usr/share
4016784 /usr
5670849 /
There you go. Clear your pacman cache as jowilkin suggested and plan to move your root fs to a larger partition. You've installed far too much for ~7GiB.

Whoops! I trusted the default partition settings

karol · 2010-04-11 00:15:24

What do you have installed? /usr is 4GB but which apps take the most space?
My 'usr/share' is <1GB and the biggest offenders are:

  245.8MiB  /festival
  141.7MiB  /soundfonts
   86.9MiB  /locale
   64.1MiB  /fonts
   54.8MiB  /xnviewmp
   54.7MiB  /doc
   53.4MiB  /man
   27.5MiB  /ghostscript
   27.0MiB  /vim
   23.7MiB  /perl5
   16.1MiB  /gtk-doc
    9.4MiB  /hydrogen

In this example, uninstalling festival and removing it from cache would help a lot. I don't dabble w/ midi atm, so soundfonts can go too.
You should check what apps you have installed and act accordingly.

I too use the default settings:

Filesystem            Size  Used Avail Use% Mounted on
/dev/sda3             7.3G  3.9G  3.1G  57% /

ideogon · 2010-04-11 00:38:27

If it's any insight, I originally installed Arch about a year ago, never really updated the system, and then did a massive update (pacman -Syu) just a week or two ago. Maybe that explains the massive pacman cache (I remember having to download nearly a gigabyte of upgraded packages).

I really hate to get into the practice of uninstalling packages just to make things fit, especially with the ever-falling price of disk space. Nevertheless:

[root@bigbird /]# du /usr | sort -rg | head -n 20
4016784 /usr
1893528 /usr/share
1296752 /usr/lib
485816  /usr/share/locale
468240  /usr/bin
228520  /usr/include
189328  /usr/share/texmf-dist
164568  /usr/share/doc
157032  /usr/share/gnome
156944  /usr/share/gnome/help
145000  /usr/lib/python2.6
123348  /usr/share/texmf-dist/fonts
121452  /usr/lib/mono
105140  /usr/share/gtk-doc
104996  /usr/share/gtk-doc/html
90464   /usr/lib/mono/gac
83300   /usr/share/texmf-var
82744   /usr/lib/python2.6/site-packages
82512   /usr/share/texmf-var/web2c
68580   /usr/share/man

/usr/share alone:

[root@bigbird share]# du /usr/share | sort -rg | head
1893528 .
485816  ./locale
189328  ./texmf-dist
164568  ./doc
157032  ./gnome
156944  ./gnome/help
123348  ./texmf-dist/fonts
105140  ./gtk-doc
104996  ./gtk-doc/html
83300   ./texmf-var

So, itappears that texmf, python 2.6, and gnome are the major culprits, but I'm not about to uninstall those!

Would it perhaps make more sense to have /usr mounted on a different file system? Is it typical to have only ~ 1 GB of free space on /root (which is what I would have removing my pacman cache)?

Last edited by ideogon (2010-04-11 00:44:31)

karol · 2010-04-11 01:12:09

156944  /usr/share/gnome/help
104996  /usr/share/gtk-doc/html

Do you really need the documentation?

485816  /usr/share/locale

400MB+?? C'mon now ...

> So, itappears that texmf, python 2.6, and gnome are the major culprits, but I'm not about to uninstall those!
And where's the rest of the space? If you add texmf, python 2.6, and gnome up it doesn't say 4GB, does it?
145000 /usr/lib/python2.6
121452 /usr/lib/mono
189328 /usr/share/texmf-dist
83300 /usr/share/texmf-var
------- + the docs above ~1GB - so where did the rest go?

Could you *please* install ncdu and copy-paste the first level of /usr/share, /usr/lib etc. like I did in my previous post? We'll do this step-by-step. 7GB is a lot and you shouldn't have to purge your pacman cache just to make things fit.

Alternatively, you can use http://bbs.archlinux.org/viewtopic.php?id=73098 'bigpkg | less' and check everything >15MB.

mikesd · 2010-04-11 01:23:46

@OP: You mentioned you had a snafu with mounting in your first post. Is there any chance you dumped files into /mnt/backups when there was nothing mounted on it? If so you may have a bunch of files hidden there. Might be worth unmounting the backup partition and checking the directory to rule it out.

@karol: ncdu is nice. Hadn't come across that utility before.

ideogon · 2010-04-11 01:26:14

karol wrote:

156944  /usr/share/gnome/help
104996  /usr/share/gtk-doc/html
Do you really need the documentation?
485816  /usr/share/locale
400MB+?? C'mon now ...

Jeez, what's your problem?

karol wrote:

> So, itappears that texmf, python 2.6, and gnome are the major culprits, but I'm not about to uninstall those!
And where's the rest of the space? If you add texmf, python 2.6, and gnome up it doesn't say 4GB, does it?
145000 /usr/lib/python2.6
121452 /usr/lib/mono
189328 /usr/share/texmf-dist
83300 /usr/share/texmf-var
------- + the docs above ~1GB - so where did the rest go?
Could you *please* install ncdu and copy-paste the first level of /usr/share, /usr/lib etc. like I did in my previous post? We'll do this step-by-step. 7GB is a lot and you shouldn't have to purge your pacman cache just to make things fit.
Alternatively, you can use http://bbs.archlinux.org/viewtopic.php?id=73098 'bigpkg | less' and check everything >15MB.

Actually, I couldn't install ncdu without first clearing some space, which I did with pacman -Scc, for better or worse, but thanks for the recommendation anyway. I'll try it out now.
*Marking this thread as [SOLVED]*

ideogon · 2010-04-11 01:28:34

mikesd wrote:

@OP: You mentioned you had a snafu with mounting in your first post. Is there any chance you dumped files into /mnt/backups when there was nothing mounted on it? If so you may have a bunch of files hidden there. Might be worth unmounting the backup partition and checking the directory to rule it out.

No, I see nothing recent in that directory (haven't used it in a while anyway). I suppose it was a simple matter of a bloated pacman cache.

karol · 2010-04-11 01:29:03

[root@bigbird /]# df -h
Filesystem Size Used Avail Use% Mounted on
/dev/sdb3 7.3G 7.3G 0 100% /

[root@bigbird /]# du --max-depth=1 -h
3.9G /usr
1.3G /var
+ a bunch of small stuff

7,3 != (3,9 + 1,3)
I'm going to bed now, happy hunting :-)

ideogon · 2010-04-11 01:43:09

Finally, with ncdu (after clearing the pacman cache, so /var is somewhat small again). There is no new insight on the 2.1 GB that I can't account for:

[root@bigbird usr]# ncdu -x /

--- / --------------------------------------------------------------------
    3.8GiB  /usr                                                          
  266.7MiB  /var
  145.2MiB  /lib
  126.5MiB  /opt
   13.8MiB  /sbin
   13.7MiB  /etc
    5.2MiB  /bin
    4.0MiB  /root
  128.0kiB  /mnt
   56.0kiB  /tmp
e  16.0kiB  /lost+found
   16.0kiB  /media
   12.0kiB  /srv
    4.0kiB  /lib64
>   0.0  B  /boot
>   0.0  B  /dev
>   0.0  B  /home
>   0.0  B  /proc
>   0.0  B  /sys

Last edited by ideogon (2010-04-11 01:45:45)

michy99 · 2010-04-11 14:24:37

ideogon wrote:

mikesd wrote:
@OP: You mentioned you had a snafu with mounting in your first post. Is there any chance you dumped files into /mnt/backups when there was nothing mounted on it? If so you may have a bunch of files hidden there. Might be worth unmounting the backup partition and checking the directory to rule it out.
No, I see nothing recent in that directory (haven't used it in a while anyway). I suppose it was a simple matter of a bloated pacman cache.

I think you may have misunderstood. If you unmount /mnt/backups you should see no files at all when you list the directory. Any files which are there will be masked when the partition is mounted.

ideogon · 2010-04-13 23:25:39

michy99 wrote:

ideogon wrote:
mikesd wrote:
@OP: You mentioned you had a snafu with mounting in your first post. Is there any chance you dumped files into /mnt/backups when there was nothing mounted on it? If so you may have a bunch of files hidden there. Might be worth unmounting the backup partition and checking the directory to rule it out.
No, I see nothing recent in that directory (haven't used it in a while anyway). I suppose it was a simple matter of a bloated pacman cache.
I think you may have misunderstood. If you unmount /mnt/backups you should see no files at all when you list the directory. Any files which are there will be masked when the partition is mounted.

You're right. I misunderstood you, and having finally tried your suggestion I find that you were absolutely right! Thank you!! This was so perplexing me how I could not account for such a significant amount of space on such a meager partition, and this buys me a lot more time to strategize what to do with this file system

I unmounted /mnt/backups and found that 1.7 GB of files were still left over, which I was able to remove. Now my file sizes add up, except only ~ 0.5G (of a 7.8G file system), which I guess is probably the 5% that ext3 is "reserving for root"?

Anyway, I like to idea of sparing some space for being able to log into root in case I fill up my system again (it certainly came in handy this time), but how much can I reasonably reduce that amount to?

trusktr · 2010-05-26 19:04:16

How exactly do you unmount something like /mnt/backups?

I'm also having this problem i believe because i have 5 GB used in root filesystem, but usr and var add up to 3.6 GB.

I think i use the unmount command somehow?

Thanks!

Last edited by trusktr (2010-05-26 19:10:13)

tomk · 2010-05-27 05:52:06

trusktr - the command is umount, and I don't see any problem described in your post. I recommend you start a new thread, with a more complete description of whatever issue is bothering you.

Acecero · 2010-05-27 06:14:12

trusktr wrote:

How exactly do you unmount something like /mnt/backups?
I'm also having this problem i believe because i have 5 GB used in root filesystem, but usr and var add up to 3.6 GB.
I think i use the unmount command somehow?
Thanks!

Another simple solution, add this to bashrc file.

alias unmount='umount'

Arch Linux

#1 2010-04-10 23:33:55

[SOLVED] Full root file system and cannot pinpoint the culprit

#2 2010-04-10 23:35:02

Re: [SOLVED] Full root file system and cannot pinpoint the culprit

#3 2010-04-10 23:46:50

Re: [SOLVED] Full root file system and cannot pinpoint the culprit

#4 2010-04-10 23:47:18

Re: [SOLVED] Full root file system and cannot pinpoint the culprit

#5 2010-04-10 23:49:57

Re: [SOLVED] Full root file system and cannot pinpoint the culprit

#6 2010-04-10 23:54:24

Re: [SOLVED] Full root file system and cannot pinpoint the culprit

#7 2010-04-10 23:55:22

Re: [SOLVED] Full root file system and cannot pinpoint the culprit

#8 2010-04-11 00:02:54

Re: [SOLVED] Full root file system and cannot pinpoint the culprit

#9 2010-04-11 00:04:37

Re: [SOLVED] Full root file system and cannot pinpoint the culprit

#10 2010-04-11 00:05:40

Re: [SOLVED] Full root file system and cannot pinpoint the culprit

#11 2010-04-11 00:15:24

Re: [SOLVED] Full root file system and cannot pinpoint the culprit

#12 2010-04-11 00:38:27

Re: [SOLVED] Full root file system and cannot pinpoint the culprit

#13 2010-04-11 01:12:09

Re: [SOLVED] Full root file system and cannot pinpoint the culprit

#14 2010-04-11 01:23:46

Re: [SOLVED] Full root file system and cannot pinpoint the culprit

#15 2010-04-11 01:26:14

Re: [SOLVED] Full root file system and cannot pinpoint the culprit

#16 2010-04-11 01:28:34

Re: [SOLVED] Full root file system and cannot pinpoint the culprit

#17 2010-04-11 01:29:03

Re: [SOLVED] Full root file system and cannot pinpoint the culprit

#18 2010-04-11 01:43:09

Re: [SOLVED] Full root file system and cannot pinpoint the culprit

#19 2010-04-11 14:24:37

Re: [SOLVED] Full root file system and cannot pinpoint the culprit

#20 2010-04-13 23:25:39

Re: [SOLVED] Full root file system and cannot pinpoint the culprit

#21 2010-05-26 19:04:16

Re: [SOLVED] Full root file system and cannot pinpoint the culprit

#22 2010-05-27 05:52:06

Re: [SOLVED] Full root file system and cannot pinpoint the culprit

#23 2010-05-27 06:14:12

Re: [SOLVED] Full root file system and cannot pinpoint the culprit

Board footer