Why an empty directory take so much space?

kokoko3k · 2012-02-07 11:04:44

Here is the problem:

[kokoko3k@Gozer ~]$ mkdir /tmp/testdir
[kokoko3k@Gozer ~]$ ls -la /tmp/|grep testdir
drwxr-xr-x   2 kokoko3k users   4096  7 feb 12.01 testdir

We have 4K, ok.

[kokoko3k@Gozer ~]$ cd /tmp/testdir/
[kokoko3k@Gozer testdir]$ while true ; do touch $RANDOM  ; ls -1 |wc -l ; done
1
2
.
.
600
^C
[kokoko3k@Gozer testdir]$ ls -la /tmp/|grep testdir
drwxr-xr-x   2 kokoko3k users  20480  7 feb 12.02 testdir

20K makes sense, it is full of files, but look here:

[kokoko3k@Gozer testdir]$ rm /tmp/testdir/*
[kokoko3k@Gozer testdir]$ ls -la /tmp/|grep testdir
drwxr-xr-x   2 kokoko3k users  20480  7 feb 12.03 testdir

...now the directory is empty and still take so much space, i dropped caches but nothing changes, the only way to free it is to rmdir and mkdir again.

filesystem is ext4 and mount options are: defaults,barrier=0
Is this... normal?

karol · 2012-02-07 15:55:35

'm /tmp/testdir/*' only removes files, not subdirectories (that may contain files).
There may be also some hidden files & directories - the ones that begin with a dot, like .X11-unix/

I duggest using e.g. du to check the used space and not ls + grep

[karol@black ~]$ sudo du -sh /tmp/
192K	/tmp/

Check, what's inside, it may simply be not empty (I have plenty in my /tmp).

kokoko3k · 2012-02-07 16:52:54

I can clearly "rmdir /tmp/testdir", that means that the directory is empty, or rmdir reports an error, but here it is the log with du, this is on another box:

koko@slimer ~  ls -la /tmp/testdir
ls: cannot access /tmp/testdir: No such file or directory

koko@slimer ~mkdir /tmp/testdir
koko@slimer /tmp/testdir $ sudo du -sh /tmp/testdir
4,0K    /tmp/testdir

koko@slimer ~cd /tmp/testdir
koko@slimer /tmp/testdir $ while true ; do touch $RANDOM  ; ls -1 |wc -l ; done
1
2
3
....
648
649
650
651
^C
koko@slimer /tmp/testdir $ sudo du -sh /tmp/testdir
20K     /tmp/testdir

koko@slimer /tmp/testdir $ cd ..
koko@slimer /tmp/testdir $ rm /tmp/testdir/*

koko@slimer /tmp/testdir $ sudo du -sh /tmp/testdir
20K     /tmp/testdir

koko@slimer /tmp $ ls /tmp/testdir/ -la
totale 44
drwxr-xr-x  2 koko users 20480  7 feb 17.43 .
drwxrwxrwt 16 root root  20480  7 feb 17.49 ..

koko@slimer /tmp $ rmdir /tmp/testdir
koko@slimer /tmp $ mkdir /tmp/testdir
koko@slimer /tmp/testdir $ sudo du -sh /tmp/testdir
4,0K     /tmp/testdir

Still, dir size become bigger when a lot of files are put into it, but never gets smaller, even after deleting all of the files.
Is it a bug?

-EDIT-
Seems to be expected...
http://serverfault.com/questions/294301 … lock-count

Last edited by kokoko3k (2012-02-07 16:58:06)

bernarcher · 2012-02-07 16:57:24

Welcome to the wonders of Unix!

You filled the testdir with a bunch of empty files. These were included in the directory structure as mere nodes. When you remove a file from a directory, only its contents vanish. The node remains in the directory structure, only being marked as free. Thus if you delete an empty file, nothing really changes space-wise.

karol · 2012-02-07 17:00:32

The inodes contain metadata, so they exist even for empty files.

@bernarcher
Are the nodes removed e.g. upon reboot or are they permanent?

kokoko3k · 2012-02-07 17:02:02

bernarcher wrote:

Welcome to the wonders of Unix!
You filled the testdir with a bunch of empty files. These were included in the directory structure as mere nodes. When you remove a file from a directory, only its contents vanish. The node remains in the directory structure, only being marked as free. Thus if you delete an empty file, nothing really changes space-wise.

Not that i'm worried about a bunch of bytes, but are you saying that the value reported by du is wrong in the sense that that space is free for other data to be written even in different directories or that space is effectively 'stolen' by the (undeleted?) inodes?

Last edited by kokoko3k (2012-02-07 17:03:06)

karol · 2012-02-07 17:29:38

I'm using tmpfs for /tmp so it gets cleaned on reboot ;P

I know you can run out of free inodes even if you have still free space left i.e. 'df -hi' will show IUse% at 100%.

I think MikeyB is right, and the space is not freed (quote from the serverfault link you posted)

When deletes happen and dir size drops so that you could free up a block, all that management would need to happen before you could actually do so. Why bother to save a few KB? Odds are you'll have to expand it later anyways.

bernarcher · 2012-02-07 17:47:57

Note: In Unix everything is a file.

The nodes list of a directory will be kept until you rmdir it (it is the main content of this directory file). Thus an empty directory always uses some amount of space. To get rid of any extra space resulting from deleted files you have to rmdir it and the mkdir again.

If ever you want to keep some files when the nodes list became too large you will have to create another directory, copy the contents there, rename the directories accordingly and remove the old one.

Blµb · 2012-02-15 10:13:56

I'm not sure, but shouldn't the space used by the directory after deleting the files be reduced later when the space is actually required? Ext* keep the files around if you delete them, they just don't show them, they are kind of "marked as deleted". (Otherwise you wouldn't be able to view recently deleted files with debugfs)

karol · 2012-02-15 16:22:53

Blµb wrote:

I'm not sure, but shouldn't the space used by the directory after deleting the files be reduced later when the space is actually required? Ext* keep the files around if you delete them, they just don't show them, they are kind of "marked as deleted". (Otherwise you wouldn't be able to view recently deleted files with debugfs)

That's another story. From what I gather, we're talking about inode metadata, not the data itself. The 'marked as removed' files are being counted as free space when you check disk usage.

Arch Linux

#1 2012-02-07 11:04:44

Why an empty directory take so much space?

#2 2012-02-07 15:55:35

Re: Why an empty directory take so much space?

#3 2012-02-07 16:52:54

Re: Why an empty directory take so much space?

#4 2012-02-07 16:57:24

Re: Why an empty directory take so much space?

#5 2012-02-07 17:00:32

Re: Why an empty directory take so much space?

#6 2012-02-07 17:02:02

Re: Why an empty directory take so much space?

#7 2012-02-07 17:29:38

Re: Why an empty directory take so much space?

#8 2012-02-07 17:47:57

Re: Why an empty directory take so much space?

#9 2012-02-15 10:13:56

Re: Why an empty directory take so much space?

#10 2012-02-15 16:22:53

Re: Why an empty directory take so much space?

Board footer