[solved] /var mysteriously filling up

tazmanian · 2013-10-10 20:06:24

My /var partition keeps filling up. I have already resized the partition several times, and it fills up in a matter of minutes. I don't understand where the space is going. Below is the relevant excerpt from df:

/dev/mapper/var        754M  716M     0 100% /var
/dev/mapper/var.cache  1.7G  1.5G   70M  96% /var/cache

I have /var/cache on a separate partition, so the pacman cache isn't an issue. df reports 716 MB used. My first suspicion was logging, but journalctl reports using a modest 5 MB:

# journalctl --disk-usage
Journals take up 5.2M on disk.

The weird thing is, du only shows 363 MB in use, which is less than half the partition:

# du -xs /var
363M    /var

Anyone have ideas on how to debug this?

Last edited by tazmanian (2013-10-12 07:29:04)

WorMzy · 2013-10-10 20:23:54

tazmanian wrote:

My /var partition keeps filling up. I have already resized the partition several times, and it fills up in a matter of minutes. I don't understand where the space is going. Below is the relevant excerpt from df:
/dev/mapper/var        754M  716M     0 100% /var
/dev/mapper/var.cache  1.7G  1.5G   70M  96% /var/cache
I have /var/cache on a separate partition, so the pacman cache isn't an issue. df reports 716 MB used. My first suspicion was logging, but journalctl reports using a modest 5 MB:
# journalctl --disk-usage
Journals take up 5.2M on disk.
The weird thing is, du only shows 363 MB in use, which is less than half the partition:
# du -xs /var
363M    /var
Anyone have ideas on how to debug this?

community/ncdu.

Last edited by WorMzy (2013-10-10 20:24:48)

hiciu · 2013-10-10 20:56:07

tazmanian wrote:

The weird thing is, du only shows 363 MB in use, which is less than half the partition:
# du -xs /var
363M    /var
Anyone have ideas on how to debug this?

In Unix you can do something like this:

#define FILE "/path/to/file"
int fd = open(FILE, O_WRONLY | O_CREAT | O_TRUNC);
remove(FILE);
char buf[1024] = {0};
while (0) {
   write(fd, buf, sizeof(buf));
}

This will create file, delete it and write to deleted file. df will report space on disk is taken, but du and ncdu report that space is empty. This technique is used when program needs to use some space as a buffer and it don't need that data in memory.

You can check for open but deleted file with `sudo lsof | grep deleted`.

Last edited by hiciu (2013-10-10 20:56:24)

graysky · 2013-10-10 21:43:39

Use ncdu:

% sudo ncdu -x /var
ncdu 1.10 ~ Use the arrow keys to navigate, press ? for help                                                                    
--- /var -----------------------------------------------------------------------------------------------------------------------
    1.1GiB [##########] /cache                                                                                                  
   32.8MiB [          ] /lib
    1.2MiB [          ] /spool
    8.0KiB [          ] /db
@   4.0KiB [          ]  lock
@   4.0KiB [          ]  mail
@   4.0KiB [          ]  log
@   4.0KiB [          ]  run
    0.0  B [          ] /tmp
e   0.0  B [          ] /opt
e   0.0  B [          ] /lost+found
e   0.0  B [          ] /local
e   0.0  B [          ] /games
e   0.0  B [          ] /empty

tazmanian · 2013-10-12 07:27:55

hiciu wrote:

You can check for open but deleted file with `sudo lsof | grep deleted`.

Thanks, this helped tremendously. I did think to check lsof for open deleted files, but didn't notice the "(deleted)" tag in the output. Looks like krunner is the culprit.

Arch Linux

#1 2013-10-10 20:06:24

[solved] /var mysteriously filling up

#2 2013-10-10 20:23:54

Re: [solved] /var mysteriously filling up

#3 2013-10-10 20:56:07

Re: [solved] /var mysteriously filling up

#4 2013-10-10 21:43:39

Re: [solved] /var mysteriously filling up

#5 2013-10-12 07:27:55

Re: [solved] /var mysteriously filling up

Board footer