Arch Linux

warped

Member

Registered: 2009-08-19

Posts: 41

slowly vanishing root space after weeks of uptime :-(

Hi *

I have server based on ArchLinux (kernel 3.3.7, AMD GA880G mono with on-board radeon).
Server has minimal set of running software (only daemons, no gfx shell).
It runs stable and I don't observe any measurable resources leaks in "top" or "ps aux".
I have issue however with slowly vanishing free space on root partition.
Root part has 5G with 3,7G of software. After reboot free space is about 1,3G.
After 20 days of uptime it has 248M of free space.
I checked where free space was allocated and it was mainly in:

1. 528MB in "/sys/devices/pci0000:00/0000:00:00.0" 

00:00.0 0600: 1022:9601
        Subsystem: 1022:9601
        Flags: bus master, 66MHz, medium devsel, latency 32
        Memory at <ignored> (64-bit, non-prefetchable)
        Capabilities: [c4] HyperTransport: Slave or Primary Interface
        Capabilities: [54] HyperTransport: UnitID Clumping
        Capabilities: [40] HyperTransport: Retry Mode
        Capabilities: [9c] HyperTransport: #1a
        Capabilities: [f8] HyperTransport: #1c

root occupation is following for this dir is:

-rw-r--r-- 1 root root      4096 Jun 26 21:54 broken_parity_status
-r--r--r-- 1 root root      4096 Jun 26 10:19 class
-rw-r--r-- 1 root root       256 Jun 26 10:19 config
-r--r--r-- 1 root root      4096 Jun 26 21:54 consistent_dma_mask_bits
-r--r--r-- 1 root root      4096 Jun 26 10:19 device
-r--r--r-- 1 root root      4096 Jun 26 21:54 dma_mask_bits
-rw------- 1 root root      4096 Jun 26 21:54 enable
-r--r--r-- 1 root root      4096 Jun 26 10:19 irq
-r--r--r-- 1 root root      4096 Jun 26 21:54 local_cpulist
-r--r--r-- 1 root root      4096 Jun 26 21:54 local_cpus
-r--r--r-- 1 root root      4096 Jun 26 21:54 modalias
-rw-r--r-- 1 root root      4096 Jun 26 21:54 msi_bus
-r--r--r-- 1 root root      4096 Jun 26 21:54 numa_node
drwxr-xr-x 2 root root         0 Jun 26 21:54 power
--w--w---- 1 root root      4096 Jun 26 21:54 remove
--w--w---- 1 root root      4096 Jun 26 21:54 rescan
-r--r--r-- 1 root root      4096 Jun 26 10:19 resource
-rw------- 1 root root 536870912 Jun 26 21:54 resource3
lrwxrwxrwx 1 root root         0 Jun 26 21:54 subsystem -> ../../../bus/pci
-r--r--r-- 1 root root      4096 Jun 26 21:54 subsystem_device
-r--r--r-- 1 root root      4096 Jun 26 21:54 subsystem_vendor
-rw-r--r-- 1 root root      4096 Jun 26 10:19 uevent
-r--r--r-- 1 root root      4096 Jun 26 10:19 vendor

   

2. 262MB in "/sys/devices/pci0000:00/0000:00:01.0/0000:01:05.0"

01:05.0 0300: 1002:9715 (prog-if 00 [VGA controller])
        Subsystem: 1458:d000
        Flags: bus master, fast devsel, latency 0, IRQ 18
        Memory at d8000000 (32-bit, prefetchable) [size=128M]
        I/O ports at de00 [size=256]
        Memory at fdce0000 (32-bit, non-prefetchable) [size=64K]
        Memory at fdb00000 (32-bit, non-prefetchable) [size=1M]
        Expansion ROM at <unassigned> [disabled]
        Capabilities: [50] Power Management version 3
        Capabilities: [a0] MSI: Enable- Count=1/1 Maskable- 64bit+
        Kernel driver in use: radeon

root occupation for this device is following:

-r--r--r-- 1 root root      4096 Jun 26 22:09 boot_vga
-rw-r--r-- 1 root root      4096 Jun 26 22:09 broken_parity_status
-r--r--r-- 1 root root      4096 Jun 26 22:09 class
-rw-r--r-- 1 root root       256 Jun 26 22:09 config
-r--r--r-- 1 root root      4096 Jun 26 22:09 consistent_dma_mask_bits
-r--r--r-- 1 root root      4096 Jun 26 22:09 device
-r--r--r-- 1 root root      4096 Jun 26 22:09 dma_mask_bits
lrwxrwxrwx 1 root root         0 Jun 26 22:09 driver -> ../../../../bus/pci/drivers/radeon
drwxr-xr-x 4 root root         0 Jun 26 22:09 drm
-rw------- 1 root root      4096 Jun 26 22:09 enable
drwxr-xr-x 3 root root         0 Jun 26 22:09 graphics
drwxr-xr-x 3 root root         0 Jun 26 10:18 i2c-1
drwxr-xr-x 3 root root         0 Jun 26 10:18 i2c-2
drwxr-xr-x 3 root root         0 Jun 26 10:18 i2c-3
drwxr-xr-x 3 root root         0 Jun 26 10:18 i2c-4
drwxr-xr-x 3 root root         0 Jun 26 10:18 i2c-5
-r--r--r-- 1 root root      4096 Jun 26 22:09 irq
-r--r--r-- 1 root root      4096 Jun 26 22:09 local_cpulist
-r--r--r-- 1 root root      4096 Jun 26 22:09 local_cpus
-r--r--r-- 1 root root      4096 Jun 26 22:09 modalias
-rw-r--r-- 1 root root      4096 Jun 26 22:09 msi_bus
-r--r--r-- 1 root root      4096 Jun 26 22:09 numa_node
drwxr-xr-x 2 root root         0 Jun 26 22:09 power
-rw-r--r-- 1 root root      4096 Jun 26 22:09 power_method
-rw-r--r-- 1 root root      4096 Jun 26 22:09 power_profile
--w--w---- 1 root root      4096 Jun 26 22:09 remove
--w--w---- 1 root root      4096 Jun 26 22:09 rescan
--w------- 1 root root      4096 Jun 26 22:09 reset
-r--r--r-- 1 root root      4096 Jun 26 22:09 resource
-rw------- 1 root root 134217728 Jun 26 22:09 resource0
-rw------- 1 root root 134217728 Jun 26 22:09 resource0_wc
-rw------- 1 root root       256 Jun 26 22:09 resource1
-rw------- 1 root root     65536 Jun 26 22:09 resource2
-rw------- 1 root root   1048576 Jun 26 22:09 resource5
-rw------- 1 root root    131072 Jun 26 22:09 rom
lrwxrwxrwx 1 root root         0 Jun 26 22:09 subsystem -> ../../../../bus/pci
-r--r--r-- 1 root root      4096 Jun 26 22:09 subsystem_device
-r--r--r-- 1 root root      4096 Jun 26 22:09 subsystem_vendor
-rw-r--r-- 1 root root      4096 Jun 26 22:09 uevent
-r--r--r-- 1 root root      4096 Jun 26 22:09 vendor

For me it looks like kind of resources leak.
Can anybody has idea how to resolve this issue ?

-br

Inxsible

Forum Fellow

From: Chicago

Registered: 2008-06-09

Posts: 9,183

Re: slowly vanishing root space after weeks of uptime :-(

Check your log file sizes. Maybe logrotate is not working up to par?
If you haven't installed it, then

# pacman -S logrotate

should fix that.

---

Forum Rules

There's no such thing as a stupid question, but there sure are a lot of inquisitive idiots !

fukawi2

Ex-Administratorino

From: .vic.au

Registered: 2007-09-28

Posts: 6,237

Website

Re: slowly vanishing root space after weeks of uptime :-(

That doesn't make sense (to me)... /sys and /dev are separate mounts, and also they are only psuedo-filesystem... They shouldn't be affecting the actual disk usage of /

How did you determine that they are the culprits? I would use `du -hx --max-depth=1` starting at / and working my way down through the results.

For example, on my work PC:

ict-phs-pc / # du -hx --max-depth=1
7.4M	./etc
12K	./media
16K	./lost+found
4.0K	./bonnie
3.8G	./usr
4.0K	./lib64
12K	./srv
8.0K	./opt
3.7M	./bin
100M	./lib
10M	./sbin
8.0K	./mnt
516K	./root
3.9G	.

(Note the lack of /dev or /sys due to the -x flag to du)

---

Are you familiar with our Forum Rules, and How To Ask Questions The Smart Way?

BlueHackers // fscanary // resticctl

karol

Archivist

Registered: 2009-05-06

Posts: 25,440

Re: slowly vanishing root space after weeks of uptime :-(

I suggest using ncdu or other similar tool https://wiki.archlinux.org/index.php/Co … y_programs instead of running 'du'
"starting at / and working my way down through the results".

warped

Member

Registered: 2009-08-19

Posts: 41

Re: slowly vanishing root space after weeks of uptime :-(

Oh,
Sorry for long  silence. I have to wait some time till my / usage grows enough....
now I don't get it all :-(

ncdu reports

    1.8GiB [          ] /usr
    1.4GiB [          ] /var
   48.9MiB [          ] /lib
   20.3MiB [          ] /home
   17.7MiB [          ] /boot
   12.8MiB [          ] /sbin
    8.5MiB [          ] /root
    7.7MiB [          ] /etc
    4.1MiB [          ] /bin
  940.0KiB [          ] /run
   56.0KiB [          ] /tmp
   12.0KiB [          ] /srv
    4.0KiB [          ] /lib64
e   4.0KiB [          ] /opt
e   4.0KiB [          ] /mnt
e   4.0KiB [          ] /lost+found
.   0.0  B [          ] /proc
    0.0  B [          ] /sys
    0.0  B [          ] /dev

df -h however reports:

rootfs          4.8G  4.0G  601M  88% /
/dev            1.9G     0  1.9G   0% /dev
run             1.9G  944K  1.9G   1% /run
/dev/sda1       4.8G  4.0G  601M  88% /
tmpfs           1.9G     0  1.9G   0% /dev/shm
tmpfs           1.9G     0  1.9G   0% /sys/fs/cgroup
tmpfs           1.0G     0  1.0G   0% /media
tmpfs           1.0G   56K  1.0G   1% /tmp
/dev/sdb1       2.7T  2.6T   14G 100% /myth
/dev/sda4       1.8T  1.7T   30G  99% /myth/video/HD
/dev/sda2       4.8G  3.2G  1.4G  70% /media/mirror.vol

Brief math says shows:
ncdu: root has 3,2+ G
df -h: root has 4G

so which one is truth ?

WorMzy

Administrator

From: Scotland

Registered: 2010-06-16

Posts: 13,579

Website

Re: slowly vanishing root space after weeks of uptime :-(

Potentially both.

If you have any files hidden beneath the other mounted filesystems, then they'll be accounted for by df, but not du (or ncdu).

For example, if you have 800MB's worth of files on your root partition's /sys folder (which is unlikely, but your first post listed two problem directories which total ~800MB, and that's amazingly the same as the discrepancy between df and du), but you have sysfs mounted over the top of it, then du won't be able to see it, but df will.

I suggest you temporarily mount --bind / onto /mnt, then run ncdu on /mnt. That'll give you a more accurate reading.

---

Sakura:-
Mobo: MSI MAG X570S TORPEDO MAX // Processor: AMD Ryzen 9 5950X @4.9GHz // GFX: AMD Radeon RX 5700 XT // RAM: 32GB (4x 8GB) Corsair DDR4 (@ 3000MHz) // Storage: 1x 3TB HDD, 6x 1TB SSD, 2x 120GB SSD, 1x 275GB M2 SSD

Making lemonade from lemons since 2015.

fukawi2

Ex-Administratorino

From: .vic.au

Registered: 2007-09-28

Posts: 6,237

Website

Re: slowly vanishing root space after weeks of uptime :-(

Also, it could be deleted files that are still open;

lsof -n | grep deleted

Part of the lsof output is size; see what that turns up for you.

---

Are you familiar with our Forum Rules, and How To Ask Questions The Smart Way?

BlueHackers // fscanary // resticctl