[Solved] /proc, /sys, Various systemd Directories: No Space?

ANOKNUSA · 2013-07-24 02:48:11

Howdy-ha, folks. So out of sheer curiosity, I decided that I'd install GNOME and KDE to see how Steam performed under their conditions. While installing KDE (and before installing GNOME), I receieved an error about no disk space remaining on the device. Incredulous, I ran "dfc" and saw plenty of space left on the drives. "dfc -a" told me this, however:

FILESYSTEM       (=) USED      FREE (-) %USED AVAILABLE     TOTAL MOUNTED ON 
rootfs           [==========----------]   48%      4.2G      8.0G /
proc             [====================]  100%        0B        0B /proc
sys              [====================]  100%        0B        0B /sys
dev              [--------------------]    0%      3.8G      3.8G /dev
run              [=-------------------]    0%      3.8G      3.8G /run
/dev/sdb2        [==========----------]   48%      4.2G      8.0G /
securityfs       [====================]  100%        0B        0B /sys/kernel/security
tmpfs            [--------------------]    0%      3.8G      3.8G /dev/shm
devpts           [====================]  100%        0B        0B /dev/pts
tmpfs            [--------------------]    0%      3.8G      3.8G /sys/fs/cgroup
cgroup           [====================]  100%        0B        0B /sys/fs/cgroup/systemd
pstore           [====================]  100%        0B        0B /sys/fs/pstore
efivarfs         [====================]  100%        0B        0B +ys/firmware/efi/efivars
cgroup           [====================]  100%        0B        0B /sys/fs/cgroup/cpuset
cgroup           [====================]  100%        0B        0B /sys/fs/cgroup/memory
cgroup           [====================]  100%        0B        0B /sys/fs/cgroup/devices
cgroup           [====================]  100%        0B        0B /sys/fs/cgroup/freezer
cgroup           [====================]  100%        0B        0B /sys/fs/cgroup/net_cls
cgroup           [====================]  100%        0B        0B /sys/fs/cgroup/blkio
cgroup           [====================]  100%        0B        0B /sys/fs/cgroup/bfqio
systemd-1        [====================]  100%        0B        0B /proc/sys/fs/binfmt_misc
systemd-1        [=-------------------]    4%    488.3M    511.0M /boot
mqueue           [====================]  100%        0B        0B /dev/mqueue
hugetlbfs        [====================]  100%        0B        0B /dev/hugepages
debugfs          [====================]  100%        0B        0B /sys/kernel/debug
configfs         [====================]  100%        0B        0B /sys/kernel/config
none             [=-------------------]    0%      3.8G      3.8G /tmp
/dev/sdb1        [=-------------------]    4%    488.3M    511.0M /boot
/dev/mapper/home [==========----------]   49%     14.6G     28.8G /home
binfmt_misc      [====================]  100%        0B        0B /proc/sys/fs/binfmt_misc

Well, that's not right. I'm not even sure what the problem is here, so Im not sure how to begin troubleshooting it. A reboot is about as far as I got, and it didn't help. Incidentally, my pacman databases seem to be screwed up as well, as the installation partially finished before spitting out the error:

$ testdb
'oxygen-icons-4.10.5-1': description file is missing
'oxygen-icons-4.10.5-1': file list is missing

So: What did I screw up this time? Are there any systemd docs in particular I should look into? At the moment my system seems to be running alright, as I was able to log in and post this, at least. I can look into the pacman docs/wiki page for clearing up the database issue when I get a chance. Thanks in advance.

Last edited by ANOKNUSA (2013-07-28 19:28:57)

karol · 2013-07-24 03:22:56

What's the output of 'df -hi'?

I get

$ dfc -a
FILESYSTEM  (=) USED      FREE (-) %USED AVAILABLE     TOTAL MOUNTED ON 
rootfs      [==========----------]   48%      3.7G      7.1G /
proc        [====================]  100%        0B        0B /proc
sysfs       [====================]  100%        0B        0B /sys
devtmpfs    [--------------------]    0%    496.2M    496.2M /dev
securityfs  [====================]  100%        0B        0B /sys/kernel/security
tmpfs       [--------------------]    0%    498.8M    498.8M /dev/shm
devpts      [====================]  100%        0B        0B /dev/pts
tmpfs       [=-------------------]    0%    498.5M    498.8M /run
tmpfs       [--------------------]    0%    498.8M    498.8M /sys/fs/cgroup
cgroup      [====================]  100%        0B        0B /sys/fs/cgroup/systemd
pstore      [====================]  100%        0B        0B /sys/fs/pstore
cgroup      [====================]  100%        0B        0B /sys/fs/cgroup/cpuset
cgroup      [====================]  100%        0B        0B +s/fs/cgroup/cpu,cpuacct
cgroup      [====================]  100%        0B        0B /sys/fs/cgroup/memory
cgroup      [====================]  100%        0B        0B /sys/fs/cgroup/devices
cgroup      [====================]  100%        0B        0B /sys/fs/cgroup/freezer
cgroup      [====================]  100%        0B        0B /sys/fs/cgroup/net_cls
cgroup      [====================]  100%        0B        0B /sys/fs/cgroup/blkio
/dev/sda3   [==========----------]   48%      3.7G      7.1G /
systemd-1   [====================]  100%        0B        0B /proc/sys/fs/binfmt_misc
mqueue      [====================]  100%        0B        0B /dev/mqueue
hugetlbfs   [====================]  100%        0B        0B /dev/hugepages
debugfs     [====================]  100%        0B        0B /sys/kernel/debug
configfs    [====================]  100%        0B        0B /sys/kernel/config
tmpfs       [=-------------------]    0%    498.7M    498.8M /tmp
/dev/sda4   [===============-----]   73%      7.9G     29.0G /home
binfmt_misc [====================]  100%        0B        0B /proc/sys/fs/binfmt_misc

and everything works fine.

/proc is a pseudo-filesystem, it doesn't occupy space and it's recreated on boot so don't bother removing anything from /proc.

ewaller · 2013-07-24 03:38:52

... in fact, both /proc and /sys are the kernel's way of communicating with user space. when you 'read' a file in those directories, a function inside the kernel serves those data to user space. When you 'write' to one of those files, a function in the kernel read the data and do something with it -- turn on/off NAT forwarding, adjust backlight brightness, etc. Read the files can tell you about memory, temperatures, error rates on the Ethernet port, etc... The point is, they are not files, they just look that way to user space.

ANOKNUSA · 2013-07-24 05:59:55

karol wrote:

/proc is a pseudo-filesystem, it doesn't occupy space and it's recreated on boot so don't bother removing anything from /proc.

ewaller wrote:

... in fact, both /proc and /sys are the kernel's way of communicating with user space.

I knew their contents were temporary, but didn't realize they didn't in fact exist; since they store vital system/session data, I figured they were more tangible then that. Anyhoo, here's what karol asked for

Filesystem       Inodes IUsed IFree IUse% Mounted on
/dev/sdb2             0     0     0     - /
dev                982K   413  982K    1% /dev
run                983K   496  983K    1% /run
tmpfs              983K     1  983K    1% /dev/shm
tmpfs              983K     9  983K    1% /sys/fs/cgroup
/dev/sdb1             0     0     0     - /boot
none               983K    17  983K    1% /tmp
/dev/mapper/home      0     0     0     - /home

I recreated this by trying to install GNOME and got this:

error: could not extract usr/lib/libspandsp.so.2 (Can't create '/usr/lib/libspandsp.so.2')
error: could not extract usr/lib/pkgconfig/spandsp.pc (Can't create '/usr/lib/pkgconfig/spandsp.pc')
error: problem occurred while installing spandsp
error: could not open file /var/lib/pacman/local/spandsp-0.0.6pre21-1/desc: No space left on device
error: could not update database entry spandsp-0.0.6pre21-1
error: could not commit transaction
error: failed to commit transaction (transaction aborted)
Errors occurred, no packages were upgraded.

My first inclination after seeing "not enough space" was to clear the pacman cache and re-sync the mirrors (stupid knee-jerk reaction); now, sync gives me this:

$ pacman -Syy
error: failed to update core (unable to lock database)
error: could not open file /var/lib/pacman/sync/extra.db.part: No space left on device
error: failed to update extra (failed to retrieve some files)
error: failed to update community (unable to lock database)
 multilib                                                                                              107.3 KiB   206K/s 00:01 [-----------------------------------------------------------------------------] 100%
error: could not rename /var/lib/pacman/sync/multilib.db.part to /var/lib/pacman/sync/multilib.db (No space left on device)
error: failed to update multilib (unexpected error)
 repo-ck                                                                                                42.7 KiB   464K/s 00:00 [-----------------------------------------------------------------------------] 100%
error: could not rename /var/lib/pacman/sync/repo-ck.db.part to /var/lib/pacman/sync/repo-ck.db (No space left on device)
error: failed to update repo-ck (unexpected error)
error: failed to synchronize any databases
error: failed to init transaction (unexpected error)

Scimmia · 2013-07-24 14:14:10

What filesystem are you using on /?

ewaller · 2013-07-24 16:22:16

ANOKNUSA wrote:

I knew their contents were temporary, but didn't realize they didn't in fact exist; since they store vital system/session data, I figured they were more tangible then that.

If you are interested, check out chapter 14 of "Linux Device Drivers" at the bottom of page 371. The top level link to the book is here. I recommend it for anyone who is curious about the innards of kernel drivers.

ANOKNUSA · 2013-07-24 19:20:45

Scimmia wrote:

What filesystem are you using on /?

BTRFS. "btrfs filesystem df" shows over half the space free, and a scrub showed no problems.

At this point, I can no longer start X, and my little test to recreate the problem above has left /var/lib/pacman/db.lck in place and it can't be deleted. I'm baffled.

EDIT: sudo now complains as well, though it still executes commands.

Last edited by ANOKNUSA (2013-07-24 19:23:26)

cfr · 2013-07-25 00:37:38

Dumb question but I assume you've checked that the relevant partition is mounted with the correct options (e.g. rw)? I guess this would be / unless you have a separate /var.

EDIT: By the way is CheckSpace enabled in pacman.conf?

Last edited by cfr (2013-07-25 00:38:40)

ngoonee · 2013-07-25 03:48:44

Have you checked for hard disc errors?

ANOKNUSA · 2013-07-25 21:19:09

@cfr: The mount options haven't changed in any way.

@ngonee: The disk is perfectly healthy.

Scimmia · 2013-07-25 23:08:57

How about posting the output of "mount"

WorMzy · 2013-07-25 23:30:58

I'm interested in the btrfs fi df output too. I've had a btrfs partition go wonky before, It assigned all free space to the metadata "pool", I had to scrub it before it realised that I was only using a <10th of the space allocated to the metadata pool and re/unallocated the space accordingly. Of course, I use the mainline kernel + git btrfs-progs (Chris Mason's), so my experiences probably aren't applicable to the average users.

ANOKNUSA · 2013-07-26 03:30:56

$ btrfs filesystem df

Data: total=7.74GB, used=3.24GB
System: total=4.00MB, used=4.00KB
Metadata: total=264.00MB, used=196.08MB

$ mount


proc on /proc type proc (rw,nosuid,nodev,noexec,relatime)
sys on /sys type sysfs (rw,nosuid,nodev,noexec,relatime)
dev on /dev type devtmpfs (rw,nosuid,relatime,size=4021028k,nr_inodes=1005257,mode=755)
run on /run type tmpfs (rw,nosuid,nodev,relatime,mode=755)
/dev/sdc2 on / type btrfs (rw,noatime,compress=lzo,ssd,discard,space_cache)
securityfs on /sys/kernel/security type securityfs (rw,nosuid,nodev,noexec,relatime)
tmpfs on /dev/shm type tmpfs (rw,nosuid,nodev)
devpts on /dev/pts type devpts (rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000)
tmpfs on /sys/fs/cgroup type tmpfs (rw,nosuid,nodev,noexec,mode=755)
cgroup on /sys/fs/cgroup/systemd type cgroup (rw,nosuid,nodev,noexec,relatime,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd)
pstore on /sys/fs/pstore type pstore (rw,nosuid,nodev,noexec,relatime)
efivarfs on /sys/firmware/efi/efivars type efivarfs (rw,nosuid,nodev,noexec,relatime)
cgroup on /sys/fs/cgroup/cpuset type cgroup (rw,nosuid,nodev,noexec,relatime,cpuset)
cgroup on /sys/fs/cgroup/memory type cgroup (rw,nosuid,nodev,noexec,relatime,memory)
cgroup on /sys/fs/cgroup/devices type cgroup (rw,nosuid,nodev,noexec,relatime,devices)
cgroup on /sys/fs/cgroup/freezer type cgroup (rw,nosuid,nodev,noexec,relatime,freezer)
cgroup on /sys/fs/cgroup/net_cls type cgroup (rw,nosuid,nodev,noexec,relatime,net_cls)
cgroup on /sys/fs/cgroup/blkio type cgroup (rw,nosuid,nodev,noexec,relatime,blkio)
cgroup on /sys/fs/cgroup/bfqio type cgroup (rw,nosuid,nodev,noexec,relatime,bfqio)
systemd-1 on /proc/sys/fs/binfmt_misc type autofs (rw,relatime,fd=33,pgrp=1,timeout=300,minproto=5,maxproto=5,direct)
systemd-1 on /boot type autofs (rw,relatime,fd=37,pgrp=1,timeout=300,minproto=5,maxproto=5,direct)
mqueue on /dev/mqueue type mqueue (rw,relatime)
hugetlbfs on /dev/hugepages type hugetlbfs (rw,relatime)
debugfs on /sys/kernel/debug type debugfs (rw,relatime)
configfs on /sys/kernel/config type configfs (rw,relatime)
none on /tmp type tmpfs (rw,relatime)
/dev/sdc1 on /boot type vfat (rw,relatime,fmask=0022,dmask=0022,codepage=437,iocharset=iso8859-1,shortname=mixed,errors=remount-ro)
/dev/mapper/home on /home type btrfs (rw,noatime,compress=lzo,ssd,discard,space_cache)

ANOKNUSA · 2013-07-26 16:52:21

So, here it is. I'm unable to do any of the following:

1) Start X
2) Mount external filesystems
3) Alter the "/" filesystem in any way (ergo, screw troubleshooting)
3a) Use a LiveCD to do so
3b) Use "single" boot mode to do so.

I can still boot, log into the shell and read system information, but the system is otherwise completely unusable. So my final question, before finally giving up and going for a fresh install, is: What is it that's reporting this disk space usage to the kernel? If every utility--df, dfc, du, ncdu, btrfs fi di, parted, ranger, pacman---all read there's plenty of space on the disk, then what the hell is telling utilities like rm and cp otherwise? What's the fundamental difference between user space tools that read data and report its size, and user space tools that manipulate that data? I have no idea, but it seems to me that somewhere in the gap between those two is where the problem must lie. Barring an answer to that or any new ideas from anyone, I'm just gonna make a backup list of what's installed from the official repos, make fresh backups of my config files in /etc, and reinstall.

WonderWoofy · 2013-07-26 18:44:00

ANOKNUSA, it doesn't sound like you have any kind of multi-device setup here, but maybe this might help?

I have been following this thread because I am genuinely interested in what is at play here. But unfortunately I don't have an answer for you. I wish you the best of luck in figuring it out. Maybe it might be a good idea to either ask on the linux-btrfs mailing list or on their IRC channel. I know people who have gotten some excellent help there, and fairly quickly.

Of course, if all else fails, at least it is mountable read-only. You can still rsync off, recreate the filesystem, and then rsync back on... a PITA I know, but it is better than total data corruption.

ANOKNUSA · 2013-07-28 19:28:30

Thanks for the feedback, WonderWoofy; I already went over that wiki page, with no luck. I wasn't able to find out what the problem was, though; instead, I just reformated the partition and restored all the system files. I did, however, save the filesystem state in an image, and might look back into the problem in a virtual machine at some point. While I didn't actually solve this problem, I consider the issue resolved anyway. Thanks for the input, everyone.

Arch Linux

#1 2013-07-24 02:48:11

[Solved] /proc, /sys, Various systemd Directories: No Space?

#2 2013-07-24 03:22:56

Re: [Solved] /proc, /sys, Various systemd Directories: No Space?

#3 2013-07-24 03:38:52

Re: [Solved] /proc, /sys, Various systemd Directories: No Space?

#4 2013-07-24 05:59:55

Re: [Solved] /proc, /sys, Various systemd Directories: No Space?

#5 2013-07-24 14:14:10

Re: [Solved] /proc, /sys, Various systemd Directories: No Space?

#6 2013-07-24 16:22:16

Re: [Solved] /proc, /sys, Various systemd Directories: No Space?

#7 2013-07-24 19:20:45

Re: [Solved] /proc, /sys, Various systemd Directories: No Space?

#8 2013-07-25 00:37:38

Re: [Solved] /proc, /sys, Various systemd Directories: No Space?

#9 2013-07-25 03:48:44

Re: [Solved] /proc, /sys, Various systemd Directories: No Space?

#10 2013-07-25 21:19:09

Re: [Solved] /proc, /sys, Various systemd Directories: No Space?

#11 2013-07-25 23:08:57

Re: [Solved] /proc, /sys, Various systemd Directories: No Space?

#12 2013-07-25 23:30:58

Re: [Solved] /proc, /sys, Various systemd Directories: No Space?

#13 2013-07-26 03:30:56

Re: [Solved] /proc, /sys, Various systemd Directories: No Space?

#14 2013-07-26 16:52:21

Re: [Solved] /proc, /sys, Various systemd Directories: No Space?

#15 2013-07-26 18:44:00

Re: [Solved] /proc, /sys, Various systemd Directories: No Space?

#16 2013-07-28 19:28:30

Re: [Solved] /proc, /sys, Various systemd Directories: No Space?

Board footer