You are not logged in.
Hello,
My LXC containers stopped working all of a sudden. Any container I try to start with lxc-start I get:
../src/lxc/cgroups/cgfsng.c: cgfsng_chown: 2009 No space left on device - Error requesting cgroup chown in new user namespace
I have plenty of space in all my partitions. I have a dozen containers defined, and I also removed some containers I wasn't using in case the "no space left" was referring to some lxc-defined limit, to no avail.
The only significant things I did before containers stopped working, was upgrading the host's kernel (6.15.11-hardened1-1-hardened) and rebooting, and moving a container manually, which I rolled back after this problem came up.
All containers are unprivileged, started by root, each with its own uid & gid range. (in blocks of 100,000). I verified that root is assigned an appropriate range in /etc/subuid and /etc/subgid.
Any hints?
Thanks in advance.
gm
Last edited by gattu_marrudu (2025-09-25 18:53:27)
Offline
I found that the problem only comes up if I set a subuid/subgid range. I am currently using this template:
lxc.include = /etc/lxc/default.conf
lxc.idmap = u 0 700000 65535
lxc.idmap = g 0 700000 65535
I normally add 100,000 to the uidmap and gidmap before creating a new container. Containers previously created with this method have been working fine until today. If I create a fresh container without this template (i.e. without the uid & gid maps) I can start it, if I use the template I can't.
Both /etc/subuid and /etc/subgid look identical:
$ cat /etc/subuid
root:100000:4000000
ste:6000000:2000000
cntadm:8000000:65536
Last edited by gattu_marrudu (2025-09-21 03:06:15)
Offline
Interesting. Did you update systemd to 258? May it be related to my issue? https://bbs.archlinux.org/viewtopic.php?id=308337
Offline
Thanks for chiming in, prMoriarty.
I downgraded systemd-{,libs-,sysvcompat-}257.9, rebooted, and all lxc are up again. So yes, it seems related to your issue.
I will follow your thread, as others have already intervened there, and update this one if eventually systemd gets patched.
Last edited by gattu_marrudu (2025-09-21 13:05:24)
Offline
Unfortunately, I am having problems again, after lxc has been working for more than one day, after downgrading to systemd*-257.
$ sudo lxc-attach eolo
lxc-attach: eolo: ../src/lxc/conf.c: userns_exec_minimal: 4599 No space left on device - Failed to unshare new user namespace
lxc-attach: eolo: ../src/lxc/conf.c: userns_exec_minimal: 4651 Message too long - Failed waiting for child process 38495" to tell us to proceed
lxc-attach: eolo: ../src/lxc/cgroups/cgfsng.c: __cgroup_attach_many: 4248 Message too long - Failed to attach to cgroup fd 11
lxc-attach: eolo: ../src/lxc/attach.c: lxc_attach: 1680 Message too long - Failed to attach cgroup
lxc-attach: eolo: ../src/lxc/attach.c: do_attach: 1332 Input/output error - Failed to prepare terminal file descriptor 8
lxc-attach: eolo: ../src/lxc/attach.c: do_attach: 1376 Failed to attach to container
I have the same container still running, and Ican log in via ssh. lxc-console is also returning a prompt. I am starting to wonder if this is really systemd specific, but also why it went away afer the systemd downgrade and it's back again now. I haven't done any upgrades or other pacman operations on the host since.
Offline
For what it's worth, I filed an issue on LXC: https://github.com/lxc/lxc/issues/4586
The LXC forum seems to be exclusively interested in Incus support and not very responsive on low-level issue reports.
Offline
I resolved this.
My /etc/sysctl.d/50-lxc.conf had
user.max_user_namespaces = 10
kernel.unprivileged_userns_clone = 1
This was allowing me to start some containers but the namespace slots got filled up quickly with other lxc-* commands, and that's why those commands stopped working some time after rebooting.
Raising the max_user_namespaces to 8192 fixed the issue.
Offline