You are not logged in.

#1 2025-09-21 02:39:47

gattu_marrudu
Member
Registered: 2008-03-29
Posts: 80

[SOLVED] Cannot start lxc containers: no space left on device

Hello,
My LXC containers stopped working all of a sudden. Any container I try to start with lxc-start I get:

../src/lxc/cgroups/cgfsng.c: cgfsng_chown: 2009 No space left on device - Error requesting cgroup chown in new user namespace

I have plenty of space in all my partitions. I have a dozen containers defined, and I also removed some containers I wasn't using in case the "no space left" was referring to some lxc-defined limit, to no avail.

The only significant things I did before containers stopped working, was upgrading the host's kernel (6.15.11-hardened1-1-hardened) and rebooting, and moving a container manually, which I rolled back after this problem came up.

All containers are unprivileged, started by root, each with its own uid & gid range. (in blocks of 100,000). I verified that root is assigned an appropriate range in /etc/subuid and /etc/subgid.

Any hints?

Thanks in advance.
gm

Last edited by gattu_marrudu (2025-09-25 18:53:27)

Offline

#2 2025-09-21 03:04:09

gattu_marrudu
Member
Registered: 2008-03-29
Posts: 80

Re: [SOLVED] Cannot start lxc containers: no space left on device

I found that the problem only comes up if I set a subuid/subgid range. I am currently using this template:

lxc.include = /etc/lxc/default.conf
lxc.idmap = u 0 700000 65535
lxc.idmap = g 0 700000 65535

I normally add 100,000 to the uidmap and gidmap before creating a new container. Containers previously created with this method have been working fine until today. If I create a fresh container without this template (i.e. without the uid & gid maps) I can start it, if I use the template I can't.

Both /etc/subuid and /etc/subgid look identical:

$ cat /etc/subuid
root:100000:4000000
ste:6000000:2000000
cntadm:8000000:65536

Last edited by gattu_marrudu (2025-09-21 03:06:15)

Offline

#3 2025-09-21 04:25:27

prMoriarty
Member
Registered: 2025-09-18
Posts: 19

Re: [SOLVED] Cannot start lxc containers: no space left on device

Interesting. Did you update systemd to 258? May it be related to my issue? https://bbs.archlinux.org/viewtopic.php?id=308337

Offline

#4 2025-09-21 13:05:08

gattu_marrudu
Member
Registered: 2008-03-29
Posts: 80

Re: [SOLVED] Cannot start lxc containers: no space left on device

Thanks for chiming in, prMoriarty.

I downgraded systemd-{,libs-,sysvcompat-}257.9, rebooted, and all lxc are up again. So yes, it seems related to your issue.

I will follow your thread, as others have already intervened there, and update this one if eventually systemd gets patched.

Last edited by gattu_marrudu (2025-09-21 13:05:24)

Offline

#5 2025-09-23 04:16:57

gattu_marrudu
Member
Registered: 2008-03-29
Posts: 80

Re: [SOLVED] Cannot start lxc containers: no space left on device

Unfortunately, I am having problems again, after lxc has been working for more than one day, after downgrading to systemd*-257.

$ sudo lxc-attach eolo
lxc-attach: eolo: ../src/lxc/conf.c: userns_exec_minimal: 4599 No space left on device - Failed to unshare new user namespace
lxc-attach: eolo: ../src/lxc/conf.c: userns_exec_minimal: 4651 Message too long - Failed waiting for child process 38495" to tell us to proceed
lxc-attach: eolo: ../src/lxc/cgroups/cgfsng.c: __cgroup_attach_many: 4248 Message too long - Failed to attach to cgroup fd 11
lxc-attach: eolo: ../src/lxc/attach.c: lxc_attach: 1680 Message too long - Failed to attach cgroup
lxc-attach: eolo: ../src/lxc/attach.c: do_attach: 1332 Input/output error - Failed to prepare terminal file descriptor 8
lxc-attach: eolo: ../src/lxc/attach.c: do_attach: 1376 Failed to attach to container

I have the same container still running, and Ican log in via ssh. lxc-console is also returning a prompt. I am starting to wonder if this is really systemd specific, but also why it went away afer the systemd downgrade and it's back again now. I haven't done any upgrades or other pacman operations on the host since.

Offline

#6 2025-09-25 14:55:43

gattu_marrudu
Member
Registered: 2008-03-29
Posts: 80

Re: [SOLVED] Cannot start lxc containers: no space left on device

For what it's worth, I filed an issue on LXC: https://github.com/lxc/lxc/issues/4586

The LXC forum seems to be exclusively interested in Incus support and not very responsive on low-level issue reports.

Offline

#7 2025-09-25 18:53:09

gattu_marrudu
Member
Registered: 2008-03-29
Posts: 80

Re: [SOLVED] Cannot start lxc containers: no space left on device

I resolved this.

My /etc/sysctl.d/50-lxc.conf had

user.max_user_namespaces = 10
kernel.unprivileged_userns_clone = 1

This was allowing me to start some containers but the namespace slots got filled up quickly with other lxc-* commands, and that's why those commands stopped working some time after rebooting.

Raising the max_user_namespaces to 8192 fixed the issue.

Offline

Board footer

Powered by FluxBB