You are not logged in.

#1 2012-10-02 06:33:35

Nanthiel
Member
From: Slovenia
Registered: 2009-09-20
Posts: 148

Systemd and LXC causing system crash

Hey everyone,

after switching to systemd, I can't use lxc-execute anymore, as it messes up the system quite a lot.

If I do this now:

# lxc-execute --name test -- echo hi

the output is:

hi
lxc-execute: No such file or directory - failed to open /proc/mounts

Before the call, /proc/mounts has the expected contents (something like mtab?), now if I try "cat /proc/mounts" it says there's no file there (but there is! A symlink pointing to /self/mounts). Now I noticed there is no "self". Actually, ALL the PID files are gone. And "ps -ef" says I need to remount /proc. That works. Why is LXC doing this? It worked fine with initscripts. Maybe something with cgroups, as the location changed from /cgroups to /sys/fs/cgroup?

Now, about messing up the system. This call, after remounting /proc, doesn't seem to have lasting problems. But this call is in a real scenario part of a program that:
1. creates a folder in /tmp and mounts another tmpfs on it
2. copies some files over
3. executes lxc in that folder (lxc-execute, no rootfs parameter set, just a few bind mounts to hide part of the filesystem)
4. unmounts the folder
5. deletes the folder

If I run it, /tmp gets abolished and is not accessible anymore.

After this, even remounting /proc doesn't help anymore. I can not open new processes any longer and must reboot the machine by force. (Or pressing CTRL-ALT-DEL in a real terminal. Systemd is still capable of launching new gettys, by the way.)

Help with debugging this would be greatly appreciated, I have no idea what's going on.

Thanks for your attention, Matej

Offline

#2 2012-10-14 07:15:29

grimpy
Member
Registered: 2012-10-14
Posts: 4

Re: Systemd and LXC causing system crash

Hi there, I have noticed simular problems while trying to start my containers when booted via systemd...
I have no idea what is causing them or how to fix it.

lxc-start -n p5
lxc-start: Invalid argument - pivot_root syscall failed
lxc-start: failed to setup pivot root
lxc-start: failed to set rootfs for 'p5'
lxc-start: failed to setup the container
lxc-start: invalid sequence number 1. expected 2
lxc-start: failed to spawn 'p5'
lxc-start: Device or resource busy - failed to remove cgroup '/sys/fs/cgroup/systemd/system/lxc/p5'

And when i try again it changes to

lxc-start -n p5
lxc-start: failed to mount '/dev/pts/8' on '/usr/lib/lxc/rootfs/dev/console'
lxc-start: failed to setup the console for 'p5'
lxc-start: failed to setup the container
lxc-start: invalid sequence number 1. expected 2
lxc-start: failed to spawn 'p5'

Also i notice that a lot of mountpounts stay behind

devpts on /usr/lib/lxc/rootfs/dev/console type devpts (rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000)
devpts on /home/MISC/LXC/p5/rootfs/dev/console type devpts (rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000)
devpts on /usr/lib/lxc/rootfs/dev/tty1 type devpts (rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000)
devpts on /home/MISC/LXC/p5/rootfs/dev/tty1 type devpts (rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000)
devpts on /usr/lib/lxc/rootfs/dev/tty2 type devpts (rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000)
devpts on /home/MISC/LXC/p5/rootfs/dev/tty2 type devpts (rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000)
devpts on /usr/lib/lxc/rootfs/dev/tty3 type devpts (rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000)
devpts on /home/MISC/LXC/p5/rootfs/dev/tty3 type devpts (rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000)
devpts on /usr/lib/lxc/rootfs/dev/tty4 type devpts (rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000)
devpts on /home/MISC/LXC/p5/rootfs/dev/tty4 type devpts (rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000)

Offline

#3 2012-10-14 09:30:21

Nanthiel
Member
From: Slovenia
Registered: 2009-09-20
Posts: 148

Re: Systemd and LXC causing system crash

I've since only found out that it probably isn't the cgroups location's fault, because Ubuntu has them at /sys/fs/cgroup. Although Ubuntu does not yet use systemd.

But I'm very far from getting it to work. I just switched to initscripts because it's vital for me that this works.

An unrelated issue: I also don't know how to get lxc-checkconfig to use "User namespace". They're enabled in Ubuntu. I tried rebuilding the kernel with the support, but the kernel config program doesn't allow me to set this, says it's experimental and unsettable or something. Manually editing the config didn't work either. But I've never built or configured my own kernel so I might've done something stupid.

Offline

#4 2012-10-14 19:26:21

grimpy
Member
Registered: 2012-10-14
Posts: 4

Re: Systemd and LXC causing system crash

Continued my investigation and figured out the libvirt manages to start my machines just fine while lxc-start failes.
I am pretty sure it indeed has something todo with the cgroups configuration just dont quiet know what yet...

Offline

#5 2012-10-15 06:55:14

hcs
Member
Registered: 2011-02-10
Posts: 2

Re: Systemd and LXC causing system crash

A "me to", I have the same lxc-start problem.
Have tried both lxc and lxc-git.

Offline

#6 2012-10-16 09:51:51

Mariusz Cyranowski
Member
From: Warsaw, Poland
Registered: 2012-10-16
Posts: 2

Re: Systemd and LXC causing system crash

I have exactly same problem with lxc-start and lxc-execute (I use systemd). I tried to move rootfs of container to other filesystem but it didn't help. Errors starts to appear just after pivot_root execution. (Invalid argument - pivot_root syscall failed)

Is there any other way to fix the problem besides waiting for lxc patch ? I'd prefer not to use libvirt.

Offline

#7 2012-10-16 14:43:23

starfry
Member
From: Surrey, UK
Registered: 2010-08-18
Posts: 170

Re: Systemd and LXC causing system crash

I'm going through the systemd change and hit the same problem. You can try this; it worked for me.

$ mount --make-rprivate /

Do that on the host *before* starting any containers.

I need to work out how to have this happen automatically on boot. With systemd I feel like I am learning to walk all over again.

Offline

#8 2012-10-16 18:44:55

Mariusz Cyranowski
Member
From: Warsaw, Poland
Registered: 2012-10-16
Posts: 2

Re: Systemd and LXC causing system crash

Your solution works for me too. Thank You.

I've got similar feelings about systemd.

Offline

#9 2012-10-17 09:45:53

starfry
Member
From: Surrey, UK
Registered: 2010-08-18
Posts: 170

Re: Systemd and LXC causing system crash

Less than 24 hours in, I can see the benefits of systemd. I think there will be pain before gain but it will be worthwhile. I've just written my first systemd service to start my LXC containers smile

update: before singing its praises too loudly in the context of LXC: currently systemd does not work inside a container. It does work on the host with sysvinit inside the container. Looking right now for a solution... will update here if/when one is found...

Last edited by starfry (2012-10-22 08:26:38)

Offline

Board footer

Powered by FluxBB