[SOLVED] Cannot start apache inside container

nticompass · 2017-12-18 20:45:05

I am running Ubuntu inside a systemd-nspawn container, and I am suddenly having issues getting apache to start (inside the container). Everything was working fine a few days ago, but now it's not. I don't know if an update inside or outside of the container has changed something.

Inside the container, I see the following:

root@container:~# apachectl -k start
/usr/sbin/apachectl: 99: ulimit: error setting limit (Operation not permitted)
Setting ulimit failed. See README.Debian for more information.
(13)Permission denied: AH00072: make_sock: could not bind to address [::]:80
(13)Permission denied: AH00072: make_sock: could not bind to address 0.0.0.0:80
no listening sockets available, shutting down
AH00015: Unable to open logs
Action '-k start' failed.
The Apache error log may have more information.

I checked using lsof and netstat and nothing else is running on ports 80/443. Also, what is that "ulimit" error? What's going on there?

I *did* use `sudo systemctl edit systemd-nspawn@my-container.service` to edit the settings to disable private networking and enable some bind mounts:

[Service]
ExecStart=
ExecStart=/usr/bin/systemd-nspawn --quiet --keep-unit --boot --link-journal=try-guest -U --settings=override --machine=%i \
        --bind-ro=/etc/resolv.conf:/etc/resolv.conf \
        --bind=/home/nticompass/Code/website:/opt/website

Yes, the apache config inside the container is setup to use `/opt/website` as its root. I have it bind mounted so I can use my IDE on my main OS to edit the files and then have the container be able to access them (without having to copy/transfer them).

I am not sure what to do here. Did something change in an update? Do I need to update a config on my main OS? Why can't the container access the ports? What is that "ulimit" error?

Last edited by nticompass (2017-12-21 01:46:25)

lo1 · 2017-12-20 13:15:56

I wouldn't know how to debug this inside an ubuntu LXC, but to be sure this has nothing to do with arch: couldn't you create an arch LXC and try the same config? At least you'll know *where* to debug this, and probably check ubuntu forums if this is a breakage caused by the latter.

graysky · 2017-12-20 13:37:57

+1 for a containerized Arch image. What do you plan to achieve by running an ubuntu container on an Arch host?

nticompass · 2017-12-20 15:41:55

graysky wrote:

+1 for a containerized Arch image. What do you plan to achieve by running an ubuntu container on an Arch host?

This container is for work and the setup scripts that my work provides assume you're using ubuntu. I guess it wouldn't be too hard to modify those to use arch instead (and even send them a pull request).

The weird thing is that it was working just fine up until a few days ago! I don't know what changed to stop it from working.

graysky · 2017-12-20 15:50:22

The rub is that this community supports Arch but you're asking for help with a Ubuntu container which is Ubuntu. It's true that your config could be causing the problem, but it's too many variables to just guess-and-check in my opinion.

nticompass · 2017-12-20 16:00:36

Yeah, I see where you're coming from. The thing is, I think the issue is on the Arch end. The container's permissions/capabilities are being limited somehow and I don't know why.

graysky · 2017-12-20 16:42:42

nticompass wrote:

The thing is, I think the issue is on the Arch end. The container's permissions/capabilities are being limited somehow and I don't know why.

You really haven't provided much to go one... lxc or docker? config files? log files?

nticompass · 2017-12-20 17:07:17

Ok, I'll do my best to post what I can to help out here. I'm using `systemd-nspawn` for the container and I'm launching it using `sudo machinectl start my-container`.

When setting it up, I followed the instructions in the Arch Wiki: https://wiki.archlinux.org/index.php/Sy … nvironment

I overrode a few settings (I changed it a little bit since what I posted originally), such as adding bound folders and disabling "private networking".

I overrode the original service using: `sudo systemctl edit systemd-nspawn@my-container.service`:

[Service]
ExecStart=
ExecStart=/usr/bin/systemd-nspawn -qbjUM %i --keep-unit --settings=override

And added the bound folders using `/etc/systemd/nspawn/my-container.nspawn`:

[Files]
BindReadOnly=/etc/resolv.conf
Bind=/home/nticompass/Code/website:/opt/website

This is the same as the configs I posted originally, I just broke out the folder binds into their own config file.

After starting the container, I ran `sudo systemctl -M my-container status apache2` and see the following:

● apache2.service - The Apache HTTP Server
   Loaded: loaded (/lib/systemd/system/apache2.service; enabled; vendor preset: enabled)
  Drop-In: /lib/systemd/system/apache2.service.d
           └─apache2-systemd.conf
   Active: failed (Result: exit-code) since Wed 2017-12-20 11:56:46 EST; 2min 27s ago
  Process: 108 ExecStart=/usr/sbin/apachectl start (code=exited, status=1/FAILURE)

*Inside* the container, running `sudo systemctl status apache2`, shows me a slightly more verbose output:

● apache2.service - The Apache HTTP Server
   Loaded: loaded (/lib/systemd/system/apache2.service; enabled; vendor preset: enabled)
  Drop-In: /lib/systemd/system/apache2.service.d
           └─apache2-systemd.conf
   Active: failed (Result: exit-code) since Wed 2017-12-20 11:56:46 EST; 3min 4s ago
  Process: 108 ExecStart=/usr/sbin/apachectl start (code=exited, status=1/FAILURE)

Dec 20 11:56:46 container apachectl[108]: (13)Permission denied: AH00072: make_sock: could not bind to address [::]:80
Dec 20 11:56:46 container apachectl[108]: (13)Permission denied: AH00072: make_sock: could not bind to address 0.0.0.0:80
Dec 20 11:56:46 container apachectl[108]: no listening sockets available, shutting down
Dec 20 11:56:46 container apachectl[108]: AH00015: Unable to open logs
Dec 20 11:56:46 container apachectl[108]: Action 'start' failed.
Dec 20 11:56:46 container apachectl[108]: The Apache error log may have more information.
Dec 20 11:56:46 container systemd[1]: apache2.service: Control process exited, code=exited status=1
Dec 20 11:56:46 container systemd[1]: Failed to start The Apache HTTP Server.
Dec 20 11:56:46 container systemd[1]: apache2.service: Unit entered failed state.
Dec 20 11:56:46 container systemd[1]: apache2.service: Failed with result 'exit-code'.

It seems something (on the Arch end) is disallowing this container to bind to privileged ports. I tried editing the permissions/capabilities of the container by adding the following to `/etc/systemd/nspawn/my-container.nspawn`:

[Exec]
Capability=CAP_NET_ADMIN CAP_NET_BIND_SERVICE

This did not change anything. Interestingly, I also cannot `ping` from inside the container:

nticompass@container ~ % ping google.com
ping: socket: Operation not permitted

Running `sudo ping google.com` doesn't work either.

I'm not sure what's going on here and why this isn't working. It was totally working just fine up into a few days ago. I have "private networking" disabled because I was having issues getting the networking to work with it enabled. And this way, I can just go to `http://127.0.0.1` and I can access the site running in the container's apache. What else can I post to try to help debug this? What other logs/configs would be helpful? I understand this is an Arch forum, and I am running Ubuntu, but I am almost 100% sure that the issue is on my Arch host, not the Ubuntu guest.

Last edited by nticompass (2017-12-20 17:16:06)

progandy · 2017-12-20 19:14:48

It seems you are running systemd-nspawn with -U. This will start an unprivileged container if possible. Before kernel 4.14.2, this wasn't possible and you got a privileged container. Since Kernel version 4.14.5 (in [core], 4.14.2 in [testing]), CONFIG_USER_NS is enabled for the root user, so root can create unprivileged containers now. This might change the behaviour of the container like prohibiting ulimit.

nticompass · 2017-12-20 20:22:39

progandy wrote:

It seems you are running systemd-nspawn with -U. This will start an unprivileged container if possible. Before kernel 4.14.2, this wasn't possible and you got a privileged container. Since Kernel version 4.14.5 (in [core], 4.14.2 in [testing]), CONFIG_USER_NS is enabled for the root user, so root can create unprivileged containers now. This might change the behaviour of the container like prohibiting ulimit.

Hmm interesting, I knew something must've changed in Arch Linux. The `-U` is being used because it's the default, so I kept it.

So, how would I go about creating a "privileged container"? Just removing the `-U` parameter didn't fix it. Do I need to somehow use `--private-users`?

EDIT: I figured it out! Thanks for the point in the right direction :-D

I removed the `-U` option from the (edited) service file and added the following to `/etc/systemd/nspawn/my-container.nspawn`:

[Exec]
PrivateUsers=off

Last edited by nticompass (2017-12-20 20:30:48)

Arch Linux

#1 2017-12-18 20:45:05

[SOLVED] Cannot start apache inside container

#2 2017-12-20 13:15:56

Re: [SOLVED] Cannot start apache inside container

#3 2017-12-20 13:37:57

Re: [SOLVED] Cannot start apache inside container

#4 2017-12-20 15:41:55

Re: [SOLVED] Cannot start apache inside container

#5 2017-12-20 15:50:22

Re: [SOLVED] Cannot start apache inside container

#6 2017-12-20 16:00:36

Re: [SOLVED] Cannot start apache inside container

#7 2017-12-20 16:42:42

Re: [SOLVED] Cannot start apache inside container

#8 2017-12-20 17:07:17

Re: [SOLVED] Cannot start apache inside container

#9 2017-12-20 19:14:48

Re: [SOLVED] Cannot start apache inside container

#10 2017-12-20 20:22:39

Re: [SOLVED] Cannot start apache inside container

Board footer