Arch Linux

abozanich

Member

Registered: 2017-04-15

Posts: 2

docker systemd socket activation: performance issue, unable to disable

Hi all,

I've been tinkering with a small side project which creates and manages ephemeral docker instances for unit/integration testing purposes and am seeing serious performance issues when running on my Arch setup that I don't see on other systems (OSX, Ubuntu).

It looks to me like for whatever reason a small number of concurrent operations are getting through to docker while the rest block.  For instance, look at the elapsed time of docker ps when running unit tests which will create three docker containers concurrently:

$ while : ; do
time sh -c 'docker ps | sed 1d | wc -l'
sleep 1
done
0
sh -c 'docker ps | sed 1d | wc -l'  0.02s user 0.00s system 92% cpu 0.018 total
0
sh -c 'docker ps | sed 1d | wc -l'  0.01s user 0.01s system 104% cpu 0.022 total
1
sh -c 'docker ps | sed 1d | wc -l'  0.01s user 0.01s system 0% cpu 6.518 total
2
sh -c 'docker ps | sed 1d | wc -l'  0.02s user 0.00s system 0% cpu 5.398 total
0
sh -c 'docker ps | sed 1d | wc -l'  0.01s user 0.01s system 0% cpu 7.637 total
1
sh -c 'docker ps | sed 1d | wc -l'  0.00s user 0.00s system 66% cpu 0.010 total
1
sh -c 'docker ps | sed 1d | wc -l'  0.02s user 0.00s system 112% cpu 0.018 total
1
sh -c 'docker ps | sed 1d | wc -l'  0.01s user 0.00s system 68% cpu 0.015 total
1
sh -c 'docker ps | sed 1d | wc -l'  0.02s user 0.01s system 112% cpu 0.021 total
1
sh -c 'docker ps | sed 1d | wc -l'  0.01s user 0.01s system 97% cpu 0.020 total
0

$ go test ./builtin/...
ok  	github.com/boz/ephemerald/builtin/pg	28.056s
ok  	github.com/boz/ephemerald/builtin/redis	21.322s
ok  	github.com/boz/ephemerald/builtin/vault	12.548s

I suspect the issue is related to "socket activation" because that is the main difference between my Arch setup and the other systems I've been testing with.

Unfortunately, I just can't figure out how to disable socket activation.  Here's my default unit files and the overrides:

$ cat /usr/lib/systemd/system/docker.socket
[Unit]
Description=Docker Socket for the API
PartOf=docker.service

[Socket]
ListenStream=/var/run/docker.sock
SocketMode=0660
SocketUser=root
SocketGroup=docker

[Install]
WantedBy=sockets.target

$ cat /usr/lib/systemd/system/docker.service
[Unit]
Description=Docker Application Container Engine
Documentation=https://docs.docker.com
After=network-online.target docker.socket firewalld.service
Wants=network-online.target
Requires=docker.socket

[Service]
Type=notify
# the default is not to use systemd for cgroups because the delegate issues still
# exists and systemd currently does not support the cgroup feature set required
# for containers run by docker
ExecStart=/usr/bin/dockerd -H fd://
ExecReload=/bin/kill -s HUP $MAINPID
LimitNOFILE=1048576
# Having non-zero Limit*s causes performance problems due to accounting overhead
# in the kernel. We recommend using cgroups to do container-local accounting.
LimitNPROC=infinity
LimitCORE=infinity
# Uncomment TasksMax if your systemd version supports it.
# Only systemd 226 and above support this version.
#TasksMax=infinity
TimeoutStartSec=0
# set delegate yes so that systemd does not reset the cgroups of docker containers
Delegate=yes
# kill only the docker process, not all processes in the cgroup
KillMode=process
# restart the docker process if it exits prematurely
Restart=on-failure
StartLimitBurst=3
StartLimitInterval=60s

[Install]
WantedBy=multi-user.target

$ cat /etc/systemd/system/docker.service.d/override.conf
[Unit]
After=
After=network-online.target firewalld.service
Requires=

[Service]
ExecStart=
ExecStart=/usr/bin/dockerd

$ cat /etc/systemd/system/docker.socket.d/override.conf
[Unit]
PartOf=

With those overrides I still get this:

$ sudo systemctl list-dependencies docker | head
docker.service
● ├─docker.socket
● ├─docker.socket
● ├─system.slice
● ├─network-online.target
● └─sysinit.target
●   ├─dev-hugepages.mount
●   ├─dev-mqueue.mount
●   ├─kmod-static-nodes.service
●   ├─ldconfig.service

And this:

$ systemctl show docker | grep docker
ExecStart={ path=/usr/bin/dockerd ; argv[]=/usr/bin/dockerd ; ignore_errors=no ; start_time=[Sat 2017-04-15 17:09:55 PDT] ; stop_time=[n/a] ; pid=27251 ; code=(null) ; status=0/0 }
ControlGroup=/system.slice/docker.service
Id=docker.service
Names=docker.service
Requires=sysinit.target system.slice docker.socket
ConsistsOf=docker.socket
After=systemd-journald.socket network-online.target docker.socket sysinit.target basic.target system.slice firewalld.service
TriggeredBy=docker.socket
Documentation=https://docs.docker.com
FragmentPath=/usr/lib/systemd/system/docker.service
DropInPaths=/etc/systemd/system/docker.service.d/override.conf

Note that the ExecStart override stuck, but the After didn't, and docker.socket shows up in a bunch of different places.

Anyone have any pointers for disabling socket activation here?

Thanks,
-Adam

tom.ty89

Member

Registered: 2012-11-15

Posts: 897

Re: docker systemd socket activation: performance issue, unable to disable

As far as I know the socket is practically "disabled" when the corresponding service is explicitly enabled, since in that way systemd will start it as soon as anything "wants" it (e.g. when it is reaching multi-user target and the service is wanted by the target).

abozanich

Member

Registered: 2017-04-15

Posts: 2

Re: docker systemd socket activation: performance issue, unable to disable

As far as I know the socket is practically "disabled" when the corresponding service is explicitly enabled, since in that way systemd will start it as soon as anything "wants" it (e.g. when it is reaching multi-user target and the service is wanted by the target).

Okay, but why does docker.service still depend on docker.socket when I removed all of the references to docker.socket?

As it turns out, my Ubuntu machine does indeed use socket activation, so that is not the issue.

Here's the results from a quick benchmark:

# Ubuntu i7-6800K CPU @ 3.40GHz, 32GB, Docker Version 17.03.1-ce

Containers  Parallelism Runtime

01          01          0.76
05          01          4.29
10          01          8.41
10          05          3.48
10          10          3.02
20          10          5.64
20          20          5.32

# OSX 10.12.4 Core i7-4960HQ @ 2.60GHz, Docker version 17.03.1-ce

  Note: Docker running in VM limited to 4 logical cores and 6GB memory:

Containers  Parallelism Runtime

01          01           1.23
05          01           9.06
10          01          18.48
10          05           5.76
10          10           6.85
20          10          17.92
20          20          26.77

# Arch Core i7-4960HQ @ 2.60GHz, 16GB, Docker Version 17.04.0-ce

Containers  Parallelism Runtime

01          01           0.76
05          01           4.65
10          01           8.74
10          05          50.29
10          10          27.21
20          10          65.05
20          20          44.40

Something is very wrong with the arch system.  For concurrent operations, Docker is 2-10 times slower on Arch than it is on a CPU and memory-limited VM on OSX.

Last edited by abozanich (2017-04-18 04:57:31)