Arch Linux

bt324

Member

Registered: 2021-10-06

Posts: 2

Docker container connection reset by peer

Hey everyone,

I'm trying setup docker and terraform using the hashicorp learn guide. After starting up the nginx container that should be accessible on 0.0.0.0:8000:

❯ docker ps
CONTAINER ID   IMAGE          COMMAND                  CREATED       STATUS       PORTS                  NAMES
76998ea5a888   f8f4ffc8092c   "/docker-entrypoint.…"   4 hours ago   Up 4 hours   0.0.0.0:8000->80/tcp   tutorial

I get the following response:

❯ curl -v 0.0.0.0:8000
*   Trying 0.0.0.0:8000...
* Connected to 0.0.0.0 (127.0.0.1) port 8000 (#0)
> GET / HTTP/1.1
> Host: 0.0.0.0:8000
> User-Agent: curl/7.79.1
> Accept: */*
>
* Recv failure: Connection reset by peer
* Closing connection 0
curl: (56) Recv failure: Connection reset by peer

For my environment I have the arch installed using the official guided archinstall running sway. I have the same issue on PC as well as laptop. I ran VMs on windows using ubuntu and suse and had no problem there. I presume there is a simple problem on networking side of containers (bridge configured incorrectly or there is a more strict form of firewall on arch by default ? ).

Extra info:
simple docker hello world test

❯ docker run hello-world

Hello from Docker!
This message shows that your installation appears to be working correctly.

To generate this message, Docker took the following steps:
 1. The Docker client contacted the Docker daemon.
 2. The Docker daemon pulled the "hello-world" image from the Docker Hub.
    (amd64)
 3. The Docker daemon created a new container from that image which runs the
    executable that produces the output you are currently reading.
 4. The Docker daemon streamed that output to the Docker client, which sent it
    to your terminal.

docker bridge

❯ docker network inspect bridge
[
    {
        "Name": "bridge",
        "Id": "cd7538d37f8d43fa4f880debda42c0f54fba00d513fb52b79c5d9a10729a99ee",
        "Created": "2021-10-10T14:52:01.359411241+02:00",
        "Scope": "local",
        "Driver": "bridge",
        "EnableIPv6": false,
        "IPAM": {
            "Driver": "default",
            "Options": null,
            "Config": [
                {
                    "Subnet": "172.30.0.0/24",
                    "Gateway": "172.30.0.1"
                }
            ]
        },
        "Internal": false,
        "Attachable": false,
        "Ingress": false,
        "ConfigFrom": {
            "Network": ""
        },
        "ConfigOnly": false,
        "Containers": {
            "76998ea5a8884ff58d0e13636f3ec43335cd44ac2a63ad108c75fc9eae584370": {
                "Name": "tutorial",
                "EndpointID": "42dc53e08ee7c952a13e8148e50002088291735a582522b4fde0916b298b67d6",
                "MacAddress": "02:42:ac:1e:00:02",
                "IPv4Address": "172.30.0.2/24",
                "IPv6Address": ""
            }
        },
        "Options": {
            "com.docker.network.bridge.default_bridge": "true",
            "com.docker.network.bridge.enable_icc": "true",
            "com.docker.network.bridge.enable_ip_masquerade": "true",
            "com.docker.network.bridge.host_binding_ipv4": "0.0.0.0",
            "com.docker.network.bridge.name": "docker0",
            "com.docker.network.driver.mtu": "1500"
        },
        "Labels": {}
    }
]

If I try to build alpine image with apk update (connect to their repos) using host driver:

❯ docker build --file Dockerfile --network host  .
Sending build context to Docker daemon   21.2MB
Step 1/3 : FROM alpine:3.14
 ---> 14119a10abf4
Step 2/3 : MAINTAINER bortyr
 ---> Using cache
 ---> 6786d6ab176f
Step 3/3 : RUN apk update     apk upgrade
 ---> Using cache
 ---> 7eca3559ab2e
Successfully built 7eca3559ab2e

Try to build alpine image with apk update (connect to their repos) using bridge driver

❯ docker build --file Dockerfile --network bridge  .
Sending build context to Docker daemon   21.2MB
Step 1/3 : FROM alpine:3.14
3.14: Pulling from library/alpine
a0d0a0d46f8b: Pull complete
Digest: sha256:e1c082e3d3c45cccac829840a25941e679c25d438cc8412c2fa221cf1a824e6a
Status: Downloaded newer image for alpine:3.14
 ---> 14119a10abf4
Step 2/3 : MAINTAINER bortyr
 ---> Running in ae67fea0d459
Removing intermediate container ae67fea0d459
 ---> 6a7a13e7a941
Step 3/3 : RUN apk update     apk upgrade
 ---> Running in 5a48db0fe170
fetch https://dl-cdn.alpinelinux.org/alpine/v3.14/main/x86_64/APKINDEX.tar.gz
ERROR: https://dl-cdn.alpinelinux.org/alpine/v3.14/main: temporary error (try again later)
WARNING: Ignoring https://dl-cdn.alpinelinux.org/alpine/v3.14/main: No such file or directory
fetch https://dl-cdn.alpinelinux.org/alpine/v3.14/community/x86_64/APKINDEX.tar.gz
ERROR: https://dl-cdn.alpinelinux.org/alpine/v3.14/community: temporary error (try again later)
WARNING: Ignoring https://dl-cdn.alpinelinux.org/alpine/v3.14/community: No such file or directory
2 errors; 14 distinct packages available
The command '/bin/sh -c apk update     apk upgrade' returned a non-zero code: 2

/etc/resolv.conf

❯ cat /etc/resolv.conf
nameserver 8.8.8.8

iptables

❯ sudo iptables -L
Chain INPUT (policy ACCEPT)
target     prot opt source               destination

Chain FORWARD (policy ACCEPT)
target     prot opt source               destination
DOCKER-USER  all  --  anywhere             anywhere
DOCKER-ISOLATION-STAGE-1  all  --  anywhere             anywhere
ACCEPT     all  --  anywhere             anywhere             ctstate RELATED,ESTABLISHED
DOCKER     all  --  anywhere             anywhere
ACCEPT     all  --  anywhere             anywhere
ACCEPT     all  --  anywhere             anywhere

Chain OUTPUT (policy ACCEPT)
target     prot opt source               destination

Chain DOCKER (1 references)
target     prot opt source               destination
ACCEPT     tcp  --  anywhere             172.30.0.2           tcp dpt:http

Chain DOCKER-ISOLATION-STAGE-1 (1 references)
target     prot opt source               destination
DOCKER-ISOLATION-STAGE-2  all  --  anywhere             anywhere
RETURN     all  --  anywhere             anywhere

Chain DOCKER-ISOLATION-STAGE-2 (1 references)
target     prot opt source               destination
DROP       all  --  anywhere             anywhere
RETURN     all  --  anywhere             anywhere

Chain DOCKER-USER (1 references)
target     prot opt source               destination
RETURN     all  --  anywhere             anywhere

ss

❯ sudo ss -plant | grep docker
LISTEN 0      4096         0.0.0.0:8000         0.0.0.0:*    users:(("docker-proxy",pid=4209,fd=4))

ip r

❯ ip r
default via 10.1.216.1 dev wlan0 proto dhcp src 10.1.218.193 metric 1024
10.1.216.0/21 dev wlan0 proto kernel scope link src 10.1.218.193 metric 1024
10.1.216.1 dev wlan0 proto dhcp scope link src 10.1.218.193 metric 1024
172.30.0.0/24 dev docker0 proto kernel scope link src 172.30.0.1 linkdown
217.113.224.36 via 10.1.216.1 dev wlan0 proto dhcp src 10.1.218.193 metric 1024
217.113.224.135 via 10.1.216.1 dev wlan0 proto dhcp src 10.1.218.193 metric 1024

networkctl list

❯ networkctl list
IDX LINK          TYPE     OPERATIONAL SETUP
  1 lo            loopback carrier     unmanaged
  2 enp0s25       ether    no-carrier  configuring
 12 docker0       bridge   no-carrier  unmanaged
 16 veth5533dfb   ether    degraded    configuring
 19 wwp0s29u1u4i6 wwan     no-carrier  configuring
 20 wlan0         wlan     routable    configured

journalctl -xeu docker.service

Oct 14 13:56:53 btx220 dockerd[937]: time="2021-10-14T13:56:53.076526369+02:00" level=info msg="Starting up"
Oct 14 13:56:53 btx220 dockerd[937]: time="2021-10-14T13:56:53.079593411+02:00" level=info msg="parsed scheme: \"unix\"" module=grpc
Oct 14 13:56:53 btx220 dockerd[937]: time="2021-10-14T13:56:53.079682618+02:00" level=info msg="scheme \"unix\" not registered, fallback to default scheme" module=grpc
Oct 14 13:56:53 btx220 dockerd[937]: time="2021-10-14T13:56:53.079762935+02:00" level=info msg="ccResolverWrapper: sending update to cc: {[{unix:///run/containerd/containerd.sock  <nil> 0 <nil>}] <nil> <nil>}" module=grpc
Oct 14 13:56:53 btx220 dockerd[937]: time="2021-10-14T13:56:53.079842568+02:00" level=info msg="ClientConn switching balancer to \"pick_first\"" module=grpc
Oct 14 13:56:53 btx220 dockerd[937]: time="2021-10-14T13:56:53.082706488+02:00" level=info msg="parsed scheme: \"unix\"" module=grpc
Oct 14 13:56:53 btx220 dockerd[937]: time="2021-10-14T13:56:53.082758262+02:00" level=info msg="scheme \"unix\" not registered, fallback to default scheme" module=grpc
Oct 14 13:56:53 btx220 dockerd[937]: time="2021-10-14T13:56:53.082822803+02:00" level=info msg="ccResolverWrapper: sending update to cc: {[{unix:///run/containerd/containerd.sock  <nil> 0 <nil>}] <nil> <nil>}" module=grpc
Oct 14 13:56:53 btx220 dockerd[937]: time="2021-10-14T13:56:53.082860142+02:00" level=info msg="ClientConn switching balancer to \"pick_first\"" module=grpc
Oct 14 13:56:53 btx220 dockerd[937]: time="2021-10-14T13:56:53.084272878+02:00" level=info msg="[graphdriver] using prior storage driver: btrfs"
Oct 14 13:56:53 btx220 dockerd[937]: time="2021-10-14T13:56:53.117182032+02:00" level=info msg="Loading containers: start."
Oct 14 13:56:53 btx220 dockerd[937]: time="2021-10-14T13:56:53.438169781+02:00" level=info msg="Removing stale sandbox e373c9fcc57ec8703f5c63cefa020f551fef07e437559a7402e7d15ba1f85f18 (76998ea5a8884ff58d0e13636f3ec43335cd44ac2a63ad108c75fc9eae584370)"
Oct 14 13:56:53 btx220 dockerd[937]: time="2021-10-14T13:56:53.478545279+02:00" level=warning msg="Error (Unable to complete atomic operation, key modified) deleting object [endpoint cd7538d37f8d43fa4f880debda42c0f54fba00d513fb52b79c5d9a10729a99ee 42dc53e08ee7c952a13e8148e50002088291735a582522b4fde0916b298b67d6], retrying...."
Oct 14 13:56:53 btx220 dockerd[937]: time="2021-10-14T13:56:53.601816097+02:00" level=info msg="Default bridge (docker0) is assigned with an IP address 172.30.0.0/24. Daemon option --bip can be used to set a preferred IP address"
Oct 14 13:56:53 btx220 dockerd[937]: time="2021-10-14T13:56:53.676745243+02:00" level=info msg="Loading containers: done."
Oct 14 13:56:53 btx220 dockerd[937]: time="2021-10-14T13:56:53.732647449+02:00" level=info msg="Docker daemon" commit=79ea9d3080 graphdriver(s)=btrfs version=20.10.9
Oct 14 13:56:53 btx220 dockerd[937]: time="2021-10-14T13:56:53.732789625+02:00" level=info msg="Daemon has completed initialization"
Oct 14 13:56:53 btx220 systemd[1]: Started Docker Application Container Engine.
░░ Subject: A start job for unit docker.service has finished successfully
░░ Defined-By: systemd
░░ Support: https://lists.freedesktop.org/mailman/listinfo/systemd-devel
░░
░░ A start job for unit docker.service has finished successfully.
░░
░░ The job identifier is 86.
Oct 14 13:56:53 btx220 dockerd[937]: time="2021-10-14T13:56:53.760171770+02:00" level=info msg="API listen on /run/docker.sock"

Some "similar" issues I found on SO: 1, 2, 3

I'd be grateful for any help here. I have little knowledge on networking so there might be an issue there as well . I also tried to look through some of the troubleshooting on the docker page of arch wiki but I'm not sure on what is the case here.

emil.s

Member

Registered: 2011-05-22

Posts: 8

Re: Docker container connection reset by peer

Hello,

Hard to tell what's going on really, especially since we have a 3rd party (Terraform), managing the whole setup. (So I don't know exactly what docker-run command that has been run).

First thing to do would be to rule out if it's a problem with the network, or an issue with Nginx (in the container).
Try accessing the Nginx in the container directly:

curl http://172.30.0.2:80

But, as you have already assumed, it seems to be some issue with the network.

One line that looks suspicious is this route:

172.30.0.0/24 dev docker0 proto kernel scope link src 172.30.0.1 linkdown

Note the "linkdown". Networkctl does also say "no-carrier", where it should say "routable".
I have never seen this. Can you enable the interface with "ip link set docker0 up"?

I would look for the root cause why docker0 is disabled. Terraform?

bt324

Member

Registered: 2021-10-06

Posts: 2

Re: Docker container connection reset by peer

Thanks emil.s for looking into it,

I tried accessing the container directly

❯ curl http://172.30.0.2:80
curl: (7) Failed to connect to 172.30.0.2 port 80 after 3047 ms: No route to host

What you mentioned about the interface sounded promising. I ran the commnad:

❯ sudo ip link set docker0 up

But even after this I still get "linkdown"

172.30.0.0/24 dev docker0 proto kernel scope link src 172.30.0.1 linkdown

To exclude terraform, I deployed the nginx container using docker command (from official nginx docs)

❯ docker run --name mynginx1 -p 80:80 -d nginx
❯ docker ps
CONTAINER ID   IMAGE     COMMAND                  CREATED         STATUS         PORTS                               NAMES
0d10b1d83c3c   nginx     "/docker-entrypoint.…"   4 seconds ago   Up 3 seconds   0.0.0.0:80->80/tcp, :::80->80/tcp   mynginx1
❯ curl 0.0.0.0:80
curl: (56) Recv failure: Connection reset by peer

The issue unfortunately remains the same.

So maybe something blocks enabling that docker0 interface ?