Arch Linux

eluffie

Member

Registered: 2021-11-01

Posts: 7

[Solved] Docker/Podman networking broken

I have the following issue: networking in docker/podman containers is broken; containers are not able to access each other (or the internet) while DNS works.

This is the minimal docker-compose file I use:

services:
  nginx:
    image: nginx
    container_name: nginx
    ports:
      - 80:80

  ping:
    image: wbitt/network-multitool
    container_name: ping
    command: tail -F /dev/null

After trying to debug docker I also tried podman (edit: not-rootless) with the same result so I think something underlying is broken but I don't know where to look. I really hope you can help me.

I already tried most if not all of the tips mentioned in this issue: https://github.com/moby/moby/issues/36151

I am also unable to access nginx from Arch (curl http://localhost:80 just fails to do anything).

You can see that name resolution is working from the ping command:

docker exec -it ping bash
bash-5.1# curl http://nginx:80
curl: (7) Failed to connect to nginx port 80 after 3053 ms: Host is unreachable
bash-5.1# ping nginx
PING nginx (172.18.0.2) 56(84) bytes of data.
From 2cc85ba6e7b2 (172.18.0.3) icmp_seq=1 Destination Host Unreachable
From 2cc85ba6e7b2 (172.18.0.3) icmp_seq=2 Destination Host Unreachable
From 2cc85ba6e7b2 (172.18.0.3) icmp_seq=3 Destination Host Unreachable
--- nginx ping statistics ---
4 packets transmitted, 0 received, +3 errors, 100% packet loss, time 3039ms

Here is the output of network inspect

docker network inspect work_default
[
    {
        "Name": "work_default",
        "Id": "58ba5fa1a5c233b76c8290016c007167b6d65f28e0d6cb77e893a7cf0c24b12c",
        "Created": "2022-07-07T15:30:17.057077926+02:00",
        "Scope": "local",
        "Driver": "bridge",
        "EnableIPv6": false,
        "IPAM": {
            "Driver": "default",
            "Options": null,
            "Config": [
                {
                    "Subnet": "172.18.0.0/16",
                    "Gateway": "172.18.0.1"
                }
            ]
        },
        "Internal": false,
        "Attachable": false,
        "Ingress": false,
        "ConfigFrom": {
            "Network": ""
        },
        "ConfigOnly": false,
        "Containers": {
            "2cc85ba6e7b2942bb06688826b471596ec055bb6cb472f372165b10b182103f9": {
                "Name": "ping",
                "EndpointID": "ea4be13b5906fdecab5af986c41e10532763651feb59b34aa85af96193e4a88b",
                "MacAddress": "02:42:ac:12:00:03",
                "IPv4Address": "172.18.0.3/16",
                "IPv6Address": ""
            },
            "732f773c59442c3739c9b6c4c543d9cb3757fdd6aee6e0c7e70f0353bbe0b1eb": {
                "Name": "nginx",
                "EndpointID": "6fc2ab0f0330fc4a0ff4343089daad38a38c35604f1f728e92a6f8bdd6bf813c",
                "MacAddress": "02:42:ac:12:00:02",
                "IPv4Address": "172.18.0.2/16",
                "IPv6Address": ""
            }
        },
        "Options": {},
        "Labels": {
            "com.docker.compose.network": "default",
            "com.docker.compose.project": "work",
            "com.docker.compose.version": "2.6.1"
        }
    }
]

docker network ls
NETWORK ID     NAME           DRIVER    SCOPE
2b1164f1da89   bridge         bridge    local
ba0d1915a1b2   host           host      local
0dd8cbd59d33   none           null      local
58ba5fa1a5c2   work_default   bridge    local

Last edited by eluffie (2023-01-07 01:50:56)

eluffie

Member

Registered: 2021-11-01

Posts: 7

Re: [Solved] Docker/Podman networking broken

I finally found the culprit.

At some point I configured dracut to allow me to decrypt LUKS via SSH adding

ip=dhcp

to the cmdline.

This creates

/run/systemd/network/91-default.network

with the following content:

# Automatically generated by systemd-network-generator

[Match]
Name=*

[Link]

[Network]
DHCP=ipv4

[DHCP]

This interferes with docker, openvpn and probably a plethora of software that creates their own routes, adapters, ... and manages them on their own.

I fixed it by creating files telling systemd to ignore certain networks (for Name={br-*,docker*,veth*,tun*}):

[Match]
Name=br-*

[Link]
Unmanaged=true

This solved the problem and now openvpn and docker networking work as expected.

Although I suspect setting ip=dhcp in the first place was unnecessary and bad as my /etc/systemd/network/* files are included in the initrd anyway (and enable DHCP themselves).