Arch Linux

kiosnk

Member

Registered: 2022-02-26

Posts: 19

[SOLVED] No internet connectivity in Docker containers

Currently, my Docker containers have no internet access. I installed it following the instructions on Arch Wiki.

"ip address" outputs the following:

1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host
       valid_lft forever preferred_lft forever
2: enp7s0f1: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc fq_codel state DOWN group default qlen 1000
    link/ether 08:97:98:67:18:98 brd ff:ff:ff:ff:ff:ff
3: wlp0s20f3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
    link/ether 28:7f:cf:33:b8:41 brd ff:ff:ff:ff:ff:ff
    inet 192.168.0.176/24 brd 192.168.0.255 scope global dynamic noprefixroute wlp0s20f3
       valid_lft 75598sec preferred_lft 75598sec
    inet6 2804:14d:1685:8002:73f7:d7ae:1882:22c7/64 scope global dynamic noprefixroute
       valid_lft 86316sec preferred_lft 71916sec
    inet6 fe80::d1a6:e455:a5d1:704c/64 scope link noprefixroute
       valid_lft forever preferred_lft forever
4: br-85bfd1abad1a: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue state DOWN group default
    link/ether 02:42:ed:be:65:70 brd ff:ff:ff:ff:ff:ff
    inet 172.18.0.1/16 brd 172.18.255.255 scope global br-85bfd1abad1a
       valid_lft forever preferred_lft forever
    inet6 fe80::42:edff:febe:6570/64 scope link
       valid_lft forever preferred_lft forever
5: docker0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue state DOWN group default
    link/ether 02:42:b2:ee:f2:02 brd ff:ff:ff:ff:ff:ff
    inet 172.17.0.1/16 brd 172.17.255.255 scope global docker0
       valid_lft forever preferred_lft forever
    inet6 2804:14d:1685:8002::1/64 scope global
       valid_lft forever preferred_lft forever
    inet6 fe80::1/64 scope link
       valid_lft forever preferred_lft forever
    inet6 fe80::42:b2ff:feee:f202/64 scope link
       valid_lft forever preferred_lft forever

I would really appreciate if someone could help me understand what's happening. I don't think it's a DNS issue because running the following also doesn't work:
"docker run busybox ping -c 1 8.8.8.8"

I've also tried to enable masquerade in firewalld but it also didn't work.
Any ideas?

Thanks

Last edited by kiosnk (2022-03-03 01:08:38)

twelveeighty

Member

Registered: 2011-09-04

Posts: 1,454

Re: [SOLVED] No internet connectivity in Docker containers

Please post actual error messages and logs, don't say "doesn't work" or "didn't work".

Your docker0 bridge state is DOWN - did you follow the instructions in the Wiki that deal specifically with that? If so, post your configs.

kiosnk

Member

Registered: 2022-02-26

Posts: 19

Re: [SOLVED] No internet connectivity in Docker containers

Please post actual error messages and logs, don't say "doesn't work" or "didn't work".

Your docker0 bridge state is DOWN - did you follow the instructions in the Wiki that deal specifically with that? If so, post your configs.

Sorry I didn't post any error message.
I'm unable to get those messages now, but for "docker run busybox ping -c 1 8.8.8.8" the error was basically the typical result we get when we ping without internet connection (1 package sent, 0 package received, 100% packet loss). That's how I knew I had no connectivity and wasn't caused by DNS (since I wasn't pinging google.com).

And yes! I did read the section you mentioned, but I believe that wasn't my problem since I checked that systemd-networkd was not running:

networkctl list
WARNING: systemd-networkd is not running, output will be incomplete.

IDX LINK            TYPE     OPERATIONAL SETUP
  1 lo              loopback n/a         unmanaged
  2 enp7s0f1        ether    n/a         unmanaged
  3 wlp0s20f3       wlan     n/a         unmanaged
  4 br-143a36472163 bridge   n/a         unmanaged
  5 docker0         bridge   n/a         unmanaged

5 links listed.

Right now I'm unable to do further tests since I can't get docker daemon running. I tried to remove it with "yay -R docker" and install it again with "yay -Syu docker" but when I try to enable the daemon I get this error:

ERRO[2022-03-02T20:18:31.729065089-03:00] failed to initialize a tracing processor "otlp"  error="no OpenTelemetry endpoint: skip plugin"
ERRO[2022-03-02T20:18:32.877099502-03:00] copy shim log                                 error="read /proc/self/fd/13: file already closed"
ERRO[2022-03-02T20:18:32.877350682-03:00] stream copy error: reading from a closed fifo
ERRO[2022-03-02T20:18:32.877363627-03:00] stream copy error: reading from a closed fifo
ERRO[2022-03-02T20:18:33.161399162-03:00] 49c472d2c23dbda99955209c59a39b074a24ec64c784002575c8980b1db70d30 cleanup: failed to delete container from containerd: no such container
ERRO[2022-03-02T20:18:33.161415383-03:00] failed to start container                     container=49c472d2c23dbda99955209c59a39b074a24ec64c784002575c8980b1db70d30 error="failed to create shim task: OCI runtime create failed: unable to retrieve OCI runtime error (open /var/run/docker/containerd/daemon/io.containerd.runtime.v2.task/moby/49c472d2c23dbda99955209c59a39b074a24ec64c784002575c8980b1db70d30/log.json: no such file or directory): exec: \"runc\": executable file not found in $PATH: unknown"

I tried removing containerd and reinstalling it along with docker again but no success: apparently there is some old container reference the uninstall didn't remove and is preventing the daemon from starting now... Gotta fix it before I can get back working on the connectivity part =/

Last edited by kiosnk (2022-03-03 00:09:09)

kiosnk

Member

Registered: 2022-02-26

Posts: 19

Re: [SOLVED] No internet connectivity in Docker containers

Solved the above and the connectivity issue!

The problem of above was that somehow I had no "runc" installed. I ignored the message about the old container and just installed "runc" and it worked.

The original issue I'm not sure how I solved, but here are some steps I took while still having the problem:
1 - "sudo firewall-cmd --list-all" and saw that "masquerade" was set to "no".
2 - then I ran "sudo firewall-cmd --permanent --add-masquerade" to set it to "yes".

This didn't work at the moment, but I kept this config (it's set to "yes" now).

After that, I also checked my daemon.json at /etc/docker/ and tried changing DNS, but that didn't have any effect.

BUT, for anyone reading this, DNS problem seems to be a common reason for no connectivity in Docker containers (I had this issue in the past). So, for anyone reading who has this problem, you could try changing /etc/docker/daemon.json to:

{
  "dns": ["8.8.8.8", "8.8.4.4"]
}

or any other DNS of your preference and see if it works (test it against "docker run busybox ping -c 8.8.8.8" AND "docker run busybox nslookup google.com" -- if it works on the first but fails on the second, then you still have a DNS problem, if it fails in both then DNS probably isn't your problem).

Since it wasn't my issue either, and I couldn't find much more to test, I decided to do a clean install of Docker once again. So, I:
1- removed docker and containerd with "yay -R docker containerd"
2- disabled all services listed in "systemctl list-dependencies | grep docker" with "systemctl disable servicename"
3- "sudo rm -rf /etc/docker" to remove any config file
4- searched for docker and containerd folders in "/usr/bin" and "/usr/share" to make sure nothing was left behind
5- installed Docker following the Wiki guide again (yay -Syu docker, systemctl enable docker, systemctl start docker) and this time it had internet!

The only logical explanation I can imagine for my case was that "sudo firewall-cmd --permanent --add-masquerade" indeed solved the problem but I had to disable and enable the services for it to work (and not only restart). Maybe it wasn't this, but I can't think of anything else I did that would actually make sense to solve the problem...

Last edited by kiosnk (2022-03-03 00:57:29)