docker containers -- no network

bins · 2015-07-20 12:38:53

hi there,

I am using the latest docker package (1:1.7.1-2) and I do not seem to have any outbound connection from inside the container:

$ ping -c3 [url=http://www.google.com]www.google.com[/url]
PING [url=http://www.google.com]www.google.com[/url] (64.233.184.103) 56(84) bytes of data.
64 bytes from wa-in-f103.1e100.net (64.233.184.103): icmp_seq=1 ttl=43 time=19.1 ms
64 bytes from wa-in-f103.1e100.net (64.233.184.103): icmp_seq=2 ttl=43 time=19.1 ms
64 bytes from wa-in-f103.1e100.net (64.233.184.103): icmp_seq=3 ttl=43 time=19.1 ms

--- [url=http://www.google.com]www.google.com[/url] ping statistics ---
3 packets transmitted, 3 received, 0% packet loss, time 2002ms
rtt min/avg/max/mdev = 19.170/19.181/19.196/0.113 ms

$ docker run -it --rm debian ping -c3 [url=http://www.google.com]www.google.com[/url]
ping: unknown host

$ docker run -it --rm debian ping -c3 8.8.8.8
PING 8.8.8.8 (8.8.8.8): 56 data bytes
--- 8.8.8.8 ping statistics ---
3 packets transmitted, 0 packets received, 100% packet loss

It's only when I pass the --net=host option that I get some network connectivity:

$ docker run -it --rm --net=host debian ping -c3 [url=http://www.google.com]www.google.com[/url]
PING [url=http://www.google.com]www.google.com[/url] (64.233.184.147): 56 data bytes
64 bytes from 64.233.184.147: icmp_seq=0 ttl=43 time=18.514 ms
64 bytes from 64.233.184.147: icmp_seq=1 ttl=43 time=18.533 ms
64 bytes from 64.233.184.147: icmp_seq=2 ttl=43 time=18.545 ms
--- [url=http://www.google.com]www.google.com[/url] ping statistics ---
3 packets transmitted, 3 packets received, 0% packet loss
round-trip min/avg/max/stddev = 18.514/18.531/18.545/0.000 ms

any idea?

here is the output of iptable:

$ iptables -L
Chain INPUT (policy ACCEPT)
target     prot opt source               destination         

Chain FORWARD (policy ACCEPT)
target     prot opt source               destination         
DOCKER     all  --  anywhere             anywhere            
ACCEPT     all  --  anywhere             anywhere             ctstate RELATED,ESTABLISHED
ACCEPT     all  --  anywhere             anywhere            
ACCEPT     all  --  anywhere             anywhere            

Chain OUTPUT (policy ACCEPT)
target     prot opt source               destination         

Chain DOCKER (1 references)
target     prot opt source               destination         

$ iptables -t nat -L
Chain PREROUTING (policy ACCEPT)
target     prot opt source               destination         
DOCKER     all  --  anywhere             anywhere             ADDRTYPE match dst-type LOCAL

Chain INPUT (policy ACCEPT)
target     prot opt source               destination         

Chain OUTPUT (policy ACCEPT)
target     prot opt source               destination         
DOCKER     all  --  anywhere            !127.0.0.0/8          ADDRTYPE match dst-type LOCAL

Chain POSTROUTING (policy ACCEPT)
target     prot opt source               destination         
MASQUERADE  all  --  172.17.0.0/16        anywhere            

Chain DOCKER (2 references)
target     prot opt source               destination

any help/hint greatly appreciated.

cheers,
-bins

edit: fix code blocks

Last edited by bins (2015-07-21 10:54:59)

firecat53 · 2015-07-20 23:46:29

I don't know exactly where your issue is, but it's definitely something with your setup. I can run that command (with ubuntu and busybox...I don't have a debian image onhand) on two different Arch machines (Docker 1.7.1) and one Ubuntu machine (Docker 1.3.2) with no issues. I only use UFW on all those machines without any separate iptables rules.

Are you running docker with any non-standard options or perhaps attached it to a bridge that isn't connected to the internet?

Also, the code tags for the BBS are

[code] and [/code]

Scott

bins · 2015-07-21 10:59:10

@firecat53 thanks for the code block syntax.

AFAIK, I don't have any non-standard options.
I start docker thru systemd with this little modification (to use the faster overlay driver):

sh> cat /etc/systemd/system/multi-user.target.wants/docker.service
[...]
[Service]
ExecStart=/usr/bin/docker -d -s overlay -H fd://

but using the archlinux default (devicemapper, so, w/o the '-s overlay' part) I also get the same outcome.

yes, I agree it must be something with iptables (as it *was* working at some point for me as well...)

I'll try installing UFW and see what gives.

bins · 2015-07-21 12:07:27

well, unfortunately, not much more luck.

here is the output of `ip a` if that helps:

sh> ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default 
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host 
       valid_lft forever preferred_lft forever

2: enp4s0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP group default qlen 1000
    link/ether 00:22:19:0e:ae:2e brd ff:ff:ff:ff:ff:ff
    inet XXX.XXX.XXX.XXX/16 brd XXX.XXX.XXX.XXX. scope global enp4s0
       valid_lft forever preferred_lft forever
    inet6 XXXXXXX/64 scope link 
       valid_lft forever preferred_lft forever

28: docker0@NONE: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue state LOWERLAYERDOWN group default 
    link/ether 00:00:00:00:00:00 brd ff:ff:ff:ff:ff:ff
    inet 172.17.42.1/16 scope global docker0
       valid_lft forever preferred_lft forever
    inet6 XXXXXXX/64 scope link 
       valid_lft forever preferred_lft forever

42: veth8072183@if41: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master docker0 state UP group default 
    link/ether b2:eb:89:a8:41:a3 brd ff:ff:ff:ff:ff:ff link-netnsid 0
    inet6 XXXXXXX/64 scope link 
       valid_lft forever preferred_lft forever

and brctl:

sh>  brctl show
bridge name	bridge id		STP enabled	interfaces
docker0		8000.b2eb89a841a3	no		veth8072183

firecat53 · 2015-07-21 15:50:53

How about your 'ip route' output? Is your default network addressing conflicting with the docker default network space? I have to run docker on a different network address than default because the combo of the WiFi addresses and my VPN addressing conflicts with the default docker networking. If iptables might be the culprit, try resetting all those settings to default.

Scott

bins · 2015-07-21 15:54:45

here it is:

sh> ip route
default via MY.PUBLIC-IP.XXX.XXX dev enp4s0  proto static 
MY.GATEWAY.0.0/16 dev enp4s0  proto kernel  scope link  src MY.PUBLIC-IP.XXX.XXX
172.17.0.0/16 dev docker0  proto kernel  scope link  src 172.17.42.1

do have I to do something?

Last edited by bins (2015-07-21 15:55:40)

firecat53 · 2015-07-21 21:47:32

Your public IP is on a different subnet than your docker network (172.17.xxx), right?

How about the output of

docker run -it --rm busybox sh -c 'ip r; cat /etc/resolv.conf'

bins · 2015-07-22 06:24:24

that would be:

sh> docker run -it --rm busybox sh -c 'ip r; cat /etc/resolv.conf'
default via 172.17.42.1 dev eth0 
172.17.0.0/16 dev eth0  proto kernel  scope link  src 172.17.0.7 
#
# /etc/resolv.conf
#

#search <yourdomain.tld>

#nameserver <ip>

# End of file
nameserver 8.8.8.8
nameserver 8.8.4.4

cles · 2015-07-22 11:18:01

Tracking a similar beast here, could you try:

sudo iptables -L -v -n

For me the first two rules on the FORWARD chain are catching nothing at all (as in, the counters for them are stuck at 0), whereas on a working docker installation the counters for those go up as you'd expect (ergo, the forwarding does work).

Now, as to why the traffic is not hitting those rules, I'm a bit at a loss...

Did you config your network with systemd-networkd btw?

cles · 2015-07-22 11:35:28

Ok, got the bastard, dude, check if forwarding for your interface is enabled

In my system (with interface name eno1), i had to:

sudo sysctl net.ipv4.conf.eno1.forwarding=1

and it started working, bam.

Same problem: https://bbs.archlinux.org/viewtopic.php?id=198075 and https://wiki.archlinux.org/index.php/In … forwarding (notice the Note)

Hope it's the same problem and it helped.

I've a feeling this is going to be a recurring problem with interactions with systemd-networkd and docker, since docker seems to assume a global forwarding flag set to 1 will do the job entirely (nope), and systemd-network assumes that if you want forwarding, you'll modify the proper file (be it .network or whatever).

TBH it does look like docker should be doing this job, but hey.

Last edited by cles (2015-07-22 12:35:41)

firecat53 · 2015-07-22 15:38:59

Docker sets this for you now. It didn't last year when Docker was new, but it has for quite a while. I don't have any manually-generated sysctl or ufw rules to set it and net.ipv4.conf.all.forwarding=1 on both my Arch laptop (NetworkManager) and Arch server (systemd-networkd with a bridge).

Scott

cles · 2015-07-22 18:20:26

Yep, it does set net.ipv4.conf.all.forwarding to 1. That's what confused me, forwarding appeared to be enabled... except the actual forward was not happening.
Apparently from systemd's 219/220 version onwards, interface-speciric forwarding gets turned off unless you configure it otherwise.

Here, I found it on the changelog:

Note that systemd-networkd manages the sysctl variable
/proc/sys/net/ipv[46]/conf/*/forwarding for each interface
it is configured for since v219. The variable controls IP
forwarding, and is a per-interface alternative to the global
/proc/sys/net/ipv[46]/ip_forward. This setting is
configurable in the IPForward= option, which defaults to
"no". This means if networkd is used for an interface it is
no longer sufficient to set the global sysctl option to turn
on IP forwarding! Instead, the .network file option
IPForward= needs to be turned on! Note that the
implementation of this behaviour was broken in v219 and has
been fixed in v220.

What I didn't know (though it does make sense I suppose) is that apparently the interface-specific setting overrides the 'all' one. I assume before it wasn't a problem because systemd-network abstained to set anything at all about forwarding unless specifically told to do so, so the global setting was respected, no problem.

Basically it seems that if you do not want systemd-network to do that, you must add

IPForward=kernel

on your interface.network file (mine being at /etc/systemd/network/eno1.network). Or so the manual says, I'll try it and see.

Last edited by cles (2015-07-22 19:53:09)

bins · 2015-07-30 09:03:10

hi guys,

*thanks*.

I indeed had some of the sysctl variables not set to 1.
ie: I had ipv4.all but not the others.
editing /etc/sysctl.d/30-ipforward.conf as advised in https://wiki.archlinux.org/index.php/In … forwarding did the trick.

*thanks* again.

-s

rudygotya · 2015-08-26 20:05:53

I had the same problem - with ipv6 enabled on my systems. In my case I noticed network outbound connections worked after I restarted docker. I diffed sysctl net values and fixed it with this configuration:

○ → cat /etc/sysctl.d/30-ipforward.conf 
net.ipv4.ip_forward=1
net.ipv6.conf.default.forwarding=1
net.ipv6.conf.all.forwarding=1

○ → cat /etc/sysctl.d/40-docker.conf 
net.ipv6.conf.docker0.accept_ra = 1
net.ipv6.conf.docker0.accept_ra_defrtr = 1
net.ipv6.conf.docker0.accept_ra_pinfo = 1
net.ipv6.conf.docker0.accept_ra_rtr_pref = 1
net.ipv6.conf.docker0.disable_ipv6 = 0

○ → cat /etc/systemd/network/ipforward.network 
[Network]
IPForward=kernel

I had the same error on two different installations so I hope this post helps someone else. I'm not sure if this is worth a bug report.

lefteris · 2015-09-01 12:09:18

Hello,

I am facing the same problem as the above poster. Network connectivity inside an Ubuntu container works only after a

systemctl restart docker.service

I am not sure what it is that changes to enable connectivity. I tried to add the same settings as the above poster posted but that did not help.

If anyone has any ideas how to figure out what is missing and why it works only after a docker restart I would be really glad.

SirMyztiq · 2015-09-01 13:33:06

lefteris wrote:

Hello,
I am facing the same problem as the above poster. Network connectivity inside an Ubuntu container works only after a
systemctl restart docker.service
I am not sure what it is that changes to enable connectivity. I tried to add the same settings as the above poster posted but that did not help.
If anyone has any ideas how to figure out what is missing and why it works only after a docker restart I would be really glad.

I have the same issue.

jetm · 2015-09-02 01:34:54

It's a known issue and it's a target to be fixed in 1.8.2 release. Check it out: Upgrade from 1.7.1 to 1.8.1 incomplete bridge settings

jdevelop · 2015-12-09 16:26:48

Doesn't work here with docker 1:1.9.1-1

- start docker service
- sysctl net.ipv4.conf.docker0.forwarding=1
- restart docker server

after that all the containers can access external network just fine

Diplomat · 2019-09-26 15:19:14

Okay, so I'm having those issues now.

This is my docker info output:

Client:
 Debug Mode: false

Server:
 Containers: 18
  Running: 0
  Paused: 0
  Stopped: 18
 Images: 3
 Server Version: 19.03.1-ce
 Storage Driver: overlay2
  Backing Filesystem: extfs
  Supports d_type: true
  Native Overlay Diff: false
 Logging Driver: json-file
 Cgroup Driver: cgroupfs
 Plugins:
  Volume: local
  Network: bridge host ipvlan macvlan null overlay
  Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
 Swarm: inactive
 Runtimes: runc
 Default Runtime: runc
 Init Binary: docker-init
 containerd version: a4bc1d432a2c33aa2eed37f338dceabb93641310.m
 runc version: 425e105d5a03fabd737a126ad93d62a9eeede87f
 init version: fec3683
 Security Options:
  seccomp
   Profile: default
 Kernel Version: 5.2.11-1-MANJARO
 Operating System: Manjaro Linux
 OSType: linux
 Architecture: x86_64
 CPUs: 12
 Total Memory: 31.34GiB
 Name: filip-pc
 ID: DJFI:HJJJ:PZQX:SNP5:KXVB:FYCO:NOGK:WJGI:6GNP:IP47:2MFM:H2NR
 Docker Root Dir: /var/lib/docker
 Debug Mode: false
 Registry: https://index.docker.io/v1/
 Labels:
 Experimental: false
 Insecure Registries:
  127.0.0.0/8
 Live Restore Enabled: false

This is my docker0 interface:

docker0: flags=4099<UP,BROADCAST,MULTICAST>  mtu 1500
        inet 172.17.0.1  netmask 255.255.0.0  broadcast 172.17.255.255
        inet6 fe80::42:d3ff:fe14:d9ba  prefixlen 64  scopeid 0x20<link>
        ether 02:42:d3:14:d9:ba  txqueuelen 0  (Ethernet)
        RX packets 87  bytes 4874 (4.7 KiB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 181  bytes 22552 (22.0 KiB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

Output of iptables -L -v -n

Chain INPUT (policy ACCEPT 622 packets, 117K bytes)
 pkts bytes target     prot opt in     out     source               destination         

Chain FORWARD (policy ACCEPT 0 packets, 0 bytes)
 pkts bytes target     prot opt in     out     source               destination         
    1    56 DOCKER-USER  all  --  *      *       0.0.0.0/0            0.0.0.0/0           
    1    56 DOCKER-ISOLATION-STAGE-1  all  --  *      *       0.0.0.0/0            0.0.0.0/0           
    0     0 ACCEPT     all  --  *      docker0  0.0.0.0/0            0.0.0.0/0            ctstate RELATED,ESTABLISHED
    0     0 DOCKER     all  --  *      docker0  0.0.0.0/0            0.0.0.0/0           
    1    56 ACCEPT     all  --  docker0 !docker0  0.0.0.0/0            0.0.0.0/0           
    0     0 ACCEPT     all  --  docker0 docker0  0.0.0.0/0            0.0.0.0/0           

Chain OUTPUT (policy ACCEPT 655 packets, 128K bytes)
 pkts bytes target     prot opt in     out     source               destination         

Chain DOCKER (1 references)
 pkts bytes target     prot opt in     out     source               destination         

Chain DOCKER-ISOLATION-STAGE-1 (1 references)
 pkts bytes target     prot opt in     out     source               destination         
    1    56 DOCKER-ISOLATION-STAGE-2  all  --  docker0 !docker0  0.0.0.0/0            0.0.0.0/0           
    1    56 RETURN     all  --  *      *       0.0.0.0/0            0.0.0.0/0           

Chain DOCKER-ISOLATION-STAGE-2 (1 references)
 pkts bytes target     prot opt in     out     source               destination         
    0     0 DROP       all  --  *      docker0  0.0.0.0/0            0.0.0.0/0           
    1    56 RETURN     all  --  *      *       0.0.0.0/0            0.0.0.0/0           

Chain DOCKER-USER (1 references)
 pkts bytes target     prot opt in     out     source               destination         
   82  5742 RETURN     all  --  *      *       0.0.0.0/0            0.0.0.0/0

I tried to set packet forwarding. I added dns records to daemon.json file... Basically every solution I found, I already tried. Ofcourse if I run container in host mode, everything works.

WorMzy · 2019-09-26 15:31:47

Please don't necrobump: https://wiki.archlinux.org/index.php/Co … bumping%22

Closing.

EDIT: and please note that Manjaro is not supported here. Please seek support on the Manjaro support channels.

Last edited by WorMzy (2019-09-26 16:02:03)

Arch Linux

#1 2015-07-20 12:38:53

docker containers -- no network

#2 2015-07-20 23:46:29

Re: docker containers -- no network

#3 2015-07-21 10:59:10

Re: docker containers -- no network

#4 2015-07-21 12:07:27

Re: docker containers -- no network

#5 2015-07-21 15:50:53

Re: docker containers -- no network

#6 2015-07-21 15:54:45

Re: docker containers -- no network

#7 2015-07-21 21:47:32

Re: docker containers -- no network

#8 2015-07-22 06:24:24

Re: docker containers -- no network

#9 2015-07-22 11:18:01

Re: docker containers -- no network

#10 2015-07-22 11:35:28

Re: docker containers -- no network

#11 2015-07-22 15:38:59

Re: docker containers -- no network

#12 2015-07-22 18:20:26

Re: docker containers -- no network

#13 2015-07-30 09:03:10

Re: docker containers -- no network

#14 2015-08-26 20:05:53

Re: docker containers -- no network

#15 2015-09-01 12:09:18

Re: docker containers -- no network

#16 2015-09-01 13:33:06

Re: docker containers -- no network

#17 2015-09-02 01:34:54

Re: docker containers -- no network

#18 2015-12-09 16:26:48

Re: docker containers -- no network

#19 2019-09-26 15:19:14

Re: docker containers -- no network

#20 2019-09-26 15:31:47

Re: docker containers -- no network

Board footer