Container 1:
# docker run -it --rm --network testnet --ip 172.18.10.10 crccheck/hello-world
Container 2:
# docker run -it --rm --network=testnet --ip 172.18.10.11 joffotron/docker-net-tools
Inside container 2:
- On the broken machine:
/ # ping -c 1 172.18.10.10
PING 172.18.10.10 (172.18.10.10): 56 data bytes
64 bytes from 172.18.10.10: seq=0 ttl=64 time=0.149 ms
--- 172.18.10.10 ping statistics ---
1 packets transmitted, 1 packets received, 0% packet loss
round-trip min/avg/max = 0.149/0.149/0.149 ms
/ # curl http://172.18.10.10:8000
curl: (7) Failed to connect to 172.18.10.10 port 8000: Host is unreachable
- On a working machine:
/ # ping -c 1 172.18.10.10
PING 172.18.10.10 (172.18.10.10): 56 data bytes
64 bytes from 172.18.10.10: seq=0 ttl=64 time=0.063 ms
--- 172.18.10.10 ping statistics ---
1 packets transmitted, 1 packets received, 0% packet loss
round-trip min/avg/max = 0.063/0.063/0.063 ms
/ # curl http://172.18.10.10:8000
<xmp>
Hello World
## .
## ## ## ==
## ## ## ## ## ===
/""""""""""""""""\___/ ===
~~~ {~~ ~~~~ ~~~ ~~~~ ~~ ~ / ===- ~~~
\______ o _,/
\ \ _,'
`'--.._\..--''
</xmp>
# docker run -it --rm --network=bridge joffotron/docker-net-tools
Then run these commands...
On the broken machine:
/ # ping -c 1 1.1.1.1
PING 1.1.1.1 (1.1.1.1): 56 data bytes
64 bytes from 1.1.1.1: seq=0 ttl=57 time=3.942 ms
--- 1.1.1.1 ping statistics ---
1 packets transmitted, 1 packets received, 0% packet loss
round-trip min/avg/max = 3.942/3.942/3.942 ms
/ # dig one.one.one.one
; <<>> DiG 9.10.4-P3 <<>> one.one.one.one
;; global options: +cmd
;; connection timed out; no servers could be reached
And on a working one:
/ # ping -c 1 1.1.1.1
PING 1.1.1.1 (1.1.1.1): 56 data bytes
64 bytes from 1.1.1.1: seq=0 ttl=56 time=8.860 ms
--- 1.1.1.1 ping statistics ---
1 packets transmitted, 1 packets received, 0% packet loss
round-trip min/avg/max = 8.860/8.860/8.860 ms
/ # dig one.one.one.one
; <<>> DiG 9.10.4-P3 <<>> one.one.one.one
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 12414
;; flags: qr rd ra; QUERY: 1, ANSWER: 2, AUTHORITY: 0, ADDITIONAL: 1
;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 1452
;; QUESTION SECTION:
;one.one.one.one. IN A
;; ANSWER SECTION:
one.one.one.one. 149 IN A 1.0.0.1
one.one.one.one. 149 IN A 1.1.1.1
;; Query time: 9 msec
;; SERVER: 1.1.1.1#53(1.1.1.1)
;; WHEN: Sun Mar 29 15:50:50 UTC 2020
;; MSG SIZE rcvd: 76
# ip route add 192.168.1.1/32 dev eth0
ip: RTNETLINK answers: Operation not permitted
]]>Inside the container add a route
ip route add 192.168.1.1/32 dev eth0
test pinging 192.168.1.1 and 1.1.1.1 .
]]>$ ping 172.18.0.1
$ ping 192.168.1.69
$ ping 192.168.1.1
container ran using:
docker run -it --rm --network=testnet --ip 172.18.10.10 joffotron/docker-net-tools
# ip link
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN qlen 1000
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
78: eth0@if79: <BROADCAST,MULTICAST,UP,LOWER_UP,M-DOWN> mtu 1500 qdisc noqueue state UP
link/ether 02:42:ac:12:0a:0a brd ff:ff:ff:ff:ff:ff
# ip address
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN qlen 1000
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
inet 127.0.0.1/8 scope host lo
valid_lft forever preferred_lft forever
78: eth0@if79: <BROADCAST,MULTICAST,UP,LOWER_UP,M-DOWN> mtu 1500 qdisc noqueue state UP
link/ether 02:42:ac:12:0a:0a brd ff:ff:ff:ff:ff:ff
inet 172.18.10.10/16 brd 172.18.255.255 scope global eth0
valid_lft forever preferred_lft forever
# ip route
default via 172.18.0.1 dev eth0
172.18.0.0/16 dev eth0 src 172.18.10.10
# ip link
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN mode DEFAULT group default qlen 1000
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
2: enp4s0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc mq state DOWN mode DEFAULT group default qlen 1000
link/ether 00:10:18:af:dc:95 brd ff:ff:ff:ff:ff:ff
3: enp6s0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP mode DEFAULT group default qlen 1000
link/ether 18:03:73:46:f4:2e brd ff:ff:ff:ff:ff:ff
4: virbr0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue state DOWN mode DEFAULT group default qlen 1000
link/ether 52:54:00:3f:06:20 brd ff:ff:ff:ff:ff:ff
5: virbr0-nic: <BROADCAST,MULTICAST> mtu 1500 qdisc fq_codel master virbr0 state DOWN mode DEFAULT group default qlen 1000
link/ether 52:54:00:3f:06:20 brd ff:ff:ff:ff:ff:ff
6: docker0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP mode DEFAULT group default
link/ether 02:42:f4:df:82:e8 brd ff:ff:ff:ff:ff:ff
7: br-f1b323ef3c02: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP mode DEFAULT group default
link/ether 02:42:d4:c1:50:1d brd ff:ff:ff:ff:ff:ff
75: vethe6ccbaa@if74: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master br-f1b323ef3c02 state UP mode DEFAULT group default
link/ether ca:a6:a7:60:8c:57 brd ff:ff:ff:ff:ff:ff link-netnsid 12
# ip address
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
inet 127.0.0.1/8 scope host lo
valid_lft forever preferred_lft forever
inet6 ::1/128 scope host
valid_lft forever preferred_lft forever
2: enp4s0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc mq state DOWN group default qlen 1000
link/ether 00:10:18:af:dc:95 brd ff:ff:ff:ff:ff:ff
3: enp6s0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP group default qlen 1000
link/ether 18:03:73:46:f4:2e brd ff:ff:ff:ff:ff:ff
inet 192.168.1.69/24 brd 192.168.1.255 scope global dynamic noprefixroute enp6s0
valid_lft 54907sec preferred_lft 54907sec
inet6 fe80::2e21:7f19:ef75:b155/64 scope link noprefixroute
valid_lft forever preferred_lft forever
4: virbr0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue state DOWN group default qlen 1000
link/ether 52:54:00:3f:06:20 brd ff:ff:ff:ff:ff:ff
inet 192.168.122.1/24 brd 192.168.122.255 scope global virbr0
valid_lft forever preferred_lft forever
5: virbr0-nic: <BROADCAST,MULTICAST> mtu 1500 qdisc fq_codel master virbr0 state DOWN group default qlen 1000
link/ether 52:54:00:3f:06:20 brd ff:ff:ff:ff:ff:ff
6: docker0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default
link/ether 02:42:f4:df:82:e8 brd ff:ff:ff:ff:ff:ff
inet 172.17.0.1/16 brd 172.17.255.255 scope global docker0
valid_lft forever preferred_lft forever
inet6 fe80::42:f4ff:fedf:82e8/64 scope link
valid_lft forever preferred_lft forever
7: br-f1b323ef3c02: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default
link/ether 02:42:d4:c1:50:1d brd ff:ff:ff:ff:ff:ff
inet 172.18.0.1/16 brd 172.18.255.255 scope global br-f1b323ef3c02
valid_lft forever preferred_lft forever
inet6 fe80::42:d4ff:fec1:501d/64 scope link
valid_lft forever preferred_lft forever
75: vethe6ccbaa@if74: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master br-f1b323ef3c02 state UP group default
link/ether ca:a6:a7:60:8c:57 brd ff:ff:ff:ff:ff:ff link-netnsid 12
inet6 fe80::c8a6:a7ff:fe60:8c57/64 scope link
valid_lft forever preferred_lft forever
# ip route
default via 192.168.1.1 dev enp6s0 proto dhcp metric 100
default via 192.168.1.1 dev enp6s0 proto dhcp src 192.168.1.69 metric 203
172.17.0.0/16 dev docker0 proto kernel scope link src 172.17.0.1
172.18.0.0/16 dev br-f1b323ef3c02 proto kernel scope link src 172.18.0.1
192.168.1.0/24 dev enp6s0 proto kernel scope link src 192.168.1.69 metric 100
192.168.1.0/24 dev enp6s0 proto dhcp scope link src 192.168.1.69 metric 203
192.168.122.0/24 dev virbr0 proto kernel scope link src 192.168.122.1 linkdown
You're clearly not using proxies.
Please post
ip link
ip address
ip route
from host.
Also run them from within a container.
Keep in mind that I do have low level networking / router knowledge but know very little about docker.
(I prefer VMs over containers)
# cat /etc/systemd/system/docker.service.d/proxy.conf
cat: /etc/systemd/system/docker.service.d/proxy.conf: No such file or directory
# systemctl show docker --property Environment
Environment=
Also I'm not using my own Dockerfiles, did you even read my post?
]]>output of
# systemctl show docker --property Environment
After reboot everything is messed up in docker:
- DNS no longer works (not even fixed by addong --dns 1.1.1.1)
- Communication between containers in the same bridge network is broken (wtf)
- Communication between bridge networks doesn't work (expected)
- Host can ping containers but they can't ping eachother...
tested with:
docker run --rm busybox ping -c 1 1.1.1.1
-> 64 bytes from 1.1.1.1: seq=0 ttl=57 time=4.160 ms
docker run --rm busybox nslookup example.com
-> nslookup: write to '192.168.1.1': No route to host
docker run --rm --dns 1.1.1.1 busybox nslookup example.com
-> nslookup: write to '1.1.1.1': No route to host
docker network create -d bridge --internal --gateway 172.18.0.1 --subnet 172.18.0.0/16 testnet
docker run -it --rm --network=testnet --ip 172.18.10.10 subfuzion/netcat -vl 8888
another terminal: docker run -i --rm --network=testnet subfuzion/netcat 172.18.10.10 8888 < /etc/passwd
-> f1b323ef3c02d58cf07a00fc1960becf1c1377399115ad4c6a117cdacd99aa20
-> Listening on [0.0.0.0] (family 0, port 8888)
-> no output
Yes, I have checked sysctl, ipv4 forwarding is enabled for all interfaces.
I did nothing else other than update from 2020-02-29 to 2020-03-24 (I update some time at the end of each month)
Update: using the arch linux archives (mirrorlist below), downgrading the packages (pacman -Syyuu) and rebooting did NOT fix the issues. Now I'm even more clueless...
Server = https://archive.archlinux.org/repos/2020/02/29/$repo/os/$arch
Update #2: Tried updating another arch machine I run and the output of the above commands is what I would expect - networking between containers is not broken and the listening side of netcat outputs the data that is read in. This is starting to feel like I'm going to have to completely remove everything docker related and do it some other way...
]]>