You are not logged in.

Pages: 1

#1 2020-03-24 14:30:15

daweed
Member
Registered: 2020-03-24
Posts: 10

Docker networking broken after pacman -Syu

So here I am doing just a regular update. Finishes fine, no errors. I stop all my containers and reboot.

After reboot everything is messed up in docker:
- DNS no longer works (not even fixed by addong --dns 1.1.1.1)
- Communication between containers in the same bridge network is broken (wtf)
- Communication between bridge networks doesn't work (expected)
- Host can ping containers but they can't ping eachother...

tested with:

docker run --rm busybox ping -c 1 1.1.1.1

-> 64 bytes from 1.1.1.1: seq=0 ttl=57 time=4.160 ms

docker run --rm busybox nslookup example.com

-> nslookup: write to '192.168.1.1': No route to host

docker run --rm --dns 1.1.1.1 busybox nslookup example.com

-> nslookup: write to '1.1.1.1': No route to host

docker network create -d bridge --internal --gateway 172.18.0.1 --subnet 172.18.0.0/16 testnet
docker run -it --rm --network=testnet --ip 172.18.10.10 subfuzion/netcat -vl 8888
another terminal: docker run -i --rm --network=testnet subfuzion/netcat 172.18.10.10 8888 < /etc/passwd

-> f1b323ef3c02d58cf07a00fc1960becf1c1377399115ad4c6a117cdacd99aa20
-> Listening on [0.0.0.0] (family 0, port 8888)
-> no output

Yes, I have checked sysctl, ipv4 forwarding is enabled for all interfaces.
I did nothing else other than update from 2020-02-29 to 2020-03-24 (I update some time at the end of each month)

Update: using the arch linux archives (mirrorlist below), downgrading the packages (pacman -Syyuu) and rebooting did NOT fix the issues. Now I'm even more clueless...

Server = https://archive.archlinux.org/repos/2020/02/29/$repo/os/$arch

Update #2: Tried updating another arch machine I run and the output of the above commands is what I would expect - networking between containers is not broken and the listening side of netcat outputs the data that is read in. This is starting to feel like I'm going to have to completely remove everything docker related and do it some other way...

Last edited by daweed (2020-03-28 19:07:01)

Offline

#2 2020-03-28 17:39:02

daweed
Member
Registered: 2020-03-24
Posts: 10

Re: Docker networking broken after pacman -Syu

Nobody??? Seriously?

Offline

#3 2020-03-28 18:58:07

WorMzy
Administrator
From: Scotland
Registered: 2010-06-16
Posts: 12,593
Website

Re: Docker networking broken after pacman -Syu

Please don't do that.

Sakura:-
Mobo: MSI MAG X570S TORPEDO MAX // Processor: AMD Ryzen 9 5950X @4.9GHz // GFX: AMD Radeon RX 5700 XT // RAM: 32GB (4x 8GB) Corsair DDR4 (@ 3000MHz) // Storage: 1x 3TB HDD, 6x 1TB SSD, 2x 120GB SSD, 1x 275GB M2 SSD

Making lemonade from lemons since 2015.

Offline

#4 2020-03-29 13:28:47

Lone_Wolf
Administrator
From: Netherlands, Europe
Registered: 2005-10-04
Posts: 13,285

Re: Docker networking broken after pacman -Syu

content of (host)   /etc/systemd/system/docker.service.d/proxy.conf and the Dockerfile for one of your containers ?

output of 

# systemctl show docker --property Environment

Disliking systemd intensely, but not satisfied with alternatives so focusing on taming systemd.

clean chroot building not flexible enough ?
Try clean chroot manager by graysky

Offline

#5 2020-03-29 13:34:50

daweed
Member
Registered: 2020-03-24
Posts: 10

Re: Docker networking broken after pacman -Syu

# cat /etc/systemd/system/docker.service.d/proxy.conf
cat: /etc/systemd/system/docker.service.d/proxy.conf: No such file or directory
# systemctl show docker --property Environment
Environment=

Also I'm not using my own Dockerfiles, did you even read my post?

Offline

#6 2020-03-29 13:54:14

Lone_Wolf
Administrator
From: Netherlands, Europe
Registered: 2005-10-04
Posts: 13,285

Re: Docker networking broken after pacman -Syu

Yes, I did and tried to figure out how you setup docker networking .

You're clearly not using proxies.

Please post 

ip link
ip address
ip route

from host.
Also run them from within a container.

Keep in mind that I do have low level networking / router knowledge but know very little about docker.
(I prefer VMs over containers)

Last edited by Lone_Wolf (2020-03-29 13:54:44)

Disliking systemd intensely, but not satisfied with alternatives so focusing on taming systemd.

clean chroot building not flexible enough ?
Try clean chroot manager by graysky

Offline

#7 2020-03-29 14:08:30

daweed
Member
Registered: 2020-03-24
Posts: 10

Re: Docker networking broken after pacman -Syu

from host:

# ip link
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN mode DEFAULT group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
2: enp4s0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc mq state DOWN mode DEFAULT group default qlen 1000
    link/ether 00:10:18:af:dc:95 brd ff:ff:ff:ff:ff:ff
3: enp6s0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP mode DEFAULT group default qlen 1000
    link/ether 18:03:73:46:f4:2e brd ff:ff:ff:ff:ff:ff
4: virbr0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue state DOWN mode DEFAULT group default qlen 1000
    link/ether 52:54:00:3f:06:20 brd ff:ff:ff:ff:ff:ff
5: virbr0-nic: <BROADCAST,MULTICAST> mtu 1500 qdisc fq_codel master virbr0 state DOWN mode DEFAULT group default qlen 1000
    link/ether 52:54:00:3f:06:20 brd ff:ff:ff:ff:ff:ff
6: docker0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP mode DEFAULT group default 
    link/ether 02:42:f4:df:82:e8 brd ff:ff:ff:ff:ff:ff
7: br-f1b323ef3c02: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP mode DEFAULT group default 
    link/ether 02:42:d4:c1:50:1d brd ff:ff:ff:ff:ff:ff
75: vethe6ccbaa@if74: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master br-f1b323ef3c02 state UP mode DEFAULT group default 
    link/ether ca:a6:a7:60:8c:57 brd ff:ff:ff:ff:ff:ff link-netnsid 12
# ip address
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host 
       valid_lft forever preferred_lft forever
2: enp4s0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc mq state DOWN group default qlen 1000
    link/ether 00:10:18:af:dc:95 brd ff:ff:ff:ff:ff:ff
3: enp6s0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP group default qlen 1000
    link/ether 18:03:73:46:f4:2e brd ff:ff:ff:ff:ff:ff
    inet 192.168.1.69/24 brd 192.168.1.255 scope global dynamic noprefixroute enp6s0
       valid_lft 54907sec preferred_lft 54907sec
    inet6 fe80::2e21:7f19:ef75:b155/64 scope link noprefixroute 
       valid_lft forever preferred_lft forever
4: virbr0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue state DOWN group default qlen 1000
    link/ether 52:54:00:3f:06:20 brd ff:ff:ff:ff:ff:ff
    inet 192.168.122.1/24 brd 192.168.122.255 scope global virbr0
       valid_lft forever preferred_lft forever
5: virbr0-nic: <BROADCAST,MULTICAST> mtu 1500 qdisc fq_codel master virbr0 state DOWN group default qlen 1000
    link/ether 52:54:00:3f:06:20 brd ff:ff:ff:ff:ff:ff
6: docker0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default 
    link/ether 02:42:f4:df:82:e8 brd ff:ff:ff:ff:ff:ff
    inet 172.17.0.1/16 brd 172.17.255.255 scope global docker0
       valid_lft forever preferred_lft forever
    inet6 fe80::42:f4ff:fedf:82e8/64 scope link 
       valid_lft forever preferred_lft forever
7: br-f1b323ef3c02: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default 
    link/ether 02:42:d4:c1:50:1d brd ff:ff:ff:ff:ff:ff
    inet 172.18.0.1/16 brd 172.18.255.255 scope global br-f1b323ef3c02
       valid_lft forever preferred_lft forever
    inet6 fe80::42:d4ff:fec1:501d/64 scope link 
       valid_lft forever preferred_lft forever
75: vethe6ccbaa@if74: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master br-f1b323ef3c02 state UP group default 
    link/ether ca:a6:a7:60:8c:57 brd ff:ff:ff:ff:ff:ff link-netnsid 12
    inet6 fe80::c8a6:a7ff:fe60:8c57/64 scope link 
       valid_lft forever preferred_lft forever
# ip route
default via 192.168.1.1 dev enp6s0 proto dhcp metric 100 
default via 192.168.1.1 dev enp6s0 proto dhcp src 192.168.1.69 metric 203 
172.17.0.0/16 dev docker0 proto kernel scope link src 172.17.0.1 
172.18.0.0/16 dev br-f1b323ef3c02 proto kernel scope link src 172.18.0.1
192.168.1.0/24 dev enp6s0 proto kernel scope link src 192.168.1.69 metric 100 
192.168.1.0/24 dev enp6s0 proto dhcp scope link src 192.168.1.69 metric 203 
192.168.122.0/24 dev virbr0 proto kernel scope link src 192.168.122.1 linkdown

Last edited by daweed (2020-03-29 14:09:10)

Offline

#8 2020-03-29 14:15:45

daweed
Member
Registered: 2020-03-24
Posts: 10

Re: Docker networking broken after pacman -Syu

from container:

container ran using:

docker run -it --rm --network=testnet --ip 172.18.10.10 joffotron/docker-net-tools
# ip link
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
78: eth0@if79: <BROADCAST,MULTICAST,UP,LOWER_UP,M-DOWN> mtu 1500 qdisc noqueue state UP 
    link/ether 02:42:ac:12:0a:0a brd ff:ff:ff:ff:ff:ff
# ip address
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
78: eth0@if79: <BROADCAST,MULTICAST,UP,LOWER_UP,M-DOWN> mtu 1500 qdisc noqueue state UP 
    link/ether 02:42:ac:12:0a:0a brd ff:ff:ff:ff:ff:ff
    inet 172.18.10.10/16 brd 172.18.255.255 scope global eth0
       valid_lft forever preferred_lft forever
# ip route
default via 172.18.0.1 dev eth0 
172.18.0.0/16 dev eth0  src 172.18.10.10

Offline

#9 2020-03-29 14:46:47

Lone_Wolf
Administrator
From: Netherlands, Europe
Registered: 2005-10-04
Posts: 13,285

Re: Docker networking broken after pacman -Syu

From inside that container, output of 

$ ping 172.18.0.1
$ ping 192.168.1.69
$ ping 192.168.1.1

Disliking systemd intensely, but not satisfied with alternatives so focusing on taming systemd.

clean chroot building not flexible enough ?
Try clean chroot manager by graysky

Offline

#10 2020-03-29 14:50:52

daweed
Member
Registered: 2020-03-24
Posts: 10

Re: Docker networking broken after pacman -Syu

Can ping the first two, but not 192.168.1.1

Offline

#11 2020-03-29 15:07:54

Lone_Wolf
Administrator
From: Netherlands, Europe
Registered: 2005-10-04
Posts: 13,285

Re: Docker networking broken after pacman -Syu

Ok , it looks like traffic does get out of the container, but is blocked further on.

Inside the container add a route

ip route add 192.168.1.1/32 dev eth0

test pinging 192.168.1.1 and 1.1.1.1 .

Disliking systemd intensely, but not satisfied with alternatives so focusing on taming systemd.

clean chroot building not flexible enough ?
Try clean chroot manager by graysky

Offline

#12 2020-03-29 15:36:35

daweed
Member
Registered: 2020-03-24
Posts: 10

Re: Docker networking broken after pacman -Syu

# ip route add 192.168.1.1/32 dev eth0
ip: RTNETLINK answers: Operation not permitted

Offline

#13 2020-03-29 15:40:32

daweed
Member
Registered: 2020-03-24
Posts: 10

Re: Docker networking broken after pacman -Syu

The thing is though... that not working is correct. It does that on a machine where docker networking is working propperly too.

Offline

#14 2020-03-29 15:51:13

daweed
Member
Registered: 2020-03-24
Posts: 10

Re: Docker networking broken after pacman -Syu

This is where it fails:

# docker run -it --rm --network=bridge joffotron/docker-net-tools

Then run these commands...

On the broken machine:

/ # ping -c 1 1.1.1.1
PING 1.1.1.1 (1.1.1.1): 56 data bytes
64 bytes from 1.1.1.1: seq=0 ttl=57 time=3.942 ms

--- 1.1.1.1 ping statistics ---
1 packets transmitted, 1 packets received, 0% packet loss
round-trip min/avg/max = 3.942/3.942/3.942 ms
/ # dig one.one.one.one

; <<>> DiG 9.10.4-P3 <<>> one.one.one.one
;; global options: +cmd
;; connection timed out; no servers could be reached

And on a working one:

/ # ping -c 1 1.1.1.1
PING 1.1.1.1 (1.1.1.1): 56 data bytes
64 bytes from 1.1.1.1: seq=0 ttl=56 time=8.860 ms

--- 1.1.1.1 ping statistics ---
1 packets transmitted, 1 packets received, 0% packet loss
round-trip min/avg/max = 8.860/8.860/8.860 ms
/ # dig one.one.one.one

; <<>> DiG 9.10.4-P3 <<>> one.one.one.one
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 12414
;; flags: qr rd ra; QUERY: 1, ANSWER: 2, AUTHORITY: 0, ADDITIONAL: 1

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 1452
;; QUESTION SECTION:
;one.one.one.one.		IN	A

;; ANSWER SECTION:
one.one.one.one.	149	IN	A	1.0.0.1
one.one.one.one.	149	IN	A	1.1.1.1

;; Query time: 9 msec
;; SERVER: 1.1.1.1#53(1.1.1.1)
;; WHEN: Sun Mar 29 15:50:50 UTC 2020
;; MSG SIZE  rcvd: 76

Offline

#15 2020-03-29 19:26:43

daweed
Member
Registered: 2020-03-24
Posts: 10

Re: Docker networking broken after pacman -Syu

Now here's something curious:

Container 1:

# docker run -it --rm --network testnet --ip 172.18.10.10 crccheck/hello-world

Container 2:

# docker run -it --rm --network=testnet --ip 172.18.10.11 joffotron/docker-net-tools

Inside container 2:

- On the broken machine:

/ # ping -c 1 172.18.10.10
PING 172.18.10.10 (172.18.10.10): 56 data bytes
64 bytes from 172.18.10.10: seq=0 ttl=64 time=0.149 ms

--- 172.18.10.10 ping statistics ---
1 packets transmitted, 1 packets received, 0% packet loss
round-trip min/avg/max = 0.149/0.149/0.149 ms
/ # curl http://172.18.10.10:8000
curl: (7) Failed to connect to 172.18.10.10 port 8000: Host is unreachable

- On a working machine:

/ # ping -c 1 172.18.10.10
PING 172.18.10.10 (172.18.10.10): 56 data bytes
64 bytes from 172.18.10.10: seq=0 ttl=64 time=0.063 ms

--- 172.18.10.10 ping statistics ---
1 packets transmitted, 1 packets received, 0% packet loss
round-trip min/avg/max = 0.063/0.063/0.063 ms
/ # curl http://172.18.10.10:8000
<xmp>
Hello World


                                       ##         .
                                 ## ## ##        ==
                              ## ## ## ## ##    ===
                           /""""""""""""""""\___/ ===
                      ~~~ {~~ ~~~~ ~~~ ~~~~ ~~ ~ /  ===- ~~~
                           \______ o          _,/
                            \      \       _,'
                             `'--.._\..--''
</xmp>

Offline

Pages: 1

Board footer

Powered by FluxBB