Why is Docker messing up my routing table and causing DNS failure?

abruptadieu · 2021-06-29 03:29:15

After installing Docker, I noticed that my DNS queries were failing. It seems that Docker is adding a routing table entry that is causing this. Why is it doing this, and how do I fix it?

here is some debugging info. First, my routing table before and after starting Docker. I'm guessing that the [pre]192.168.1.0/24 dev br-918c82870cac ...[/pre] entry is at fault.

[sherman@airplane ~]$ ip route
default via 192.168.1.1 dev enp2s0 proto dhcp src 192.168.1.204 metric 1024
192.168.1.0/24 dev enp2s0 proto kernel scope link src 192.168.1.204
192.168.1.1 dev enp2s0 proto dhcp scope link src 192.168.1.204 metric 1024
[sherman@airplane ~]$
[sherman@airplane ~]$ sudo  systemctl start docker
[sudo] password for sherman:
[sherman@airplane ~]$
[sherman@airplane ~]$
[sherman@airplane ~]$ ip route
default via 192.168.1.1 dev enp2s0 proto dhcp src 192.168.1.204 metric 1024
172.17.0.0/16 dev docker0 proto kernel scope link src 172.17.0.1 linkdown
192.168.1.0/24 dev enp2s0 proto kernel scope link src 192.168.1.204
192.168.1.0/24 dev br-918c82870cac proto kernel scope link src 192.168.1.1 linkdown
192.168.1.1 dev enp2s0 proto dhcp scope link src 192.168.1.204 metric 1024

Some DNS queries before and after starting Docker. Using different domains to avoid cached responses from systemd-resolved

[sherman@airplane ~]$ resolvectl
Global
           Protocols: +LLMNR +mDNS -DNSOverTLS DNSSEC=no/unsupported
    resolv.conf mode: stub
Fallback DNS Servers: 1.1.1.1 9.9.9.10 8.8.8.8 2606:4700:4700::1111 2620:fe::10 2001:4860:4860::8888

Link 2 (enp2s0)
    Current Scopes: DNS LLMNR/IPv4 LLMNR/IPv6
         Protocols: +DefaultRoute +LLMNR -mDNS -DNSOverTLS DNSSEC=no/unsupported
Current DNS Server: 192.168.1.1
       DNS Servers: 192.168.1.1
[sherman@airplane ~]$
[sherman@airplane ~]$ host google.com
google.com has address 142.250.191.142
google.com has IPv6 address 2607:f8b0:4009:818::200e
google.com mail is handled by 10 aspmx.l.google.com.
google.com mail is handled by 20 alt1.aspmx.l.google.com.
google.com mail is handled by 30 alt2.aspmx.l.google.com.
google.com mail is handled by 40 alt3.aspmx.l.google.com.
google.com mail is handled by 50 alt4.aspmx.l.google.com.
[sherman@airplane ~]$
[sherman@airplane ~]$
[sherman@airplane ~]$
[sherman@airplane ~]$ host google.com
google.com has address 142.250.191.142
google.com has IPv6 address 2607:f8b0:4009:818::200e
google.com mail is handled by 20 alt1.aspmx.l.google.com.
google.com mail is handled by 30 alt2.aspmx.l.google.com.
google.com mail is handled by 10 aspmx.l.google.com.
google.com mail is handled by 50 alt4.aspmx.l.google.com.
google.com mail is handled by 40 alt3.aspmx.l.google.com.
[sherman@airplane ~]$
[sherman@airplane ~]$ host facebook.com
facebook.com has address 157.240.18.35
facebook.com has IPv6 address 2a03:2880:f127:283:face:b00c:0:25de
facebook.com mail is handled by 10 smtpin.vvv.facebook.com.
[sherman@airplane ~]$
[sherman@airplane ~]$ sudo  systemctl start docker
[sudo] password for sherman:
[sherman@airplane ~]$
[sherman@airplane ~]$
[sherman@airplane ~]$ host reddit.com
Host reddit.com not found: 2(SERVFAIL)

[sherman@airplane ~]$ resolvectl
Global
           Protocols: +LLMNR +mDNS -DNSOverTLS DNSSEC=no/unsupported
    resolv.conf mode: stub
Fallback DNS Servers: 1.1.1.1 9.9.9.10 8.8.8.8 2606:4700:4700::1111 2620:fe::10 2001:4860:4860::8888

Link 2 (enp2s0)
    Current Scopes: DNS LLMNR/IPv4 LLMNR/IPv6
         Protocols: +DefaultRoute +LLMNR -mDNS -DNSOverTLS DNSSEC=no/unsupported
Current DNS Server: 192.168.1.1
       DNS Servers: 192.168.1.1

Link 3 (br-918c82870cac)
Current Scopes: none
     Protocols: -DefaultRoute +LLMNR -mDNS -DNSOverTLS DNSSEC=no/unsupported

Link 4 (docker0)
Current Scopes: none
     Protocols: -DefaultRoute +LLMNR -mDNS -DNSOverTLS DNSSEC=no/unsupported
[sherman@airplane ~]$
[sherman@airplane ~]$
[sherman@airplane ~]$ sudo  systemctl stop docker
Warning: Stopping docker.service, but it can still be activated by:
  docker.socket
[sherman@airplane ~]$
[sherman@airplane ~]$
[sherman@airplane ~]$ resolvectl
Global
           Protocols: +LLMNR +mDNS -DNSOverTLS DNSSEC=no/unsupported
    resolv.conf mode: stub
Fallback DNS Servers: 1.1.1.1 9.9.9.10 8.8.8.8 2606:4700:4700::1111 2620:fe::10 2001:4860:4860::8888

Link 2 (enp2s0)
    Current Scopes: DNS LLMNR/IPv4 LLMNR/IPv6
         Protocols: +DefaultRoute +LLMNR -mDNS -DNSOverTLS DNSSEC=no/unsupported
Current DNS Server: 192.168.1.1
       DNS Servers: 192.168.1.1

Link 3 (br-918c82870cac)
Current Scopes: none
     Protocols: -DefaultRoute +LLMNR -mDNS -DNSOverTLS DNSSEC=no/unsupported

Link 4 (docker0)
Current Scopes: none
     Protocols: -DefaultRoute +LLMNR -mDNS -DNSOverTLS DNSSEC=no/unsupported
[sherman@airplane ~]$
[sherman@airplane ~]$ host reddit.com
Host reddit.com not found: 2(SERVFAIL)
[sherman@airplane ~]$ host slashdot.org
Host slashdot.org not found: 2(SERVFAIL)
[sherman@airplane ~]$
[sherman@airplane ~]$
[sherman@airplane ~]$ host slashdot.org
Host slashdot.org not found: 2(SERVFAIL)

tcpdump output matching the above DNS queries, showing that after starting Docker, the systemd-resolved sends the queries over lo0 instead of the external interface, causing them to fail.

[sherman@airplane ~]$ sudo tcpdump -n -vvv -i any port 53
[sudo] password for sherman:
tcpdump: data link type LINUX_SLL2
tcpdump: listening on any, link-type LINUX_SLL2 (Linux cooked v2), snapshot length 262144 bytes

21:37:23.311811 lo    In  IP (tos 0x0, ttl 64, id 28286, offset 0, flags [none], proto UDP (17), length 56)
    127.0.0.1.35659 > 127.0.0.53.53: [bad udp cksum 0xfe6b -> 0x69c4!] 63601+ A? google.com. (28)
21:37:23.311927 lo    In  IP (tos 0x0, ttl 1, id 15901, offset 0, flags [DF], proto UDP (17), length 72)
    127.0.0.53.53 > 127.0.0.1.35659: [bad udp cksum 0xfe7b -> 0xd970!] 63601 q: A? google.com. 1/0/0 google.com. [4m38s] A 142.250.191.142 (44)
21:37:23.312129 lo    In  IP (tos 0x0, ttl 64, id 28287, offset 0, flags [none], proto UDP (17), length 56)
    127.0.0.1.37241 > 127.0.0.53.53: [bad udp cksum 0xfe6b -> 0x724b!] 59809+ AAAA? google.com. (28)
21:37:23.312218 lo    In  IP (tos 0x0, ttl 1, id 15902, offset 0, flags [DF], proto UDP (17), length 84)
    127.0.0.53.53 > 127.0.0.1.37241: [bad udp cksum 0xfe87 -> 0xa95a!] 59809 q: AAAA? google.com. 1/0/0 google.com. [4m38s] AAAA 2607:f8b0:4009:818::200e (56)
21:37:23.312371 lo    In  IP (tos 0x0, ttl 64, id 28288, offset 0, flags [none], proto UDP (17), length 56)
    127.0.0.1.53398 > 127.0.0.53.53: [bad udp cksum 0xfe6b -> 0x5b02!] 49626+ MX? google.com. (28)
21:37:23.312508 lo    In  IP (tos 0x0, ttl 1, id 15903, offset 0, flags [DF], proto UDP (17), length 164)
    127.0.0.53.53 > 127.0.0.1.53398: [bad udp cksum 0xfed7 -> 0x57ac!] 49626 q: MX? google.com. 5/0/0 google.com. [9m38s] MX alt1.aspmx.l.google.com. 20, google.com. [9m38s] MX alt2.aspmx.l.google.com. 30, google.com. [9m38s] MX aspmx.l.google.com. 10, google.com. [9m38s] MX alt4.aspmx.l.google.com. 50, google.com. [9m38s] MX alt3.aspmx.l.google.com. 40 (136)
21:37:34.570743 lo    In  IP (tos 0x0, ttl 64, id 29146, offset 0, flags [none], proto UDP (17), length 58)
    127.0.0.1.33800 > 127.0.0.53.53: [bad udp cksum 0xfe6d -> 0x0c29!] 62963+ A? facebook.com. (30)
21:37:34.570872 enp2s0 Out IP (tos 0x0, ttl 64, id 28248, offset 0, flags [none], proto UDP (17), length 69)
    192.168.1.204.58418 > 192.168.1.1.53: [bad udp cksum 0xac60 -> 0x02f0!] 51201+ [1au] A? facebook.com. ar: . OPT UDPsize=512 (41)
21:37:34.588975 enp2s0 In  IP (tos 0x0, ttl 64, id 20396, offset 0, flags [DF], proto UDP (17), length 85)
    192.168.1.1.53 > 192.168.1.204.58418: [udp sum ok] 51201 q: A? facebook.com. 1/0/1 facebook.com. [3m46s] A 157.240.18.35 ar: . OPT UDPsize=4096 (57)
21:37:34.589054 lo    In  IP (tos 0x0, ttl 1, id 15980, offset 0, flags [DF], proto UDP (17), length 74)
    127.0.0.53.53 > 127.0.0.1.33800: [bad udp cksum 0xfe7d -> 0x1a7f!] 62963 q: A? facebook.com. 1/0/0 facebook.com. [3m46s] A 157.240.18.35 (46)
21:37:34.589280 lo    In  IP (tos 0x0, ttl 64, id 29148, offset 0, flags [none], proto UDP (17), length 58)
    127.0.0.1.48941 > 127.0.0.53.53: [bad udp cksum 0xfe6d -> 0x282b!] 40625+ AAAA? facebook.com. (30)
21:37:34.589421 enp2s0 Out IP (tos 0x0, ttl 64, id 28251, offset 0, flags [none], proto UDP (17), length 69)
    192.168.1.204.41954 > 192.168.1.1.53: [bad udp cksum 0xac60 -> 0x51f6!] 47408+ [1au] AAAA? facebook.com. ar: . OPT UDPsize=512 (41)
21:37:34.607485 enp2s0 In  IP (tos 0x0, ttl 64, id 20398, offset 0, flags [DF], proto UDP (17), length 97)
    192.168.1.1.53 > 192.168.1.204.41954: [udp sum ok] 47408 q: AAAA? facebook.com. 1/0/1 facebook.com. [2m19s] AAAA 2a03:2880:f127:283:face:b00c:0:25de ar: . OPT UDPsize=4096 (69)
21:37:34.607558 lo    In  IP (tos 0x0, ttl 1, id 15982, offset 0, flags [DF], proto UDP (17), length 86)
    127.0.0.53.53 > 127.0.0.1.48941: [bad udp cksum 0xfe89 -> 0xcfc4!] 40625 q: AAAA? facebook.com. 1/0/0 facebook.com. [2m19s] AAAA 2a03:2880:f127:283:face:b00c:0:25de (58)
21:37:34.607744 lo    In  IP (tos 0x0, ttl 64, id 29154, offset 0, flags [none], proto UDP (17), length 58)
    127.0.0.1.56873 > 127.0.0.53.53: [bad udp cksum 0xfe6d -> 0x48b0!] 24381+ MX? facebook.com. (30)
21:37:34.607849 enp2s0 Out IP (tos 0x0, ttl 64, id 28257, offset 0, flags [none], proto UDP (17), length 69)
    192.168.1.204.60138 > 192.168.1.1.53: [bad udp cksum 0xac60 -> 0x4150!] 33499+ [1au] MX? facebook.com. ar: . OPT UDPsize=512 (41)
21:37:34.647541 enp2s0 In  IP (tos 0x0, ttl 64, id 20399, offset 0, flags [DF], proto UDP (17), length 96)
    192.168.1.1.53 > 192.168.1.204.60138: [udp sum ok] 33499 q: MX? facebook.com. 1/0/1 facebook.com. [1h] MX smtpin.vvv.facebook.com. 10 ar: . OPT UDPsize=4096 (68)
21:37:34.647729 lo    In  IP (tos 0x0, ttl 1, id 15988, offset 0, flags [DF], proto UDP (17), length 85)
    127.0.0.53.53 > 127.0.0.1.56873: [bad udp cksum 0xfe88 -> 0xae27!] 24381 q: MX? facebook.com. 1/0/0 facebook.com. [1h] MX smtpin.vvv.facebook.com. 10 (57)
21:38:13.471343 lo    In  IP (tos 0x0, ttl 64, id 35389, offset 0, flags [none], proto UDP (17), length 56)
    127.0.0.1.43379 > 127.0.0.53.53: [bad udp cksum 0xfe6b -> 0x0b17!] 14074+ A? reddit.com. (28)
21:38:13.471463 lo    In  IP (tos 0x0, ttl 64, id 39395, offset 0, flags [none], proto UDP (17), length 67)
    192.168.1.204.41934 > 192.168.1.1.53: [bad udp cksum 0xac5e -> 0xe2bf!] 36341+ [1au] A? reddit.com. ar: . OPT UDPsize=512 (39)
21:38:13.471504 lo    In  IP (tos 0x0, ttl 1, id 19331, offset 0, flags [DF], proto UDP (17), length 56)
    127.0.0.53.53 > 127.0.0.1.43379: [bad udp cksum 0xfe6b -> 0x8694!] 14074 ServFail* q: A? reddit.com. 0/0/0 (28)
21:39:09.086356 lo    In  IP (tos 0x0, ttl 64, id 46175, offset 0, flags [none], proto UDP (17), length 56)
    127.0.0.1.41125 > 127.0.0.53.53: [bad udp cksum 0xfe6b -> 0xdd13!] 28107+ A? reddit.com. (28)
21:39:09.086461 lo    In  IP (tos 0x0, ttl 64, id 43063, offset 0, flags [none], proto UDP (17), length 67)
    192.168.1.204.49573 > 192.168.1.1.53: [bad udp cksum 0xac5e -> 0x3293!] 8267+ [1au] A? reddit.com. ar: . OPT UDPsize=512 (39)
21:39:09.086501 lo    In  IP (tos 0x0, ttl 1, id 34632, offset 0, flags [DF], proto UDP (17), length 56)
    127.0.0.53.53 > 127.0.0.1.41125: [bad udp cksum 0xfe6b -> 0x5891!] 28107 ServFail* q: A? reddit.com. 0/0/0 (28)
21:39:30.933561 lo    In  IP (tos 0x0, ttl 64, id 50122, offset 0, flags [none], proto UDP (17), length 58)
    127.0.0.1.47414 > 127.0.0.53.53: [bad udp cksum 0xfe6d -> 0x83d3!] 11271+ A? slashdot.org. (30)
21:39:30.933655 lo    In  IP (tos 0x0, ttl 64, id 47830, offset 0, flags [none], proto UDP (17), length 69)
    192.168.1.204.45823 > 192.168.1.1.53: [bad udp cksum 0xac60 -> 0x01ab!] 56677+ [1au] A? slashdot.org. ar: . OPT UDPsize=512 (41)
21:39:30.933692 lo    In  IP (tos 0x0, ttl 1, id 34906, offset 0, flags [DF], proto UDP (17), length 58)
    127.0.0.53.53 > 127.0.0.1.47414: [bad udp cksum 0xfe6d -> 0xff50!] 11271 ServFail* q: A? slashdot.org. 0/0/0 (30)
21:40:09.643750 lo    In  IP (tos 0x0, ttl 64, id 51821, offset 0, flags [none], proto UDP (17), length 58)
    127.0.0.1.45293 > 127.0.0.53.53: [bad udp cksum 0xfe6d -> 0xb6e8!] 315+ A? slashdot.org. (30)
21:40:09.643865 lo    In  IP (tos 0x0, ttl 64, id 56983, offset 0, flags [none], proto UDP (17), length 69)
    192.168.1.204.57758 > 192.168.1.1.53: [bad udp cksum 0xac60 -> 0x0fa0!] 41169+ [1au] A? slashdot.org. ar: . OPT UDPsize=512 (41)
21:40:09.643902 lo    In  IP (tos 0x0, ttl 1, id 41445, offset 0, flags [DF], proto UDP (17), length 58)
    127.0.0.53.53 > 127.0.0.1.45293: [bad udp cksum 0xfe6d -> 0x3266!] 315 ServFail* q: A? slashdot.org. 0/0/0 (30)

progandy · 2021-06-29 07:48:14

I have no idea why that happens, but

Using different domains to avoid cached responses from systemd-resolved

You can flush the cache with

resolvectl flush-caches

Edit: systemd-resolved should listen on 127.0.0.53 for applications talking to servers from /etc/resolv.conf and then forward the queries to the real dns servers.

Last edited by progandy (2021-06-29 07:50:34)

Arch Linux

#1 2021-06-29 03:29:15

Why is Docker messing up my routing table and causing DNS failure?

#2 2021-06-29 07:48:14

Re: Why is Docker messing up my routing table and causing DNS failure?

Board footer