Arch Linux

george55

Member

Registered: 2011-08-10

Posts: 68

DNS resolving comes to a crawl once Docker is started

I have this pretty odd problem. DNS resolving becomes incredibly slow, but only after I start my Docker container. I am not talking about DNS resolving inside the container, but about DNS resolving on my machine itself. The weird thing is that a) this started when I changed ISP and started using a new router, but b) it only happens after I start the Docker container... until I start the container everything is perfectly fine, and c) even when I stop the Docker container, the problem persists.

Here is the procedure:

- I boot my machine. Go online. DNS is fast.
- I manually start docker (systemctl start docker). DNS is fast.
- I start a docker container (nothing special, it's the same container I have used for ages and nothing has changed in this container recently.
- Once the container has started, DNS starts acting up (extremely slow)

Here is what I have tried:

- I stop the container. DNS remains slow.
- I stop docker altogether (systemctl stop docker). DNS remains slow.
- I restart iptables (which gets changed by Docker), and load my old ruleset. DNS remains slow.
- Instead of using 8.8.8.8 in my resolv.conf, I installed a local DNS server (unbound). Nothing improves.
- I try reconnecting to my network. No changes.
- I tried with a number of very different containers; problem always occurs

The only thing that helps is rebooting my machine altogether. DNS will be fast. Until I start a docker container.

I tried reconnecting to my previous router. I reboot, start docker, and... no problem anymore! So it's something that is linked to the new router + docker... but how?!

Example ping before starting the container:

time ping -c 1 google.co.uk
PING google.co.uk (172.217.170.35) 56(84) bytes of data.
64 bytes from jnb02s03-in-f3.1e100.net (172.217.170.35): icmp_seq=1 ttl=57 time=26.9 ms

--- google.co.uk ping statistics ---
1 packets transmitted, 1 received, 0% packet loss, time 0ms
rtt min/avg/max/mdev = 26.893/26.893/26.893/0.000 ms

real	0m0.062s
user	0m0.023s
sys	0m0.009s

Timeouts will happen after I have started up the container:

time ping -c 1 google.co.uk
ping: google.co.uk: Temporary failure in name resolution

real	0m10.031s
user	0m0.014s
sys	0m0.006s

Here are some screenshots from the network tab in Firefox:

Before starting the container:

After starting the container:

I am at a loss here. It doesn't seem to be just docker - but a particular container. But after I stop it, the problem persists. But the problem only started when I got a new router, which is a very odd combination of symptoms. What could be the cause here, anybody any idea?

Using wicd for networking, i3 window manager, no desktop environment, unbound for dns (but I had been using Google DNS before installing this, same results). Kernel 5.1.7-arch1-1-ARCH.

Last edited by george55 (2019-06-08 18:52:44)

george55

Member

Registered: 2011-08-10

Posts: 68

Re: DNS resolving comes to a crawl once Docker is started

Here is some additional output. This is a series of dig commands that I am running AFTER starting a docker container. As you can see: a) it doesn't matter if I use my local DNS server or Google's DNS server, results are the same, and b) the response is very intermittent... often there is a time-out, but sometimes I do get a response, which varies from pretty slow to very decent (but that may well be the result of it being cached after a successful attempt).

george@home ~  $ time dig example.org @8.8.8.8

; <<>> DiG 9.14.2 <<>> example.org @8.8.8.8
;; global options: +cmd
;; connection timed out; no servers could be reached

real	0m15.020s
user	0m0.012s
sys	0m0.006s
george@home ~  $ time dig example.org @8.8.8.8

; <<>> DiG 9.14.2 <<>> example.org @8.8.8.8
;; global options: +cmd
;; connection timed out; no servers could be reached

real	0m15.022s
user	0m0.016s
sys	0m0.006s
george@home ~  $ time dig example.org @8.8.8.8

; <<>> DiG 9.14.2 <<>> example.org @8.8.8.8
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 36890
;; flags: qr rd ra ad; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 1

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 512
;; QUESTION SECTION:
;example.org.			IN	A

;; ANSWER SECTION:
example.org.		16609	IN	A	93.184.216.34

;; Query time: 20 msec
;; SERVER: 8.8.8.8#53(8.8.8.8)
;; WHEN: Sun Jun 09 08:45:50 SAST 2019
;; MSG SIZE  rcvd: 56


real	0m10.038s
user	0m0.007s
sys	0m0.010s
george@home ~  $ time dig example.org

; <<>> DiG 9.14.2 <<>> example.org
;; global options: +cmd
;; connection timed out; no servers could be reached

real	0m18.019s
user	0m0.009s
sys	0m0.009s
george@home ~  $ time dig example.org

; <<>> DiG 9.14.2 <<>> example.org
;; global options: +cmd
;; connection timed out; no servers could be reached

real	0m18.021s
user	0m0.010s
sys	0m0.010s
george@home ~  $ time dig example.org

; <<>> DiG 9.14.2 <<>> example.org
;; global options: +cmd
;; connection timed out; no servers could be reached

real	0m18.024s
user	0m0.009s
sys	0m0.011s
george@home ~  $ time dig example.org

; <<>> DiG 9.14.2 <<>> example.org
;; global options: +cmd
;; connection timed out; no servers could be reached

real	0m18.021s
user	0m0.010s
sys	0m0.009s
george@home ~  $ time dig example.org

; <<>> DiG 9.14.2 <<>> example.org
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 34938
;; flags: qr rd ra ad; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 1

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 4096
;; QUESTION SECTION:
;example.org.			IN	A

;; ANSWER SECTION:
example.org.		86399	IN	A	93.184.216.34

;; Query time: 741 msec
;; SERVER: ::1#53(::1)
;; WHEN: Sun Jun 09 08:47:35 SAST 2019
;; MSG SIZE  rcvd: 56


real	0m0.765s
user	0m0.014s
sys	0m0.006s
george@home ~  $ time dig example.org

; <<>> DiG 9.14.2 <<>> example.org
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 56931
;; flags: qr rd ra ad; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 1

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 4096
;; QUESTION SECTION:
;example.org.			IN	A

;; ANSWER SECTION:
example.org.		86396	IN	A	93.184.216.34

;; Query time: 0 msec
;; SERVER: ::1#53(::1)
;; WHEN: Sun Jun 09 08:47:38 SAST 2019
;; MSG SIZE  rcvd: 56


real	0m0.028s
user	0m0.007s
sys	0m0.011s
george@home ~  $ time dig example.org

; <<>> DiG 9.14.2 <<>> example.org
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 61122
;; flags: qr rd ra ad; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 1

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 4096
;; QUESTION SECTION:
;example.org.			IN	A

;; ANSWER SECTION:
example.org.		86393	IN	A	93.184.216.34

;; Query time: 3 msec
;; SERVER: ::1#53(::1)
;; WHEN: Sun Jun 09 08:47:41 SAST 2019
;; MSG SIZE  rcvd: 56


real	0m0.024s
user	0m0.013s
sys	0m0.007s