resolving domain with dig is fast but using ping is slow

masenko · 2023-10-06 00:55:47

When i run `dig duckduckgo.com @10.0.1.1` it returns an ip in 10ms, and when i `ping duckduckgo.com` it shows a latency of 22ms,
but the time between new lines is sometimes up to 8 seconds, it can vary a lot. This is the output of `resolvectl monitor`:

→ Q: duckduckgo.com IN A
→ Q: duckduckgo.com IN AAAA
← S: success
← A: duckduckgo.com IN A 52.142.124.215

→ Q: 215.124.142.52.in-addr.arpa IN PTR
← S: attempts-max-reached

Every ping after the initial one, shows this:

→ Q: 215.124.142.52.in-addr.arpa IN PTR
← S: attempts-max-reached

This also happens with discord.com, but not google.com.

I have linked /etc/resolv.conf to /run/systemd/resolve/stub-resolv.conf
Here's my config:

#/etc/systemd/resolved.conf
[Resolve]
DNS=10.0.1.1

I have also tried different dns servers, such as 8.8.8.8 and 1.1.1.1 and i have managed to reproduce this on 5 other machines, running these versions:

254.5-1-arch
247.3-7+deb11u4
252.12-1~deb12u1

mpan · 2023-10-06 02:42:14

There is some confusion here.

You compare the query time⁽¹⁾ to 10.0.1.1 with the latency to duckduckgo.com. These are two different servers. The corresponding `ping` command would be `ping 10.0.1.1`.

Also 8 seconds is between what lines?
____
⁽¹⁾ Which is not the same as latency, but that is not important for the misunderstanding I point to.

Last edited by mpan (2023-10-06 02:44:29)

masenko · 2023-10-06 05:49:17

mpan wrote:

You compare the query time⁽¹⁾ to 10.0.1.1 with the latency to duckduckgo.com. These are two different servers. The corresponding `ping` command would be `ping 10.0.1.1`.

I'm not comparing dig and ping, i'm just stating that both ping and dig show low latency, which means that both my dns server and internet connection are not at fault here.

Edit:

mpan wrote:

Also 8 seconds is between what lines?

Sorry for not being more clear here, I meant the new lines of text that ping outputs. so for example:

ping duckduckgo.com
PING duckduckgo.com (52.142.124.215) 56(84) bytes of data.
64 bytes from 52.142.124.215 (52.142.124.215): icmp_seq=1 ttl=112 time=25.4 ms
(between each of these new pings, is a delay of 3 to 8 seconds)
64 bytes from 52.142.124.215 (52.142.124.215): icmp_seq=2 ttl=112 time=24.0 ms
(ex: 3 seconds)
64 bytes from 52.142.124.215 (52.142.124.215): icmp_seq=3 ttl=112 time=24.5 ms
(ex: 8 seconds)
64 bytes from 52.142.124.215 (52.142.124.215): icmp_seq=4 ttl=112 time=28.7 ms
64 bytes from 52.142.124.215 (52.142.124.215): icmp_seq=5 ttl=112 time=24.3 ms

Last edited by masenko (2023-10-06 05:52:41)

seth · 2023-10-06 06:04:47

I'm not comparing dig and ping

but mpan wrote:

You compare … to 10.0.1.1 with … duckduckgo.com

ping -c10 10.0.1.1

masenko · 2023-10-06 06:27:25

seth wrote:

ping -c10 10.0.1.1

Are you telling me to ping 10.0.1.1? If so, i'm afraid i'll have to wait until the issue pops up again.

Also, I still don't get the "comparing" thing. I'm showing that looking up the domain via my dns server, is fast. And the command `ping duckduckgo.com` also automatically looks up the domain so it can ping the ip address. But weirdly, even though dig gives a fast answer, and ping can also resolve the ip address fast, it still has a delay in between each new line. Am i making any sense?

seth · 2023-10-06 06:34:45

Am I making any sense?

No.

10.0.1.1 is a private IP, most likely in your LAN
duckduckgo.com resolves to 52.142.124.215, a public IP in Dublin - for all we know there could be one or two oceans between those locations, perhaps some desperate despote blew up a relevant landline or you just happen to be a Vodafuck consumer, god knows - and now most of the packages simply drop.

Ping resovles the IP *once*, right before it starts pinging it. Your dig queries to a local IP are completely irrelevant to your ping findings.

masenko · 2023-10-06 06:45:28

seth wrote:

Ping resovles the IP *once*, right before it starts pinging it. Your dig queries to a local IP are completely irrelevant to your ping findings.

Oh i thought ping might be looking up the domain every time because i keep seeing "attempts-max-reached" in the resolvectl. I can't find anything on that, do you know what the response means?

And i'll be waiting for the next time it happens so i can run `ping 10.0.1.1`

seth · 2023-10-06 06:58:44

I suspect that "ping 10.0.1.1" is gonna be your _gateway and fast.
Rather "tracepath duckduckgo.com" or "tracepath 52.142.124.215" to see where things stall (if) and run "ping -c10 duckduckgo.com" to get a summary.

ping might attempt a reverse lookup (actually I think it does, but though it'd not be successful for ddg/in your case), but that's orthogonal to the issue at hand - the dig reply still tells something completely different since the reverse lookup isn't cached.
You'd want

dig -x 52.142.124.215

Edit:

ping -n duckduckgo.com

will prevent the host lookup for the response.

Last edited by seth (2023-10-06 07:10:57)

mpan · 2023-10-06 07:10:59

As for the delays between ping results are displayed: does it happen if you pass -n option to ping?

By default ping does a reverse DNS lookup for each report line to tell the name of the server. I think there is no read timeout set, so indeed you may see long delays between consecutive pings. Option -n tells ping to always report the numerical address.

Another possibility is a large number of lost packets. That will be reported at the end of ping invocation.

masenko · 2023-10-06 09:03:16

Yes, i tested with -n before and it works without delays. But why does this show up every time i run `ping duckduckgo.com` without -n:

→ Q: 215.124.142.52.in-addr.arpa IN PTR
← S: attempts-max-reached

Does this have nothing to do with the big delay? if not, why does `ping google.com` have no delay, and why doesn't it print any of these lines "attempts-max-reached"?
Also, there's no lost packets.

I'm sorry if i'm having trouble to understand you two but it feels like there is some miscommunication.

Last edited by masenko (2023-10-06 09:03:58)

seth · 2023-10-06 13:28:11

The delay *might* be the reverse lookup, we don't know.
=> You want to test the dig reverse lookup performance.

google will resolve a different IP (obviously) and probably also succeed on a reverse lookup, so differences there might be due to a whole slew of things.
This is why mpan initially pointed out that you're comparing two vastly different things.

Compare the reverse lookup for the google IP w/ that for the DDG IP.

mpan · 2023-10-06 17:16:05

Comparing dig’s query time using local DNS and latency to an external server is comparing apples to oranges. But long delays between pings is a separate problem.

ping is doing a reverse lookup on each line reported and it seems a response is not coming for that long.

Do `dig -x IP_ADDR` on the IP address reported by ping and see, how long it takes.

Also, what exactly is 10.0.1.1 and why do you use it instead of common DNS services?

seth · 2023-10-06 18:12:53

Also, what exactly is 10.0.1.1 and why do you use it instead of common DNS services?

itr, sanity check:

nmap -p 53 _gateway

Raynman · 2023-10-07 11:34:39

mpan wrote:

Comparing dig’s query time using local DNS and latency to an external server is comparing apples to oranges. But long delays between pings is a separate problem.

That is the problem (reason for this thread) and nobody was comparing those things directly. Sure, the title might give the wrong impression and the OP wasn't the best (no actual question or explicit "this is the problem and these are things I thought to look at to try and narrow down the cause"), but that was further explained with the second and third post.

masenko was only comparing the delay between ping output lines against the (total/combined) time taken by things that ping might be spending this time on, like resolving the domain and waiting for a reply. That wasn't enough to explain the delay (hence this thread), but to say that it doesn't even make any sense...? Remember that this was before being told that ping would only resolve the destination IP once; while it wasn't entirely accurate, the idea was pretty similar to your suggestion to check dig's query time for a reverse lookup.

mpan wrote:

Also, what exactly is 10.0.1.1 and why do you use it instead of common DNS services?

masenko wrote:

I have also tried different dns servers, such as 8.8.8.8 and 1.1.1.1

seth · 2023-10-07 12:15:59

resolving domain with dig is fast but using ping is slow wrote:

When i run `dig duckduckgo.com @10.0.1.1` it returns an ip in 10ms, and when i `ping duckduckgo.com` it shows a latency of 22ms,
but the time between new lines is sometimes up to 8 seconds, it can vary a lot.

There's no question in the OP but the response to the above might as well be "so?"

The problem in masenko's tests is that they're changing too many variables at once - local ./. wide area network and forward ./. reverse lookup.
The local DNS might be caching and occlude more general DNS issues and the DNS changes might or not have a systemd-resolved restart.

The ping delays *might* be reverse lookups, yes. They also might be dropped packages because of a wonky link.

It also doesn't help that the problem seems intermittent and that resolved has a verly peculiar stance towards DNS cascades.
But the things that invariably need to happen to narrow this down are

dig @10.0.1.1 -x duckduckgo.com # does reverse lookup at 10.0.1.1 work?
dig @8.8.8.8 -x duckduckgo.com # does reverse lookup at google work?
ping -nc8 duckduckgo.com # is ping fast w/o host discovery
ping -c8 duckduckgo.com # is ping slow w/  host discovery - both statistics will also tell about lost packages
# EDIT: =============== this  should™ be covered by the direct dig
nmap -p53 10.0.1.1 # is this a usable DNS at all or will resolved fall over to google/cloudflare
# ======================= /EDIT
ip a # what is the network stack

Most likely causes:
1. WAN errors (because of the intermittent nature)
2. optional VPN (because of MTU issues)
3. bogus DNS @10.0.1.1 (but then the intermittent nature makes that unlikely, except resolved locking you in/out of a bogus DNS)

Last edited by seth (2023-10-07 12:17:56)

masenko · 2023-10-09 14:22:01

seth wrote:

dig @10.0.1.1 -x duckduckgo.com # does reverse lookup at 10.0.1.1 work?
dig @8.8.8.8 -x duckduckgo.com # does reverse lookup at google work?
ping -nc8 duckduckgo.com # is ping fast w/o host discovery
ping -c8 duckduckgo.com # is ping slow w/  host discovery - both statistics will also tell about lost packages
# EDIT: =============== this  should™ be covered by the direct dig
nmap -p53 10.0.1.1 # is this a usable DNS at all or will resolved fall over to google/cloudflare
# ======================= /EDIT
ip a # what is the network stack

dig @10.0.1.1 -x duckduckgo.com
;; AUTHORITY SECTION:
in-addr.arpa.		3600	IN	SOA	b.in-addr-servers.arpa. nstld.iana.org. 2022092328 1800 900 604800 3600

;; Query time: 20 msec
;; SERVER: 10.0.1.1#53(10.0.1.1) (UDP)

dig @8.8.8.8 -x duckduckgo.com
;; AUTHORITY SECTION:
in-addr.arpa.		3570	IN	SOA	b.in-addr-servers.arpa. nstld.iana.org. 2022092328 1800 900 604800 3600

;; Query time: 16 msec
;; SERVER: 8.8.8.8#53(8.8.8.8) (UDP)

ping -nc8 duckduckgo.com
PING duckduckgo.com (52.142.124.215) 56(84) bytes of data.
64 bytes from 52.142.124.215: icmp_seq=1 ttl=112 time=28.8 ms
64 bytes from 52.142.124.215: icmp_seq=2 ttl=112 time=25.7 ms
64 bytes from 52.142.124.215: icmp_seq=3 ttl=112 time=26.9 ms
64 bytes from 52.142.124.215: icmp_seq=4 ttl=112 time=22.3 ms
64 bytes from 52.142.124.215: icmp_seq=5 ttl=112 time=22.9 ms
64 bytes from 52.142.124.215: icmp_seq=6 ttl=112 time=25.0 ms
64 bytes from 52.142.124.215: icmp_seq=7 ttl=112 time=24.1 ms
64 bytes from 52.142.124.215: icmp_seq=8 ttl=112 time=23.9 ms

--- duckduckgo.com ping statistics ---
8 packets transmitted, 8 received, 0% packet loss, time 7010ms
rtt min/avg/max/mdev = 22.345/24.943/28.804/2.000 ms
(no delays between pings)

ping -c8 duckduckgo.com
PING duckduckgo.com (52.142.124.215) 56(84) bytes of data.
64 bytes from 52.142.124.215 (52.142.124.215): icmp_seq=1 ttl=112 time=26.6 ms
64 bytes from 52.142.124.215 (52.142.124.215): icmp_seq=2 ttl=112 time=35.3 ms
64 bytes from 52.142.124.215 (52.142.124.215): icmp_seq=3 ttl=112 time=30.0 ms
64 bytes from 52.142.124.215 (52.142.124.215): icmp_seq=4 ttl=112 time=29.9 ms
64 bytes from 52.142.124.215 (52.142.124.215): icmp_seq=5 ttl=112 time=31.5 ms
64 bytes from 52.142.124.215 (52.142.124.215): icmp_seq=6 ttl=112 time=29.5 ms
64 bytes from 52.142.124.215 (52.142.124.215): icmp_seq=7 ttl=112 time=40.1 ms
64 bytes from 52.142.124.215: icmp_seq=8 ttl=112 time=26.8 ms

--- duckduckgo.com ping statistics ---
8 packets transmitted, 8 received, 0% packet loss, time 54218ms
rtt min/avg/max/mdev = 26.607/31.227/40.108/4.212 ms
(long delays between pings)

nmap -p53 10.0.1.1
Nmap scan report for _gateway (10.0.1.1)
Host is up (0.00015s latency).
PORT   STATE SERVICE
53/tcp open  domain

ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host noprefixroute 
       valid_lft forever preferred_lft forever
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP group default qlen 1000
    link/ether 00:00:00:00:00:00 brd ff:ff:ff:ff:ff:ff
    altname enp6s0
    inet 10.0.1.100/24 brd 10.0.1.255 scope global dynamic noprefixroute eth0
       valid_lft 63931sec preferred_lft 63931sec

seth · 2023-10-09 17:44:37

So you're not loosing any packages, 10.0.1.1 is your gateway, there's an open DNS and the reverse lookup slows down ping.

The reverse dig has no significant query time on that DNS, does it take very long on 10.0.1.1 still?
(That's btw. not the entire dig output, is it?)

masenko · 2023-10-11 14:13:37

seth wrote:

The reverse dig has no significant query time on that DNS, does it take very long on 10.0.1.1 still?
(That's btw. not the entire dig output, is it?)

Sorry i don't get the question, and yes that's not the entire dig output.

seth · 2023-10-11 14:15:48

"dig @10.0.1.1 -x duckduckgo.com" reports "Query time: 20 msec" but how long did/does it take before the command returns?
Also always post the entire output, not random selections of it.

masenko · 2023-10-12 15:43:11

seth wrote:

"dig @10.0.1.1 -x duckduckgo.com" reports "Query time: 20 msec" but how long did/does it take before the command returns?
Also always post the entire output, not random selections of it.

first time running the command:

; <<>> DiG 9.18.19 <<>> @10.0.1.1 -x duckduckgo.com
; (1 server found)
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NXDOMAIN, id: 10559
;; flags: qr rd ra; QUERY: 1, ANSWER: 0, AUTHORITY: 1, ADDITIONAL: 1

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 1232
;; QUESTION SECTION:
;com.duckduckgo.in-addr.arpa.	IN	PTR

;; AUTHORITY SECTION:
in-addr.arpa.		3600	IN	SOA	b.in-addr-servers.arpa. nstld.iana.org. 2022092365 1800 900 604800 3600

;; Query time: 176 msec
;; SERVER: 10.0.1.1#53(10.0.1.1) (UDP)
;; WHEN: Thu Oct 12 17:40:52 CEST 2023
;; MSG SIZE  rcvd: 124

second time:

; <<>> DiG 9.18.19 <<>> @10.0.1.1 -x duckduckgo.com
; (1 server found)
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NXDOMAIN, id: 1524
;; flags: qr rd ra; QUERY: 1, ANSWER: 0, AUTHORITY: 1, ADDITIONAL: 1

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 1232
;; QUESTION SECTION:
;com.duckduckgo.in-addr.arpa.	IN	PTR

;; AUTHORITY SECTION:
in-addr.arpa.		3567	IN	SOA	b.in-addr-servers.arpa. nstld.iana.org. 2022092365 1800 900 604800 3600

;; Query time: 0 msec
;; SERVER: 10.0.1.1#53(10.0.1.1) (UDP)
;; WHEN: Thu Oct 12 17:41:25 CEST 2023
;; MSG SIZE  rcvd: 124

seth · 2023-10-12 20:13:33

So 10.0.1.1 correctly and promptly (second attempt is cached) responds NXDOMAIN for the ddg reverse lookup.
You didn't address how long the response (beyond the query time) takes, is it instant?

Is the "problem" limited to ping?

type ping
ping -i 1 -c 3 duckduckgo.com

masenko · 2023-10-16 03:18:58

seth wrote:

So 10.0.1.1 correctly and promptly (second attempt is cached) responds NXDOMAIN for the ddg reverse lookup.
You didn't address how long the response (beyond the query time) takes, is it instant?

Yes, both commands almost immediately print an output.

seth wrote:

Is the "problem" limited to ping?
type ping
ping -i 1 -c 3 duckduckgo.com

type ping
ping is /usr/bin/ping

ping -i 1 -c 3 duckduckgo.com
PING duckduckgo.com (52.142.124.215) 56(84) bytes of data.
64 bytes from 52.142.124.215 (52.142.124.215): icmp_seq=1 ttl=114 time=23.3 ms
64 bytes from 52.142.124.215 (52.142.124.215): icmp_seq=2 ttl=114 time=26.7 ms
64 bytes from 52.142.124.215: icmp_seq=3 ttl=114 time=23.8 ms

--- duckduckgo.com ping statistics ---
3 packets transmitted, 3 received, 0% packet loss, time 15300ms
rtt min/avg/max/mdev = 23.252/24.601/26.742/1.530 ms

Sorry for my late replies!

seth · 2023-10-16 07:41:58

So not just a "mis"configured ping interval…

My last-best-guess would be to disable resolved, apparently the configured DNS doesn't matter and 10.0.1.1 seems to be a caching local DNS stub, so there's no point in using resolved anyway.

masenko · 2023-10-16 13:27:39

seth wrote:

So not just a "mis"configured ping interval…
My last-best-guess would be to disable resolved, apparently the configured DNS doesn't matter and 10.0.1.1 seems to be a caching local DNS stub, so there's no point in using resolved anyway.

I tried only putting `nameserver 10.0.1.1` in /etc/resolv.conf to see if that would work but it didn't change anything. I'm using resolved because I've found that that's the only stable way to keep /etc/resolv.conf correct.

Thank you for all the help though!

seth · 2023-10-16 14:40:53

I've found that that's the only stable way to keep /etc/resolv.conf correct

What? How? Why?

How do you configure the network to being with?
Please post the output of

find /etc/systemd -type l -exec test -f {} \; -print | awk -F'/' '{ printf ("%-40s | %s\n", $(NF-0), $(NF-1)) }' | sort -f

Then disable systemd-resolved and make sure it's not auto-used by NM.
Editing the resolv.conf doesn't do anything about that - you'd have to remove it from nsswitch.conf, but for now we best just disable it entirely.

Arch Linux

#1 2023-10-06 00:55:47

resolving domain with dig is fast but using ping is slow

#2 2023-10-06 02:42:14

Re: resolving domain with dig is fast but using ping is slow

#3 2023-10-06 05:49:17

Re: resolving domain with dig is fast but using ping is slow

#4 2023-10-06 06:04:47

Re: resolving domain with dig is fast but using ping is slow

#5 2023-10-06 06:27:25

Re: resolving domain with dig is fast but using ping is slow

#6 2023-10-06 06:34:45

Re: resolving domain with dig is fast but using ping is slow

#7 2023-10-06 06:45:28

Re: resolving domain with dig is fast but using ping is slow

#8 2023-10-06 06:58:44

Re: resolving domain with dig is fast but using ping is slow

#9 2023-10-06 07:10:59

Re: resolving domain with dig is fast but using ping is slow

#10 2023-10-06 09:03:16

Re: resolving domain with dig is fast but using ping is slow

#11 2023-10-06 13:28:11

Re: resolving domain with dig is fast but using ping is slow

#12 2023-10-06 17:16:05

Re: resolving domain with dig is fast but using ping is slow

#13 2023-10-06 18:12:53

Re: resolving domain with dig is fast but using ping is slow

#14 2023-10-07 11:34:39

Re: resolving domain with dig is fast but using ping is slow

#15 2023-10-07 12:15:59

Re: resolving domain with dig is fast but using ping is slow

#16 2023-10-09 14:22:01

Re: resolving domain with dig is fast but using ping is slow

#17 2023-10-09 17:44:37

Re: resolving domain with dig is fast but using ping is slow

#18 2023-10-11 14:13:37

Re: resolving domain with dig is fast but using ping is slow

#19 2023-10-11 14:15:48

Re: resolving domain with dig is fast but using ping is slow

#20 2023-10-12 15:43:11

Re: resolving domain with dig is fast but using ping is slow

#21 2023-10-12 20:13:33

Re: resolving domain with dig is fast but using ping is slow

#22 2023-10-16 03:18:58

Re: resolving domain with dig is fast but using ping is slow

#23 2023-10-16 07:41:58

Re: resolving domain with dig is fast but using ping is slow

#24 2023-10-16 13:27:39

Re: resolving domain with dig is fast but using ping is slow

#25 2023-10-16 14:40:53

Re: resolving domain with dig is fast but using ping is slow

Board footer