Intermittent DNS resolution problems

jernst · 2019-02-22 22:44:14

I'm seeing intermittent DNS hostname resolution issues. It's usually like this:

attempt to access example.com (well, some other domain; doesn't matter what)
times out
second attempt: works
sometimes it takes more than two attempts.

This is irrespective of client app and (I think) protocol, and happens from web browser/http to curl and ping.

This only occurs on my Arch box, not on any other boxes on the same LAN. DNS is resolved, for all of them, via dnsmasq that's running on some other local box and that delegates to the ISP's DNS. I'm running systemd-resolved with defaults plus "DNSStubListener=no"

It does not seem to be a DNS lookup problem per se, because "dig example.com" works even while apps time out. It's not a connectivity problem either, because I can ping the IP addresses (but not the DNS name) of example.com.

So it appears to me /etc/nsswitch.conf might be the culprit. It is currently:

hosts: files mymachines myhostname resolve [!UNAVAIL=return] dns

which I believe I copied off the wiki at some point, but I cannot currently find it there. (Does this make sense? I'm unclear about the UNAVAIL=return, I'm not quite understanding what the man page says. It may be working better without that, but I'm not certain.)

systemd-resolved messages:

Feb 22 22:25:54 ubos-ec2 systemd-resolved[677]: Using degraded feature set (UDP+EDNS0) for DNS server xx.xx.xx.xx.
Feb 22 22:25:54 ubos-ec2 systemd-resolved[677]: DNSSEC validation failed for question example.com IN DS: failed-auxiliary
Feb 22 22:25:54 ubos-ec2 systemd-resolved[677]: DNSSEC validation failed for question example.com IN SOA: failed-auxiliary

Not sure what those are supposed to mean either. The domain in question does not have DNSSEC setup.

Any ideas? Thank you!

seth · 2019-02-23 08:29:20

plus "DNSStubListener=no"

Why? Is there some other dns provider (dnsmasq) involved?

The DNS server you redacted is broken, add "DNSSEC=off" to resolved.conf - the default is allow-downgrade which will attempt DNSSEC and only resort to the unsecure query if DNSSEC isn't available. You DNS doesn't provide DNSSEC and (very most likely) misimplements EDNS what's going to break the downgrade. iow: actually kick that DNS server.

jernst · 2019-02-23 17:58:27

Thanks, Seth: There is currently no other DNS server involved. (There used to be when I ran this box as a router/NAT with dnsmasq. I think that's why I set "DNSStubListener=no", and I left it because I didn't see how it gets in the way of traditional DNS lookups, even after dnsmasq was gone. Am I wrong about that?)

I guess my basic problem is that I don't understand how how DNS resolution works with 127.0.0.53. Here is what I think happens:

A hostname lookup is needed by something like curl example.com.
The GNU C library reads /etc/nsswitch.conf, and tries out one method at a time in the "hosts" line.
It gets to the "resolve" entry. Now what happens here exactly? Does this do a DNS query to 127.0.0.53 port 53? Or ...? And if so, what does systemd-resolved do there exactly? Does DNSStubListener=no disable (?) this step? Or how does it impact it?
Eventually we get to the "dns" entry and that performs a DNS query to whatever is in /etc/resolv.conf? Which in my case is a symlink to /run/systemd/resolve/resolv.conf, so it goes back to systemd-resolved, just like the resolve entry? (that seems to make little sense, so I am sure I'm wrong in some substantial way) Or maybe we never get there because of "[!UNAVAIL=return]"?

Also, does each of those attempt DNSSEC first, and fall back to non-DNSSEC if it doesn't work out? (And what consistitues a "does not work out"?). If I set DNSSEC=off, does this impact just step 3 or also step 4?

If somebody can enlighten me (thank you!), then I will write this up for the wiki for the next guy.

seth · 2019-02-23 20:54:57

You should not have the stub on 127.0.0.53 (because of the setting) so the query should go straight to the DNS server configured in /run/systemd/resolve/resolv.conf (/etc/resolv.conf symlinks for clients which don't use nsswitch and because systemd's default stance towards NIH implementations…)
This server is asked via DNSSEC and apparently broken (can't say since we don't know the IP), so the failsafe to regular DNS seems to fail. The next thing should be that resolved asks the next DNS server in the line and actually continue to use that for every further request (what is very resolved specific and a deviation from all former DNS implementations, usually each process would start at the top, but w/ the resolved daemon, there's just one process to make actual DNS queries)

What ultimately resolves the domain (another server or if unavailable ultimately the broken one which reacts different on subsequent queries - or whether resolved infers that DNSSEC is probably somehow broken and drops that idea) is hard to say, but you could wireshark the traffic and see for yourself.

Things you should be aware of are "nmap -p 53 127.0.0.53" and "cat /run/systemd/resolve/resolv.conf" (ie. the amount and order of what DNS servers are configured)
Whether "DNSStubListener … gets in the way of traditional DNS lookups" depends on what you define as "traditional DNS lookups", but it should™ not impede your system at all. That said: I'm personally not much of a fan of the resolved approach…

jernst · 2019-02-23 21:32:54

Hey, I appreciate this. This gives me enough material to go looking. I figure I need to read up on DNSSEC first, and, as you suggest, watch actual traffic go around. Might even learn something in the process ;-) Thanks.

Arch Linux

#1 2019-02-22 22:44:14

Intermittent DNS resolution problems

#2 2019-02-23 08:29:20

Re: Intermittent DNS resolution problems

#3 2019-02-23 17:58:27

Re: Intermittent DNS resolution problems

#4 2019-02-23 20:54:57

Re: Intermittent DNS resolution problems

#5 2019-02-23 21:32:54

Re: Intermittent DNS resolution problems

Board footer