Arch Linux

bhe69

Member

Registered: 2024-01-11

Posts: 13

How do I debug systemd-resolved?

Hello everyone,

as I repeatedly had the problem that domains no longer are resolved I tried to find the reason. In the meantime I fond the root cause - a bug introduced with the latest version of systemd. Nonetheless, I found systemd-resolved to be very tightlipped as to reveal the problem.

$ ping 9.9.9.9
PING 9.9.9.9 (9.9.9.9) 56(84) bytes of data.
64 bytes from 9.9.9.9: icmp_seq=1 ttl=59 time=15.3 ms
[....]
$ ping spiegel.de
ping: spiegel.de: Temporary failure in name resolution

So the network is up, but dns resolution is the problem.

$ resolvectl status
Global
           Protocols: +LLMNR +mDNS +DNSOverTLS DNSSEC=no/unsupported
    resolv.conf mode: uplink
  Current DNS Server: 1.2.3.4#dns.redacted.inv
         DNS Servers: 1.2.3.4#dns.redacted.inv 9.9.9.9#dns.quad9.net
Fallback DNS Servers: 1.1.1.1#cloudflare-dns.com 9.9.9.9#dns.quad9.net
                      8.8.8.8#dns.google 2606:4700:4700::1111#cloudflare-dns.com
                      2620:fe::9#dns.quad9.net 2001:4860:4860::8888#dns.google

Link 2 (enp0s25)
    Current Scopes: none
         Protocols: -DefaultRoute +LLMNR +mDNS +DNSOverTLS DNSSEC=no/unsupported
     Default Route: no

Link 3 (wlp3s0)
    Current Scopes: DNS LLMNR/IPv4 LLMNR/IPv6 mDNS/IPv4 mDNS/IPv6
         Protocols: +DefaultRoute +LLMNR +mDNS +DNSOverTLS DNSSEC=no/unsupported
Current DNS Server: 192.168.2.1
       DNS Servers: 192.168.2.1
     Default Route: yes

Ok, but which DNS server is actually used? Global or Link 3?

$ resolvectl show-server-state
Server: 1.2.3.4#dns.redacted.inv          
                                     Type: system
                   Verified feature level: TLS+EDNS0
                   Possible feature level: TLS+EDNS0
                              DNSSEC Mode: no
                         DNSSEC Supported: yes
       Maximum UDP fragment size received: 512
                      Failed UDP attempts: 0
                      Failed TCP attempts: 0
                    Seen truncated packet: no
                 Seen OPT RR getting lost: no
                    Seen RRSIG RR missing: no
                      Seen invalid packet: no
                   Server dropped DO flag: no
                                                    
Server: 9.9.9.9#dns.quad9.net                
                              Type: system
            Verified feature level: TLS+EDNS0
            Possible feature level: TLS+EDNS0
                       DNSSEC Mode: no
                  DNSSEC Supported: yes
Maximum UDP fragment size received: 512
               Failed UDP attempts: 0
               Failed TCP attempts: 0
             Seen truncated packet: no
          Seen OPT RR getting lost: no
             Seen RRSIG RR missing: no
               Seen invalid packet: no
            Server dropped DO flag: no

[...]
                                                    
Server: 192.168.2.1                          
                              Type: link
                         Interface: wlp3s0
                   Interface Index: 3
            Verified feature level: n/a
            Possible feature level: TLS+EDNS0
                       DNSSEC Mode: no
                  DNSSEC Supported: yes
Maximum UDP fragment size received: 512
               Failed UDP attempts: 0
               Failed TCP attempts: 0
             Seen truncated packet: no
          Seen OPT RR getting lost: no
             Seen RRSIG RR missing: no
               Seen invalid packet: no
            Server dropped DO flag: no

                                           
Still no proper info on which server is in use.

$ resolvectl query spiegel.de
spiegel.de: resolve call failed: All attempts to contact name servers or networks failed

Gee, I thought so, but still not very helpful. Which servers did you try and why didn't they respond?

So to sum it up: How can I find out which DNS server is actually used, and how can I find out (the next time an error occurs, which is no software bug) what the exact problem is?

Thanks for your help!

sipak

Member

Registered: 2019-02-03

Posts: 50

Re: How do I debug systemd-resolved?

Or you could get ride of resolved and use the glibc resolver, or openresolv, or dnscrypt, etc.

systemctl disable systemd-resolved.service

You can use $ dig yahoo.com to see what dns server is being used.

If using NetworkManager, you can also setup the default DNS that it will automatically use to modify /etc/resolv.conf
Many of use simply setup resolv.conf and chattr +i it.

---

A problem to do what everybody else does without questioning. A danger to go against the way things are just because. Too much or too little, ivory towers of perfection or functional mess... Balance is what this world needs. Selective, not the middle ground. Objectivity and idealism, but within a pragmatic scope. - Minimalism is achieved through efficiency, not deficiency.

Whoracle

Member

Registered: 2010-11-02

Posts: 191

Re: How do I debug systemd-resolved?

AFAIK systemd-resolved uses the DNS server from whichever link is the default route, so in your case Link 3 - 192.168.2.1

You can go through them manually and see which one fails to make sure:

dig +short spiegel.de @192.168.2.1   # start with this, it's most likely to be the culprit
dig +short spiegel.de @1.2.3.4
dig +short spiegel.de @9.9.9.9
dig +short spiegel.de @1.1.1.1
dig +short spiegel.de @8.8.8.8

ETA:

spiegel.de: resolve call failed: All attempts to contact name servers or networks failed

I can't contact 1.2.3.4 either, so maybe in your setup 1.2.3.4 is used first. The question is also where that came from.

cat /etc/systemd/resolved.conf
cat /etc/resolv.conf

Or does your DHCP server push that? (unlikely, I know)

Last edited by Whoracle (2025-08-08 07:47:04)

tomaszc

Member

Registered: 2010-09-11

Posts: 21

Re: How do I debug systemd-resolved?

This is probably a bug in the latest version of systemd: https://github.com/systemd/systemd/issues/38509

Edit:
Sorry, I didn't read that you had already found this bug.

Last edited by tomaszc (2025-08-08 13:16:21)

cryptearth

Member

Registered: 2024-02-03

Posts: 1,831

Re: How do I debug systemd-resolved?

some time ago I manually added google, cloudflare and quad9 with DoH/DoT to my routers config and since experienced sporadic issues
fun fact: I most notice this for github.com - but other domains are affected as well, although way less often
I maybe give it a crosscheck with DoH/DoT disabled and a local resolver if the issues is github, the dns servers or the TLS

side note about TLS: I once had issues because for some reason tls used /random only instead of /urandom and I ran low on entropy
could this be related that the actual cause of the failure could be the crypto and the system running low on entropy and thereby causing timeouts? just a wild guess in the blue

bhe69

Member

Registered: 2024-01-11

Posts: 13

Re: How do I debug systemd-resolved?

You can use $ dig yahoo.com to see what dns server is being used.

Thanks, good idea. For the record: dig yahoo.com +tls shall be used when DNS over TLS is used.

AFAIK systemd-resolved uses the DNS server from whichever link is the default route, so in your case Link 3 - 192.168.2.1

For the record: It uses the globally defined DNS server, as dig yahoo.com +tls shows.

cat /etc/resolv.conf

This shows that the DNS servers from the global section should be used, only as a last measure it falls back to the default route DNS. BTW: as the DNS server I use is private I redacted it to 1.2.3.4

nameserver 1.2.3.4
nameserver 9.9.9.9
nameserver 192.168.2.1
search .

Whoracle

Member

Registered: 2010-11-02

Posts: 191

Re: How do I debug systemd-resolved?

nameserver 1.2.3.4
nameserver 9.9.9.9
nameserver 192.168.2.1
search .

Just to make sure I understand correctly: That is the FULL output of your /etc/resolv.conf? What does stat /etc/resolv.conf show?

bhe69

Member

Registered: 2024-01-11

Posts: 13

Re: How do I debug systemd-resolved?

nameserver 1.2.3.4
nameserver 9.9.9.9
nameserver 192.168.2.1
search .

Just to make sure I understand correctly: That is the FULL output of your /etc/resolv.conf? What does stat /etc/resolv.conf show?

Not quite, I skipped the comments:

# This is /run/systemd/resolve/resolv.conf managed by man:systemd-resolved(8).
# Do not edit.
[...]

stat /etc/resolv.conf shows

  File: /etc/resolv.conf -> /run/systemd/resolve/resolv.conf
  Size: 32        	Blocks: 0          IO Block: 4096   symbolic link
Device: 8,10	Inode: 948204      Links: 1
Access: (0777/lrwxrwxrwx)  Uid: (    0/    root)   Gid: (    0/    root)