[SOLVED] DNS resolution starts failing after a random period

gtf21 · 2022-04-04 11:17:20

Symptoms:

I'll connect to e.g. my wifi network, after some time I can't access websites (firefox says "cannot find that site", apps like slack stop working and say I'm offline, etc.). I can ping IP addresses (like 1.1.1.1) but when I try to ping a name like google.com, I get:

ping: google.com: Temporary failure in name resolution

I can fix this problem by bringing the network interface for my wifi card down and up again (and, I think, by turning off and on the VPN interface I use). Nothing else works, I have tried `sudo resolvectl flush-caches`, `sudo resolvectl dns {2,7} 1.1.1.1` to set the servers, editing /etc/systemd/resolve.conf and adding the servers in followed by restarting `systemd-resolved`.

Background on my setup:

I have two network interfaces, one for my wifi card (wlp...) and one for my VPN which is managed by the mullvad-daemon service (wg-mullvad). Other than that I'm using the standard systemd-networkd setup (using netctl-auto, systemd-resolved, etc.) rather than something like NetworkManager.

Main question

I'm really not sure what the source of the error is here -- when I try to look at the logs for systemd-resolved I don't really get any more info. I'm also not totally sure how to debug, since the output of `sudo resolvectl status` and `systemd-resolve --status` both show 1.1.1.1 as one of my DNS servers.

I'd like to know (a) if anyone has an idea of what might be causing this; and (b) if you have any hints about how to debug it because I'm totally at a loss.

Any and all help much appreciated.

Last edited by gtf21 (2022-04-09 15:49:18)

Lone_Wolf · 2022-04-04 11:55:33

netctl-auto, systemd-resolved

Are you using dhcpcd with netctl ?

If yes, you have atleast 2 competing dns resolvers, dhpcd and systemd-resolved .

See https://wiki.archlinux.org/title/System … omatically for more info .

gtf21 · 2022-04-04 13:00:25

Lone_Wolf wrote:

Are you using dhcpcd with netctl ?

I don't think so, dhcpcd is installed, but systemctl status dhcpcd shows it is disabled.

seth · 2022-04-04 13:21:01

and, I think, by turning off and on the VPN interface I use

What if you skip the VPN entirely? Same situation?

1.1.1.1 as one of my DNS servers

<longepicrantaboutresolvedthatwouldgetmebanned> - what's the actual output?
Also (when things are good and broken) compare

ip a
ip r
stat /etc/resolve.conf
cat /etc/resolve.conf
nslookup google.com
drill google.com

Finally:

standard systemd-networkd setup (using netctl-auto, systemd-resolved, etc.)

Yeah, no.
netctl and systemd-networkd are not the same and will compete against each other.

find /etc/systemd -type l -exec test -f {} \; -print | awk -F'/' '{ printf ("%-40s | %s\n", $(NF-0), $(NF-1)) }' | sort -f

gtf21 · 2022-04-04 13:51:12

Ok so I'm now in the somewhat fascinating position that my terminal (freshly opened) isn't working and my browser is.

For example, in my browser, I can open google.com, but `ping google.com` results in "Temporary failure in name resolution". Oddly, other things also work like zoom.

So, some output:

Firstly, I don't have `/etc/resolve.conf`, but I do have `resolv.conf`, so I've stat'd that instead (just in case).

~ > resolvectl status  
Global
           Protocols: +LLMNR +mDNS -DNSOverTLS DNSSEC=no/unsupported
    resolv.conf mode: foreign
  Current DNS Server: 1.1.1.1
         DNS Servers: 1.1.1.1
Fallback DNS Servers: 1.1.1.1#cloudflare-dns.com 9.9.9.10#dns.quad9.net
                      8.8.8.8#dns.google 2606:4700:4700::1111#cloudflare-dns.com
                      2620:fe::10#dns.quad9.net 2001:4860:4860::8888#dns.google

Link 2 (wlp0s20f3)
    Current Scopes: DNS LLMNR/IPv4 LLMNR/IPv6
         Protocols: +DefaultRoute +LLMNR -mDNS -DNSOverTLS DNSSEC=no/unsupported
Current DNS Server: 1.1.1.1
       DNS Servers: 1.1.1.1

Link 11 (wg-mullvad)
Current Scopes: none
     Protocols: -DefaultRoute +LLMNR -mDNS -DNSOverTLS DNSSEC=no/unsupported

~ > ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host 
       valid_lft forever preferred_lft forever
2: wlp0s20f3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
    link/ether 40:ec:99:20:9c:cf brd ff:ff:ff:ff:ff:ff
    inet 192.168.1.154/24 brd 192.168.1.255 scope global dynamic noprefixroute wlp0s20f3
       valid_lft 76805sec preferred_lft 66005sec
    inet6 2a00:23c8:4a86:2701:42ec:99ff:fe20:9ccf/64 scope global dynamic mngtmpaddr 
       valid_lft 283sec preferred_lft 103sec
    inet6 fe80::42ec:99ff:fe20:9ccf/64 scope link 
       valid_lft forever preferred_lft forever
11: wg-mullvad: <POINTOPOINT,UP,LOWER_UP> mtu 1380 qdisc noqueue state UNKNOWN group default qlen 1000
    link/none 
    inet 10.114.114.19/32 scope global wg-mullvad
       valid_lft forever preferred_lft forever
    inet6 fc00:bbbb:bbbb:bb01::33:7212/128 scope global 
       valid_lft forever preferred_lft forever

default via 192.168.1.254 dev wlp0s20f3 proto dhcp src 192.168.1.154 metric 3002 
10.64.0.1 dev wg-mullvad proto static 
192.168.1.0/24 dev wlp0s20f3 proto dhcp scope link src 192.168.1.154 metric 3002

~ > drill google.com
;; ->>HEADER<<- opcode: QUERY, rcode: NOERROR, id: 36342
;; flags: qr rd ra ; QUERY: 1, ANSWER: 1, AUTHORITY: 4, ADDITIONAL: 0 
;; QUESTION SECTION:
;; google.com.	IN	A

;; ANSWER SECTION:
google.com.	49	IN	A	216.58.212.206

;; AUTHORITY SECTION:
google.com.	171903	IN	NS	ns4.google.com.
google.com.	171903	IN	NS	ns1.google.com.
google.com.	171903	IN	NS	ns3.google.com.
google.com.	171903	IN	NS	ns2.google.com.

;; ADDITIONAL SECTION:

;; Query time: 8 msec
;; SERVER: fc00:bbbb:bbbb:bb01::1
;; WHEN: Mon Apr  4 14:45:53 2022
;; MSG SIZE  rcvd: 116

~ > stat /etc/resolv.conf
  File: /etc/resolv.conf
  Size: 118       	Blocks: 8          IO Block: 4096   regular file
Device: 0,27	Inode: 488         Links: 1
Access: (0644/-rw-r--r--)  Uid: (    0/    root)   Gid: (    0/    root)
Access: 2022-04-04 12:04:16.230314343 +0100
Modify: 2022-04-04 12:04:16.210314246 +0100
Change: 2022-04-04 12:04:16.210314246 +0100
 Birth: 2020-07-04 23:19:21.977324388 +0100
~ > stat /etc/resolve.conf
stat: cannot statx '/etc/resolve.conf': No such file or directory

With respect to the VPN, I'm not sure exactly how to bypass it. Mullvad configures some sort of firewall rules which I don't entirely know how to access which prevent traffic from leaking outside the VPN when it's connected except to local addresses.

gtf21 · 2022-04-04 13:53:51

Incidentally, I disabled the VPN and then everything started working, so it does seem to be an issue with how that interface is configured (or, it could be that disabling the VPN interface causes _something_ to refresh its configuration). I haven't yet re-enabled it and here's a copy of all the above output:

~ > resolvectl status
Global
           Protocols: +LLMNR +mDNS -DNSOverTLS DNSSEC=no/unsupported
    resolv.conf mode: foreign
  Current DNS Server: 1.1.1.1
         DNS Servers: 1.1.1.1
Fallback DNS Servers: 1.1.1.1#cloudflare-dns.com 9.9.9.10#dns.quad9.net
                      8.8.8.8#dns.google 2606:4700:4700::1111#cloudflare-dns.com
                      2620:fe::10#dns.quad9.net 2001:4860:4860::8888#dns.google

Link 2 (wlp0s20f3)
    Current Scopes: DNS LLMNR/IPv4 LLMNR/IPv6
         Protocols: +DefaultRoute +LLMNR -mDNS -DNSOverTLS DNSSEC=no/unsupported
Current DNS Server: 1.1.1.1
       DNS Servers: 1.1.1.1

~ > ip a             
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host 
       valid_lft forever preferred_lft forever
2: wlp0s20f3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
    link/ether 40:ec:99:20:9c:cf brd ff:ff:ff:ff:ff:ff
    inet 192.168.1.154/24 brd 192.168.1.255 scope global dynamic noprefixroute wlp0s20f3
       valid_lft 76280sec preferred_lft 65480sec
    inet6 2a00:23c8:4a86:2701:42ec:99ff:fe20:9ccf/64 scope global dynamic mngtmpaddr 
       valid_lft 251sec preferred_lft 71sec
    inet6 fe80::42ec:99ff:fe20:9ccf/64 scope link 
       valid_lft forever preferred_lft forever

~ > ip r
default via 192.168.1.254 dev wlp0s20f3 proto dhcp src 192.168.1.154 metric 3002 
192.168.1.0/24 dev wlp0s20f3 proto dhcp scope link src 192.168.1.154 metric 3002

~ > drill google.com
;; ->>HEADER<<- opcode: QUERY, rcode: NOERROR, id: 62896
;; flags: qr rd ra ; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 0 
;; QUESTION SECTION:
;; google.com.	IN	A

;; ANSWER SECTION:
google.com.	221	IN	A	142.250.187.238

;; AUTHORITY SECTION:

;; ADDITIONAL SECTION:

;; Query time: 13 msec
;; SERVER: 192.168.1.254
;; WHEN: Mon Apr  4 14:53:27 2022
;; MSG SIZE  rcvd: 44

gtf21 · 2022-04-04 14:02:00

I added DNS servers explicitly in the mullvad app and that suddenly made my terminal work but I'm really confused by this because:

1/ it wasn't blocking access to the DNS servers before, so if they were configured they should still work
2/ this hasn't changed the output of resolvectl status so I'm not sure what's changed.

seth · 2022-04-04 14:08:20

stat: cannot statx '/etc/resolve.conf': No such file or directory

Sorry, typo'd and copyfailed… This meant "/etc/resolv.conf" for both statements.

The original drill response comes from fc00:bbbb:bbbb:bb01::1 which is an IPv6 host in the mullvad segment, the second one from 192.168.1.254 in your wlp0s20f3 segment.
Not surprisingly, https://wiki.archlinux.org/title/Mullvad#DNS_leaks
However, the original drill also succeeds (you receive a different IP, but 216.58.212.206 and 142.250.187.238 are both Google LLC)

resolved acts as a client to resolv.conf, but we've not seen the contents of the latter (in either case) and it's unclear where fc00:bbbb:bbbb:bb01::1 stems from.

=> Outputs of "cat /etc/resolv.conf" and "nslookup google.com".
Also /etc/nsswitch.conf

Can you "ping -4 google.com" when "ping google.com" fails?

gtf21 · 2022-04-04 14:56:10

So, I disabled the "custom DNS" in the mullvad GUI and this is definitely where the problem lies because then `ping` etc. stop working (although, curiously, Firefox keeps working but maybe it is caching something). When I try `ping -4 google.com` that still fails.

~ > cat /etc/resolv.conf
# Generated by resolvconf
domain home
nameserver 10.64.0.1
nameserver fc00:bbbb:bbbb:bb01::1
nameserver 192.168.1.254

~ > stat /etc/resolv.conf
  File: /etc/resolv.conf
  Size: 118       	Blocks: 8          IO Block: 4096   regular file
Device: 0,27	Inode: 488         Links: 1
Access: (0644/-rw-r--r--)  Uid: (    0/    root)   Gid: (    0/    root)
Access: 2022-04-04 15:53:07.106667453 +0100
Modify: 2022-04-04 15:53:07.103334119 +0100
Change: 2022-04-04 15:53:07.103334119 +0100
 Birth: 2020-07-04 23:19:21.977324388 +0100

So it seems that, quite weirdly, the mullvad DNS servers are what's failing?

gtf21 · 2022-04-04 14:59:57

Additionally, here's a result of a DNS lookup followed by a failing ping

~ > nslookup google.com
Server:		10.64.0.1
Address:	10.64.0.1#53

Non-authoritative answer:
Name:	google.com
Address: 216.58.213.14
Name:	google.com
Address: 2a00:1450:4009:816::200e

~ > ping google.com
ping: google.com: Temporary failure in name resolution

~ > ping 216.58.213.14
PING 216.58.213.14 (216.58.213.14) 56(84) bytes of data.
64 bytes from 216.58.213.14: icmp_seq=1 ttl=116 time=21.4 ms

~ > ping 2a00:1450:4009:816::200e
PING 2a00:1450:4009:816::200e(2a00:1450:4009:816::200e) 56 data bytes
64 bytes from 2a00:1450:4009:816::200e: icmp_seq=1 ttl=116 time=20.3 ms

It feels like it can't be a failure then of the DNS servers, drill gives me the following:

~ > drill google.com
;; ->>HEADER<<- opcode: QUERY, rcode: NOERROR, id: 47864
;; flags: qr rd ra ; QUERY: 1, ANSWER: 1, AUTHORITY: 4, ADDITIONAL: 2 
;; QUESTION SECTION:
;; google.com.	IN	A

;; ANSWER SECTION:
google.com.	238	IN	A	216.58.213.14

;; AUTHORITY SECTION:
google.com.	58997	IN	NS	ns1.google.com.
google.com.	58997	IN	NS	ns4.google.com.
google.com.	58997	IN	NS	ns3.google.com.
google.com.	58997	IN	NS	ns2.google.com.

;; ADDITIONAL SECTION:
ns1.google.com.	70832	IN	A	216.239.32.10
ns3.google.com.	70780	IN	A	216.239.36.10

;; Query time: 11 msec
;; SERVER: fc00:bbbb:bbbb:bb01::1
;; WHEN: Mon Apr  4 15:59:36 2022
;; MSG SIZE  rcvd: 148

seth · 2022-04-04 20:12:43

What if you disable (and stop) systemd-resolved?

gtf21 · 2022-04-09 15:47:46

So I disabled systemd-resolved a few days ago and it seems that I'm no longer having any issues. What's weird about this is that I didn't manually enable it or anything recently, AFAIK, yet this is recently emergent behaviour. I'll mark as resolved for now, hopefully it won't come back.

Arch Linux

#1 2022-04-04 11:17:20