You are not logged in.
My ISP provides a DNS server by DHCP, however it's faulty, many records are outdated, and resolve to the wrong address. I don't get resolution failure, but a wrong IP address, resulting timeouts in the client applications. I tried many things to make the Network Manager established connections to use my preferred DNS (for testing, it can be 8.8.8.8). I decided to rely on `systemd-resolved`, `dnsmasq` is not active. I followed the wiki article https://wiki.archlinux.org/index.php/Systemd-resolved. `/etc/resolv.conf` points to `/run/systemd/resolve/stub-resolv.conf`. In `/etc/systemd/resolved.conf` I have the followings:
DNS=8.8.8.8
FallbackDNS=
Domains=~.
DNSSEC=yes
DNSOverTLS=yes
MulticastDNS=no
Cache=no
And `/etc/NetworkManager/conf.d/dns.conf`:
[main]
dns=systemd-resolved
[connection]
ignore-auto-dns=true
mdns=no
The output of `resolvectl status`:
Global
Protocols: +LLMNR -mDNS +DNSOverTLS DNSSEC=yes/supported
resolv.conf mode: stub
Current DNS Server: 8.8.8.8
DNS Servers: 8.8.8.8
DNS Domain: ~.
Link 2 (enp0s25)
Current Scopes: none
Protocols: -DefaultRoute +LLMNR -mDNS +DNSOverTLS DNSSEC=yes/supported
Link 3 (wlp3s0)
Current Scopes: DNS LLMNR/IPv4 LLMNR/IPv6
Protocols: +DefaultRoute +LLMNR -mDNS +DNSOverTLS DNSSEC=yes/supported
Current DNS Server: 2a02:908:2:b::1
DNS Servers: 192.168.0.1 2a02:908:2:a::1 2a02:908:2:b::1
DNS Domain: home
Where `192.168.0.1 ` is the router from the ISP. I expected that `ignore-auto-dns=true` fixes the problem, I also tried `dns=none`. After each config change I restarted the systemd-resolved and NetworkManager services, I also tried to down-up the connection. However 192.168.0.1 remains there for wlp3s0.
Last edited by deeenes (2021-04-24 20:22:01)
Offline
I did some progress and collected more information, however now there is a discrepancy between results returned by systemd-resolved and plain resolv.conf. First of all, I could remove the scope specific DNS by adding the following section to /etc/NetworkManager/conf.d/dns.conf:
[global-dns-domain-*]
servers=127.0.0.53
In this case systemd-resolved forwards all DNS queries to 8.8.8.8. I use uniprot.org as an example to show the difference in the DNS response. If I use resolvectl query uniprot.org or dig uniprot.org the response is 128.175.245.202:
$ dig uniprot.org
; <<>> DiG 9.16.11 <<>> uniprot.org
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 46725
;; flags: qr rd ra; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 1
;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 65494
;; QUESTION SECTION:
;uniprot.org. IN A
;; ANSWER SECTION:
uniprot.org. 0 IN A 128.175.245.202
;; Query time: 0 msec
;; SERVER: 127.0.0.53#53(127.0.0.53)
;; WHEN: Wed Apr 21 02:52:49 CEST 2021
;; MSG SIZE rcvd: 56
If I manually write `nameserver 8.8.8.8` into `/etc/resolv.conf` I get the following result (which is actually correct):
$ dig uniprot.org
; <<>> DiG 9.16.11 <<>> uniprot.org
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 6221
;; flags: qr rd ra; QUERY: 1, ANSWER: 2, AUTHORITY: 0, ADDITIONAL: 1
;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 1232
; COOKIE: c243ef09b9f304bbb7ea48dd607f778d3cd8e314d8dcc862 (good)
;; QUESTION SECTION:
;uniprot.org. IN A
;; ANSWER SECTION:
uniprot.org. 2440 IN A 193.62.193.81
uniprot.org. 2440 IN A 128.175.240.206
;; Query time: 136 msec
;; SERVER: 8.8.8.8#53(8.8.8.8)
;; WHEN: Wed Apr 21 02:53:33 CEST 2021
;; MSG SIZE rcvd: 100
Then I turned on the debug log for systemd-resolved by creating a file `/etc/systemd/system/systemd-resolved.service.d/debug.conf` with the following contents:
[Service]
Environment=SYSTEMD_LOG_LEVEL=debug
And then restarted the service by:
systemctl daemon-reload
systemctl restart systemd-resolved.service
I tried resolving uniprot.org with or without DNSSEC, I show the debug log for the latter as the result is the same:
Apr 21 17:16:10 x systemd-resolved[518724]: varlink-15: New incoming message: {"method":"io.systemd.Resolve.ResolveHostname","parameters":{"name":"www.uniprot.org"}}
Apr 21 17:16:10 x systemd-resolved[518724]: varlink-15: varlink: changing state idle-server → processing-method
Apr 21 17:16:10 x systemd-resolved[518724]: idn2_lookup_u8: www.uniprot.org → www.uniprot.org
Apr 21 17:16:10 x systemd-resolved[518724]: Looking up RR for www.uniprot.org IN A.
Apr 21 17:16:10 x systemd-resolved[518724]: Looking up RR for www.uniprot.org IN AAAA.
Apr 21 17:16:10 x systemd-resolved[518724]: Cache miss for www.uniprot.org IN AAAA
Apr 21 17:16:10 x systemd-resolved[518724]: Transaction 58968 for <www.uniprot.org IN AAAA> scope dns on */*.
Apr 21 17:16:10 x systemd-resolved[518724]: Using feature level UDP+EDNS0 for transaction 58968.
Apr 21 17:16:10 x systemd-resolved[518724]: Using DNS server 8.8.8.8 for transaction 58968.
Apr 21 17:16:10 x systemd-resolved[518724]: Sending query packet with id 58968 of size 44.
Apr 21 17:16:10 x systemd-resolved[518724]: Cache miss for www.uniprot.org IN A
Apr 21 17:16:10 x systemd-resolved[518724]: Transaction 33940 for <www.uniprot.org IN A> scope dns on */*.
Apr 21 17:16:10 x systemd-resolved[518724]: Using feature level UDP+EDNS0 for transaction 33940.
Apr 21 17:16:10 x systemd-resolved[518724]: Using DNS server 8.8.8.8 for transaction 33940.
Apr 21 17:16:10 x systemd-resolved[518724]: Sending query packet with id 33940 of size 44.
Apr 21 17:16:10 x systemd-resolved[518724]: varlink-15: varlink: changing state processing-method → pending-method
Apr 21 17:16:10 x systemd-resolved[518724]: Processing incoming packet on transaction 33940 (rcode=SUCCESS).
Apr 21 17:16:10 x systemd-resolved[518724]: Transaction 33940 for <www.uniprot.org IN A> on scope dns on */* now complete with <success> from network (unsigned).
Apr 21 17:16:10 x systemd-resolved[518724]: Processing incoming packet on transaction 58968 (rcode=SUCCESS).
Apr 21 17:16:10 x systemd-resolved[518724]: Transaction 58968 for <www.uniprot.org IN AAAA> on scope dns on */* now complete with <success> from network (unsigned).
Apr 21 17:16:10 x systemd-resolved[518724]: Freeing transaction 33940.
Apr 21 17:16:10 x systemd-resolved[518724]: Following CNAME/DNAME www.uniprot.org → uniprot.org.
Apr 21 17:16:10 x systemd-resolved[518724]: varlink-15: Sending message: {"parameters":{"addresses":[{"ifindex":0,"family":2,"address":[128,175,245,202]}],"name":"uniprot.org","flags":1}}
Apr 21 17:16:10 x systemd-resolved[518724]: varlink-15: varlink: changing state pending-method → idle-server
Apr 21 17:16:10 x systemd-resolved[518724]: Freeing transaction 58968.
Apr 21 17:16:10 x systemd-resolved[518724]: varlink-15: Got POLLHUP from socket.
Apr 21 17:16:10 x systemd-resolved[518724]: varlink-15: varlink: changing state idle-server → pending-disconnect
Apr 21 17:16:10 x systemd-resolved[518724]: varlink-15: varlink: changing state pending-disconnect → processing-disconnect
Apr 21 17:16:10 x systemd-resolved[518724]: varlink-15: varlink: changing state processing-disconnect → disconnected
Apr 21 17:16:10 x systemd-resolved[518724]: Got DNS stub UDP query packet for id 19222
Apr 21 17:16:10 x systemd-resolved[518724]: Looking up RR for www.uniprot.org IN A.
Apr 21 17:16:10 x systemd-resolved[518724]: Cache miss for www.uniprot.org IN A
Apr 21 17:16:10 x systemd-resolved[518724]: Transaction 20883 for <www.uniprot.org IN A> scope dns on */*.
Apr 21 17:16:10 x systemd-resolved[518724]: Using feature level UDP+EDNS0 for transaction 20883.
Apr 21 17:16:10 x systemd-resolved[518724]: Using DNS server 8.8.8.8 for transaction 20883.
Apr 21 17:16:10 x systemd-resolved[518724]: Sending query packet with id 20883 of size 44.
Apr 21 17:16:10 x systemd-resolved[518724]: Processing query...
Apr 21 17:16:10 x systemd-resolved[518724]: Got DNS stub UDP query packet for id 20010
Apr 21 17:16:10 x systemd-resolved[518724]: Looking up RR for www.uniprot.org IN AAAA.
Apr 21 17:16:10 x systemd-resolved[518724]: Cache miss for www.uniprot.org IN AAAA
Apr 21 17:16:10 x systemd-resolved[518724]: Transaction 9603 for <www.uniprot.org IN AAAA> scope dns on */*.
Apr 21 17:16:10 x systemd-resolved[518724]: Using feature level UDP+EDNS0 for transaction 9603.
Apr 21 17:16:10 x systemd-resolved[518724]: Using DNS server 8.8.8.8 for transaction 9603.
Apr 21 17:16:10 x systemd-resolved[518724]: Sending query packet with id 9603 of size 44.
Apr 21 17:16:10 x systemd-resolved[518724]: Processing query...
Apr 21 17:16:10 x systemd-resolved[518724]: Processing incoming packet on transaction 20883 (rcode=SUCCESS).
Apr 21 17:16:10 x systemd-resolved[518724]: Transaction 20883 for <www.uniprot.org IN A> on scope dns on */* now complete with <success> from network (unsigned).
Apr 21 17:16:10 x systemd-resolved[518724]: Following CNAME/DNAME www.uniprot.org → uniprot.org.
Apr 21 17:16:10 x systemd-resolved[518724]: Sending response packet with id 19222 on interface 1/AF_INET of size 74.
Apr 21 17:16:10 x systemd-resolved[518724]: Freeing transaction 20883.
Apr 21 17:16:10 x systemd-resolved[518724]: Processing incoming packet on transaction 9603 (rcode=SUCCESS).
Apr 21 17:16:10 x systemd-resolved[518724]: Transaction 9603 for <www.uniprot.org IN AAAA> on scope dns on */* now complete with <success> from network (unsigned).
Apr 21 17:16:10 x systemd-resolved[518724]: Following CNAME/DNAME www.uniprot.org → uniprot.org.
Apr 21 17:16:10 x systemd-resolved[518724]: Sending response packet with id 20010 on interface 1/AF_INET of size 58.
As we see systemd-resolved used the server 8.8.8.8 and returned the address 128.175.245.202, the contents of the response is not in the log.
Then I wrote manually `nameserver 8.8.8.8` into `/etc/resolv.conf` and executed `dig +trace uniprot.org`:
$ dig +trace uniprot.org
; <<>> DiG 9.16.11 <<>> +trace uniprot.org
;; global options: +cmd
. 67981 IN NS f.root-servers.net.
. 67981 IN NS c.root-servers.net.
. 67981 IN NS a.root-servers.net.
. 67981 IN NS i.root-servers.net.
. 67981 IN NS h.root-servers.net.
. 67981 IN NS m.root-servers.net.
. 67981 IN NS j.root-servers.net.
. 67981 IN NS k.root-servers.net.
. 67981 IN NS g.root-servers.net.
. 67981 IN NS d.root-servers.net.
. 67981 IN NS e.root-servers.net.
. 67981 IN NS l.root-servers.net.
. 67981 IN NS b.root-servers.net.
. 515527 IN RRSIG NS 8 0 518400 20210504050000 20210421040000 14631 . R+51QCa2sAXghb90AZQmS9q/OKngU8+EV4MunjlHnVZwBKKcKXsuhMPj Ly3oys8TiWVoSf7gz1HM/J9OcwqwVHWrfvGEAo7PA6m2mZPJR8NuV9G9 GmLIBQcqc9s960mmAyk5gFmA7tmqM30psy/W7DN2DwHQzre6Nv0jyjiB E5WZsZURhodHEkxjv3Bgl2qmCsdFuKy8Ea/0IgneNVbnGeLV1FaxSnkn aV4/DvrIE9oL2lELzLcX/MBEMO/ELSM96p9B9CDsp7tJHz4J5vtStzVm uB5Sbj6OpcBqAH+cn6KEqDGfLKzKHMSB6tvUaRiiQy3aN0oyyttX78QJ DeIIVg==
;; Received 553 bytes from 8.8.8.8#53(8.8.8.8) in 46 ms
uniprot.org. 414 IN A 193.62.193.81
uniprot.org. 414 IN A 128.175.240.206
;; Received 100 bytes from 2001:500:2f::f#53(f.root-servers.net) in 66 ms
The same output with `/etc/resolv.conf` pointing to `127.0.0.53` (systemd-resolved):
$ dig +trace uniprot.org
; <<>> DiG 9.16.11 <<>> +trace uniprot.org
;; global options: +cmd
. 67142 IN NS g.root-servers.net.
. 67142 IN NS i.root-servers.net.
. 67142 IN NS k.root-servers.net.
. 67142 IN NS a.root-servers.net.
. 67142 IN NS b.root-servers.net.
. 67142 IN NS d.root-servers.net.
. 67142 IN NS l.root-servers.net.
. 67142 IN NS h.root-servers.net.
. 67142 IN NS j.root-servers.net.
. 67142 IN NS f.root-servers.net.
. 67142 IN NS e.root-servers.net.
. 67142 IN NS c.root-servers.net.
. 67142 IN NS m.root-servers.net.
;; Received 239 bytes from 127.0.0.53#53(127.0.0.53) in 46 ms
uniprot.org. 85605 IN NS dns101.register.com.
uniprot.org. 85605 IN NS dns102.register.com.
;; Received 154 bytes from 2001:7fd::1#53(k.root-servers.net) in 86 ms
uniprot.org. 85805 IN NS dns101.register.com.
uniprot.org. 85805 IN NS dns102.register.com.
;; BAD (HORIZONTAL) REFERRAL
;; Received 154 bytes from 207.204.40.61#53(dns101.register.com) in 26 ms
uniprot.org. 817 IN A 128.175.240.206
uniprot.org. 817 IN A 193.62.193.81
;; Received 100 bytes from 207.204.40.61#53(dns101.register.com) in 26 ms
Actually I've just noticed this response is the same. Even more weird, if I call dig without `+trace` the response is different:
$ dig uniprot.org
; <<>> DiG 9.16.11 <<>> uniprot.org
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 26724
;; flags: qr rd ra; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 1
;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 65494
;; QUESTION SECTION:
;uniprot.org. IN A
;; ANSWER SECTION:
uniprot.org. 0 IN A 128.175.245.202
;; Query time: 0 msec
;; SERVER: 127.0.0.53#53(127.0.0.53)
;; WHEN: Wed Apr 21 17:48:02 CEST 2021
;; MSG SIZE rcvd: 56
Although all responses are supposed to origin from 8.8.8.8, the results are different. I am not sure what is behind.
(I will try to change the topic title as now the problem seems to be different.)
Last edited by deeenes (2021-04-21 15:50:49)
Offline
Hi,
Did you really notice requests going to your ISP router, even though if IP is there in Link 3 ?
I have the same setup, but requests do indeed go through the "Global" ones set through systemd-resolved. Furthermore, since +DNSOverTLS is set, wouldn't it block all requests to the ISP router (which probably does not support it) ?
Offline
Did you really notice requests going to your ISP router, even though if IP is there in Link 3 ?
Actually this I haven't checked, but I am already sure the requests go to 8.8.8.8
Offline
Summarizing the current status: 8.8.8.8, the TLD and authoritative servers all return the correct response, but systemd-resolved returns a wrong response. The caching for systemd-resolved is disabled and the cache size is zero. This is confirmed by the debug log, as well as the fact that systemd-resolved queries 8.8.8.8 at each lookup. The question: where the wrong answer is coming from?
Edit: systemd-resolved doesn't query 8.8.8.8 at each lookup, but only for example if I do `dig +trace` or the query comes from firefox (I guess it does something similar like +trace). A simple resolvectl query is answered without contacting any server.
Last edited by deeenes (2021-04-21 23:25:36)
Offline
Further investigating the issue, I've done a traffic dump with wireshark (using filter udp.port == 53), and realized that the queries returning wrong responses actually don't perform any communication to servers outside. It means systemd-resolved finds out the wrong answer from somewhere else. So I again turned on the debugging and I see this in the log:
Apr 21 20:06:40 x systemd-resolved[589765]: Successfully acquired requested service name.
Apr 21 20:06:40 x systemd-resolved[589765]: Got message type=method_return sender=org.freedesktop.DBus destination=:1.1001 path=n/a interface=n/a member=n/a cookie=5 reply_cookie=3 signature=n/a error-name=n/a error-me
ssage=n/a
Apr 21 20:06:40 x systemd-resolved[589765]: Match type='signal',sender='org.freedesktop.login1',path='/org/freedesktop/login1',interface='org.freedesktop.login1.Manager',member='PrepareForSleep' successfully installed.
Apr 21 20:06:40 x kernel: audit: type=1130 audit(1619028400.147:1887): pid=1 uid=0 auid=4294967295 ses=4294967295 msg='unit=systemd-resolved comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? re
s=success'
Apr 21 20:06:46 x systemd-resolved[589765]: Got message type=method_call sender=:1.1002 destination=org.freedesktop.resolve1 path=/org/freedesktop/resolve1 interface=org.freedesktop.resolve1.Manager member=ResolveHostn
ame cookie=2 reply_cookie=0 signature=isit error-name=n/a error-message=n/a
Apr 21 20:06:46 x systemd-resolved[589765]: idn2_lookup_u8: uniprot.org → uniprot.org
Apr 21 20:06:46 x systemd-resolved[589765]: Looking up RR for uniprot.org IN A.
Apr 21 20:06:46 x systemd-resolved[589765]: Looking up RR for uniprot.org IN AAAA.
Apr 21 20:06:46 x systemd-resolved[589765]: Sent message type=method_call sender=n/a destination=org.freedesktop.DBus path=/org/freedesktop/DBus interface=org.freedesktop.DBus member=AddMatch cookie=4 reply_cookie=0 si
gnature=s error-name=n/a error-message=n/a
Apr 21 20:06:46 x systemd-resolved[589765]: Sent message type=method_call sender=n/a destination=org.freedesktop.DBus path=/org/freedesktop/DBus interface=org.freedesktop.DBus member=GetNameOwner cookie=5 reply_cookie=
0 signature=s error-name=n/a error-message=n/a
Apr 21 20:06:46 x systemd-resolved[589765]: Got message type=method_return sender=org.freedesktop.DBus destination=:1.1001 path=n/a interface=n/a member=n/a cookie=7 reply_cookie=5 signature=s error-name=n/a error-mess
age=n/a
Apr 21 20:06:46 x systemd-resolved[589765]: Sent message type=method_return sender=n/a destination=:1.1002 path=n/a interface=n/a member=n/a cookie=6 reply_cookie=2 signature=a(iiay)st error-name=n/a error-message=n/a
Apr 21 20:06:46 x systemd-resolved[589765]: Sent message type=method_call sender=n/a destination=org.freedesktop.DBus path=/org/freedesktop/DBus interface=org.freedesktop.DBus member=RemoveMatch cookie=7 reply_cookie=0
signature=s error-name=n/a error-message=n/a
Apr 21 20:06:46 x systemd-resolved[589765]: Got message type=method_return sender=org.freedesktop.DBus destination=:1.1001 path=n/a interface=n/a member=n/a cookie=6 reply_cookie=4 signature=n/a error-name=n/a error-me
ssage=n/a
Apr 21 20:06:50 x systemd-resolved[589765]: Got DNS stub UDP query packet for id 24871
Apr 21 20:06:50 x systemd-resolved[589765]: Looking up RR for uniprot.org IN A.
Apr 21 20:06:50 x systemd-resolved[589765]: Sending response packet with id 24871 on interface 1/AF_INET of size 56.
Apr 21 20:06:50 x systemd-resolved[589765]: Processing query...
Apr 21 20:07:00 x systemd-resolved[589765]: n/a: New incoming connection.
Apr 21 20:07:00 x systemd-resolved[589765]: n/a: Connections of user 1000: 0 (of 576 max)
Apr 21 20:07:00 x systemd-resolved[589765]: varlink-15: varlink: setting state idle-server
There is no transaction ID assigned to this lookup, the answer is not shown, and I have no idea whether any of these messages gives information how the answer was obtained (I don't know what are all those dbus messages about).
Offline
I checked the dbus communication by `dbus-monitor --system --monitor > dbus-dump-resctl.mon` and found that `resolvectl query uniprot.org` asks systemd-resolved by dbus, and systemd-resolved doesn't query any server but gives a wrong answer (the IP address in the dbus dump appears as 4 hex numbers; at me systemd-resolved had the dbus name "1.1001" and to find out the actual process behind, I used the command `ps -q $(dbus-send --system --print-reply --dest=org.freedesktop.DBus /org/freedesktop/DBus org.freedesktop.DBus.GetConnectionUnixProcessID 'string::1.1001' | tail -n 1 | awk '{print $NF}')`). I also checked the traffic on the lo interface, and found that dig also queries systemd-resolved and gets the same answer. The log messages from systemd-resolved in during this process tell absolutely nothing about the origin and contents of the answer:
Apr 21 22:00:19 x systemd-resolved[589765]: Got DNS stub UDP query packet for id 2156
Apr 21 22:00:19 x systemd-resolved[589765]: Looking up RR for uniprot.org IN A.
Apr 21 22:00:19 x systemd-resolved[589765]: Sending response packet with id 2156 on interface 1/AF_INET of size 56.
Last edited by deeenes (2021-04-21 20:23:27)
Offline
After my last update, I wrote to the systemd-devel mailing list: https://lists.freedesktop.org/archives/ … 46363.html. I immediately got the correct answer, the wrong address is in /etc/hosts. I was shocked to see a webpage in the hosts file, it crossed my mind before, but I haven't even checked, I found it so much unlikely. Instead I've done hours of troubleshooting to find a very banal answer
Offline