systemd-networkd: Wired interface does not use systemd-resolved DNS

ondoho · 2019-09-19 07:16:25

I have stripped down my networking settings as much as possible. No wireless, no DHCP.
My only wired network is configured like this:

cat /etc/systemd/network/20-wired.network 
[Match]
Name=eth0
[Network]
Address=192.168.0.10/24
Gateway=192.168.0.1

systemd-resolved is not configured per se, but since /etc/resolv.conf is not a symlink it provides nameservers.
/etc/resolv.conf is populated by a script that gets current DNS servers from opennic.org.

Now the journal gets flooded with this message:

systemd-networkd[316]: eth0: Configured

every 10-30s.

And resolvectl says:

Global
       LLMNR setting: yes
MulticastDNS setting: yes
  DNSOverTLS setting: no
      DNSSEC setting: allow-downgrade
    DNSSEC supported: yes
  Current DNS Server: 2a00:1a28:1157:3ef::2
         DNS Servers: 2a00:1a28:1157:3ef::2
                      2a00:f826:8:2::195
                      2a01:4f9:c010:475c::1
                      2a03:4000:2a:351:1889:79ff:fe46:60d0
                      2a03:4000:28:365::1
                      2001:470:1f15:b80::53
                      2a02:a00:2000:22::2
                      2a01:4f8:192:43a5::2
                      2a0d:3b80::5054:ff:feec:4b41
                      2a01:7e01::f03c:91ff:febc:322
                      2a03:b0c0:3:e0::67:1001
                      2a00:dcc0:dead:b242::42
                      2001:41d0:701:1100::21b5
                      51.15.98.97
Fallback DNS Servers: 1.1.1.1
                      9.9.9.10
                      8.8.8.8
                      2606:4700:4700::1111
                      2620:fe::10
                      2001:4860:4860::8888
          DNSSEC NTA: 10.in-addr.arpa
                      16.172.in-addr.arpa
                      168.192.in-addr.arpa
                      17.172.in-addr.arpa
                      18.172.in-addr.arpa
                      19.172.in-addr.arpa
                      20.172.in-addr.arpa
                      21.172.in-addr.arpa
                      22.172.in-addr.arpa
                      23.172.in-addr.arpa
                      24.172.in-addr.arpa
                      25.172.in-addr.arpa
                      26.172.in-addr.arpa
                      27.172.in-addr.arpa
                      28.172.in-addr.arpa
                      29.172.in-addr.arpa
                      30.172.in-addr.arpa
                      31.172.in-addr.arpa
                      corp
                      d.f.ip6.arpa
                      home
                      internal
                      intranet
                      lan
                      local
                      private
                      test

Link 2 (eth0)
      Current Scopes: DNS LLMNR/IPv4 LLMNR/IPv6
DefaultRoute setting: yes
       LLMNR setting: yes
MulticastDNS setting: no
  DNSOverTLS setting: no
      DNSSEC setting: allow-downgrade
    DNSSEC supported: yes
  Current DNS Server: 2001:14b8:1000::1
         DNS Servers: 2001:14b8:1000::1
                      2001:14b8:1000::2

This means that the DNS servers from /etc/resolv.conf are being picked up, but eth0 then uses its own (from my ISP).

How can I tell systemd-resolved and/or systemd-networkd to use only the global DNS servers?

Stellarator · 2019-09-19 08:33:20

Try to turn off DNSDefaultRoute:

cat /etc/systemd/network/20-wired.network 
[Match]
Name=eth0
[Network]
Address=192.168.0.10/24
Gateway=192.168.0.1
DNSDefaultRoute=no

Ideally your /etc/systemd/resolved.conf.d/something.conf should contain:

...
[Resolve]
...
Domains=~.
...

But perhaps the first setup will be enough.

EDIT:

Fix a typo in the second block (#5)

Last edited by Stellarator (2019-09-22 12:39:53)

ondoho · 2019-09-19 20:24:58

Stellarator wrote:

DNSDefaultRoute=no

This breaks name resolution, regardless of the second option you posted.
The 'resolvectl' output is the same as above, except for 'DefaultRoute setting: no' changing to 'yes' for eth0.

I forgot to mention in my last post: Name resolution works OK, it just seems a little slow.
But of course I do want to use those alternative DNS servers which have been working just fine for years!

I wish I knew what's going on; For both cases (DNSDefaultRoute=no or DNSDefaultRoute=yes) I get the same:

systemctl status -n99999 systemd-networkd
● systemd-networkd.service - Network Service
   Loaded: loaded (/usr/lib/systemd/system/systemd-networkd.service; enabled; vendor preset: enabled)
   Active: active (running) since Thu 2019-09-19 23:06:00 EEST; 12s ago
     Docs: man:systemd-networkd.service(8)
 Main PID: 79877 (systemd-network)
   Status: "Processing requests..."
    Tasks: 1
   Memory: 1.4M
   CGroup: /system.slice/systemd-networkd.service
           └─79877 /usr/lib/systemd/systemd-networkd

Sep 19 23:06:00 arch systemd[1]: Starting Network Service...
Sep 19 23:06:00 arch systemd-networkd[79877]: eth0: Gained IPv6LL
Sep 19 23:06:00 arch systemd-networkd[79877]: Enumeration completed
Sep 19 23:06:00 arch systemd[1]: Started Network Service.
Sep 19 23:06:00 arch systemd-networkd[79877]: eth0: IPv6 successfully enabled
Sep 19 23:06:01 arch systemd-networkd[79877]: eth0: DHCPv6 address 2001:14ba:9ff:3200::1/128 timeout preferred 1800 valid 3600
Sep 19 23:06:03 arch systemd-networkd[79877]: eth0: Configured

last line keeps repeating every 10-30s.

Brocellous · 2019-09-19 21:39:25

Considering your log shows a DHCPv6 configured address, but you have no DHCPv6 or IPv6AcceptRA options in your config, you are picking up RDNSS server addresses from a DHCPv6 server started by SLAAC.

Set DHCPV6.UseDNS=no and IPV6ACCEPTRA.UseDNS=no in your .network if you don't want that.

Unless the DHCPv6 server is badly misbehaving, something else is causing the configured message to repeat.

ondoho · 2019-09-20 05:58:40

Okay, "Domains=.~" from post #2 should've been "Domains=~.", and I changed "DHCPV6.UseDNS=no and IPV6ACCEPTRA.UseDNS=no" to
[DHCPV6]
UseDNS=no
[IPV6ACCEPTRA]
UseDNS=no

- but even so, I could never get name resolution to work properly in this scenario.
I tried all possible permutations of everything proposed here and then some.

It weirds me out, because this has been working OK for a long time.
I guess some updates to systemd...

The one thing I did not look at is networkd.conf.

Anyhow, here's a workaround or solution:

remove /etc/resolv.conf
create /etc/systemd/network/20-wired.network.d/dns.conf with the nameservers from /etc/resolv.conf

all other edits undone
/etc/systemd/network/20-wired.network reads

[Match]
Name=eth0
[Network]
Address=192.168.0.10/24
Gateway=192.168.0.1

again

The upside is that resolved or networkd don't seem to go looking for my ISP's namservers anymore.

...But I'm still getting these messages:

Sep 19 23:06:03 arch systemd-networkd[79877]: eth0: Configured

every 10-30s.
What's that all about?

loqs · 2019-09-21 00:12:31

ondoho wrote:

...But I'm still getting these messages:
Sep 19 23:06:03 arch systemd-networkd[79877]: eth0: Configured
every 10-30s.
What's that all about?

https://github.com/systemd/systemd/issues/13595 it would appear the interface is still being configured by RA.

Stellarator · 2019-09-22 13:14:55

ondoho wrote:

Okay, "Domains=.~" from post #2 should've been "Domains=~."...

My bad (a typo is fixed).

And wait a moment... DNSDefaultRoute=no should not break name resolution if you have DNS servers globally configured and your device can reach them (and even if you don't have ANY configured DNS servers systemd-resolved should fallback to its FallbackDNS list). The most likely reason of this behavior is bad configuration of the IP stack. Your first post showed a bunch of IPv6 DNS servers and meanwhile the network configuration clearly states that only IPv4 address is (manually) configured on the interface. There is a side discussion related to the RA issues but nobody asked a good question: does your IPv6 stack work at all (or work correctly)? Can you ping mentioned DNS servers?

ip link
ip address
ip route
ip -6 route
for the review...

This excerpt provides a detailed (albeit a little confusing) explanation of DNS query routing logic. Please, check especially the second part of the section. I would not be so convinced if I had not used dual-stack and systemd-networkd/systemd-resolved pair by myself (with global DNS servers configured as primary and DNS servers received by DHCPv4 means (I use them solely to resolve domains in local networks)).

p.s. I also assume that you're not in situation where multiple network managers are fighting for your soul(?)

ondoho · 2019-09-22 14:22:06

@loqs: Thank you, that looks relevant. It seems my ISP does that much more often than every 200s...
But I will leave it aside for a moment, because I agree with @Stellarator that something is amiss here:

Stellarator wrote:

DNSDefaultRoute=no should not break name resolution if you have DNS servers globally configured and your device can reach them (and even if you don't have ANY configured DNS servers systemd-resolved should fallback to its FallbackDNS list).

Yet that's exactly what happens, every time.
I don't know about "and your device can reach them" though.

Can you ping mentioned DNS servers?

The opennic IPv6 servers? yes.

ip link
ip address
ip route
ip -6 route
for the review...

$ ip link
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN mode DEFAULT group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UP mode DEFAULT group default qlen 1000
    link/ether XX:YY:XX:YY:XX:YY brd ff:ff:ff:ff:ff:ff

$ ip address
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host 
       valid_lft forever preferred_lft forever
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UP group default qlen 1000
    link/ether XX:YY:XX:YY:XX:YY brd ff:ff:ff:ff:ff:ff
    inet 192.168.0.10/24 brd 192.168.0.255 scope global eth0
       valid_lft forever preferred_lft forever
    inet6 2001:14ba:9ff:3200::1/128 scope global dynamic noprefixroute 
       valid_lft 3395sec preferred_lft 1595sec
    inet6 2001:14ba:9ff:3200:dacb:8aff:fec1:b3ae/64 scope global dynamic mngtmpaddr noprefixroute 
       valid_lft 17992sec preferred_lft 11242sec
    inet6 fe80::dacb:8aff:fec1:b3ae/64 scope link 
       valid_lft forever preferred_lft forever

$ ip route
default via 192.168.0.1 dev eth0 proto static 
192.168.0.0/24 dev eth0 proto kernel scope link src 192.168.0.10 

$ ip -6 route
::1 dev lo proto kernel metric 256 pref medium
2001:14ba:9ff:3200::/64 dev eth0 proto ra metric 1024 expires 17980sec pref medium
2001:14ba:9ff:3200::/56 via fe80::125f:49ff:fe6a:3341 dev eth0 proto ra metric 1024 expires 17980sec pref medium
fe80::/64 dev eth0 proto kernel metric 256 pref medium
default via fe80::125f:49ff:fe6a:3341 dev eth0 proto ra metric 1024 expires 1780sec pref medium

When i set IPv6AcceptRA=false an interesting thing happens:

$ ping archive.org
ping: archive.org: Temporary failure in name resolution
$ ping 2a03:b0c0:3:e0::67:1001
ping: connect: Network is unreachable
$ ping 8.8.8.8
PING 8.8.8.8 (8.8.8.8) 56(84) bytes of data.
64 bytes from 8.8.8.8: icmp_seq=1 ttl=54 time=19.2 ms
64 bytes from 8.8.8.8: icmp_seq=2 ttl=54 time=19.4 ms
64 bytes from 8.8.8.8: icmp_seq=3 ttl=54 time=18.2 ms
64 bytes from 8.8.8.8: icmp_seq=4 ttl=54 time=16.8 ms
64 bytes from 8.8.8.8: icmp_seq=5 ttl=54 time=17.0 ms

--- 8.8.8.8 ping statistics ---
5 packets transmitted, 5 received, 0% packet loss, time 4006ms
rtt min/avg/max/mdev = 16.762/18.115/19.394/1.081 ms
$ systemctl status systemd-networkd.service
● systemd-networkd.service - Network Service
   Loaded: loaded (/usr/lib/systemd/system/systemd-networkd.service; enabled; vendor preset: enabled)
   Active: active (running) since Sun 2019-09-22 17:11:01 EEST; 1min 33s ago
     Docs: man:systemd-networkd.service(8)
 Main PID: 16413 (systemd-network)
   Status: "Processing requests..."
    Tasks: 1
   Memory: 1.4M
   CGroup: /system.slice/systemd-networkd.service
           └─16413 /usr/lib/systemd/systemd-networkd

Sep 22 17:11:00 arch systemd[1]: Starting Network Service...
Sep 22 17:11:01 arch systemd-networkd[16413]: eth0: Gained IPv6LL
Sep 22 17:11:01 arch systemd-networkd[16413]: Enumeration completed
Sep 22 17:11:01 arch systemd[1]: Started Network Service.
Sep 22 17:11:01 arch systemd-networkd[16413]: eth0: IPv6 successfully enabled
Sep 22 17:11:01 arch systemd-networkd[16413]: eth0: Configured

________________

p.s. I also assume that you're not in situation where multiple network managers are fighting for your soul(?)

Quite sure about that.
I uninstalled netctl long ago, networkmanager probably never was installed in the first place, and I also uninstalled dhcp.
pacman -Ss network manager|grep installed returns nothing.
But I have been messing around with networking stuff for years now, it's quite possible I took a wrong turn somewhere and there's some bad config residue...

Stellarator · 2019-09-22 17:05:05

Huh, I don't see any criminal. So the suspicious target is systemd-resolved and name resolution as a whole. Would you like to try a more "classic" approach? Like:

ln -sf /run/systemd/resolve/stub-resolv.conf /etc/resolv.conf

and put something like

[Resolve]
DNS=8.8.8.8 9.9.9.9
DNSSEC=no
Domains=~.

into /etc/systemd/resolved.conf.d/something.conf
where manipulation with DNSSEC is optional (just to cut off excess) and 8.8.8.8 9.9.9.9 is to start from and then change to something from your IPv6 list.

No so many things come to my mind (like verifying /etc/gai.conf and /etc/nsswitch.conf)... Some blocks from firewall / ISP side? But you've said that latest version of systemd is likely the reason...

ondoho · 2019-09-23 06:31:00

^ Additionally I also edited /etc/systemd/network/20-wired.network:

[Match]
Name=eth0
[Network]
Address=192.168.0.10/24
Gateway=192.168.0.1
IPv6AcceptRA=false
DNSDefaultRoute=false

Together, that worked!

But then I tried my IPv6 nameservers, and it failed!

So as long as I only use IPv4 nameservers, this approach works (also the RA flooding is gone now).

What's wrong with IPv6?

Here's some output with IPv6 DNS servers, and no name resolution:

$ resolvectl
Global
       LLMNR setting: yes
MulticastDNS setting: yes
  DNSOverTLS setting: no
      DNSSEC setting: no
    DNSSEC supported: no
  Current DNS Server: 2001:470:1f15:b80::53
         DNS Servers: 2a00:f826:8:2::195
                      2a01:4f9:c010:475c::1
                      2a03:4000:2a:351:1889:79ff:fe46:60d0
                      2a00:f826:8:1::254
                      2001:470:1f15:b80::53
Fallback DNS Servers: 1.1.1.1
                      9.9.9.10
                      8.8.8.8
                      2606:4700:4700::1111
                      2620:fe::10
                      2001:4860:4860::8888
          DNS Domain: ~.
          DNSSEC NTA: 10.in-addr.arpa
                      16.172.in-addr.arpa
                      168.192.in-addr.arpa
                      17.172.in-addr.arpa
                      18.172.in-addr.arpa
                      19.172.in-addr.arpa
                      20.172.in-addr.arpa
                      21.172.in-addr.arpa
                      22.172.in-addr.arpa
                      23.172.in-addr.arpa
                      24.172.in-addr.arpa
                      25.172.in-addr.arpa
                      26.172.in-addr.arpa
                      27.172.in-addr.arpa
                      28.172.in-addr.arpa
                      29.172.in-addr.arpa
                      30.172.in-addr.arpa
                      31.172.in-addr.arpa
                      corp
                      d.f.ip6.arpa
                      home
                      internal
                      intranet
                      lan
                      local
                      private
                      test

Link 2 (eth0)
      Current Scopes: LLMNR/IPv4 LLMNR/IPv6
DefaultRoute setting: no
       LLMNR setting: yes
MulticastDNS setting: no
  DNSOverTLS setting: no
      DNSSEC setting: no
    DNSSEC supported: no

$ journalctl -b -u systemd-resolved.service -p 7|grep -v /etc/hosts
{snip}
Sep 23 09:20:58 arch systemd[1]: Stopped Network Name Resolution.
Sep 23 09:20:58 arch systemd[1]: Starting Network Name Resolution...
Sep 23 09:20:58 arch systemd-resolved[18015]: Positive Trust Anchors:
Sep 23 09:20:58 arch systemd-resolved[18015]: . IN DS 19036 8 2 49aac11d7b6f6446702e54a1607371607a1a41855200fd2ce1cdde32f24e8fb5
Sep 23 09:20:58 arch systemd-resolved[18015]: . IN DS 20326 8 2 e06d44b80b8f1d39a95c0b0d7c65d08458e880409bbc683457104237c7f8ec8d
Sep 23 09:20:58 arch systemd-resolved[18015]: Negative trust anchors: 10.in-addr.arpa 16.172.in-addr.arpa 17.172.in-addr.arpa 18.172.in-addr.arpa 19.172.in-addr.arpa 20.172.in-addr.arpa 21.172.in-addr.arpa 22.172.in-addr.arpa 23.172.in-addr.arpa 24.172.in-addr.arpa 25.172.in-addr.arpa 26.172.in-addr.arpa 27.172.in-addr.arpa 28.172.in-addr.arpa 29.172.in-addr.arpa 30.172.in-addr.arpa 31.172.in-addr.arpa 168.192.in-addr.arpa d.f.ip6.arpa corp home internal intranet lan local private test
Sep 23 09:20:58 arch systemd-resolved[18015]: Using system hostname 'arch'.
Sep 23 09:20:58 arch systemd[1]: Started Network Name Resolution.

Stellarator · 2019-09-23 08:36:07

You need some mechanism to configure IPv6 and its routing. You can try to do it manually using ip command (you have all needed parts in #8). And if it works, then it remains to deal with RA and a company (DHCPv6?).

Your setup should look something like this:

ip addr add 2001:14ba:9ff:3200:dacb:8aff:fec1:b3ae/64 dev eth0
ip -6 route add 2000::/3 via fe80::125f:49ff:fe6a:3341

(but, please, verify and adapt to your current conditions)

ondoho · 2019-09-23 16:56:54

With the setup from my previous post I cannot ping IPv6 addresses at all.
Name resolution works, but it seems to always choose IPv4 addresses.

After issuing commands:

Stellarator wrote:

ip addr add 2001:14ba:9ff:3200:dacb:8aff:fec1:b3ae/64 dev eth0
ip -6 route add 2000::/3 via fe80::125f:49ff:fe6a:3341 dev eth0

I can ping IPv6 addresses, too, and name resolution seems to prefer IPv6 addresses.

But what do those commands do?
What are these addresses? They are not nameservers afaics.
And how can I tell systemd-{networkd,resolved} to always configure the network this way?

Anyhow, thank you so far. Networking really is my weakest point in Linux administration.

Arch Linux

#1 2019-09-19 07:16:25

systemd-networkd: Wired interface does not use systemd-resolved DNS

#2 2019-09-19 08:33:20

Re: systemd-networkd: Wired interface does not use systemd-resolved DNS

EDIT:

#3 2019-09-19 20:24:58

Re: systemd-networkd: Wired interface does not use systemd-resolved DNS

#4 2019-09-19 21:39:25

Re: systemd-networkd: Wired interface does not use systemd-resolved DNS

#5 2019-09-20 05:58:40

Re: systemd-networkd: Wired interface does not use systemd-resolved DNS

#6 2019-09-21 00:12:31

Re: systemd-networkd: Wired interface does not use systemd-resolved DNS

#7 2019-09-22 13:14:55

Re: systemd-networkd: Wired interface does not use systemd-resolved DNS

#8 2019-09-22 14:22:06

Re: systemd-networkd: Wired interface does not use systemd-resolved DNS

#9 2019-09-22 17:05:05

Re: systemd-networkd: Wired interface does not use systemd-resolved DNS

#10 2019-09-23 06:31:00

Re: systemd-networkd: Wired interface does not use systemd-resolved DNS

#11 2019-09-23 08:36:07

Re: systemd-networkd: Wired interface does not use systemd-resolved DNS

#12 2019-09-23 16:56:54

Re: systemd-networkd: Wired interface does not use systemd-resolved DNS

Board footer