Arch Linux

Dryden

Member

Registered: 2023-10-30

Posts: 14

[Solved] WireGuard breaks with NetworkManager, systemd-resolved

My workstation running linux 6.5.9.arch2-1, networkmanager 1.44.2-3, systemd-resolvconf 254.5-1, and wireguard-tools 1.0.20210914-1 and using systemd-resolved exhibits the following behavior.

Without

#/etc/NetworkManager/conf.d/unmanaged.conf
[keyfile]
unmanaged-devices=type:wireguard

• Normal connectivity via VPN server and SOCKS proxy when connected to home network---the router runs OpenWRT firmware---with Wireguard interface enabled on workstation

• Normal connectivity (minus VPN and SOCKS proxy) when tethered to mobile phone (which runs a graphical VPN client with WireGuard backend) with WireGuard interface disabled on workstation---I have observed the same problem on other networks, but this is the only one available to test at the moment

• Broken DNS when tethered to mobile phone (which runs a graphical VPN client with WireGuard backend) with WireGuard interface enabled on workstation

After first setting

#/etc/NetworkManager/conf.d/unmanaged.conf
[keyfile]
unmanaged-devices=type:wireguard

per the wiki
and after

nmcli general reload

but before rebooting

• Regardless of network (home or tethered to mobile), connections are normal (WireGuard connection to VPN server) only if a SOCKS proxy is not used

After rebooting with

#/etc/NetworkManager/conf.d/unmanaged.conf
[keyfile]
unmanaged-devices=type:wireguard

• WireGuard interface does not come up, so there is no VPN connection

Other contents of

#/etc/NetworkManager/conf.d/

are:

#/etc/NetworkManager/conf.d/dns.conf
[main]
dns=systemd-resolved

#/etc/NetworkManager/conf.d/rc-manager.conf
#[main]
#rc-manager=resolvconf

Outputs after rebooting:

find /etc/systemd -type l -exec test -f {} \; -print | awk -F'/' '{ printf("%-40s | %s\n", $(NF-0), $(NF-1)) }' | sort -f
apparmor.service                         | multi-user.target.wants
cups.path                                | multi-user.target.wants
cups.service                             | multi-user.target.wants
cups.service                             | printers.target.wants
cups.socket                              | sockets.target.wants
dbus-org.freedesktop.nm-dispatcher.service | system
dbus-org.freedesktop.resolve1.service    | system
dbus-org.freedesktop.timesync1.service   | system
getty@tty1.service                       | getty.target.wants
libvirtd-ro.socket                       | sockets.target.wants
libvirtd.service                         | multi-user.target.wants
libvirtd.socket                          | sockets.target.wants
NetworkManager.service                   | multi-user.target.wants
NetworkManager-wait-online.service       | network-online.target.wants
p11-kit-server.socket                    | sockets.target.wants
pipewire-pulse.socket                    | sockets.target.wants
pipewire-session-manager.service         | user
pipewire.socket                          | sockets.target.wants
privoxy.service                          | multi-user.target.wants
remote-fs.target                         | multi-user.target.wants
sshd.service                             | multi-user.target.wants
systemd-resolved.service                 | multi-user.target.wants
systemd-resolved.service                 | sysinit.target.wants
systemd-timesyncd.service                | sysinit.target.wants
ufw.service                              | multi-user.target.wants
virtlockd.socket                         | sockets.target.wants
virtlogd.socket                          | sockets.target.wants
wireplumber.service                      | pipewire.service.wants

resolvectl status
Global
           Protocols: +LLMNR +mDNS -DNSOverTLS DNSSEC=no/unsupported
    resolv.conf mode: stub
  Current DNS Server: redacted-remote-server-IPv4-2
         DNS Servers: redacted-remote-server-IPv4-1 redacted-remote-server-IPv4-2 redacted-remote-server-IPv6-1
Fallback DNS Servers: redacted

Link 2 (enp0s31f6)
    Current Scopes: none
         Protocols: -DefaultRoute +LLMNR +mDNS -DNSOverTLS DNSSEC=no/unsupported

Link 3 (wlp2s0)
    Current Scopes: DNS LLMNR/IPv4 LLMNR/IPv6 mDNS/IPv4 mDNS/IPv6
         Protocols: +DefaultRoute +LLMNR +mDNS -DNSOverTLS DNSSEC=no/unsupported
Current DNS Server: redacted-router-IPv4
       DNS Servers: redacted-router-IPv4 redacted-router-IPv6 
        DNS Domain: lan

ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host noprefixroute 
       valid_lft forever preferred_lft forever
2: enp0s31f6: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc fq_codel state DOWN group default qlen 1000
    link/ether redacted-MAC-1 brd ff:ff:ff:ff:ff:ff
3: wlp2s0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
    link/ether redacted-MAC-2 brd ff:ff:ff:ff:ff:ff
    inet redacted-LAN-IPv4-1/24 brd redacted-LAN-IPv4-2 scope global dynamic noprefixroute wlp2s0
       valid_lft 40310sec preferred_lft 40310sec
    inet6 redacted-IPv6-1/128 scope global dynamic noprefixroute 
       valid_lft 40309sec preferred_lft 40309sec
    inet6 redacted-IPv6-2/128 scope global dynamic noprefixroute 
       valid_lft 40309sec preferred_lft 40309sec
    inet6 redacted-IPv6-2/64 scope global dynamic noprefixroute 
       valid_lft 322698sec preferred_lft 322698sec
    inet6 redacted-IPv6-1/64 scope global noprefixroute 
       valid_lft forever preferred_lft forever
    inet6 redacted-IPv6-3/64 scope link noprefixroute 
       valid_lft forever preferred_lft forever
5: wg-interface: <POINTOPOINT,NOARP> mtu 1420 qdisc noop state DOWN group default qlen 1000
    link/none

ip r
default via redacted-router-IPv4 dev wlp2s0 proto dhcp src 192.168.redacted.1 metric 600 
192.168.redacted.0/24 dev wlp2s0 proto kernel scope link src redacted-LAN-IPv4-1 metric 600

I would be very grateful for assistance in achieving a configuration that brings up the WireGuard interface on boot, but does not disrupt DNS or SOCKS proxying, regardless of the network to which my workstation is connected.

Last edited by Dryden (2023-11-09 15:42:51)

-thc

Member

Registered: 2017-03-15

Posts: 502

Re: [Solved] WireGuard breaks with NetworkManager, systemd-resolved

What kind of tethering? USB? Bluetooth? WiFi?

How did you configure the WireGuard connection and how do you connect to it?

Dryden

Member

Registered: 2023-10-30

Posts: 14

Re: [Solved] WireGuard breaks with NetworkManager, systemd-resolved

What kind of tethering? USB? Bluetooth? WiFi?

I connect to the mobile via WiFi tethering. In other words, I broadcast a WiFi hotspot from the mobile to which I then connect my workstation. But the networking problems have occurred in the past when connecting to, e.g., home routers and public WiFi networks. Indeed, I have never had completely normal networking behavior on this workstation for any network other than my home network.

How did you configure the WireGuard connection and how do you connect to it?

As far as I can remember, I configured it with NetworkManager by importing a file provided by my VPN service:

nmcli connection import type wireguard file redacted

I configured the interface before I began using systemd-resolved, which I began to do because I could not get DNS to work when switching networks. At that time, I believe that I had dnsmasq installed and enabled, which I no longer have.
NetworkManager brings up the WireGuard interface at boot unless I set the interface to be unmanaged by NetworkManager (which also disables manually connecting to it with nmcli).

-thc

Member

Registered: 2017-03-15

Posts: 502

Re: [Solved] WireGuard breaks with NetworkManager, systemd-resolved

O.K. - please make sure that "/etc/wireguard" is empty and remove "/etc/NetworkManager/conf.d/unmanaged.conf".

NetworkManager will not automatically start it's own WireGuard connections until they are configured to do so.

"resolveconf" is in your context only useful for "wg-quick" activated WireGuard connections - you can remove it (and "/etc/NetworkManager/conf.d/rc-manager.conf").

Make sure your firewall ruleset doesn't interfere with DNS resolution.

Your "resolvectl" status looks O.K.

A phone with WiFi tethering works like a WiFi internet router for your PC - including the VPN running on the PC. VPN on the phone should be inactive.

Last edited by -thc (2023-10-30 17:04:36)

Dryden

Member

Registered: 2023-10-30

Posts: 14

Re: [Solved] WireGuard breaks with NetworkManager, systemd-resolved

O.K. - please make sure that "/etc/wireguard" is empty and remove "/etc/NetworkManager/conf.d/unmanaged.conf".

Thank you. "/etc/wireguard" was and is empty and I have removed "unmanaged.conf."

"resolveconf" is in your context only useful for "wg-quick" activated WireGuard connections - you can remove it (and "/etc/NetworkManager/conf.d/rc-manager.conf").

Is the suggestion to uninstall systemd-resolvconf? As far as I know, the only resolvconf I have is "/etc/resolv.conf," which is symlinked to "run/systemd/resolve/stub-resolv.conf" and has the following contents:

nameserver 127.0.0.53
options edns0 trust-ad
search .

Make sure your firewall ruleset doesn't interfere with DNS resolution.

My only explicitly configured firewall rule, the only one that shows up on a ufw status report, allows tcp on my ssh port from the LAN.

A phone with WiFi tethering works like a WiFi internet router for your PC - including the VPN running on the PC. VPN on the phone should be inactive.

Thank you. I tried connecting just now with the VPN disabled on the phone, but the behavior seems the same as when it is enabled. Also, I have another workstation (that unfortunately runs Windows) with a graphical, WireGuard-based VPN client that exhibits the desired connection behavior (WireGuard, VPN, SOCKS) on all networks, including when using my phone as a WiFi hotspot while the phone has the VPN enabled. 

I'm grateful to you for taking the time to investigate this with me.

-thc

Member

Registered: 2017-03-15

Posts: 502

Re: [Solved] WireGuard breaks with NetworkManager, systemd-resolved

Is the suggestion to uninstall systemd-resolvconf?

Yes.
The "/etc/resolv.conf" is O.K.

Thank you. I tried connecting just now with the VPN disabled on the phone, but the behavior seems the same as when it is enabled. Also, I have another workstation (that unfortunately runs Windows) with a graphical, WireGuard-based VPN client that exhibits the desired connection behavior (WireGuard, VPN, SOCKS) on all networks, including when using my phone as a WiFi hotspot while the phone has the VPN enabled.

From now on I will always assume that your phone's VPN is off.

To make sure I understand you correctly - is the following correct:

- You have an imported WireGuard configuration from a VPN provider in NetworkManager with "systemd-resolved" as a DNS cache.
- If you are at home in your home network (WiFi) everything works as expected.
- If you are abroad, the WiFi tethering to your phone via mobile data breaks name resolution.

In the case of three yeses: What does your "/etc/systemd/resolved.conf" look like?

Dryden

Member

Registered: 2023-10-30

Posts: 14

Re: [Solved] WireGuard breaks with NetworkManager, systemd-resolved

Thank you. I uninstalled systemd-resolved, as well as netctl, which required it. After rebooting, there seems to be no difference in behavior.

To make sure I understand you correctly - is the following correct:

- You have an imported WireGuard configuration from a VPN provider in NetworkManager with "systemd-resolved" as a DNS cache.
- If you are at home in your home network (WiFi) everything works as expected.
- If you are abroad, the WiFi tethering to your phone via mobile data breaks name resolution.

In the case of three yeses: What does your "/etc/systemd/resolved.conf" look like?

Yes x3

#/etc/systemd/resolved.conf
[Resolve]
DNS= VPN-IPv4-1#VPN-URL VPN-IPv4-2#VPN-URL VPN-IPv6#VPN-URL
FallbackDNS=Popular-IPv4-1#Popular-URL Popular-IPv4-2#Popular-URL Popular-IPv6-1#Popular-URL Popular-IPv6-2#Popular-URL

-thc

Member

Registered: 2017-03-15

Posts: 502

Re: [Solved] WireGuard breaks with NetworkManager, systemd-resolved

Thank you. I uninstalled systemd-resolved, as well as netctl, which required it. After rebooting, there seems to be no difference in behavior.

Don't expect it to - we're still eliminating possible side-effects.

#/etc/systemd/resolved.conf
[Resolve]
DNS= VPN-IPv4-1#VPN-URL VPN-IPv4-2#VPN-URL VPN-IPv6#VPN-URL
FallbackDNS=Popular-IPv4-1#Popular-URL Popular-IPv4-2#Popular-URL Popular-IPv6-1#Popular-URL Popular-IPv6-2#Popular-URL

That looks like a common misunderstanding of "FallbackDNS" in systemd-resolved. Whenever no link-specific DNS servers are available the global ones are used instead. If those are empty too (default), the fallback servers will be contacted. Your configuration is the functional equivalent of

#/etc/systemd/resolved.conf
[Resolve]
DNS= VPN-IPv4-1#VPN-URL VPN-IPv4-2#VPN-URL VPN-IPv6#VPN-URL
FallbackDNS=

because the fallback servers will never be used.

Secondly the setting of the VPN DNS servers should be done either by the VPN endpoint (e.g. OpenVPN via push/DHCP) or by local configuration (e.g. WireGuard)
I use the configuration

#/etc/systemd/resolved.conf
[Resolve]
DNS=
FallbackDNS=

because I want to know when something goes wrong with the link-specific DNS.

Thirdly are those VPN DNS servers open and publicly available - outside the VPN? If not, do not use them in this way.

My recommendation: Remove the VPN DNS servers from the global systemd-resolved configuration and add them to the NetworkManager WireGuard connection.

Dryden

Member

Registered: 2023-10-30

Posts: 14

Re: [Solved] WireGuard breaks with NetworkManager, systemd-resolved

Thank you for correcting my misunderstanding.

Thirdly are those VPN DNS servers open and publicly available - outside the VPN?

Yes

My recommendation: Remove the VPN DNS servers from the global systemd-resolved configuration and add them to the NetworkManager WireGuard connection.

Thank you. After doing this and restarting systemd-resolved and doing a general reload of nmcli, but not rebooting---I have a lot of projects open---the behavior is still the same. I tried three different VPN-provided DNS servers in the NetworkManager WireGuard profile one by one.

-thc

Member

Registered: 2017-03-15

Posts: 502

Re: [Solved] WireGuard breaks with NetworkManager, systemd-resolved

Please do the following in both scenarios (at home, working and via hotspot, not working):

resolvectl
drill wiki.archlinux.org
drill wiki.archlinux.org @8.8.8.8
resolvectl query wiki.archlinux.org

and post the output here.

Dryden

Member

Registered: 2023-10-30

Posts: 14

Re: [Solved] WireGuard breaks with NetworkManager, systemd-resolved

Please do the following in both scenarios (at home, working and via hotspot, not working):

resolvectl
drill wiki.archlinux.org
drill wiki.archlinux.org @8.8.8.8
resolvectl query wiki.archlinux.org

and post the output here.

Scenario 1 (home network, working)

resolvectl
Global
         Protocols: +LLMNR +mDNS -DNSOverTLS DNSSEC=no/unsupported
  resolv.conf mode: stub

Link 2 (enp0s31f6)
    Current Scopes: none
         Protocols: -DefaultRoute +LLMNR +mDNS -DNSOverTLS DNSSEC=no/unsupported

Link 3 (wlp2s0)
    Current Scopes: DNS LLMNR/IPv4 LLMNR/IPv6 mDNS/IPv4 mDNS/IPv6
         Protocols: -DefaultRoute +LLMNR +mDNS -DNSOverTLS DNSSEC=no/unsupported
Current DNS Server: redacted-router-IPv4
       DNS Servers: redacted-router-IPv4
        DNS Domain: lan

Link 9 (wg)
    Current Scopes: DNS
         Protocols: +DefaultRoute +LLMNR +mDNS -DNSOverTLS DNSSEC=no/unsupported
Current DNS Server: redacted-wg.nmconnection-IPv4
       DNS Servers: redacted-wg.nmconnection-IPv4
        DNS Domain: ~.

drill wiki.archlinux.org
;; ->>HEADER<<- opcode: QUERY, rcode: NOERROR, id: 32205
;; flags: qr rd ra ; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 0 
;; QUESTION SECTION:
;; wiki.archlinux.org.	IN	A

;; ANSWER SECTION:
wiki.archlinux.org.	3600	IN	A	135.181.27.174

;; AUTHORITY SECTION:

;; ADDITIONAL SECTION:

;; Query time: 228 msec
;; SERVER: 127.0.0.53
;; WHEN: Thu Nov  2 20:14:10 2023
;; MSG SIZE  rcvd: 52

drill wiki.archlinux.org @8.8.8.8
;; ->>HEADER<<- opcode: QUERY, rcode: NOERROR, id: 13974
;; flags: qr rd ra ; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 0 
;; QUESTION SECTION:
;; wiki.archlinux.org.	IN	A

;; ANSWER SECTION:
wiki.archlinux.org.	3575	IN	A	135.181.27.174

;; AUTHORITY SECTION:

;; ADDITIONAL SECTION:

;; Query time: 22 msec
;; SERVER: 8.8.8.8
;; WHEN: Thu Nov  2 20:14:35 2023
;; MSG SIZE  rcvd: 52

resolvectl query wiki.archlinux.org
wiki.archlinux.org: 2a01:4f9:c010:a4eb::1      -- link: wg
                    135.181.27.174             -- link: wg

-- Information acquired via protocol DNS in 179.1ms.
-- Data is authenticated: no; Data was acquired via local or encrypted transport: no
-- Data from: cache network

Scenario 2 (via hotspot, not working)

resolvectl
Global
         Protocols: +LLMNR +mDNS -DNSOverTLS DNSSEC=no/unsupported
  resolv.conf mode: stub

Link 2 (enp0s31f6)
    Current Scopes: none
         Protocols: -DefaultRoute +LLMNR +mDNS -DNSOverTLS DNSSEC=no/unsupported

Link 3 (wlp2s0)
    Current Scopes: LLMNR/IPv4 LLMNR/IPv6 mDNS/IPv4 mDNS/IPv6
         Protocols: -DefaultRoute +LLMNR +mDNS -DNSOverTLS DNSSEC=no/unsupported

Link 10 (wg)
    Current Scopes: DNS
         Protocols: +DefaultRoute +LLMNR +mDNS -DNSOverTLS DNSSEC=no/unsupported
Current DNS Server: redacted-wg.nmconnection-IPv4
       DNS Servers: redacted-wg.nmconnection-IPv4
        DNS Domain: ~.

drill wiki.archlinux.org
Error: error sending query: Could not send or receive, because of network error

drill wiki.archlinux.org @8.8.8.8
Error: error sending query: Could not send or receive, because of network error

resolvectl query wiki.archlinux.org
wiki.archlinux.org: resolve call failed: Connection timed out

-thc

Member

Registered: 2017-03-15

Posts: 502

Re: [Solved] WireGuard breaks with NetworkManager, systemd-resolved

Please post the output of the following commands in Scenario 2 (hotspot) - first with WireGuard disabled and secondly with enabled VPN.

ip a
ip route
ip rule
resolvectl

At first you should see an address for the WiFi interface with the correct routing table and the "empty" rule table - three lookup rules for local, main and default. "resolvectl" should point to the hotspot DNS.

In the second case "ip rule" should contain two additional rules - one of them points to a five digit routing table number ([...] lookup NNNNN).
The command

ip route list table NNNNN

should yield one rule - to route everything to the WireGuard interface.

Dryden

Member

Registered: 2023-10-30

Posts: 14

Re: [Solved] WireGuard breaks with NetworkManager, systemd-resolved

Please post the output of the following commands in Scenario 2 (hotspot) - first with WireGuard disabled and secondly with enabled VPN.

ip a
ip route
ip rule
resolvectl

At first you should see an address for the WiFi interface with the correct routing table and the "empty" rule table - three lookup rules for local, main and default. "resolvectl" should point to the hotspot DNS.

In the second case "ip rule" should contain two additional rules - one of them points to a five digit routing table number ([...] lookup NNNNN).
The command

ip route list table NNNNN

should yield one rule - to route everything to the WireGuard interface.

Hotspot with WireGuard disabled:

ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host noprefixroute 
       valid_lft forever preferred_lft forever
2: enp0s31f6: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc fq_codel state DOWN group default qlen 1000
    link/ether redacted-MAC-1 brd ff:ff:ff:ff:ff:ff
3: wlp2s0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
    link/ether redacted-MAC-2 brd ff:ff:ff:ff:ff:ff
    inet redacted-LAN-IPv4-1/24 brd redacted-LAN-IPv4-2 scope global dynamic noprefixroute wlp2s0
       valid_lft 3055sec preferred_lft 3055sec
    inet6 redacted-LAN-IPv6-1/64 scope link noprefixroute 
       valid_lft forever preferred_lft forever

ip route
default via redacted-LAN-IPv4-3 dev wlp2s0 proto dhcp src redacted-LAN-IPv4-1 metric 600 
redacted-LAN-IPv4-4/24 dev wlp2s0 proto kernel scope link src redacted-LAN-IPv4-1 metric 600

ip rule
0:	from all lookup local
32766:	from all lookup main
32767:	from all lookup default

resolvectl
Global
         Protocols: +LLMNR +mDNS -DNSOverTLS DNSSEC=no/unsupported
  resolv.conf mode: stub

Link 2 (enp0s31f6)
    Current Scopes: none
         Protocols: -DefaultRoute +LLMNR +mDNS -DNSOverTLS DNSSEC=no/unsupported

Link 3 (wlp2s0)
    Current Scopes: DNS LLMNR/IPv4 LLMNR/IPv6 mDNS/IPv4 mDNS/IPv6
         Protocols: +DefaultRoute +LLMNR +mDNS -DNSOverTLS DNSSEC=no/unsupported
Current DNS Server: redacted-LAN-IPv4-3
       DNS Servers: redacted-LAN-IPv4-3

Hotspot with enabled VPN:

ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host noprefixroute 
       valid_lft forever preferred_lft forever
2: enp0s31f6: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc fq_codel state DOWN group default qlen 1000
    link/ether redacted-MAC-1 brd ff:ff:ff:ff:ff:ff
3: wlp2s0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
    link/ether redacted-MAC-2 brd ff:ff:ff:ff:ff:ff
    inet redacted-LAN-IPv4-1/24 brd redacted-LAN-IPv4-2 scope global dynamic noprefixroute wlp2s0
       valid_lft 2835sec preferred_lft 2835sec
    inet6 redacted-LAN-IPv6-1/64 scope link noprefixroute 
       valid_lft forever preferred_lft forever
11: wg: <POINTOPOINT,NOARP,UP,LOWER_UP> mtu 1420 qdisc noqueue state UNKNOWN group default qlen 1000
    link/none 
    inet redacted-external-IPv4/32 scope global noprefixroute wg
       valid_lft forever preferred_lft forever
    inet6 redacted-external-IPv6-1/128 scope global noprefixroute 
       valid_lft forever preferred_lft forever
    inet6 redacted-external-IPv6-2/64 scope link noprefixroute 
       valid_lft forever preferred_lft forever

ip route
default via redacted-LAN-IPv4-3 dev wlp2s0 proto dhcp src redacted-LAN-IPv4-1 metric 600 
redacted-LAN-IPv4-4/24 dev wlp2s0 proto kernel scope link src redacted-LAN-IPv4-1 metric 600

ip rule
0:	from all lookup local
30814:	from all lookup main suppress_prefixlength 0
30815:	not from all fwmark 0xcb4d lookup NNNNN
32766:	from all lookup main
32767:	from all lookup default

resolvectl
Global
         Protocols: +LLMNR +mDNS -DNSOverTLS DNSSEC=no/unsupported
  resolv.conf mode: stub

Link 2 (enp0s31f6)
    Current Scopes: none
         Protocols: -DefaultRoute +LLMNR +mDNS -DNSOverTLS DNSSEC=no/unsupported

Link 3 (wlp2s0)
    Current Scopes: LLMNR/IPv4 LLMNR/IPv6 mDNS/IPv4 mDNS/IPv6
         Protocols: -DefaultRoute +LLMNR +mDNS -DNSOverTLS DNSSEC=no/unsupported

Link 11 (wg)
    Current Scopes: DNS
         Protocols: +DefaultRoute +LLMNR +mDNS -DNSOverTLS DNSSEC=no/unsupported
Current DNS Server: redacted-wg.nmconnection-IPv4
       DNS Servers: redacted-wg.nmconnection-IPv4
        DNS Domain: ~.

ip route list table NNNNN
default dev wg proto static scope link metric 20050

-thc

Member

Registered: 2017-03-15

Posts: 502

Re: [Solved] WireGuard breaks with NetworkManager, systemd-resolved

Well - this is unexpected. Everything looks O.K. but it doesn't work anyway.

Since you never mentioned it explicitly: In scenario 2 with WireGuard enabled - can you ping/access hosts by IPv4 address?

Dryden

Member

Registered: 2023-10-30

Posts: 14

Re: [Solved] WireGuard breaks with NetworkManager, systemd-resolved

In scenario 2 with WireGuard enabled - can you ping/access hosts by IPv4 address?

In that scenario the only IPv4 addresses that I have succeeded in pinging are the LAN address of the device broadcasting the hotspot and the external VPN server specified in the nmconnection file. I cannot ping other external addresses, including the VPN's DNS server.

For the next few hours I am in a location where I can test behavior on a WiFi network other than my home network and my phone's hotspot. The behavior looks the same to me. As far as I can tell, the only relevant difference between this network and my phone's hotspot when running 'ip a,' 'ip route,' 'ip rule,' and 'resolvectl' is this:

Phone with WireGuard enabled:

resolvectl
...
Link 3 (wlp2s0)
    Current Scopes: LLMNR/IPv4 LLMNR/IPv6 mDNS/IPv4 mDNS/IPv6
         Protocols: -DefaultRoute +LLMNR +mDNS -DNSOverTLS DNSSEC=no/unsupported
...

 
Non-home WiFi network with WireGuard enabled:

resolvectl
...
Link 3 (wlp2s0)
    Current Scopes: DNS LLMNR/IPv4 LLMNR/IPv6 mDNS/IPv4 mDNS/IPv6
         Protocols: -DefaultRoute +LLMNR +mDNS -DNSOverTLS DNSSEC=no/unsupported
     Current DNS server: redacted-router-IPv4
     DNS Servers: redacted-router-IPv4
     DNS Domain: home
...

Last edited by Dryden (2023-11-05 19:21:57)

-thc

Member

Registered: 2017-03-15

Posts: 502

Re: [Solved] WireGuard breaks with NetworkManager, systemd-resolved

In that scenario the only IPv4 addresses that I have succeeded in pinging are the LAN address of the device broadcasting the hotspot and the external VPN server specified in the nmconnection file. I cannot ping other external addresses, including the VPN's DNS server.

This means that every packet entering the WireGuard tunnel gets dropped. Since your setup works fine in your home network, I suspect your phone's hotspot - can you test this with another phone's hotspot?

For the next few hours I am in a location where I can test behavior on a WiFi network other than my home network and my phone's hotspot. The behavior looks the same to me. As far as I can tell, the only relevant difference between this network and my phone's hotspot when running 'ip a,' 'ip route,' 'ip rule,' and 'resolvectl' is this:

Phone with WireGuard enabled:

resolvectl
...
Link 3 (wlp2s0)
    Current Scopes: LLMNR/IPv4 LLMNR/IPv6 mDNS/IPv4 mDNS/IPv6
         Protocols: -DefaultRoute +LLMNR +mDNS -DNSOverTLS DNSSEC=no/unsupported
...

 
Non-home WiFi network with WireGuard enabled:

resolvectl
...
Link 3 (wlp2s0)
    Current Scopes: DNS LLMNR/IPv4 LLMNR/IPv6 mDNS/IPv4 mDNS/IPv6
         Protocols: -DefaultRoute +LLMNR +mDNS -DNSOverTLS DNSSEC=no/unsupported
     Current DNS server: redacted-router-IPv4
     DNS Servers: redacted-router-IPv4
     DNS Domain: home
...

This is expected. The DHCP server of this network adds a "search domain" ("home") and systemd-resolved retains the DNS server for that purpose. In your case above the hotspot DHCP server doesn't set a search domain and it's DNS server gets superseded.

Dryden

Member

Registered: 2023-10-30

Posts: 14

Re: [Solved] WireGuard breaks with NetworkManager, systemd-resolved

This means that every packet entering the WireGuard tunnel gets dropped. Since your setup works fine in your home network, I suspect your phone's hotspot - can you test this with another phone's hotspot?

Thank you. I tested just now using another phone's hotspot and using someone else's residential WiFi network. Neither of these has any remarkable features. The ping behavior is the same: the hotspot is reachable in the one case, the router in the other, and the only reachable external IPv4 is the VPN server.
The only differences (aside from MAC and IP) that I notice in 'ip a,' 'ip route,' 'ip rule,' and 'resolvectl' outputs between the two phone hotspots when my VPN interface is up on my workstation are that in the second (someone else's phone hotspot) 'ip a' lists two different inet6 outputs under the WiFi interface (wlp2s0) instead of one and 'ip route' has 'metric 2600' in the first line of output rather than 'metric 600.'

-thc

Member

Registered: 2017-03-15

Posts: 502

Re: [Solved] WireGuard breaks with NetworkManager, systemd-resolved

So basically your setup only works at your home? This is really strange.

One remaining possibility is an IPv4/IPv6 dual stack problem (WireGuard endpoint as a name, IPv6/IPv4 name resolving, IPv4/IPv6 not reachable).

Dryden

Member

Registered: 2023-10-30

Posts: 14

Re: [Solved] WireGuard breaks with NetworkManager, systemd-resolved

So basically your setup only works at your home? This is really strange.

Yes. Home is the only place where I have observed the setup working as expected.

One remaining possibility is an IPv4/IPv6 dual stack problem (WireGuard endpoint as a name, IPv6/IPv4 name resolving, IPv4/IPv6 not reachable).

Thanks. Do you have an idea about how to address that? I've been very grateful for your persistence in attempting to resolve this issue.

-thc

Member

Registered: 2017-03-15

Posts: 502

Re: [Solved] WireGuard breaks with NetworkManager, systemd-resolved

Do you have an idea about how to address that?

I would like to take a look at

ip a
ip route
ip rule
resolvectl

in Scenario 1 (home without WireGuard VPN and with working WIreGuard VPN).

Dryden

Member

Registered: 2023-10-30

Posts: 14

Re: [Solved] WireGuard breaks with NetworkManager, systemd-resolved

I would like to take a look at

ip a
ip route
ip rule
resolvectl

in Scenario 1 (home without WireGuard VPN and with working WIreGuard VPN).

Home network with WireGuard VPN profile disabled:

ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host noprefixroute 
       valid_lft forever preferred_lft forever
2: enp0s31f6: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc fq_codel state DOWN group default qlen 1000
    link/ether redacted-MAC-1 brd ff:ff:ff:ff:ff:ff
3: wlp2s0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
    link/ether redacted-MAC-2 brd ff:ff:ff:ff:ff:ff
    inet redacted-LAN-IPv4-1/24 brd redacted-LAN-IPv4-2 scope global dynamic noprefixroute wlp2s0
       valid_lft 38265sec preferred_lft 38265sec
    inet6 redacted-LAN-IPv6-1/128 scope global dynamic noprefixroute 
       valid_lft 38266sec preferred_lft 38266sec
    inet6 redacted-LAN-IPv6-2/128 scope global dynamic noprefixroute 
       valid_lft 38266sec preferred_lft 38266sec
    inet6 redacted-LAN-IPv6-3/64 scope global dynamic noprefixroute 
       valid_lft 494802sec preferred_lft 494802sec
    inet6 redacted-LAN-IPv6-4/64 scope global noprefixroute 
       valid_lft forever preferred_lft forever
    inet6 redacted-LAN-IPv6-5/64 scope link noprefixroute 
       valid_lft forever preferred_lft forever

ip route
default via redacted-LAN-IPv4-3 dev wlp2s0 proto dhcp src redacted-LAN-IPv4-1 metric 600 
redacted-LAN-IPv4-4/24 dev wlp2s0 proto kernel scope link src redacted-LAN-IPv4-1 metric 600

ip rule
0:	from all lookup local
32766:	from all lookup main
32767:	from all lookup default

resolvectl
Global
         Protocols: +LLMNR +mDNS -DNSOverTLS DNSSEC=no/unsupported
  resolv.conf mode: stub

Link 2 (enp0s31f6)
    Current Scopes: none
         Protocols: -DefaultRoute +LLMNR +mDNS -DNSOverTLS DNSSEC=no/unsupported

Link 3 (wlp2s0)
    Current Scopes: DNS LLMNR/IPv4 LLMNR/IPv6 mDNS/IPv4 mDNS/IPv6
         Protocols: +DefaultRoute +LLMNR +mDNS -DNSOverTLS DNSSEC=no/unsupported
Current DNS Server: redacted-LAN-IPv4-3
       DNS Servers: redacted-LAN-IPv4-3 redacted-LAN-IPv6-6
        DNS Domain: lan

Home network with WireGuard VPN profile enabled:

ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host noprefixroute 
       valid_lft forever preferred_lft forever
2: enp0s31f6: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc fq_codel state DOWN group default qlen 1000
    link/ether redacted-MAC-1 brd ff:ff:ff:ff:ff:ff
3: wlp2s0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
    link/ether redacted-MAC-2 brd ff:ff:ff:ff:ff:ff
    inet redacted-LAN-IPv4-1/24 brd redacted-LAN-IPv4-2 scope global dynamic noprefixroute wlp2s0
       valid_lft 38222sec preferred_lft 38222sec
    inet6 redacted-LAN-IPv6-1/128 scope global dynamic noprefixroute 
       valid_lft 38223sec preferred_lft 38223sec
    inet6 redacted-LAN-IPv6-2/128 scope global dynamic noprefixroute 
       valid_lft 38223sec preferred_lft 38223sec
    inet6 redacted-LAN-IPv6-3/64 scope global dynamic noprefixroute 
       valid_lft 494759sec preferred_lft 494759sec
    inet6 redacted-LAN-IPv6-4/64 scope global noprefixroute 
       valid_lft forever preferred_lft forever
    inet6 redacted-LAN-IPv6-5/64 scope link noprefixroute 
       valid_lft forever preferred_lft forever
12: wg: <POINTOPOINT,NOARP,UP,LOWER_UP> mtu 1420 qdisc noqueue state UNKNOWN group default qlen 1000
    link/none 
    inet redacted-external-IPv4-1/32 scope global noprefixroute wg
       valid_lft forever preferred_lft forever
    inet6 redacted-external-IPv6-1/128 scope global noprefixroute 
       valid_lft forever preferred_lft forever
    inet6 redacted-external-IPv6-2/64 scope link noprefixroute 
       valid_lft forever preferred_lft forever

ip route
default via redacted-LAN-IPv4-3 dev wlp2s0 proto dhcp src redacted-LAN-IPv4-1 metric 600 
redacted-LAN-IPv4-4/24 dev wlp2s0 proto kernel scope link src redacted-LAN-IPv4-1 metric 600

ip rule
0:	from all lookup local
30814:	from all lookup main suppress_prefixlength 0
30815:	not from all fwmark 0xcb4d lookup 52045
32766:	from all lookup main
32767:	from all lookup default

resolvectl
Global
         Protocols: +LLMNR +mDNS -DNSOverTLS DNSSEC=no/unsupported
  resolv.conf mode: stub

Link 2 (enp0s31f6)
    Current Scopes: none
         Protocols: -DefaultRoute +LLMNR +mDNS -DNSOverTLS DNSSEC=no/unsupported

Link 3 (wlp2s0)
    Current Scopes: DNS LLMNR/IPv4 LLMNR/IPv6 mDNS/IPv4 mDNS/IPv6
         Protocols: -DefaultRoute +LLMNR +mDNS -DNSOverTLS DNSSEC=no/unsupported
Current DNS Server: redacted-LAN-IPv4-3
       DNS Servers: redacted-LAN-IPv4-3
        DNS Domain: lan

Link 12 (wg)
    Current Scopes: DNS
         Protocols: +DefaultRoute +LLMNR +mDNS -DNSOverTLS DNSSEC=no/unsupported
Current DNS Server: redacted-wg.nmconnection-IPv4
       DNS Servers: redacted-wg.nmconnection-IPv4
        DNS Domain: ~.

-thc

Member

Registered: 2017-03-15

Posts: 502

Re: [Solved] WireGuard breaks with NetworkManager, systemd-resolved

The most obvious difference is that your home network seems to have a working IPv6 connection.

Does the "Endpoint" field of the WireGuard "Peer" contain an IPv4 address, an IPv6 address or a name?

If it's a name - con you resolve it (with disabled VPN):

drill vpnendpoint.some.domain
drill vpnendpoint.some.domain -t AAAA

Can you explain those mysterious global IPv6 addresses with a /128 prefix? I somehow missed the one on the WireGuard interface earlier in this thread - is this address part of the original WireGuard configuration for that provider?

Dryden

Member

Registered: 2023-10-30

Posts: 14

Re: [Solved] WireGuard breaks with NetworkManager, systemd-resolved

Does the "Endpoint" field of the WireGuard "Peer" contain an IPv4 address, an IPv6 address or a name?

The 'Endpoint' field, both in '.nmconnection' and in the config file provided by the VPN that I used to configure the interface, contains: [IPv6]:port

Can you explain those mysterious global IPv6 addresses with a /128 prefix? I somehow missed the one on the WireGuard interface earlier in this thread - is this address part of the original WireGuard configuration for that provider?

The global IPv6 with '/128' on the WireGuard interface matches the IPv6 address in the 'address' field of the config file provided by the VPN that I used to configure the interface. The second IPv6 with '/128' on the WireGuard interface, the link one, does not correspond to anything in that config file.
I do not know what the global IPv6 addresses with '/128' on wlp2s0 are, but redacted-LAN-IPv6-4 is a longer variant of redacted-LAN-IPv6-1 and redacted-LAN-IPv6-3 is a longer variant of redacted-LAN-IPv6-2.

-thc

Member

Registered: 2017-03-15

Posts: 502

Re: [Solved] WireGuard breaks with NetworkManager, systemd-resolved

The 'Endpoint' field, both in '.nmconnection' and in the config file provided by the VPN that I used to configure the interface, contains: [IPv6]:port

This is
A. the reason it doesn't work with your hotspot (and possibly the other two test networks too)
B. really, really strange

This VPN configuration will only work when the physical network interface has a valid global IPv6 address (2000::/3 - at the present beginning with "2") and access to an IPv6 router.

The global IPv6 with '/128' on the WireGuard interface matches the IPv6 address in the 'address' field of the config file provided by the VPN that I used to configure the interface. The second IPv6 with '/128' on the WireGuard interface, the link one, does not correspond to anything in that config file.
I do not know what the global IPv6 addresses with '/128' on wlp2s0 are, but redacted-LAN-IPv6-4 is a longer variant of redacted-LAN-IPv6-1 and redacted-LAN-IPv6-3 is a longer variant of redacted-LAN-IPv6-2.

The second one will begin with "fe80" and is a link local address (LLA) - every IPv6 capable (non-loopback) interface has at least this one. The first one surprised me but it's possibly correct.

Dryden

Member

Registered: 2023-10-30

Posts: 14

Re: [Solved] WireGuard breaks with NetworkManager, systemd-resolved

This is
A. the reason it doesn't work with your hotspot (and possibly the other two test networks too)
B. really, really strange

This VPN configuration will only work when the physical network interface has a valid global IPv6 address (2000::/3 - at the present beginning with "2") and access to an IPv6 router.

Thank you very much for sticking with this long enough to isolate the problem. Would you be so kind as to tell me what the 'Endpoint' field should look like in order to achieve a working configuration across networks/hotspots? Should it contain a name, for example, or an IPv4 address, instead of an IPv6 address?