Arch Linux

jfk

Member

Registered: 2017-12-11

Posts: 108

[SOLVED] Wireguard stopped working after ISP switch

I use a simple wireguard config to establish a tunnel to my VPS:

[Interface]
Address = 10.0.0.5/32
PrivateKey = 
DNS = 46.28.201.21, 46.28.201.22

[Peer]
PublicKey = 
AllowedIPs = 0.0.0.0/0, ::/0
Endpoint = 
PersistentKeepalive = 25

My local interface is set with systemd-networkd (wired and static):

[Match]
Name=eno1

[Network]
Address=192.168.1.10/24
Gateway=192.168.1.1
DNS=192.168.1.1

Routing is done through wireguard-tools.

This worked fine until I changed my ISP / router. As the tunnel on the wireguard Android app has not been affected, I believe there's somehow an issue with my DNS resolution on Arch. The only change I can pinpoint (which doesn't seem relevant) is that my new ISP uses IPV6 WAN values.

I use systemd-resolved with systemd-resolvconf. When the wireguard interface is up I got:

$ resolvectl status
Global
           Protocols: +LLMNR +mDNS -DNSOverTLS DNSSEC=no/unsupported
    resolv.conf mode: stub
Fallback DNS Servers: 1.1.1.1#cloudflare-dns.com 9.9.9.9#dns.quad9.net 8.8.8.8#dns.google 2606:4700:4700::1111#cloudflare-dns.com 2620:fe::9#dns.quad9.net 2001:4860:4860::8888#dns.google

Link 2 (eno1)
    Current Scopes: DNS LLMNR/IPv4 LLMNR/IPv6
         Protocols: +DefaultRoute +LLMNR -mDNS -DNSOverTLS DNSSEC=no/unsupported
Current DNS Server: 192.168.1.1
       DNS Servers: 192.168.1.1 2001:1700:a00::14 2001:1700:a00::15

Link 5 (8700g)
    Current Scopes: DNS
         Protocols: +DefaultRoute +LLMNR +mDNS -DNSOverTLS DNSSEC=no/unsupported
Current DNS Server: 46.28.201.21
       DNS Servers: 46.28.201.21 46.28.201.22
        DNS Domain: ~.

Along with this resolv.conf:

#  This file is part of systemd.
#
#  systemd is free software; you can redistribute it and/or modify it under the
#  terms of the GNU Lesser General Public License as published by the Free
#  Software Foundation; either version 2.1 of the License, or (at your option)
#  any later version.
#
# Entries in this file show the compile time defaults. Local configuration
# should be created by either modifying this file (or a copy of it placed in
# /etc/ if the original file is shipped in /usr/), or by creating "drop-ins" in
# the /etc/systemd/resolved.conf.d/ directory. The latter is generally
# recommended. Defaults can be restored by simply deleting the main
# configuration file and all drop-ins located in /etc/.
#
# Use 'systemd-analyze cat-config systemd/resolved.conf' to display the full config.
#
# See resolved.conf(5) for details.

[Resolve]
# Some examples of DNS servers which may be used for DNS= and FallbackDNS=:
# Cloudflare: 1.1.1.1#cloudflare-dns.com 1.0.0.1#cloudflare-dns.com 2606:4700:4700::1111#cloudflare-dns.com 2606:4700:4700::1001#cloudflare-dns.com
# Google:     8.8.8.8#dns.google 8.8.4.4#dns.google 2001:4860:4860::8888#dns.google 2001:4860:4860::8844#dns.google
# Quad9:      9.9.9.9#dns.quad9.net 149.112.112.112#dns.quad9.net 2620:fe::fe#dns.quad9.net 2620:fe::9#dns.quad9.net
#FallbackDNS=
#Domains=
#DNSSEC=no
#DNSOverTLS=no
#MulticastDNS=yes
#LLMNR=yes
#Cache=yes
#CacheFromLocalhost=no
#DNSStubListener=yes
#DNSStubListenerExtra=
#ReadEtcHosts=yes
#ResolveUnicastSingleLabel=no
#StaleRetentionSec=0

Wireguard logs:

$ sudo systemctl status wg-quick@8700g
● wg-quick@8700g.service - WireGuard via wg-quick(8) for 8700g
     Loaded: loaded (/usr/lib/systemd/system/wg-quick@.service; enabled; preset: disabled)
     Active: active (exited) since Tue 2024-11-12 18:42:01 CET; 2s ago
 Invocation: f7fa6011f14b4c1c993913598c01a4cb
       Docs: man:wg-quick(8)
             man:wg(8)
             https://www.wireguard.com/
             https://www.wireguard.com/quickstart/
             https://git.zx2c4.com/wireguard-tools/about/src/man/wg-quick.8
             https://git.zx2c4.com/wireguard-tools/about/src/man/wg.8
    Process: 3172 ExecStart=/usr/bin/wg-quick up 8700g (code=exited, status=0/SUCCESS)
   Main PID: 3172 (code=exited, status=0/SUCCESS)
   Mem peak: 5.6M
        CPU: 45ms

nov 12 18:42:01 8700g wg-quick[3172]: [#] ip -6 route add ::/0 dev 8700g table 51820
nov 12 18:42:01 8700g wg-quick[3172]: [#] ip -6 rule add not fwmark 51820 table 51820
nov 12 18:42:01 8700g wg-quick[3172]: [#] ip -6 rule add table main suppress_prefixlength 0
nov 12 18:42:01 8700g wg-quick[3212]: [#] ip6tables-restore -n
nov 12 18:42:01 8700g wg-quick[3172]: [#] ip -4 route add 0.0.0.0/0 dev 8700g table 51820
nov 12 18:42:01 8700g wg-quick[3172]: [#] ip -4 rule add not fwmark 51820 table 51820
nov 12 18:42:01 8700g wg-quick[3172]: [#] ip -4 rule add table main suppress_prefixlength 0
nov 12 18:42:01 8700g wg-quick[3172]: [#] sysctl -q net.ipv4.conf.all.src_valid_mark=1
nov 12 18:42:01 8700g wg-quick[3222]: [#] iptables-restore -n
nov 12 18:42:01 8700g systemd[1]: Finished WireGuard via wg-quick(8) for 8700g.

Last edited by jfk (2024-11-14 15:03:42)

-thc

Member

Registered: 2017-03-15

Posts: 754

Re: [SOLVED] Wireguard stopped working after ISP switch

Everything looks O.K. to me - what you didn't mention is what exactly is not working.

Two things worth mentioning: Are you aware that wg-quick uses "resolvconf" to update DNS information? Since your "resolvectl" info seems correct you probably are. The WireGuard DNS servers are not public - is that part of your VPN scheme?

jfk

Member

Registered: 2017-12-11

Posts: 108

Re: [SOLVED] Wireguard stopped working after ISP switch

what you didn't mention is what exactly is not working.

The connection to the "server" works. So I deduced it was DNS-related.

Two things worth mentioning: Are you aware that wg-quick uses "resolvconf" to update DNS information? Since your "resolvectl" info seems correct you probably are. The WireGuard DNS servers are not public - is that part of your VPN scheme?

1) Yes I'm aware. But if I'm not mistaken it's done through systemd-resolvconf or openresolv, right?
2) Yes my VPS provider requires the use of those DNS. The resolution won't work with any other address.

Last edited by jfk (2024-11-12 19:11:41)

-thc

Member

Registered: 2017-03-15

Posts: 754

Re: [SOLVED] Wireguard stopped working after ISP switch

Yep - you need systemd-resolvconf for that.

When you are connected via WireGuard:

ping -c 4 8.8.8.8
ping -c 4 2001:4860:4860::8888
resolvectl query dns.google

jfk

Member

Registered: 2017-12-11

Posts: 108

Re: [SOLVED] Wireguard stopped working after ISP switch

ping is not usable within my VPS provider networking.

$ resolvectl query dns.google
dns.google: 8.8.4.4                            -- link: 8700g
            8.8.8.8                            -- link: 8700g
            2001:4860:4860::8888               -- link: 8700g
            2001:4860:4860::8844               -- link: 8700g

-- Information acquired via protocol DNS in 30.2ms.
-- Data is authenticated: no; Data was acquired via local or encrypted transport: no
-- Data from: network

-thc

Member

Registered: 2017-03-15

Posts: 754

Re: [SOLVED] Wireguard stopped working after ISP switch

So ICMP messages are filtered and DNS is working.

If you know what's not working - can you resolve the address of the resource (via "resolvectl query") that's not working.

jfk

Member

Registered: 2017-12-11

Posts: 108

Re: [SOLVED] Wireguard stopped working after ISP switch

OK looks like my IMAP et IM protocols go through. But I can't display anything web-related on my browsers, access Arch repos or SSH [edit: statement corrected below].

ufw is disabled.

Last edited by jfk (2024-11-12 21:57:13)

-thc

Member

Registered: 2017-03-15

Posts: 754

Re: [SOLVED] Wireguard stopped working after ISP switch

If you want to access a SSH host - do you use a DNS name and can you "resolvectl query" this name?

Can you "resolvectl query" the current Arch repo servers name?

jfk

Member

Registered: 2017-12-11

Posts: 108

Re: [SOLVED] Wireguard stopped working after ISP switch

If you want to access a SSH host - do you use a DNS name and can you "resolvectl query" this name?

No, I should have precised indeed: I use an IPV4 address of the form:

ssh user@ip -p

This seems to definitely exclude my intuition that DNS is somehow at stake in this issue. I didn't realise at once, sry.

Can you "resolvectl query" the current Arch repo servers name?

Actually yes, I need to correct my previous report (got a different input after reloading both systemd-networkd and wg-quick). I can reach for instance:

$ resolvectl query mirror.init7.net
mirror.init7.net: 109.202.202.202              -- link: 8700g
                  2001:1620::1620              -- link: 8700g

-- Information acquired via protocol DNS in 32.3ms.
-- Data is authenticated: no; Data was acquired via local or encrypted transport: no
-- Data from: network

pacman -Syu works but in sub-optimal speed (~5% of my bandwidth). However

yay firefox

doesn't. It's "pending" like webpages on my browser.

Last edited by jfk (2024-11-12 22:09:25)

-thc

Member

Registered: 2017-03-15

Posts: 754

Re: [SOLVED] Wireguard stopped working after ISP switch

Can we agree for now that this seems not to be DNS related?

Does everything work if you disable WireGuard?

Is the WireGuard-"server" a VPS that you control? If yes, do you run an IP filter on it? If yes, is it aware of your changed IP source addresses?

Last edited by -thc (2024-11-13 06:20:28)

jfk

Member

Registered: 2017-12-11

Posts: 108

Re: [SOLVED] Wireguard stopped working after ISP switch

Does everything work if you disable WireGuard?

Yes.

Is the WireGuard-"server" a VPS that you control? If yes, do you run an IP filter on it? If yes, is it aware of your changed IP source addresses?

Yes I am the admin. I don't run IP filter. What makes me doubt that there's something wrong on the VPS side is the fact that the .conf works when I set it through the wireguard Android app.

More info (from the VPS):

# wg show
interface: wg0
  public key: 
  private key: (hidden)
  listening port: 51820

peer: 
  endpoint: [IPV4]:63437
  allowed ips: 10.0.0.5/32
  latest handshake: 1 minute, 9 seconds ago 
  transfer: 13.21 GiB received, 501.74 GiB sent

# cat wg0.conf
[Interface]
PrivateKey = 
Address = 10.0.0.1/24
ListenPort = 51820
PostUp = iptables -A FORWARD -i wg0 -j ACCEPT; iptables -t nat -A POSTROUTING -o ens160 -j MASQUERADE
PostDown = iptables -D FORWARD -i wg0 -j ACCEPT; iptables -t nat -D POSTROUTING -o ens160 -j MASQUERADE

[Peer]
#1
PublicKey = 
AllowedIPs = 10.0.0.2/32

Etc.

Last edited by jfk (2024-11-13 11:58:49)

-thc

Member

Registered: 2017-03-15

Posts: 754

Re: [SOLVED] Wireguard stopped working after ISP switch

I use an IPV4 address of the form:

ssh user@ip -p

Try the following: Temporarily enter this IPv4 address only in the "Allowed IPs" field of your "client"-WireGuard configuration and start WireGuard.

If you can connect to the SSH-host, check there if the established connection originates from the IP of your WireGuard-"server":

 sudo ss -t -n -p

jfk

Member

Registered: 2017-12-11

Posts: 108

Re: [SOLVED] Wireguard stopped working after ISP switch

Is this what you mean?

[Interface]
Address = 10.0.0.5/32
PrivateKey = 
DNS = 46.28.201.21, 46.28.201.22

[Peer]
PublicKey = 
AllowedIPs = IP
PersistentKeepalive = 25

If yes, I got:

ssh: connect to host IP port PORT: No route to host

-thc

Member

Registered: 2017-03-15

Posts: 754

Re: [SOLVED] Wireguard stopped working after ISP switch

My mistake - in a case like this one has to allow the WireGuard endpoint explicitly:

AllowedIPs = 10.0.0.1/32, IP/32

Last edited by -thc (2024-11-13 13:22:11)

jfk

Member

Registered: 2017-12-11

Posts: 108

Re: [SOLVED] Wireguard stopped working after ISP switch

Still the same error.

$ sudo wg-quick up 8700g
[#] ip link add 8700g type wireguard
[#] wg setconf 8700g /dev/fd/63
[#] ip -4 address add 10.0.0.5/32 dev 8700g
[#] ip link set mtu 1420 up dev 8700g
[#] resolvconf -a 8700g -m 0 -x
[#] ip -4 route add IP/32 dev 8700g
[#] ip -4 route add 10.0.0.1/32 dev 8700g

ssh: connect to host IP port PORT: No route to host

-thc

Member

Registered: 2017-03-15

Posts: 754

Re: [SOLVED] Wireguard stopped working after ISP switch

Are the routes even active after connecting?

ip -4 route

jfk

Member

Registered: 2017-12-11

Posts: 108

Re: [SOLVED] Wireguard stopped working after ISP switch

$ ip -4 route
default via 192.168.1.1 dev eno1 proto static 
10.0.0.1 dev 8700g scope link 
IP dev 8700g scope link 
192.168.1.0/24 dev eno1 proto kernel scope link src 192.168.1.10 

-thc

Member

Registered: 2017-03-15

Posts: 754

Re: [SOLVED] Wireguard stopped working after ISP switch

I'm stumped. This looks like something inside your kernel/IP/netfilter-subsystem is deliberately blocking traffic to this IP - but only via WireGuard.

Does the journal offer any insights?

Last edited by -thc (2024-11-13 13:52:22)

-thc

Member

Registered: 2017-03-15

Posts: 754

Re: [SOLVED] Wireguard stopped working after ISP switch

Revisiting this thread lead to a possible explanation that would fit most of effects described here: A MTU/Path MTU problem that only arises via VPN tunnel. Does lowering the WireGuard MTU to 1400 change anything?

jfk

Member

Registered: 2017-12-11

Posts: 108

Re: [SOLVED] Wireguard stopped working after ISP switch

Revisiting this thread lead to a possible explanation that would fit most of effects described here: A MTU/Path MTU problem that only arises via VPN tunnel. Does lowering the WireGuard MTU to 1400 change anything?

Nice take, this solved the "connection" issue, thank you.

As I like to understand the root cause of such issues, why changing my ISP/router would have necessitated this modification? Do you believe the IP protocol of the router plays a role after all? Or does my previous ISP bandwidth was not high enough for me to notice this sub-optimal setting?

Last edited by jfk (2024-11-14 17:27:29)

-thc

Member

Registered: 2017-03-15

Posts: 754

Re: [SOLVED] Wireguard stopped working after ISP switch

I'm glad we finally found the culprit .

The standard MTU for Ethernet (1500 bytes/Ethernet v2) means a packet on the Ethernet wire can transport a maximum of 1500 bytes payload data - the 18 bytes for the Ethernet header are outside (1518 bytes maximum total).

This payload is an IPv4 or IPv6 packet - their headers run at 16 bytes (IPv4) or 36 bytes (IPv6) minimum but may be longer. IPv4 defines "options" and IPv6 "extension headers" that can extend the header length. Subtract the IP header and you have a new payload.

This payload is an UDP or TCP packet. Their headers are 4 bytes (UDP) or 16 bytes (TCP). Subtract that and you have the application payload.

Which in case of a VPN packet is a complete IPv4/IPv6-packet on its own - and the circus starts again...

WireGuard subtracts 80 bytes per default - which is generous considering a typical combined TCP/IPv6 header runs at 52 bytes. But since your provider and/or router may choose to add options and extensions of their own or have lower MTUs due to their transport media this may not be enough.

And also important in your case: Because ICMP is blocked, the automatic "Path MTU" discovery doesn't work.

Last edited by -thc (2024-11-14 13:22:56)

jfk

Member

Registered: 2017-12-11

Posts: 108

Re: [SOLVED] Wireguard stopped working after ISP switch

OK I see.

Thanks again for your replies.