Arch Linux

andyturfer

Member

Registered: 2021-01-08

Posts: 104

Wireguard sometimes not working after rebooting

I have WireGuard running on a "server" at home. Most of the time everything works just fine. However, sometimes after rebooting, "clients" can't connect to anything or ping anything (by hostname or IP address).

When this happens (which is about 1 in every 5 reboots), if I "sudo systemctl restart NetworkManager" on the server, then everything starts working again.

I'm not running systemd-resolved or systemd-networkd.

My knowledge of networking is quite limited, not sure where to begin diagnosing this issue.

-thc

Member

Registered: 2017-03-15

Posts: 998

Re: Wireguard sometimes not working after rebooting

We can't help you with this kind of vague and (for me) contradictory information.

What is this exact nature of the "serving" machine? Why is it a WireGuard client? Why is a "server" rebooted that often?
It sounds like one of your laptops maintains an auto-start WireGuard VPN connection and serves as a router for the other laptops.

If your "knowledge of networking is quite limited" how did you manage to set up this non-basic configuration?

cryptearth

Member

Registered: 2024-02-03

Posts: 1,726

Re: Wireguard sometimes not working after rebooting

Why is a "server" rebooted that often?

off-topic:
I do update and reboot my servers at least once a month with the plan to at least check for uodates every to every other weekend.
So, from what I get from OP the local vpn server is "some machine" which, likely as you mentioned, serves as a router with an established vpn to some remote endpoint - and from that isn't any different from other systems of OP hence it's updated and reboited regular.
Or to ask the question upside-down: Why does a server has to be up 24/7? Yes there are some techniques to hot-patch even the kernel in place which reduces to needs for actual full reboots - but sometimes a full reboot proper is just required.
fun fact: Have you ever tried what happen when you shutdown a system hosted in a data center instead of rebooting it? From my experience: In the best case the systems seem to receive frequent wake/power-on signals so a server shut down would just start back up. In the worst case some power monitor immediately raises a ticket you get charged on because "a technician had to go to the machine and restart it physical after proper checking".

andyturfer

Member

Registered: 2021-01-08

Posts: 104

Re: Wireguard sometimes not working after rebooting

We can't help you with this kind of vague and (for me) contradictory information.

What is this exact nature of the "serving" machine? Why is it a WireGuard client? Why is a "server" rebooted that often?
It sounds like one of your laptops maintains an auto-start WireGuard VPN connection and serves as a router for the other laptops.

If your "knowledge of networking is quite limited" how did you manage to set up this non-basic configuration?

Hopefully this will make my question less vague:

I have a mini PC running at home. I use it to access my home network via WireGuard. It is behind a NAT. This is the WireGuard setup on the mini PC:

[andy:bastion] ~ $ sudo cat /etc/wireguard/homeserver.conf

PrivateKey = <some_private_key>
Address = 10.0.2.1/32

# DNS only works if systemd-resolved is installed and active.
# DNS=10.20.1.2

ListenPort = 51820

# This is required if running behind a NAT.
PostUp = iptables -A FORWARD -i %i -j ACCEPT; iptables -t nat -A POSTROUTING -o enp1s0 -j MASQUERADE
PostDown = iptables -D FORWARD -i %i -j ACCEPT; iptables -t nat -D POSTROUTING -o enp1s0 -j MASQUERADE


# -----------------------------------------------------------------------------------------------------
# Clients
# -----------------------------------------------------------------------------------------------------

# Laptop - 2024-09-24
# -------------------
[Peer]
PublicKey = <some_public_key>
PresharedKey = <some_preshared_key>
AllowedIPs = 10.0.2.2/32
...
... (other "clients" included here for mobile phone etc)
...

Other information that may be relevant:

[andy:bastion] ~ $ sudo systemctl status systemd-resolved.service 

○ systemd-resolved.service - Network Name Resolution
     Loaded: loaded (/usr/lib/systemd/system/systemd-resolved.service; disabled; preset: enabled)
     Active: inactive (dead)
       Docs: man:systemd-resolved.service(8)
             man:org.freedesktop.resolve1(5)
             https://systemd.io/WRITING_NETWORK_CONFIGURATION_MANAGERS
             https://systemd.io/WRITING_RESOLVER_CLIENTS

From what I can tell, I'm using NetworkManager, and shouldn't be using systemd-resolved or systemd-networked:

[andy:bastion] ~ $ sudo nmcli
enp1s0: connected to Wired connection 1
        "Realtek RTL8111/8168/8211/8411"
        ethernet (r8169), 00:CE:39:C1:6A:A3, hw, mtu 1500
        ip4 default
        inet4 10.20.1.5/24
        route4 10.20.1.0/24 metric 100
        route4 default via 10.20.1.1 metric 100
        inet6 fe80::4c4e:1a5f:d542:4909/64
        route6 fe80::/64 metric 1024

lo: connected (externally) to lo
        "lo"
        loopback (unknown), 00:00:00:00:00:00, sw, mtu 65536
        inet4 127.0.0.1/8
        inet6 ::1/128

homeserver: connected (externally) to homeserver
        "homeserver"
        wireguard, sw, mtu 1420
        inet4 10.0.2.1/32
        route4 10.0.2.2/32 metric 0

p2p-dev-wlp0s20f3: disconnected
        "p2p-dev-wlp0s20f3"
        wifi-p2p, hw

DNS configuration:
        servers: 10.20.1.2
        domains: home.lan
        interface: enp1s0
        

Use "nmcli device show" to get complete information about known devices and
"nmcli connection show" to get an overview on active connection profiles.

Consult nmcli(1) and nmcli-examples(7) manual pages for complete usage details.

My home router port-forwards UDP port 51820 to 10.20.1.5 (which is my mini PC that WireGuard is configured on).

I connect from a "client" (i.e. my laptop connected to my mobile phone's hot-spot) with:

$ sudo wg-quick up home

As mentioned in my original post, everything works fine most of the time. Because I'm running Arch Linux (without a GUI) on the mini PC, there are often updates, and I often reboot it (and yes, I understand that many folks think you shouldn't run Arch as a server, but it's just something I wanted to try out).

About 1 in every 5 times after rebooting, when I connect to my mini PC from my laptop using WireGuard, it just doesn't work (can't ping anything, either FQDNs or IP addresses). Restarting Network Manager with "sudo systemctl restart NetworkManager" resolves the issue.

I am thinking about writing a systemd unit file that restarts NetworkManager about 30 seconds after rebooting, but I feel that this approach would be "dirty". I would rather understand why it's happening and implement a proper fix.

I'm just wondering if anyone else has come across this problem or has any ideas about what is causing this and/or how this could be resolved.

-thc

Member

Registered: 2017-03-15

Posts: 998

Re: Wireguard sometimes not working after rebooting

[andy:bastion] ~ $ sudo cat /etc/wireguard/homeserver.conf

PrivateKey = <some_private_key>
Address = 10.0.2.1/32

# DNS only works if systemd-resolved is installed and active.
# DNS=10.20.1.2

ListenPort = 51820

# This is required if running behind a NAT.
PostUp = iptables -A FORWARD -i %i -j ACCEPT; iptables -t nat -A POSTROUTING -o enp1s0 -j MASQUERADE
PostDown = iptables -D FORWARD -i %i -j ACCEPT; iptables -t nat -D POSTROUTING -o enp1s0 -j MASQUERADE


# -----------------------------------------------------------------------------------------------------
# Clients
# -----------------------------------------------------------------------------------------------------

# Laptop - 2024-09-24
# -------------------
[Peer]
PublicKey = <some_public_key>
PresharedKey = <some_preshared_key>
AllowedIPs = 10.0.2.2/32
...
... (other "clients" included here for mobile phone etc)
...

That comment about the DNS entry is misleading. On a WireGuard endpoint like yours the DNS server is already present (10.20.1.2) and WireGuard doesn't need to do anything about it. On the client side it's another matter - but depends on the WireGuard connection method.

[andy:bastion] ~ $ sudo nmcli
enp1s0: connected to Wired connection 1
        "Realtek RTL8111/8168/8211/8411"
        ethernet (r8169), 00:CE:39:C1:6A:A3, hw, mtu 1500
        ip4 default
        inet4 10.20.1.5/24
        route4 10.20.1.0/24 metric 100
        route4 default via 10.20.1.1 metric 100
        inet6 fe80::4c4e:1a5f:d542:4909/64
        route6 fe80::/64 metric 1024

lo: connected (externally) to lo
        "lo"
        loopback (unknown), 00:00:00:00:00:00, sw, mtu 65536
        inet4 127.0.0.1/8
        inet6 ::1/128

homeserver: connected (externally) to homeserver
        "homeserver"
        wireguard, sw, mtu 1420
        inet4 10.0.2.1/32
        route4 10.0.2.2/32 metric 0

p2p-dev-wlp0s20f3: disconnected
        "p2p-dev-wlp0s20f3"
        wifi-p2p, hw

DNS configuration:
        servers: 10.20.1.2
        domains: home.lan
        interface: enp1s0
        

Use "nmcli device show" to get complete information about known devices and
"nmcli connection show" to get an overview on active connection profiles.

Consult nmcli(1) and nmcli-examples(7) manual pages for complete usage details.

$ sudo wg-quick up home

You may not know that NetworkManager has native, internal WireGuard support and can also manage those connections.

I assume from the configuration file on your server and the command for your client that you activate WireGuard on the server with a wg-quick systemd unit.

That's a little bit like having two different network managements at the same time. Either tell NetworkManager to ignore (wg-quick-)WireGuard

# /etc/NetworkManager/conf.d/unmanaged.conf 
[keyfile]
unmanaged-devices=interface-name:homeserver

or switch to NetworkManager WireGuard.

andyturfer

Member

Registered: 2021-01-08

Posts: 104

Re: Wireguard sometimes not working after rebooting

That's a little bit like having two different network managements at the same time. Either tell NetworkManager to ignore (wg-quick-)WireGuard

# /etc/NetworkManager/conf.d/unmanaged.conf 
[keyfile]
unmanaged-devices=interface-name:homeserver

or switch to NetworkManager WireGuard.

Thanks for this - very good to know. Unfortunately the problem still persisted after making the WireGuard connection "unmanaged". I probably misconfigured a whole bunch of things...

-thc

Member

Registered: 2017-03-15

Posts: 998

Re: Wireguard sometimes not working after rebooting

Then you have to closely monitor the WireGuard startup after every boot. Starting WireGuard is nothing more than adding a virtual network adapter - AFAIK there are no network packets involved. It should leave a hint in the logs.