Arch Linux

genji

Member

Registered: 2023-08-13

Posts: 45

Trying to create VPN gateway but traffic is being dropped

Hi everyone, I have a paid WindScribe VPN that I want to act as a gateway for other devices on my network, so all they have to do is use it as a gateway and won't need individual connections per device.

But something is wrong.  After the tunnel goes up, it exhibits very weird behavior - First of all speeds are atrocious, on Windows I'll get 500mbps and this thing seems to indicate more like 100mbps down.  But more problematic, is the gateway doesn't actually work.  Something goes wrong after it sends the first sputters of data it seems like.  A page will start loading, partially load a tiny amount of data, then freeze up and stall out.  Also tests seem to indicate I either can't upload at all, or it's being severely crippled for some reason..  If you can spot anything I'm doing wrong, I'd love the assist, thank you

Before VPN goes up:

  13/08/2023   17:29.42   /home/mobaxterm  ssh genji@192.168.0.111
X11 forwarding request failed on channel 0
Last login: Sun Aug 13 15:33:00 2023 from 192.168.0.3
[genji@gateway-wind ~]$
[genji@gateway-wind ~]$ ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host noprefixroute
       valid_lft forever preferred_lft forever
2: ens18: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UP group default qlen 1000
    link/ether 4e:ba:86:1d:c6:94 brd ff:ff:ff:ff:ff:ff
    altname enp0s18
    inet 192.168.0.111/24 brd 192.168.0.255 scope global noprefixroute ens18
       valid_lft forever preferred_lft forever
    inet6 fe80::e22b:695f:5839:fac4/64 scope link
       valid_lft forever preferred_lft forever
[genji@gateway-wind ~]$ ping google.ca
PING google.ca (142.251.41.67) 56(84) bytes of data.
64 bytes from yyz10s20-in-f3.1e100.net (142.251.41.67): icmp_seq=1 ttl=118 time=8.17 ms
64 bytes from yyz10s20-in-f3.1e100.net (142.251.41.67): icmp_seq=2 ttl=118 time=8.05 ms
64 bytes from yyz10s20-in-f3.1e100.net (142.251.41.67): icmp_seq=3 ttl=118 time=8.24 ms
64 bytes from yyz10s20-in-f3.1e100.net (142.251.41.67): icmp_seq=4 ttl=118 time=8.06 ms

--- google.ca ping statistics ---
4 packets transmitted, 4 received, 0% packet loss, time 3003ms
rtt min/avg/max/mdev = 8.049/8.128/8.235/0.077 ms
[genji@gateway-wind ~]$ speedtest-cli --secure
Retrieving speedtest.net configuration...
Testing from Datacamp Limited (178.249.214.67)...
Retrieving speedtest.net server list...
Selecting best server based on ping...
Hosted by Beanfield Metroconnect (Toronto, ON) [3.45 km]: 14.789 ms
Testing download speed................................................................................
Download: 594.17 Mbit/s
Testing upload speed......................................................................................................
Upload: 248.51 Mbit/s
[genji@gateway-wind ~]$

Everything's fine as far as I can tell at this point - the Arch installation is relatively new and doesn't have much else on it so far so no major configuration changes or anything

Now once I activate my VPN:

[root@gateway-wind ~]# wg-quick up wg0
[#] ip link add wg0 type wireguard
[#] wg setconf wg0 /dev/fd/63
[#] ip -4 address add 100.74.132.217/32 dev wg0
[#] ip link set mtu 1420 up dev wg0
[#] wg set wg0 fwmark 51820
[#] ip -6 route add ::/0 dev wg0 table 51820
[#] ip -6 rule add not fwmark 51820 table 51820
[#] ip -6 rule add table main suppress_prefixlength 0
[#] nft -f /dev/fd/63
[#] ip -4 route add 0.0.0.0/0 dev wg0 table 51820
[#] ip -4 rule add not fwmark 51820 table 51820
[#] ip -4 rule add table main suppress_prefixlength 0
[#] sysctl -q net.ipv4.conf.all.src_valid_mark=1
[#] nft -f /dev/fd/63
[root@gateway-wind ~]# ping google.ca
PING google.ca (142.251.41.67) 56(84) bytes of data.
64 bytes from yyz10s20-in-f3.1e100.net (142.251.41.67): icmp_seq=1 ttl=121 time=8.67 ms
64 bytes from yyz10s20-in-f3.1e100.net (142.251.41.67): icmp_seq=2 ttl=121 time=8.66 ms
64 bytes from yyz10s20-in-f3.1e100.net (142.251.41.67): icmp_seq=3 ttl=121 time=8.42 ms

--- google.ca ping statistics ---
3 packets transmitted, 3 received, 0% packet loss, time 2004ms
rtt min/avg/max/mdev = 8.424/8.584/8.671/0.113 ms
[root@gateway-wind ~]# curl icanhazip.com
204.187.100.198
[root@gateway-wind ~]# sysctl -a |grep forward
net.ipv4.conf.all.bc_forwarding = 0
net.ipv4.conf.all.forwarding = 1
net.ipv4.conf.all.mc_forwarding = 0
net.ipv4.conf.default.bc_forwarding = 0
net.ipv4.conf.default.forwarding = 1
net.ipv4.conf.default.mc_forwarding = 0
net.ipv4.conf.ens18.bc_forwarding = 0
net.ipv4.conf.ens18.forwarding = 1
net.ipv4.conf.ens18.mc_forwarding = 0
net.ipv4.conf.lo.bc_forwarding = 0
net.ipv4.conf.lo.forwarding = 1
net.ipv4.conf.lo.mc_forwarding = 0
net.ipv4.conf.wg0.bc_forwarding = 0
net.ipv4.conf.wg0.forwarding = 1
net.ipv4.conf.wg0.mc_forwarding = 0
net.ipv4.ip_forward = 1
net.ipv4.ip_forward_update_priority = 1
net.ipv4.ip_forward_use_pmtu = 0
net.ipv6.conf.all.forwarding = 0
net.ipv6.conf.all.mc_forwarding = 0
net.ipv6.conf.default.forwarding = 0
net.ipv6.conf.default.mc_forwarding = 0
net.ipv6.conf.ens18.forwarding = 0
net.ipv6.conf.ens18.mc_forwarding = 0
net.ipv6.conf.lo.forwarding = 0
net.ipv6.conf.lo.mc_forwarding = 0
net.ipv6.conf.wg0.forwarding = 0
net.ipv6.conf.wg0.mc_forwarding = 0
[root@gateway-wind ~]# speedtest-cli --secure
Retrieving speedtest.net configuration...
Testing from Amanah Tech (204.187.100.198)...
Retrieving speedtest.net server list...
Selecting best server based on ping...
Hosted by Primus (Toronto, ON) [15.06 km]: 11.622 ms
Testing download speed................................................................................
Download: 129.67 Mbit/s
Testing upload speed......................................................................................................
Upload: 0.23 Mbit/s
[root@gateway-wind ~]#

As you can see, up until I begin the actual speedtest, it looks promising at first..  Traffic is passing through.  curl-ing an ip check website shows I'm using WindScribe (that's not my IP).  ping results are great, sub 10 ms.  Yet something is wrong beneath the surface..  The speedtest-cli dots stop mid-way through the check and reports download speed as only 129Mb/s, which I suspect is because the throughput actually started off at high speed but then broke mid-way through resulting in a low result.  And the upload is clearly completely broken 100% here..

And putting any device to use it as a gateway doesn't work either, it exhibits similar behavior - pages start to load then stop mid-way through

A real head scratcher, what did I do wrong here?  Thank you Arch forum

edit: just in case it's asked, here's the wg0.conf that was generated by WindScribe

[root@gateway-wind wireguard]# cat wg0.conf
[Interface]
PrivateKey = <censored>
Address = 100.74.132.217/32

[Peer]
PublicKey = <censored>
AllowedIPs = 0.0.0.0/0, ::/0
Endpoint = yyz-72-wg.whiskergalaxy.com:1194
PresharedKey = <censored>
PersistentKeepalive = 25

I added PersistentKeepalive = 25 to see if it would fix the problem after trying my best to Google things, but it had zero effect

Last edited by genji (2023-08-13 22:21:45)

2ManyDogs

Forum Fellow

Registered: 2012-01-15

Posts: 4,645

Re: Trying to create VPN gateway but traffic is being dropped

Please edit your post and use [ code ] tags (not quote tags) when posting output. This makes the output easier to read and provides a scroll box for long output.

https://wiki.archlinux.org/title/Genera … s_and_code
https://bbs.archlinux.org/help.php#bbcode

-thc

Member

Registered: 2017-03-15

Posts: 516

Re: Trying to create VPN gateway but traffic is being dropped

Your IP config and pure IPv4 forwarding suggest a pure IPv4 setup. Consider changing wg0.conf accordingly:

AllowedIPs = 0.0.0.0/0

Your problem sounds MTU related to me - can you test the following setting in the interface section of wg0.conf:

MTU = 1280

genji

Member

Registered: 2023-08-13

Posts: 45

Re: Trying to create VPN gateway but traffic is being dropped

Thanks so much for your reply! I appreciate it very much and am pleased to report your MTU suggestion DID seem to have fix the download/upload problem - once it was lowered, indeed, the upload finally started working and relatively fast I suppose, download seems decent too, which is great news and did work for me!  I also found this related reddit thread saying this fix was required for many people as well after a random person offered it as a solution: https://www.reddit.com/r/WireGuard/comm … questions/

I also was coming to report, before even reading your reply at all, that in the meantime I had tried OpenVPN instead of WireGuard and it TOO seemed to have been working with no problems! 

Maybe this means OpenVPN is more forgiving or does some kind of MTU adjustment automatically that WireGuard isn't?

Anyhow, after I tried your solution, I am now back to using WireGuard with the MTU = 1280 line set.

However - the gateway aspect sadly still isn't working for other computers on my network.  I feel it must be close and I'm missing something obvious or have something backwards!  The devices that attempt to use the PC as a gateway can reach everything on the local network with no problems, any 192.168.0.x device via web browser and ping and ssh all 100% good to go - there's just no outbound traffic permitted by the looks of it now.

So, I visited the Arch Wiki and went back to re-read this section: https://wiki.archlinux.org/title/Internet_sharing

and I think it comes down to maybe section 2.3, enabling NAT - I don't think I've figured out the right combination of commands.

I tried the Wiki's instructions for nft because people told me before nft is better to learn if you don't know anything yet:

# cat /proc/sys/net/ipv4/ip_forward
1
# nft add table inet nat
# nft add chain inet nat postrouting '{ type nat hook postrouting priority 100 ; }'
# nft add rule inet nat postrouting oifname ens18 masquerade

But this results in no change - any device who's using the host as a gateway still can only access 192.168.0.x devices (and with no issues)

I rebooted and thought maybe ens18 was supposed to be wg0:

[root@gateway-wind ~]# wg-quick up wg0
[#] ip link add wg0 type wireguard
[#] wg setconf wg0 /dev/fd/63
[#] ip -4 address add 100.70.254.9/32 dev wg0
[#] ip link set mtu 1350 up dev wg0
[#] wg set wg0 fwmark 51820
[#] ip -4 route add 0.0.0.0/0 dev wg0 table 51820
[#] ip -4 rule add not fwmark 51820 table 51820
[#] ip -4 rule add table main suppress_prefixlength 0
[#] sysctl -q net.ipv4.conf.all.src_valid_mark=1
[#] nft -f /dev/fd/63
[root@gateway-wind ~]# speedtest --secure
Retrieving speedtest.net configuration...
Testing from Unknown (149.57.28.169)...
Retrieving speedtest.net server list...
Selecting best server based on ping...
Hosted by Netcrawler.ca Internet (Ajax, ON) [529.78 km]: 11.02 ms
Testing download speed................................................................................
Download: 410.62 Mbit/s
Testing upload speed......................................................................................................
Upload: 195.34 Mbit/s
[root@gateway-wind ~]# nft add table inet nat
[root@gateway-wind ~]# nft add chain inet nat postrouting '{ type nat hook postrouting priority 100 ; }'
[root@gateway-wind ~]# nft add rule inet nat postrouting oifname wg0 masquerade

And no change - the bugger has no access to the general internet still - devices using the host as a gateway can still access only 192.168.0.x devices

It must be something super obvious and it's extremely close, just need public internet access now, not just 192.168.0.x

Thanks again for your response, THC

-thc

Member

Registered: 2017-03-15

Posts: 516

Re: Trying to create VPN gateway but traffic is being dropped

Maybe this means OpenVPN is more forgiving or does some kind of MTU adjustment automatically that WireGuard isn't?

The MTU issue is more related to the underlying connection (your internet connection). If this connection has it's own small overhead, the default 80 bytes for wireguard (1500 minus 1420) are possibly not enough. You can try reducing the default 1420 by 8 bytes steps until you reach the full speed - or you can keep 1280 . OpenVPN is not immune to this - I had to reduce the default OpenVPN MTU one time to get it working.

However - the gateway aspect sadly still isn't working for other computers on my network.  I feel it must be close and I'm missing something obvious or have something backwards!

You need two things (on your router/gateway) for this to work: Packet forwarding and NAT. While packet forwarding via sysctl

net.ipv4.conf.all.forwarding = 1

is "directionless" - NAT is not. You have to masquerade all packets coming in from ens18 and leaving via wg0. You do this via

nft add rule inet nat postrouting oifname wg0 masquerade

and not masquerading packets leaving ens18 like you did. Be aware that your MTU issue may "resurface" on your clients.

genji

Member

Registered: 2023-08-13

Posts: 45

Re: Trying to create VPN gateway but traffic is being dropped

You have to masquerade all packets coming in from ens18 and leaving via wg0. You do this via

nft add rule inet nat postrouting oifname wg0 masquerade

and not masquerading packets leaving ens18 like you did. Be aware that your MTU issue may "resurface" on your clients.

Thanks so much for responding again.  I do believe I tried this, though, it was the last line in my code blocks in the post above, and on my last attempt I arrived at that conclusion as something to just "try" automatically and it does not resolve the problem - still only 192.168.0.x devices are reachable, nothing on the internet.  I can't even ping 4.2.2.1 for example

Last edited by genji (2023-08-15 16:25:41)

-thc

Member

Registered: 2017-03-15

Posts: 516

Re: Trying to create VPN gateway but traffic is being dropped

I somehow missed that. Now that your router box seems to have the correct configuration - are the clients configured correctly (gateway, DNS)?

Please post the output of

nft list table inet nat

of your router box.