Arch Linux

Superhansel

Member

Registered: 2018-03-10

Posts: 14

[✓] WireGuard/Network: Re: MTU and tunnel through IPv4 vs IPv6 network

I debugged the strangest thing in the last couple of hours. I managed to find the solution in the end but I don't really understand why the problem occurs in the first place. Here is quick summary:

I configured a WireGuard connection between a VPS and my Home PC like this:

VPS config:

[Interface]
PrivateKey = <Redacted>
Address = 10.78.250.1/24,fd93:8048:956a:3ffb::1/64
ListenPort = 51820

[Peer]
PublicKey = <Redacted>
AllowedIPs = 10.78.250.20/32, fd93:8048:956a:3ffb::20/128

Home PC config:

[Interface]
PrivateKey = <Redacted>
Address = 10.78.250.20/24, fd93:8048:956a:3ffb::20/64

[Peer]
Endpoint = vpn.myvps.tld:51820
PublicKey = <Redacted>
AllowedIPs = 10.78.250.1/32,fd93:8048:956a:3ffb::1/128, 1.2.3.4/32, 2001:db8::1/128

Don't mind the two placeholder IPs (1.2.3.4/32, 2001:db8::1/128). These are just placeholders for the actual IP's that  I am tunneling through the WireGuard connection.

So far - so good. Notice that I initiate the wireguard connection through my Home PC and use a DNS name for it (vpn.myvps.tld) which has both A and AAAA records set. Wireguard seems to prefer IPv6 so it always establishes the connection through the IPv6 address.

The problem:
On my home PC I wasn't able to establish HTTPS connections to the tunneled IP (2001:db8::1). I was able to ping the IP and also establish a HTTP connection (although it only contained a small payload with a Location redirect). As soon as I tried to fetch data through HTTPS, the connection seemingly "frooze" and it simply came stuck on my Home PC. I eventually ran a tcpdump on both ends (on the wg0 interface) and compared them. And the lines were perfectly equal to one another until real payload was actually transferred through the connection. On the VPS side I got these to lines:

12:20:36.908796 IP6 2001:db8::1.443 > fd93:8048:956a:3ffb::20.48920: Flags [P.], seq 1:2697, ack 518, win 21, options [nop,nop,TS val 3080899161 ecr 3600758243], length 2696
12:20:36.908815 IP6 2001:db8::1.443 > fd93:8048:956a:3ffb::20.48920: Flags [P.], seq 2697:3757, ack 518, win 21, options [nop,nop,TS val 3080899161 ecr 3600758243], length 1060

Whereas on the client side, only the second line showed up:

14:20:36.945921 IP6 2001:db8::1.443 > fd93:8048:956a:3ffb::20.48920: Flags [P.], seq 2697:3757, ack 518, win 21, options [nop,nop,TS val 3080899161 ecr 3600758243], length 1060

So a whole segment didn't arrived by the looks of it. This ultimately seems to be a fragmentation issue because...

...the solution:
I eventually realized that this seems to be caused because the VPS uses jumbo frames:

VPS interface MTU:    (9001 WAN/8921 wg0)
Home PC interface MTU (1500 LAN/1420 wg0)

Once I configures the VPS wg0 interface to use an MTU of 1420, everything just worked fine.

My question:
What I don't understand: When I established the WireGuard tunnel to my VPS using its public IPv4 address, everything works fine, dispite the VPS is configured to use jumbo frames on its WAN and wg0 interface. The problem only manifests when the WireGuard tunnel is established over an IPv6 connection. But why?

From what I understand the headers of IPv6 packets are (obviously) bigger than that of IPv4 packets which in turn could lead to performance issues because each UDP datagram (used by WireGuard) has less space to fit the actual payload, and it may could cause inefficient wrapping of the tunneled network traffic. However, why does it cause a whole TCP segment to disappear (consistently, I want to add!)? And why does it only do so when establishing the Tunnel through IPv6?

(P.S. it doesn't seem to be due to the segment length of 2696 bytes that exceeds the MTU. The IPv4 tunnel connection works perfectly fine with segments of even 5472 bytes)

I hope someone who has a better understanding of networks can help me understand

Last edited by Superhansel (2024-07-01 18:53:52)

-thc

Member

Registered: 2017-03-15

Posts: 775

Re: [✓] WireGuard/Network: Re: MTU and tunnel through IPv4 vs IPv6 network

What you seem to miss is that every hop between your PC and the VPS (try "tracepath vpn.myvps.tld") has an individual MTU value based on the underlying physical layer. The lowest value on the path ("path MTU discovery") should assure transport of packets without fragmentation.

Because the mechanisms involved are different between IPv6 and IPv6 I can only assume that at some point in the path IPv6 packets get dropped because of a failed MTU negotiation.

This even could be the last hop between a SP router and your VPS.

Last edited by -thc (2024-06-30 17:20:10)

Superhansel

Member

Registered: 2018-03-10

Posts: 14

Re: [✓] WireGuard/Network: Re: MTU and tunnel through IPv4 vs IPv6 network

Thank you! The "black hole connection" described on the linked Wikipedia page is exactly what I experienced. When I find time I'll check whether ICMP messages targeting the VPS are blocked/dropped for some reason (ICMP echo was definitely working, but others might not). I'll report my findings here.

Superhansel

Member

Registered: 2018-03-10

Posts: 14

Re: [✓] WireGuard/Network: Re: MTU and tunnel through IPv4 vs IPv6 network

It was indeed a missing firewall rule (aka security group in AWS terms) for ICMPv6 traffic. I thought I had allowed that but I didn't. Despite what the documentation says, it wasn't sufficient for me to just allow "type 2" ICMPv6 pakets. Once I allowed all of them it worked. Thanks again!