[SOLVED] IPv6 connectivity issues behind Arch router with WG tunnel

avs · 2024-04-19 21:26:11

Dear Community,

I have a home network gateway router on Arch with IPv4-only provider and Wireguard tunnel to IPv6 broker.
And there is Arch host behind the router.
This configuration exists for some years, but with time I found, what some sites do not work well from behind the gateway.
The most prominent examples are t.me, home.netatmo.com, while they do work via IPv4.
On the other side, google.com works well with both IPv4 and IPv6.

Here is an example of failure for t.me from host behind the router

[user@host-behind-router ~]$ curl -6 https://t.me -v
* Host t.me:443 was resolved.
* IPv6: 2001:67c:4e8:f004::9
* IPv4: (none)
*   Trying [2001:67c:4e8:f004::9]:443...
* Connected to t.me (2001:67c:4e8:f004::9) port 443
* ALPN: curl offers h2,http/1.1
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
*  CAfile: /etc/ssl/certs/ca-certificates.crt
*  CApath: none
* Recv failure: Connection reset by peer
* OpenSSL SSL_connect: Connection reset by peer in connection to t.me:443
* Closing connection
curl: (35) Recv failure: Connection reset by peer

And successful connection from router itself:

[user@router ~]# curl -6 https://t.me -v
* Host t.me:443 was resolved.
* IPv6: 2001:67c:4e8:f004::9
* IPv4: (none)
*   Trying [2001:67c:4e8:f004::9]:443...
* Connected to t.me (2001:67c:4e8:f004::9) port 443
* ALPN: curl offers h2,http/1.1
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
*  CAfile: /etc/ssl/certs/ca-certificates.crt
*  CApath: none
* TLSv1.3 (IN), TLS handshake, Server hello (2):
* TLSv1.3 (IN), TLS handshake, Encrypted Extensions (8):
* TLSv1.3 (IN), TLS handshake, Certificate (11):
* TLSv1.3 (IN), TLS handshake, CERT verify (15):
* TLSv1.3 (IN), TLS handshake, Finished (20):
* TLSv1.3 (OUT), TLS change cipher, Change cipher spec (1):
* TLSv1.3 (OUT), TLS handshake, Finished (20):
* SSL connection using TLSv1.3 / TLS_AES_256_GCM_SHA384 / x25519 / RSASSA-PSS
* ALPN: server accepted h2
* Server certificate:
*  subject: CN=*.t.me
*  start date: Oct  6 19:50:31 2023 GMT
*  expire date: Nov  6 19:50:31 2024 GMT
*  subjectAltName: host "t.me" matched cert's "t.me"
*  issuer: C=US; ST=Arizona; L=Scottsdale; O=GoDaddy.com, Inc.; OU=http://certs.godaddy.com/repository/; CN=Go Daddy Secure Certificate Authority - G2
*  SSL certificate verify ok.
*   Certificate level 0: Public key type RSA (2048/112 Bits/secBits), signed using sha256WithRSAEncryption
*   Certificate level 1: Public key type RSA (2048/112 Bits/secBits), signed using sha256WithRSAEncryption
*   Certificate level 2: Public key type RSA (2048/112 Bits/secBits), signed using sha256WithRSAEncryption
* using HTTP/2
* [HTTP/2] [1] OPENED stream for https://t.me/
* [HTTP/2] [1] [:method: GET]
* [HTTP/2] [1] [:scheme: https]
* [HTTP/2] [1] [:authority: t.me]
* [HTTP/2] [1] [:path: /]
* [HTTP/2] [1] [user-agent: curl/8.7.1]
* [HTTP/2] [1] [accept: */*]
> GET / HTTP/2
> Host: t.me
> User-Agent: curl/8.7.1
> Accept: */*
>
* Request completely sent off
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
* old SSL session ID is stale, removing
< HTTP/2 302
< server: nginx/1.18.0
< date: Fri, 19 Apr 2024 20:07:44 GMT
< content-type: text/html; charset=UTF-8
< content-length: 0
< set-cookie: stel_ssid=1c43a4bb5297e8b32e_18041809465154706482; expires=Sat, 20 Apr 2024 20:07:44 GMT; path=/; samesite=None; secure; HttpOnly
< pragma: no-cache
< cache-control: no-store
< location: //telegram.org/
< strict-transport-security: max-age=35768000
<
* Connection #0 to host t.me left intact

IPv4 works well on the host and router.
Here is an example from the host:

[user@host ~]$ curl -4 https://t.me -v
* Host t.me:443 was resolved.
* IPv6: (none)
* IPv4: 149.154.167.99
*   Trying 149.154.167.99:443...
* Connected to t.me (149.154.167.99) port 443
* ALPN: curl offers h2,http/1.1
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
*  CAfile: /etc/ssl/certs/ca-certificates.crt
*  CApath: none
* TLSv1.3 (IN), TLS handshake, Server hello (2):
* TLSv1.3 (IN), TLS handshake, Encrypted Extensions (8):
* TLSv1.3 (IN), TLS handshake, Certificate (11):
* TLSv1.3 (IN), TLS handshake, CERT verify (15):
* TLSv1.3 (IN), TLS handshake, Finished (20):
* TLSv1.3 (OUT), TLS change cipher, Change cipher spec (1):
* TLSv1.3 (OUT), TLS handshake, Finished (20):
* SSL connection using TLSv1.3 / TLS_AES_256_GCM_SHA384 / x25519 / RSASSA-PSS
* ALPN: server accepted h2
* Server certificate:
*  subject: CN=*.t.me
*  start date: Oct  6 19:50:31 2023 GMT
*  expire date: Nov  6 19:50:31 2024 GMT
*  subjectAltName: host "t.me" matched cert's "t.me"
*  issuer: C=US; ST=Arizona; L=Scottsdale; O=GoDaddy.com, Inc.; OU=http://certs.godaddy.com/repository/; CN=Go Daddy Secure Certificate Authority - G2
*  SSL certificate verify ok.
*   Certificate level 0: Public key type RSA (2048/112 Bits/secBits), signed using sha256WithRSAEncryption
*   Certificate level 1: Public key type RSA (2048/112 Bits/secBits), signed using sha256WithRSAEncryption
*   Certificate level 2: Public key type RSA (2048/112 Bits/secBits), signed using sha256WithRSAEncryption
* using HTTP/2
* [HTTP/2] [1] OPENED stream for https://t.me/
* [HTTP/2] [1] [:method: GET]
* [HTTP/2] [1] [:scheme: https]
* [HTTP/2] [1] [:authority: t.me]
* [HTTP/2] [1] [:path: /]
* [HTTP/2] [1] [user-agent: curl/8.7.1]
* [HTTP/2] [1] [accept: */*]
> GET / HTTP/2
> Host: t.me
> User-Agent: curl/8.7.1
> Accept: */*
>
* Request completely sent off
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
* old SSL session ID is stale, removing
< HTTP/2 302
< server: nginx/1.18.0
< date: Fri, 19 Apr 2024 20:10:42 GMT
< content-type: text/html; charset=UTF-8
< content-length: 0
< set-cookie: stel_ssid=6ea5d94a1c3d3a27a0_3931572385564620564; expires=Sat, 20 Apr 2024 20:10:42 GMT; path=/; samesite=None; secure; HttpOnly
< pragma: no-cache
< cache-control: no-store
< location: //telegram.org/
< strict-transport-security: max-age=35768000
<
* Connection #0 to host t.me left intact

Unfortunately, I can not find good explanation for such behaviour.
IPv6 ICMP is not restricted to router nor to host.

tcpdump on IPv6 interface of the router shows following:
- for successful connection from router

No.     Time           Source                Destination           Protocol Length Info
      1 0.000000       router::1      2001:67c:4e8:f004::9  TCP      80     47832 → 443 [SYN] Seq=0 Win=32640 Len=0 MSS=1360 SACK_PERM TSval=441334520 TSecr=0 WS=512

Frame 1: 80 bytes on wire (640 bits), 80 bytes captured (640 bits)
Raw packet data
Internet Protocol Version 6, Src: router::1, Dst: 2001:67c:4e8:f004::9
Transmission Control Protocol, Src Port: 47832, Dst Port: 443, Seq: 0, Len: 0

No.     Time           Source                Destination           Protocol Length Info
      2 0.077340       2001:67c:4e8:f004::9  router::1      TCP      80     443 → 47832 [SYN, ACK] Seq=0 Ack=1 Win=28560 Len=0 MSS=1440 SACK_PERM TSval=3147458445 TSecr=441334520 WS=1024

Frame 2: 80 bytes on wire (640 bits), 80 bytes captured (640 bits)
Raw packet data
Internet Protocol Version 6, Src: 2001:67c:4e8:f004::9, Dst: router::1
Transmission Control Protocol, Src Port: 443, Dst Port: 47832, Seq: 0, Ack: 1, Len: 0

No.     Time           Source                Destination           Protocol Length Info
      3 0.077378       router::1      2001:67c:4e8:f004::9  TCP      72     47832 → 443 [ACK] Seq=1 Ack=1 Win=32768 Len=0 TSval=441334597 TSecr=3147458445

Frame 3: 72 bytes on wire (576 bits), 72 bytes captured (576 bits)
Raw packet data
Internet Protocol Version 6, Src: router::1, Dst: 2001:67c:4e8:f004::9
Transmission Control Protocol, Src Port: 47832, Dst Port: 443, Seq: 1, Ack: 1, Len: 0

No.     Time           Source                Destination           Protocol Length Info
      4 0.079584       router::1      2001:67c:4e8:f004::9  TLSv1.3  589    Client Hello (SNI=t.me)

Frame 4: 589 bytes on wire (4712 bits), 589 bytes captured (4712 bits)
Raw packet data
Internet Protocol Version 6, Src: router::1, Dst: 2001:67c:4e8:f004::9
Transmission Control Protocol, Src Port: 47832, Dst Port: 443, Seq: 1, Ack: 1, Len: 517
Transport Layer Security

No.     Time           Source                Destination           Protocol Length Info
      5 0.116248       2001:67c:4e8:f004::9  router::1      TCP      72     443 → 47832 [ACK] Seq=1 Ack=518 Win=29696 Len=0 TSval=3147458466 TSecr=441334600

Frame 5: 72 bytes on wire (576 bits), 72 bytes captured (576 bits)
Raw packet data
Internet Protocol Version 6, Src: 2001:67c:4e8:f004::9, Dst: router::1
Transmission Control Protocol, Src Port: 443, Dst Port: 47832, Seq: 1, Ack: 518, Len: 0

No.     Time           Source                Destination           Protocol Length Info
      6 0.116275       2001:67c:4e8:f004::9  router::1      TLSv1.3  4168   Server Hello, Change Cipher Spec, Application Data

Frame 6: 4168 bytes on wire (33344 bits), 4168 bytes captured (33344 bits)
Raw packet data
Internet Protocol Version 6, Src: 2001:67c:4e8:f004::9, Dst: router::1
Transmission Control Protocol, Src Port: 443, Dst Port: 47832, Seq: 1, Ack: 518, Len: 4096
Transport Layer Security

No.     Time           Source                Destination           Protocol Length Info
      7 0.116290       router::1      2001:67c:4e8:f004::9  TCP      72     47832 → 443 [ACK] Seq=518 Ack=4097 Win=32256 Len=0 TSval=441334636 TSecr=3147458466

Frame 7: 72 bytes on wire (576 bits), 72 bytes captured (576 bits)
Raw packet data
Internet Protocol Version 6, Src: router::1, Dst: 2001:67c:4e8:f004::9
Transmission Control Protocol, Src Port: 47832, Dst Port: 443, Seq: 518, Ack: 4097, Len: 0

No.     Time           Source                Destination           Protocol Length Info
      8 0.118207       2001:67c:4e8:f004::9  router::1      TLSv1.3  1636   Application Data, Application Data, Application Data

Frame 8: 1636 bytes on wire (13088 bits), 1636 bytes captured (13088 bits)
Raw packet data
Internet Protocol Version 6, Src: 2001:67c:4e8:f004::9, Dst: router::1
Transmission Control Protocol, Src Port: 443, Dst Port: 47832, Seq: 4097, Ack: 518, Len: 1564
[2 Reassembled TCP Segments (5126 bytes): #6(3922), #8(1204)]
Transport Layer Security
Transport Layer Security

No.     Time           Source                Destination           Protocol Length Info
      9 0.118243       router::1      2001:67c:4e8:f004::9  TCP      72     47832 → 443 [ACK] Seq=518 Ack=5661 Win=32256 Len=0 TSval=441334638 TSecr=3147458467

Frame 9: 72 bytes on wire (576 bits), 72 bytes captured (576 bits)
Raw packet data
Internet Protocol Version 6, Src: router::1, Dst: 2001:67c:4e8:f004::9
Transmission Control Protocol, Src Port: 47832, Dst Port: 443, Seq: 518, Ack: 5661, Len: 0

No.     Time           Source                Destination           Protocol Length Info
     10 0.119851       router::1      2001:67c:4e8:f004::9  TLSv1.3  152    Change Cipher Spec, Application Data

Frame 10: 152 bytes on wire (1216 bits), 152 bytes captured (1216 bits)
Raw packet data
Internet Protocol Version 6, Src: router::1, Dst: 2001:67c:4e8:f004::9
Transmission Control Protocol, Src Port: 47832, Dst Port: 443, Seq: 518, Ack: 5661, Len: 80
Transport Layer Security

No.     Time           Source                Destination           Protocol Length Info
     11 0.120193       router::1      2001:67c:4e8:f004::9  TLSv1.3  158    Application Data

Frame 11: 158 bytes on wire (1264 bits), 158 bytes captured (1264 bits)
Raw packet data
Internet Protocol Version 6, Src: router::1, Dst: 2001:67c:4e8:f004::9
Transmission Control Protocol, Src Port: 47832, Dst Port: 443, Seq: 598, Ack: 5661, Len: 86
Transport Layer Security

No.     Time           Source                Destination           Protocol Length Info
     12 0.120341       router::1      2001:67c:4e8:f004::9  TLSv1.3  126    Application Data

Frame 12: 126 bytes on wire (1008 bits), 126 bytes captured (1008 bits)
Raw packet data
Internet Protocol Version 6, Src: router::1, Dst: 2001:67c:4e8:f004::9
Transmission Control Protocol, Src Port: 47832, Dst Port: 443, Seq: 684, Ack: 5661, Len: 54
Transport Layer Security

No.     Time           Source                Destination           Protocol Length Info
     13 0.156164       2001:67c:4e8:f004::9  router::1      TCP      72     443 → 47832 [ACK] Seq=5661 Ack=738 Win=29696 Len=0 TSval=3147458476 TSecr=441334640

Frame 13: 72 bytes on wire (576 bits), 72 bytes captured (576 bits)
Raw packet data
Internet Protocol Version 6, Src: 2001:67c:4e8:f004::9, Dst: router::1
Transmission Control Protocol, Src Port: 443, Dst Port: 47832, Seq: 5661, Ack: 738, Len: 0

No.     Time           Source                Destination           Protocol Length Info
     14 0.156164       2001:67c:4e8:f004::9  router::1      TLSv1.3  343    Application Data

Frame 14: 343 bytes on wire (2744 bits), 343 bytes captured (2744 bits)
Raw packet data
Internet Protocol Version 6, Src: 2001:67c:4e8:f004::9, Dst: router::1
Transmission Control Protocol, Src Port: 443, Dst Port: 47832, Seq: 5661, Ack: 738, Len: 271
Transport Layer Security

No.     Time           Source                Destination           Protocol Length Info
     15 0.156198       2001:67c:4e8:f004::9  router::1      TLSv1.3  343    Application Data

Frame 15: 343 bytes on wire (2744 bits), 343 bytes captured (2744 bits)
Raw packet data
Internet Protocol Version 6, Src: 2001:67c:4e8:f004::9, Dst: router::1
Transmission Control Protocol, Src Port: 443, Dst Port: 47832, Seq: 5932, Ack: 738, Len: 271
Transport Layer Security

No.     Time           Source                Destination           Protocol Length Info
     16 0.156198       2001:67c:4e8:f004::9  router::1      TLSv1.3  143    Application Data

Frame 16: 143 bytes on wire (1144 bits), 143 bytes captured (1144 bits)
Raw packet data
Internet Protocol Version 6, Src: 2001:67c:4e8:f004::9, Dst: router::1
Transmission Control Protocol, Src Port: 443, Dst Port: 47832, Seq: 6203, Ack: 738, Len: 71
Transport Layer Security

No.     Time           Source                Destination           Protocol Length Info
     17 0.156490       router::1      2001:67c:4e8:f004::9  TCP      72     47832 → 443 [ACK] Seq=738 Ack=6274 Win=32256 Len=0 TSval=441334676 TSecr=3147458476

Frame 17: 72 bytes on wire (576 bits), 72 bytes captured (576 bits)
Raw packet data
Internet Protocol Version 6, Src: router::1, Dst: 2001:67c:4e8:f004::9
Transmission Control Protocol, Src Port: 47832, Dst Port: 443, Seq: 738, Ack: 6274, Len: 0

No.     Time           Source                Destination           Protocol Length Info
     18 0.156555       router::1      2001:67c:4e8:f004::9  TLSv1.3  103    Application Data

Frame 18: 103 bytes on wire (824 bits), 103 bytes captured (824 bits)
Raw packet data
Internet Protocol Version 6, Src: router::1, Dst: 2001:67c:4e8:f004::9
Transmission Control Protocol, Src Port: 47832, Dst Port: 443, Seq: 738, Ack: 6274, Len: 31
Transport Layer Security

No.     Time           Source                Destination           Protocol Length Info
     19 0.162519       2001:67c:4e8:f004::9  router::1      TLSv1.3  353    Application Data

Frame 19: 353 bytes on wire (2824 bits), 353 bytes captured (2824 bits)
Raw packet data
Internet Protocol Version 6, Src: 2001:67c:4e8:f004::9, Dst: router::1
Transmission Control Protocol, Src Port: 443, Dst Port: 47832, Seq: 6274, Ack: 738, Len: 281
Transport Layer Security

No.     Time           Source                Destination           Protocol Length Info
     20 0.163073       router::1      2001:67c:4e8:f004::9  TLSv1.3  96     Application Data

Frame 20: 96 bytes on wire (768 bits), 96 bytes captured (768 bits)
Raw packet data
Internet Protocol Version 6, Src: router::1, Dst: 2001:67c:4e8:f004::9
Transmission Control Protocol, Src Port: 47832, Dst Port: 443, Seq: 769, Ack: 6555, Len: 24
Transport Layer Security

No.     Time           Source                Destination           Protocol Length Info
     21 0.164160       router::1      2001:67c:4e8:f004::9  TCP      72     47832 → 443 [FIN, ACK] Seq=793 Ack=6555 Win=32256 Len=0 TSval=441334684 TSecr=3147458478

Frame 21: 72 bytes on wire (576 bits), 72 bytes captured (576 bits)
Raw packet data
Internet Protocol Version 6, Src: router::1, Dst: 2001:67c:4e8:f004::9
Transmission Control Protocol, Src Port: 47832, Dst Port: 443, Seq: 793, Ack: 6555, Len: 0

No.     Time           Source                Destination           Protocol Length Info
     22 0.196372       2001:67c:4e8:f004::9  router::1      TCP      72     443 → 47832 [ACK] Seq=6555 Ack=794 Win=29696 Len=0 TSval=3147458486 TSecr=441334677

Frame 22: 72 bytes on wire (576 bits), 72 bytes captured (576 bits)
Raw packet data
Internet Protocol Version 6, Src: 2001:67c:4e8:f004::9, Dst: router::1
Transmission Control Protocol, Src Port: 443, Dst Port: 47832, Seq: 6555, Ack: 794, Len: 0

No.     Time           Source                Destination           Protocol Length Info
     23 0.196433       2001:67c:4e8:f004::9  router::1      TCP      72     443 → 47832 [FIN, ACK] Seq=6555 Ack=794 Win=29696 Len=0 TSval=3147458486 TSecr=441334677

Frame 23: 72 bytes on wire (576 bits), 72 bytes captured (576 bits)
Raw packet data
Internet Protocol Version 6, Src: 2001:67c:4e8:f004::9, Dst: router::1
Transmission Control Protocol, Src Port: 443, Dst Port: 47832, Seq: 6555, Ack: 794, Len: 0

No.     Time           Source                Destination           Protocol Length Info
     24 0.196449       router::1      2001:67c:4e8:f004::9  TCP      72     47832 → 443 [ACK] Seq=794 Ack=6556 Win=32256 Len=0 TSval=441334716 TSecr=3147458486

Frame 24: 72 bytes on wire (576 bits), 72 bytes captured (576 bits)
Raw packet data
Internet Protocol Version 6, Src: router::1, Dst: 2001:67c:4e8:f004::9
Transmission Control Protocol, Src Port: 47832, Dst Port: 443, Seq: 794, Ack: 6556, Len: 0

and for failed connection from host

No.     Time           Source                Destination           Protocol Length Info
      1 0.000000       host::2    2001:67c:4e8:f004::9  TCP      80     32954 → 443 [SYN] Seq=0 Win=26820 Len=0 MSS=8940 SACK_PERM TSval=4014101112 TSecr=0 WS=128

Frame 1: 80 bytes on wire (640 bits), 80 bytes captured (640 bits)
Raw packet data
Internet Protocol Version 6, Src: host::2, Dst: 2001:67c:4e8:f004::9
Transmission Control Protocol, Src Port: 32954, Dst Port: 443, Seq: 0, Len: 0

No.     Time           Source                Destination           Protocol Length Info
      2 0.045358       2001:67c:4e8:f004::9  host::2    TCP      80     443 → 32954 [SYN, ACK] Seq=0 Ack=1 Win=28560 Len=0 MSS=1440 SACK_PERM TSval=3941946430 TSecr=4014101112 WS=1024

Frame 2: 80 bytes on wire (640 bits), 80 bytes captured (640 bits)
Raw packet data
Internet Protocol Version 6, Src: 2001:67c:4e8:f004::9, Dst: host::2
Transmission Control Protocol, Src Port: 443, Dst Port: 32954, Seq: 0, Ack: 1, Len: 0

No.     Time           Source                Destination           Protocol Length Info
      3 0.045803       host::2    2001:67c:4e8:f004::9  TCP      72     32954 → 443 [ACK] Seq=1 Ack=1 Win=26880 Len=0 TSval=4014101158 TSecr=3941946430

Frame 3: 72 bytes on wire (576 bits), 72 bytes captured (576 bits)
Raw packet data
Internet Protocol Version 6, Src: host::2, Dst: 2001:67c:4e8:f004::9
Transmission Control Protocol, Src Port: 32954, Dst Port: 443, Seq: 1, Ack: 1, Len: 0

No.     Time           Source                Destination           Protocol Length Info
      4 0.048000       host::2    2001:67c:4e8:f004::9  TLSv1    589    Client Hello (SNI=t.me)

Frame 4: 589 bytes on wire (4712 bits), 589 bytes captured (4712 bits)
Raw packet data
Internet Protocol Version 6, Src: host::2, Dst: 2001:67c:4e8:f004::9
Transmission Control Protocol, Src Port: 32954, Dst Port: 443, Seq: 1, Ack: 1, Len: 517
Transport Layer Security

No.     Time           Source                Destination           Protocol Length Info
      5 0.080672       2001:67c:4e8:f004::9  host::2    TCP      72     443 → 32954 [ACK] Seq=1 Ack=518 Win=29696 Len=0 TSval=3941946441 TSecr=4014101160

Frame 5: 72 bytes on wire (576 bits), 72 bytes captured (576 bits)
Raw packet data
Internet Protocol Version 6, Src: 2001:67c:4e8:f004::9, Dst: host::2
Transmission Control Protocol, Src Port: 443, Dst Port: 32954, Seq: 1, Ack: 518, Len: 0

No.     Time           Source                Destination           Protocol Length Info
      6 0.181640       2001:67c:4e8:f004::9  host::2    SSL      208    [TCP Previous segment not captured] , Continuation Data

Frame 6: 208 bytes on wire (1664 bits), 208 bytes captured (1664 bits)
Raw packet data
Internet Protocol Version 6, Src: 2001:67c:4e8:f004::9, Dst: host::2
Transmission Control Protocol, Src Port: 443, Dst Port: 32954, Seq: 5525, Ack: 518, Len: 136
Transport Layer Security

No.     Time           Source                Destination           Protocol Length Info
      7 0.181976       host::2    2001:67c:4e8:f004::9  TCP      84     [TCP Dup ACK 3#1] 32954 → 443 [ACK] Seq=518 Ack=1 Win=26880 Len=0 TSval=4014101294 TSecr=3941946441 SLE=5525 SRE=5661

Frame 7: 84 bytes on wire (672 bits), 84 bytes captured (672 bits)
Raw packet data
Internet Protocol Version 6, Src: host::2, Dst: 2001:67c:4e8:f004::9
Transmission Control Protocol, Src Port: 32954, Dst Port: 443, Seq: 518, Ack: 1, Len: 0

No.     Time           Source                Destination           Protocol Length Info
      8 60.082823      2001:67c:4e8:f004::9  host::2    TCP      72     443 → 32954 [FIN, ACK] Seq=5661 Ack=518 Win=29696 Len=0 TSval=3941961442 TSecr=4014101294

Frame 8: 72 bytes on wire (576 bits), 72 bytes captured (576 bits)
Raw packet data
Internet Protocol Version 6, Src: 2001:67c:4e8:f004::9, Dst: host::2
Transmission Control Protocol, Src Port: 443, Dst Port: 32954, Seq: 5661, Ack: 518, Len: 0

No.     Time           Source                Destination           Protocol Length Info
      9 60.083386      host::2    2001:67c:4e8:f004::9  TCP      84     [TCP Dup ACK 3#2] 32954 → 443 [ACK] Seq=518 Ack=1 Win=26880 Len=0 TSval=4014161195 TSecr=3941946441 SLE=5525 SRE=5662

Frame 9: 84 bytes on wire (672 bits), 84 bytes captured (672 bits)
Raw packet data
Internet Protocol Version 6, Src: host::2, Dst: 2001:67c:4e8:f004::9
Transmission Control Protocol, Src Port: 32954, Dst Port: 443, Seq: 518, Ack: 1, Len: 0

No.     Time           Source                Destination           Protocol Length Info
     10 121.125012     host::2    2001:67c:4e8:f004::9  TCP      84     [TCP Keep-Alive] 32954 → 443 [ACK] Seq=517 Ack=1 Win=26880 Len=0 TSval=4014222237 TSecr=3941946441 SLE=5525 SRE=5662

Frame 10: 84 bytes on wire (672 bits), 84 bytes captured (672 bits)
Raw packet data
Internet Protocol Version 6, Src: host::2, Dst: 2001:67c:4e8:f004::9
Transmission Control Protocol, Src Port: 32954, Dst Port: 443, Seq: 517, Ack: 1, Len: 0

No.     Time           Source                Destination           Protocol Length Info
     11 121.160666     2001:67c:4e8:f004::9  host::2    TCP      72     [TCP Keep-Alive ACK] 443 → 32954 [ACK] Seq=5662 Ack=518 Win=29696 Len=0 TSval=3941976711 TSecr=4014161195

Frame 11: 72 bytes on wire (576 bits), 72 bytes captured (576 bits)
Raw packet data
Internet Protocol Version 6, Src: 2001:67c:4e8:f004::9, Dst: host::2
Transmission Control Protocol, Src Port: 443, Dst Port: 32954, Seq: 5662, Ack: 518, Len: 0

No.     Time           Source                Destination           Protocol Length Info
     12 182.564969     host::2    2001:67c:4e8:f004::9  TCP      84     [TCP Keep-Alive] 32954 → 443 [ACK] Seq=517 Ack=1 Win=26880 Len=0 TSval=4014283677 TSecr=3941946441 SLE=5525 SRE=5662

Frame 12: 84 bytes on wire (672 bits), 84 bytes captured (672 bits)
Raw packet data
Internet Protocol Version 6, Src: host::2, Dst: 2001:67c:4e8:f004::9
Transmission Control Protocol, Src Port: 32954, Dst Port: 443, Seq: 517, Ack: 1, Len: 0

No.     Time           Source                Destination           Protocol Length Info
     13 182.597419     2001:67c:4e8:f004::9  host::2    TCP      60     443 → 32954 [RST] Seq=1 Win=0 Len=0

Frame 13: 60 bytes on wire (480 bits), 60 bytes captured (480 bits)
Raw packet data
Internet Protocol Version 6, Src: 2001:67c:4e8:f004::9, Dst: host::2
Transmission Control Protocol, Src Port: 443, Dst Port: 32954, Seq: 1, Len: 0

Any advice what can be wrong with network configuration on the router would be appreciated.

Last edited by avs (2024-04-20 12:08:32)

just4arch · 2024-04-19 22:15:34

MSS on host looks suspicious, are you using jumbo frames?
Depending on your firewall you can try clamping mss to pmtu on the router for forwarded packets.

Maybe check with tracepath.

avs · 2024-04-19 22:52:23

I do use jumbo frames between router and host.
I hoped what MSS clamping is an ugly past of IPv4.

tracepath shows correct pmtu discovery though:

[user@host ~]$ tracepath 2001:67c:4e8:f004::9
 1?: [LOCALHOST]                        0.015ms pmtu 9000
 1:  _gateway                                              1.172ms
 1:  _gateway                                              0.792ms
 2:  _gateway                                              0.817ms pmtu 1420
 2:  X.ipv6.magic.ungleich.ch                              15.205ms
 3:  X.loves.ipv6.at.ungleich.ch                           29.182ms
 4:  2001:1700:3500:2::11                                 30.701ms
 5:  no reply
 6:  be3011.ccr51.zrh02.atlas.cogentco.com                35.227ms asymm  9
 7:  no reply
 8:  no reply
 9:  no reply
10:  be2278.rcr21.b038092-0.ams03.atlas.cogentco.com      66.910ms asymm 13
11:  no reply
12:  no reply
13:  no reply
...

Adding following:

nft add rule inet filter forward tcp flags syn tcp option maxseg size set rt mtu

~~seems solved the connectivity issue to t.me, but~~ connection to home.netatmo.com fails.
~~One step forward is good sign :-)~~

Last edited by avs (2024-04-20 10:47:07)

just4arch · 2024-04-20 10:20:55

I hoped what MSS clamping is an ugly past of IPv4.

Yep, been bitten by that too - it's still ugly but it works.

nft add rule inet filter forward tcp flags syn tcp option maxseg size set rt mtu

Depending on your network, you could restrict that to the internet interfaces.

Can you provide updated data for home.netatmo.com?

Last edited by just4arch (2024-04-20 10:23:30)

avs · 2024-04-20 11:06:50

Unfortunately no progress. My test for t.me was false positive, I made a typo and curl worked with IPv4.
home.netatmo.com does not work as well.

Here is what I did:
- executed tests on Arch host2 with standard MTU of 1500. Both sites does not work.
- set MTU to 1420 on host2 and both sites start to work
- Tested MTU black hole at http://icmpcheckv6.popcount.org/

According to the black hole test (MTU 1500 and 1420 gave same results):
ICMP path MTU message was successfully delivered to you.
Looks like IP fragments failed to be delivered to you.

Will do tcpdumps on the tunnel, I suspect what clamping does not work.

home.netatmo.com from host2, MTU 1500, site does not load:

13:10:22.910095 IP6 host2.51704 > 2620:1ec:46::60.https: Flags [S], seq 1719781683, win 33120, options [mss 1440,sackOK,TS val 4219632143 ecr 0,nop,wscale 7], length 0
13:10:22.934347 IP6 2620:1ec:46::60.https > host2.51704: Flags [S.], seq 4148194110, ack 1719781684, win 64766, options [mss 1390,sackOK,TS val 1686127480 ecr 4219632143,nop,wscale 7], length 0
13:10:22.935206 IP6 host2.51704 > 2620:1ec:46::60.https: Flags [.], ack 1, win 259, options [nop,nop,TS val 4219632168 ecr 1686127480], length 0
13:10:22.937203 IP6 host2.51704 > 2620:1ec:46::60.https: Flags [P.], seq 1:518, ack 1, win 259, options [nop,nop,TS val 4219632170 ecr 1686127480], length 517
13:10:22.957028 IP6 2620:1ec:46::60.https > host2.51704: Flags [.], ack 518, win 502, options [nop,nop,TS val 1686127503 ecr 4219632170], length 0
13:10:22.957028 IP6 2620:1ec:46::60.https > host2.51704: Flags [P.], seq 1:100, ack 518, win 502, options [nop,nop,TS val 1686127503 ecr 4219632170], length 99
13:10:22.958327 IP6 host2.51704 > 2620:1ec:46::60.https: Flags [.], ack 100, win 259, options [nop,nop,TS val 4219632191 ecr 1686127503], length 0
13:10:22.958623 IP6 host2.51704 > 2620:1ec:46::60.https: Flags [P.], seq 518:1041, ack 100, win 259, options [nop,nop,TS val 4219632192 ecr 1686127503], length 523
13:10:22.988406 IP6 2620:1ec:46::60.https > host2.51704: Flags [P.], seq 2856:4196, ack 1041, win 502, options [nop,nop,TS val 1686127531 ecr 4219632192], length 1340
13:10:22.988406 IP6 2620:1ec:46::60.https > host2.51704: Flags [P.], seq 4196:4602, ack 1041, win 502, options [nop,nop,TS val 1686127532 ecr 4219632192], length 406
13:10:22.989508 IP6 host2.51704 > 2620:1ec:46::60.https: Flags [.], ack 100, win 259, options [nop,nop,TS val 4219632222 ecr 1686127503,nop,nop,sack 1 {2856:4196}], length 0
13:10:22.989534 IP6 host2.51704 > 2620:1ec:46::60.https: Flags [.], ack 100, win 259, options [nop,nop,TS val 4219632222 ecr 1686127503,nop,nop,sack 1 {2856:4602}], length 0
13:10:27.973819 IP6 2620:1ec:46::60.https > host2.51704: Flags [F.], seq 4602, ack 1041, win 502, options [nop,nop,TS val 1686132504 ecr 4219632222], length 0
13:10:27.989895 IP6 host2.51704 > 2620:1ec:46::60.https: Flags [.], ack 100, win 259, options [nop,nop,TS val 4219637223 ecr 1686127503,nop,nop,sack 1 {2856:4603}], length 0
13:10:52.502967 IP6 2620:1ec:bdf::60.https > host2.48824: Flags [P.], seq 1188663580:1188663619, ack 1007438794, win 502, options [nop,nop,TS val 2818791498 ecr 2669761404], length 39
13:10:52.502968 IP6 2620:1ec:bdf::60.https > host2.48824: Flags [FP.], seq 39:63, ack 1, win 502, options [nop,nop,TS val 2818791498 ecr 2669761404], length 24
13:10:52.566976 IP6 host2.48824 > 2620:1ec:bdf::60.https: Flags [.], ack 64, win 1050, options [nop,nop,TS val 2669851296 ecr 2818791498], length 0
13:10:52.567256 IP6 host2.48824 > 2620:1ec:bdf::60.https: Flags [F.], seq 1, ack 64, win 1050, options [nop,nop,TS val 2669851296 ecr 2818791498], length 0
13:10:52.606029 IP6 2620:1ec:bdf::60.https > host2.48824: Flags [.], ack 2, win 502, options [nop,nop,TS val 2818791603 ecr 2669851296], length 0
13:11:28.072871 IP6 host2.51704 > 2620:1ec:46::60.https: Flags [.], ack 100, win 259, options [nop,nop,TS val 4219697306 ecr 1686127503,nop,nop,sack 1 {2856:4603}], length 0
13:11:28.091505 IP6 2620:1ec:46::60.https > host2.51704: Flags [.], ack 1041, win 502, options [nop,nop,TS val 1686192636 ecr 4219637223], length 0
13:12:29.085724 IP6 host2.51704 > 2620:1ec:46::60.https: Flags [.], ack 100, win 259, options [nop,nop,TS val 4219758319 ecr 1686127503,nop,nop,sack 1 {2856:4603}], length 0
13:12:29.112238 IP6 2620:1ec:46::60.https > host2.51704: Flags [R], seq 4148194210, win 0, length 0

home.netatmo.com from host2, MTU 1420, site loads:

13:14:01.117047 IP6 host2.58794 > 2620:1ec:bdf::60.https: Flags [S], seq 89278018, win 32640, options [mss 1360,sackOK,TS val 2670039848 ecr 0,nop,wscale 7], length 0
13:14:01.165284 IP6 2620:1ec:bdf::60.https > host2.58794: Flags [S.], seq 2266987105, ack 89278019, win 64766, options [mss 1390,sackOK,TS val 232190287 ecr 2670039848,nop,wscale 7], length 0
13:14:01.166376 IP6 host2.58794 > 2620:1ec:bdf::60.https: Flags [.], ack 1, win 255, options [nop,nop,TS val 2670039898 ecr 232190287], length 0
13:14:01.168466 IP6 host2.58794 > 2620:1ec:bdf::60.https: Flags [P.], seq 1:518, ack 1, win 255, options [nop,nop,TS val 2670039900 ecr 232190287], length 517
13:14:01.259728 IP6 2620:1ec:bdf::60.https > host2.58794: Flags [.], ack 518, win 502, options [nop,nop,TS val 232190368 ecr 2670039900], length 0
13:14:01.259728 IP6 2620:1ec:bdf::60.https > host2.58794: Flags [P.], seq 1:100, ack 518, win 502, options [nop,nop,TS val 232190368 ecr 2670039900], length 99
13:14:01.260698 IP6 host2.58794 > 2620:1ec:bdf::60.https: Flags [.], ack 100, win 255, options [nop,nop,TS val 2670039992 ecr 232190368], length 0
13:14:01.261066 IP6 host2.58794 > 2620:1ec:bdf::60.https: Flags [P.], seq 518:1041, ack 100, win 255, options [nop,nop,TS val 2670039992 ecr 232190368], length 523
13:14:01.308428 IP6 2620:1ec:bdf::60.https > host2.58794: Flags [.], seq 100:1448, ack 1041, win 502, options [nop,nop,TS val 232190437 ecr 2670039992], length 1348
13:14:01.308580 IP6 2620:1ec:bdf::60.https > host2.58794: Flags [P.], seq 1448:2796, ack 1041, win 502, options [nop,nop,TS val 232190437 ecr 2670039992], length 1348
13:14:01.308610 IP6 2620:1ec:bdf::60.https > host2.58794: Flags [.], seq 2796:4144, ack 1041, win 502, options [nop,nop,TS val 232190437 ecr 2670039992], length 1348
13:14:01.308745 IP6 2620:1ec:bdf::60.https > host2.58794: Flags [P.], seq 4144:4196, ack 1041, win 502, options [nop,nop,TS val 232190437 ecr 2670039992], length 52
13:14:01.308774 IP6 2620:1ec:bdf::60.https > host2.58794: Flags [P.], seq 4196:4602, ack 1041, win 502, options [nop,nop,TS val 232190439 ecr 2670039992], length 406
13:14:01.309886 IP6 host2.58794 > 2620:1ec:bdf::60.https: Flags [.], ack 4144, win 249, options [nop,nop,TS val 2670040041 ecr 232190437], length 0
13:14:01.311075 IP6 host2.58794 > 2620:1ec:bdf::60.https: Flags [.], ack 4602, win 250, options [nop,nop,TS val 2670040042 ecr 232190437], length 0
13:14:01.311794 IP6 host2.58794 > 2620:1ec:bdf::60.https: Flags [P.], seq 1041:1115, ack 4602, win 250, options [nop,nop,TS val 2670040042 ecr 232190437], length 74
13:14:01.311809 IP6 host2.58794 > 2620:1ec:bdf::60.https: Flags [P.], seq 1115:1201, ack 4602, win 250, options [nop,nop,TS val 2670040043 ecr 232190437], length 86
13:14:01.311968 IP6 host2.58794 > 2620:1ec:bdf::60.https: Flags [P.], seq 1201:1263, ack 4602, win 250, options [nop,nop,TS val 2670040043 ecr 232190437], length 62
13:14:01.353145 IP6 2620:1ec:bdf::60.https > host2.58794: Flags [.], ack 1263, win 502, options [nop,nop,TS val 232190482 ecr 2670040042], length 0
13:14:01.353245 IP6 2620:1ec:bdf::60.https > host2.58794: Flags [P.], seq 4602:4681, ack 1263, win 502, options [nop,nop,TS val 232190482 ecr 2670040042], length 79
13:14:01.353245 IP6 2620:1ec:bdf::60.https > host2.58794: Flags [P.], seq 4681:4760, ack 1263, win 502, options [nop,nop,TS val 232190483 ecr 2670040042], length 79
13:14:01.353316 IP6 2620:1ec:bdf::60.https > host2.58794: Flags [P.], seq 4760:4831, ack 1263, win 502, options [nop,nop,TS val 232190483 ecr 2670040042], length 71
13:14:01.354473 IP6 host2.58794 > 2620:1ec:bdf::60.https: Flags [.], ack 4831, win 250, options [nop,nop,TS val 2670040086 ecr 232190482], length 0
13:14:01.354497 IP6 host2.58794 > 2620:1ec:bdf::60.https: Flags [P.], seq 1263:1294, ack 4831, win 250, options [nop,nop,TS val 2670040086 ecr 232190482], length 31
13:14:01.425875 IP6 2620:1ec:bdf::60.https > host2.58794: Flags [.], seq 4831:6179, ack 1294, win 502, options [nop,nop,TS val 232190551 ecr 2670040086], length 1348
13:14:01.426021 IP6 2620:1ec:bdf::60.https > host2.58794: Flags [P.], seq 6179:7527, ack 1294, win 502, options [nop,nop,TS val 232190551 ecr 2670040086], length 1348
13:14:01.426021 IP6 2620:1ec:bdf::60.https > host2.58794: Flags [P.], seq 7527:8186, ack 1294, win 502, options [nop,nop,TS val 232190551 ecr 2670040086], length 659
13:14:01.426021 IP6 2620:1ec:bdf::60.https > host2.58794: Flags [P.], seq 8186:8217, ack 1294, win 502, options [nop,nop,TS val 232190552 ecr 2670040086], length 31
13:14:01.427231 IP6 host2.58794 > 2620:1ec:bdf::60.https: Flags [.], ack 7527, win 250, options [nop,nop,TS val 2670040158 ecr 232190551], length 0
13:14:01.427513 IP6 host2.58794 > 2620:1ec:bdf::60.https: Flags [.], ack 8217, win 250, options [nop,nop,TS val 2670040159 ecr 232190551], length 0
13:14:01.427536 IP6 host2.58794 > 2620:1ec:bdf::60.https: Flags [P.], seq 1294:1318, ack 8217, win 250, options [nop,nop,TS val 2670040159 ecr 232190551], length 24
13:14:01.428132 IP6 host2.58794 > 2620:1ec:bdf::60.https: Flags [F.], seq 1318, ack 8217, win 250, options [nop,nop,TS val 2670040159 ecr 232190551], length 0
13:14:01.467324 IP6 2620:1ec:bdf::60.https > host2.58794: Flags [F.], seq 8217, ack 1318, win 502, options [nop,nop,TS val 232190599 ecr 2670040159], length 0
13:14:01.468268 IP6 host2.58794 > 2620:1ec:bdf::60.https: Flags [.], ack 8218, win 250, options [nop,nop,TS val 2670040200 ecr 232190599], length 0
13:14:01.724656 IP6 host2.58794 > 2620:1ec:bdf::60.https: Flags [F.], seq 1318, ack 8218, win 250, options [nop,nop,TS val 2670040455 ecr 232190599], length 0
13:14:01.778486 IP6 2620:1ec:bdf::60.https > host2.58794: Flags [.], ack 1319, win 502, options [nop,nop,TS val 232190905 ecr 2670040455], length 0

It seems clamping does reduce MSS but not deep enough.

Last edited by avs (2024-04-20 11:30:43)

avs · 2024-04-20 12:07:37

After properly configuring nftables MTU clamping works well

	chain forward { # handle 2
		type filter hook forward priority filter; policy drop;
		meta l4proto ipv6-icmp accept # handle 32
		oifname "ug-ipv6" tcp flags syn tcp option maxseg size set rt mtu # handle 36
		iifname { "br-lan2", "br-lan3", "br-lan9" } oifname { "br-wan", "ug-ipv6" } accept # handle 35
		iifname { "br-wan", "ug-ipv6" } oifname { "br-lan2", "br-lan3", "br-lan9" } ct state { established, related } accept # handle 24
		iifname { "br-lan2", "br-lan5", "br-lan9" } oifname { "br-lan2", "br-lan5", "br-lan9" } accept # handle 27
	}

Initially I added clamping at the end of forward chain (after handle 27).
And the rule was never hit as accepts above intercept all traffic.
At handle 36 it works as expected.

Both t.me and home.netatmo.com work well.

Arch Linux

#1 2024-04-19 21:26:11

[SOLVED] IPv6 connectivity issues behind Arch router with WG tunnel

#2 2024-04-19 22:15:34

Re: [SOLVED] IPv6 connectivity issues behind Arch router with WG tunnel

#3 2024-04-19 22:52:23

Re: [SOLVED] IPv6 connectivity issues behind Arch router with WG tunnel

#4 2024-04-20 10:20:55

Re: [SOLVED] IPv6 connectivity issues behind Arch router with WG tunnel

#5 2024-04-20 11:06:50

Re: [SOLVED] IPv6 connectivity issues behind Arch router with WG tunnel

#6 2024-04-20 12:07:37

Re: [SOLVED] IPv6 connectivity issues behind Arch router with WG tunnel

Board footer