You are not logged in.

Pages: 1

#1 2014-04-23 05:56:44

tazmanian
Member
Registered: 2009-10-25
Posts: 39

[solved] TCP outbound bandwidth issues w/ OpenVPN

I am running OpenVPN in TUN mode over UDP, and I am having bandwidth issues when sending TCP data through the TUN device. Oddly, receiving TCP data works fine, as does sending and receiving UDP data through the TUN device. Sending TCP to the same remote host works fine when not tunnelled.

I used tshark to look at what happens when sending TCP over the tunnel, and shortly after the TCP handshake, I see a long period of silence (262 ms) after the receiver has ACKed everything, before the sender resumes sending. Subsequently, the sender starts waiting for acks on every packet, even though the receive window has grown.

Any ideas on what is going on?

EDIT: Added solution in quote box below to make it easier to find, since this thread has continued beyond the identification of the solution.

Solution is to revert to a pre-3.14 kernel. There is a kernel bug that affects TCP performance over TUN devices. A patch has been issued and should land upstream with kernel 3.15.

Update: This issue is now fixed in Arch Linux: the patch has been backported into linux-3.14.2-1.


Details follow...

The local system is Arch Linux running openvpn-2.3.3 on linux-3.14.1-1. The remote is Ubuntu 12.04 (precise) running openvpn-2.2.1 on linux-3.2.0-49-generic.

Here is the output of iperf testing TCP through the tunnel. Outbound bandwidth is abysmal, but inbound reaches 20 Mbps.

$ iperf -c 192.168.12.1 -i 2 -t 10 -l 1374 -L 5002 -r -m
------------------------------------------------------------
Server listening on TCP port 5002
TCP window size: 85.3 KByte (default)
------------------------------------------------------------
------------------------------------------------------------
Client connecting to 192.168.12.1, TCP port 5001
TCP window size: 45.0 KByte (default)
------------------------------------------------------------
[  5] local 192.168.12.2 port 40465 connected with 192.168.12.1 port 5001
[ ID] Interval       Transfer     Bandwidth
[  5]  0.0- 2.0 sec  56.4 KBytes   231 Kbits/sec
[  5]  2.0- 4.0 sec  41.6 KBytes   170 Kbits/sec
[  5]  4.0- 6.0 sec  0.00 Bytes  0.00 bits/sec
[  5]  6.0- 8.0 sec  61.7 KBytes   253 Kbits/sec
[  5]  8.0-10.0 sec  16.1 KBytes  66.0 Kbits/sec
[  5]  0.0-11.7 sec   177 KBytes   124 Kbits/sec
[  5] MSS size 1328 bytes (MTU 1368 bytes, unknown interface)
[  4] local 192.168.12.2 port 5002 connected with 192.168.12.1 port 54021
[  4]  0.0- 2.0 sec  4.54 MBytes  19.1 Mbits/sec
[  4]  2.0- 4.0 sec  4.57 MBytes  19.2 Mbits/sec
[  4]  4.0- 6.0 sec  5.12 MBytes  21.5 Mbits/sec
[  4]  6.0- 8.0 sec  4.85 MBytes  20.4 Mbits/sec
[  4]  8.0-10.0 sec  5.10 MBytes  21.4 Mbits/sec
[  4]  0.0-10.0 sec  24.2 MBytes  20.3 Mbits/sec
[  4] MSS size 1328 bytes (MTU 1368 bytes, unknown interface)

I also used iperf test UDP through the tunnel with 1374-byte packets (any larger and the packets would fail to traverse the tunnel, presumably because of MTU issues). Both directions reach 30 Mbps easily with negligible packet loss/reordering.

Below is the first 6 seconds of the tshark trace for the above iperf test for outbound TCP. Weird things that are happening include long periods of silence after the receiver has ACKed everything (e.g., 262 ms between packets 27 and 28). Subsequently, the sender starts waiting for acks on every packet, even though the receive window has grown (packets 29-52).

$ tshark -n -i tun1 port 5001
Capturing on 'tun1'
  1   0.000000 192.168.12.2 -> 192.168.12.1 TCP 40465 > 5001 [SYN] Seq=0 Win=29200 Len=0 MSS=1460 SACK_PERM=1 TSval=14512326 TSecr=0 WS=128
  2   0.036964 192.168.12.1 -> 192.168.12.2 TCP 5001 > 40465 [SYN, ACK] Seq=0 Ack=1 Win=14480 Len=0 MSS=1340 SACK_PERM=1 TSval=1996759218 TSecr=14512326 WS=4
  3   0.036991 192.168.12.2 -> 192.168.12.1 TCP 40465 > 5001 [ACK] Seq=1 Ack=1 Win=29312 Len=0 TSval=14512337 TSecr=1996759218
  4   0.037070 192.168.12.2 -> 192.168.12.1 TCP 40465 > 5001 [PSH, ACK] Seq=1 Ack=1 Win=29312 Len=24 TSval=14512337 TSecr=1996759218
  5   0.037086 192.168.12.2 -> 192.168.12.1 TCP 40465 > 5001 [ACK] Seq=25 Ack=1 Win=29312 Len=1328 TSval=14512337 TSecr=1996759218
  6   0.037106 192.168.12.2 -> 192.168.12.1 TCP 40465 > 5001 [ACK] Seq=1353 Ack=1 Win=29312 Len=1328 TSval=14512337 TSecr=1996759218
  7   0.037110 192.168.12.2 -> 192.168.12.1 TCP 40465 > 5001 [ACK] Seq=2681 Ack=1 Win=29312 Len=1328 TSval=14512337 TSecr=1996759218
  8   0.037117 192.168.12.2 -> 192.168.12.1 TCP 40465 > 5001 [ACK] Seq=4009 Ack=1 Win=29312 Len=1328 TSval=14512337 TSecr=1996759218
  9   0.037122 192.168.12.2 -> 192.168.12.1 TCP 40465 > 5001 [ACK] Seq=5337 Ack=1 Win=29312 Len=1328 TSval=14512337 TSecr=1996759218
 10   0.037127 192.168.12.2 -> 192.168.12.1 TCP 40465 > 5001 [ACK] Seq=6665 Ack=1 Win=29312 Len=1328 TSval=14512337 TSecr=1996759218
 11   0.037138 192.168.12.2 -> 192.168.12.1 TCP 40465 > 5001 [ACK] Seq=7993 Ack=1 Win=29312 Len=1328 TSval=14512338 TSecr=1996759218
 12   0.037142 192.168.12.2 -> 192.168.12.1 TCP 40465 > 5001 [ACK] Seq=9321 Ack=1 Win=29312 Len=1328 TSval=14512338 TSecr=1996759218
 13   0.037148 192.168.12.2 -> 192.168.12.1 TCP 40465 > 5001 [ACK] Seq=10649 Ack=1 Win=29312 Len=1328 TSval=14512338 TSecr=1996759218
 14   0.074371 192.168.12.1 -> 192.168.12.2 TCP 5001 > 40465 [ACK] Seq=1 Ack=25 Win=14480 Len=0 TSval=1996759227 TSecr=14512337
 15   0.074398 192.168.12.2 -> 192.168.12.1 TCP 40465 > 5001 [ACK] Seq=11977 Ack=1 Win=29312 Len=1328 TSval=14512349 TSecr=1996759227
 16   0.074525 192.168.12.1 -> 192.168.12.2 TCP 5001 > 40465 [ACK] Seq=1 Ack=1353 Win=17376 Len=0 TSval=1996759227 TSecr=14512337
 17   0.074540 192.168.12.2 -> 192.168.12.1 TCP 40465 > 5001 [ACK] Seq=13305 Ack=1 Win=29312 Len=1328 TSval=14512349 TSecr=1996759227
 18   0.074549 192.168.12.1 -> 192.168.12.2 TCP 5001 > 40465 [ACK] Seq=1 Ack=2681 Win=20272 Len=0 TSval=1996759227 TSecr=14512337
 19   0.074580 192.168.12.1 -> 192.168.12.2 TCP 5001 > 40465 [ACK] Seq=1 Ack=4009 Win=23168 Len=0 TSval=1996759227 TSecr=14512337
 20   0.075164 192.168.12.1 -> 192.168.12.2 TCP 5001 > 40465 [ACK] Seq=1 Ack=5337 Win=26064 Len=0 TSval=1996759227 TSecr=14512337
 21   0.075179 192.168.12.1 -> 192.168.12.2 TCP 5001 > 40465 [ACK] Seq=1 Ack=6665 Win=28960 Len=0 TSval=1996759227 TSecr=14512337
 22   0.075213 192.168.12.1 -> 192.168.12.2 TCP 5001 > 40465 [ACK] Seq=1 Ack=7993 Win=31856 Len=0 TSval=1996759227 TSecr=14512337
 23   0.075418 192.168.12.1 -> 192.168.12.2 TCP 5001 > 40465 [ACK] Seq=1 Ack=9321 Win=34752 Len=0 TSval=1996759227 TSecr=14512338
 24   0.075439 192.168.12.1 -> 192.168.12.2 TCP 5001 > 40465 [ACK] Seq=1 Ack=10649 Win=37648 Len=0 TSval=1996759227 TSecr=14512338
 25   0.075473 192.168.12.1 -> 192.168.12.2 TCP 5001 > 40465 [ACK] Seq=1 Ack=11977 Win=40544 Len=0 TSval=1996759227 TSecr=14512338
 26   0.111499 192.168.12.1 -> 192.168.12.2 TCP 5001 > 40465 [ACK] Seq=1 Ack=13305 Win=42244 Len=0 TSval=1996759236 TSecr=14512349
 27   0.111538 192.168.12.1 -> 192.168.12.2 TCP 5001 > 40465 [ACK] Seq=1 Ack=14633 Win=42244 Len=0 TSval=1996759236 TSecr=14512349
 28   0.347154 192.168.12.2 -> 192.168.12.1 TCP 40465 > 5001 [ACK] Seq=14633 Ack=1 Win=29312 Len=1328 TSval=14512431 TSecr=1996759236
 29   0.384316 192.168.12.1 -> 192.168.12.2 TCP 5001 > 40465 [ACK] Seq=1 Ack=15961 Win=42244 Len=0 TSval=1996759304 TSecr=14512431
 30   0.620498 192.168.12.2 -> 192.168.12.1 TCP 40465 > 5001 [ACK] Seq=15961 Ack=1 Win=29312 Len=1328 TSval=14512513 TSecr=1996759304
 31   0.657656 192.168.12.1 -> 192.168.12.2 TCP 5001 > 40465 [ACK] Seq=1 Ack=17289 Win=42244 Len=0 TSval=1996759373 TSecr=14512513
 32   0.893831 192.168.12.2 -> 192.168.12.1 TCP 40465 > 5001 [ACK] Seq=17289 Ack=1 Win=29312 Len=1328 TSval=14512595 TSecr=1996759373
 33   0.930995 192.168.12.1 -> 192.168.12.2 TCP 5001 > 40465 [ACK] Seq=1 Ack=18617 Win=42244 Len=0 TSval=1996759441 TSecr=14512595
 34   1.167165 192.168.12.2 -> 192.168.12.1 TCP 40465 > 5001 [ACK] Seq=18617 Ack=1 Win=29312 Len=1328 TSval=14512677 TSecr=1996759441
 35   1.204354 192.168.12.1 -> 192.168.12.2 TCP 5001 > 40465 [ACK] Seq=1 Ack=19945 Win=42244 Len=0 TSval=1996759509 TSecr=14512677
 36   1.440511 192.168.12.2 -> 192.168.12.1 TCP 40465 > 5001 [ACK] Seq=19945 Ack=1 Win=29312 Len=1328 TSval=14512759 TSecr=1996759509
 37   1.477776 192.168.12.1 -> 192.168.12.2 TCP 5001 > 40465 [ACK] Seq=1 Ack=21273 Win=42244 Len=0 TSval=1996759578 TSecr=14512759
 38   1.713826 192.168.12.2 -> 192.168.12.1 TCP 40465 > 5001 [ACK] Seq=21273 Ack=1 Win=29312 Len=1328 TSval=14512841 TSecr=1996759578
 39   1.750971 192.168.12.1 -> 192.168.12.2 TCP 5001 > 40465 [ACK] Seq=1 Ack=22601 Win=42244 Len=0 TSval=1996759646 TSecr=14512841
 40   1.987166 192.168.12.2 -> 192.168.12.1 TCP 40465 > 5001 [ACK] Seq=22601 Ack=1 Win=29312 Len=1328 TSval=14512923 TSecr=1996759646
 41   2.024296 192.168.12.1 -> 192.168.12.2 TCP 5001 > 40465 [ACK] Seq=1 Ack=23929 Win=42244 Len=0 TSval=1996759714 TSecr=14512923
 42   2.260494 192.168.12.2 -> 192.168.12.1 TCP 40465 > 5001 [ACK] Seq=23929 Ack=1 Win=29312 Len=1328 TSval=14513005 TSecr=1996759714
 43   2.297850 192.168.12.1 -> 192.168.12.2 TCP 5001 > 40465 [ACK] Seq=1 Ack=25257 Win=42244 Len=0 TSval=1996759783 TSecr=14513005
 44   2.533853 192.168.12.2 -> 192.168.12.1 TCP 40465 > 5001 [ACK] Seq=25257 Ack=1 Win=29312 Len=1328 TSval=14513087 TSecr=1996759783
 45   2.571144 192.168.12.1 -> 192.168.12.2 TCP 5001 > 40465 [ACK] Seq=1 Ack=26585 Win=42244 Len=0 TSval=1996759851 TSecr=14513087
 46   2.807183 192.168.12.2 -> 192.168.12.1 TCP 40465 > 5001 [PSH, ACK] Seq=26585 Ack=1 Win=29312 Len=1328 TSval=14513169 TSecr=1996759851
 47   2.844493 192.168.12.1 -> 192.168.12.2 TCP 5001 > 40465 [ACK] Seq=1 Ack=27913 Win=42244 Len=0 TSval=1996759919 TSecr=14513169
 48   3.080495 192.168.12.2 -> 192.168.12.1 TCP 40465 > 5001 [ACK] Seq=27913 Ack=1 Win=29312 Len=1328 TSval=14513251 TSecr=1996759919
 49   3.117984 192.168.12.1 -> 192.168.12.2 TCP 5001 > 40465 [ACK] Seq=1 Ack=29241 Win=42244 Len=0 TSval=1996759988 TSecr=14513251
 50   3.353834 192.168.12.2 -> 192.168.12.1 TCP 40465 > 5001 [ACK] Seq=29241 Ack=1 Win=29312 Len=1328 TSval=14513333 TSecr=1996759988
 51   3.391030 192.168.12.1 -> 192.168.12.2 TCP 5001 > 40465 [ACK] Seq=1 Ack=30569 Win=42244 Len=0 TSval=1996760056 TSecr=14513333
 52   3.391065 192.168.12.2 -> 192.168.12.1 TCP 40465 > 5001 [ACK] Seq=30569 Ack=1 Win=29312 Len=1328 TSval=14513344 TSecr=1996760056
 53   3.391074 192.168.12.2 -> 192.168.12.1 TCP 40465 > 5001 [ACK] Seq=31897 Ack=1 Win=29312 Len=1328 TSval=14513344 TSecr=1996760056
 54   3.428165 192.168.12.1 -> 192.168.12.2 TCP 5001 > 40465 [ACK] Seq=1 Ack=31897 Win=42244 Len=0 TSval=1996760065 TSecr=14513344
 55   3.428198 192.168.12.2 -> 192.168.12.1 TCP 40465 > 5001 [ACK] Seq=33225 Ack=1 Win=29312 Len=1328 TSval=14513355 TSecr=1996760065
 56   3.428207 192.168.12.2 -> 192.168.12.1 TCP 40465 > 5001 [ACK] Seq=34553 Ack=1 Win=29312 Len=1328 TSval=14513355 TSecr=1996760065
 57   3.428232 192.168.12.1 -> 192.168.12.2 TCP 5001 > 40465 [ACK] Seq=1 Ack=33225 Win=42244 Len=0 TSval=1996760065 TSecr=14513344
 58   3.428249 192.168.12.2 -> 192.168.12.1 TCP 40465 > 5001 [ACK] Seq=35881 Ack=1 Win=29312 Len=1328 TSval=14513355 TSecr=1996760065
 59   3.428256 192.168.12.2 -> 192.168.12.1 TCP 40465 > 5001 [ACK] Seq=37209 Ack=1 Win=29312 Len=1328 TSval=14513355 TSecr=1996760065
 60   3.465413 192.168.12.1 -> 192.168.12.2 TCP 5001 > 40465 [ACK] Seq=1 Ack=34553 Win=42244 Len=0 TSval=1996760075 TSecr=14513355
 61   3.465445 192.168.12.2 -> 192.168.12.1 TCP 40465 > 5001 [ACK] Seq=38537 Ack=1 Win=29312 Len=1328 TSval=14513366 TSecr=1996760075
 62   3.465455 192.168.12.2 -> 192.168.12.1 TCP 40465 > 5001 [PSH, ACK] Seq=39865 Ack=1 Win=29312 Len=1328 TSval=14513366 TSecr=1996760075
 63   3.465541 192.168.12.1 -> 192.168.12.2 TCP 5001 > 40465 [ACK] Seq=1 Ack=35881 Win=42244 Len=0 TSval=1996760075 TSecr=14513355
 64   3.465557 192.168.12.2 -> 192.168.12.1 TCP 40465 > 5001 [ACK] Seq=41193 Ack=1 Win=29312 Len=1328 TSval=14513366 TSecr=1996760075
 65   3.465564 192.168.12.2 -> 192.168.12.1 TCP 40465 > 5001 [ACK] Seq=42521 Ack=1 Win=29312 Len=1328 TSval=14513366 TSecr=1996760075
 66   3.465590 192.168.12.1 -> 192.168.12.2 TCP 5001 > 40465 [ACK] Seq=1 Ack=37209 Win=41388 Len=0 TSval=1996760075 TSecr=14513355
 67   3.465609 192.168.12.2 -> 192.168.12.1 TCP 40465 > 5001 [ACK] Seq=43849 Ack=1 Win=29312 Len=1328 TSval=14513366 TSecr=1996760075
 68   3.465620 192.168.12.2 -> 192.168.12.1 TCP 40465 > 5001 [ACK] Seq=45177 Ack=1 Win=29312 Len=1328 TSval=14513366 TSecr=1996760075
 69   3.465637 192.168.12.1 -> 192.168.12.2 TCP 5001 > 40465 [ACK] Seq=1 Ack=38537 Win=42244 Len=0 TSval=1996760075 TSecr=14513355
 70   3.465654 192.168.12.2 -> 192.168.12.1 TCP 40465 > 5001 [ACK] Seq=46505 Ack=1 Win=29312 Len=1328 TSval=14513366 TSecr=1996760075
 71   3.465660 192.168.12.2 -> 192.168.12.1 TCP 40465 > 5001 [ACK] Seq=47833 Ack=1 Win=29312 Len=1328 TSval=14513366 TSecr=1996760075
 72   3.502405 192.168.12.1 -> 192.168.12.2 TCP 5001 > 40465 [ACK] Seq=1 Ack=39865 Win=42244 Len=0 TSval=1996760084 TSecr=14513366
 73   3.502452 192.168.12.2 -> 192.168.12.1 TCP 40465 > 5001 [ACK] Seq=49161 Ack=1 Win=29312 Len=1328 TSval=14513377 TSecr=1996760084
 74   3.502548 192.168.12.1 -> 192.168.12.2 TCP 5001 > 40465 [ACK] Seq=1 Ack=41193 Win=42244 Len=0 TSval=1996760084 TSecr=14513366
 75   3.502558 192.168.12.2 -> 192.168.12.1 TCP 40465 > 5001 [ACK] Seq=50489 Ack=1 Win=29312 Len=1328 TSval=14513377 TSecr=1996760084
 76   3.502564 192.168.12.1 -> 192.168.12.2 TCP 5001 > 40465 [ACK] Seq=1 Ack=42521 Win=42244 Len=0 TSval=1996760084 TSecr=14513366
 77   3.502581 192.168.12.2 -> 192.168.12.1 TCP 40465 > 5001 [ACK] Seq=51817 Ack=1 Win=29312 Len=1328 TSval=14513377 TSecr=1996760084
 78   3.502748 192.168.12.1 -> 192.168.12.2 TCP 5001 > 40465 [ACK] Seq=1 Ack=43849 Win=42244 Len=0 TSval=1996760084 TSecr=14513366
 79   3.502771 192.168.12.2 -> 192.168.12.1 TCP 40465 > 5001 [ACK] Seq=53145 Ack=1 Win=29312 Len=1328 TSval=14513377 TSecr=1996760084
 80   3.503049 192.168.12.1 -> 192.168.12.2 TCP 5001 > 40465 [ACK] Seq=1 Ack=45177 Win=42244 Len=0 TSval=1996760084 TSecr=14513366
 81   3.503063 192.168.12.2 -> 192.168.12.1 TCP 40465 > 5001 [ACK] Seq=54473 Ack=1 Win=29312 Len=1328 TSval=14513377 TSecr=1996760084
 82   3.503250 192.168.12.1 -> 192.168.12.2 TCP 5001 > 40465 [ACK] Seq=1 Ack=46505 Win=42244 Len=0 TSval=1996760084 TSecr=14513366
 83   3.503264 192.168.12.2 -> 192.168.12.1 TCP 40465 > 5001 [PSH, ACK] Seq=55801 Ack=1 Win=29312 Len=1328 TSval=14513377 TSecr=1996760084
 84   3.503322 192.168.12.1 -> 192.168.12.2 TCP 5001 > 40465 [ACK] Seq=1 Ack=47833 Win=42244 Len=0 TSval=1996760084 TSecr=14513366
 85   3.503330 192.168.12.2 -> 192.168.12.1 TCP 40465 > 5001 [ACK] Seq=57129 Ack=1 Win=29312 Len=1328 TSval=14513377 TSecr=1996760084
 86   3.503335 192.168.12.1 -> 192.168.12.2 TCP 5001 > 40465 [ACK] Seq=1 Ack=49161 Win=42244 Len=0 TSval=1996760084 TSecr=14513366
 87   3.503341 192.168.12.2 -> 192.168.12.1 TCP 40465 > 5001 [ACK] Seq=58457 Ack=1 Win=29312 Len=1328 TSval=14513377 TSecr=1996760084
 88   3.539707 192.168.12.1 -> 192.168.12.2 TCP 5001 > 40465 [ACK] Seq=1 Ack=51817 Win=42244 Len=0 TSval=1996760093 TSecr=14513377
 89   3.539730 192.168.12.1 -> 192.168.12.2 TCP 5001 > 40465 [ACK] Seq=1 Ack=54473 Win=42244 Len=0 TSval=1996760093 TSecr=14513377
 90   3.540206 192.168.12.1 -> 192.168.12.2 TCP 5001 > 40465 [ACK] Seq=1 Ack=57129 Win=42244 Len=0 TSval=1996760093 TSecr=14513377
 91   3.540328 192.168.12.1 -> 192.168.12.2 TCP 5001 > 40465 [ACK] Seq=1 Ack=59785 Win=42244 Len=0 TSval=1996760094 TSecr=14513377
 92   3.773837 192.168.12.2 -> 192.168.12.1 TCP 40465 > 5001 [ACK] Seq=59785 Ack=1 Win=29312 Len=1328 TSval=14513459 TSecr=1996760094
 93   3.811056 192.168.12.1 -> 192.168.12.2 TCP 5001 > 40465 [ACK] Seq=1 Ack=61113 Win=42244 Len=0 TSval=1996760161 TSecr=14513459
 94   4.047156 192.168.12.2 -> 192.168.12.1 TCP 40465 > 5001 [ACK] Seq=61113 Ack=1 Win=29312 Len=1328 TSval=14513541 TSecr=1996760161
 95   4.084261 192.168.12.1 -> 192.168.12.2 TCP 5001 > 40465 [ACK] Seq=1 Ack=62441 Win=42244 Len=0 TSval=1996760229 TSecr=14513541
 96   4.320497 192.168.12.2 -> 192.168.12.1 TCP 40465 > 5001 [ACK] Seq=62441 Ack=1 Win=29312 Len=1328 TSval=14513623 TSecr=1996760229
 97   4.357729 192.168.12.1 -> 192.168.12.2 TCP 5001 > 40465 [ACK] Seq=1 Ack=63769 Win=42244 Len=0 TSval=1996760298 TSecr=14513623
 98   4.593842 192.168.12.2 -> 192.168.12.1 TCP 40465 > 5001 [ACK] Seq=63769 Ack=1 Win=29312 Len=1328 TSval=14513705 TSecr=1996760298
 99   4.631070 192.168.12.1 -> 192.168.12.2 TCP 5001 > 40465 [ACK] Seq=1 Ack=65097 Win=42244 Len=0 TSval=1996760366 TSecr=14513705
100   4.867178 192.168.12.2 -> 192.168.12.1 TCP 40465 > 5001 [ACK] Seq=65097 Ack=1 Win=29312 Len=1328 TSval=14513787 TSecr=1996760366
101   4.904431 192.168.12.1 -> 192.168.12.2 TCP 5001 > 40465 [ACK] Seq=1 Ack=66425 Win=42244 Len=0 TSval=1996760434 TSecr=14513787
102   5.140491 192.168.12.2 -> 192.168.12.1 TCP 40465 > 5001 [ACK] Seq=66425 Ack=1 Win=29312 Len=1328 TSval=14513869 TSecr=1996760434
103   5.177699 192.168.12.1 -> 192.168.12.2 TCP 5001 > 40465 [ACK] Seq=1 Ack=67753 Win=42244 Len=0 TSval=1996760503 TSecr=14513869
104   5.413831 192.168.12.2 -> 192.168.12.1 TCP 40465 > 5001 [ACK] Seq=67753 Ack=1 Win=29312 Len=1328 TSval=14513951 TSecr=1996760503
105   5.451026 192.168.12.1 -> 192.168.12.2 TCP 5001 > 40465 [ACK] Seq=1 Ack=69081 Win=42244 Len=0 TSval=1996760571 TSecr=14513951
106   5.687178 192.168.12.2 -> 192.168.12.1 TCP 40465 > 5001 [PSH, ACK] Seq=69081 Ack=1 Win=29312 Len=1328 TSval=14514033 TSecr=1996760571
107   5.724462 192.168.12.1 -> 192.168.12.2 TCP 5001 > 40465 [ACK] Seq=1 Ack=70409 Win=42244 Len=0 TSval=1996760639 TSecr=14514033
108   5.960502 192.168.12.2 -> 192.168.12.1 TCP 40465 > 5001 [ACK] Seq=70409 Ack=1 Win=29312 Len=1328 TSval=14514115 TSecr=1996760639
109   5.997666 192.168.12.1 -> 192.168.12.2 TCP 5001 > 40465 [ACK] Seq=1 Ack=71737 Win=42244 Len=0 TSval=1996760708 TSecr=14514115

Outbound TCP performs fine when not tunnelled:

$ iperf -c [REDACTED] -i 2 -t 10 -l 1374 -m
------------------------------------------------------------
Client connecting to [REDACTED], TCP port 5001
TCP window size: 85.0 KByte (default)
------------------------------------------------------------
[  3] local 192.168.85.120 port 38701 connected with [REDACTED] port 5001
[ ID] Interval       Transfer     Bandwidth
[  3]  0.0- 2.0 sec  3.44 MBytes  14.4 Mbits/sec
[  3]  2.0- 4.0 sec  3.23 MBytes  13.6 Mbits/sec
[  3]  4.0- 6.0 sec  3.34 MBytes  14.0 Mbits/sec
[  3]  6.0- 8.0 sec  3.34 MBytes  14.0 Mbits/sec
[  3]  8.0-10.0 sec  3.23 MBytes  13.6 Mbits/sec
[  3]  0.0-10.0 sec  16.6 MBytes  13.9 Mbits/sec
[  3] MSS size 1448 bytes (MTU 1500 bytes, ethernet)

Local OpenVPN config:

dev tun1
client
proto udp
lport 1195
rport 1194

remote [REDACTED]
resolv-retry infinite
float

tls-client
ca ca.crt
cert client.crt
key client.key

remote-cert-tls server
crl-verify crl.pem

cipher AES-128-CBC
auth SHA256
reneg-sec 60

user nobody
group nobody

comp-lzo no
mtu-disc yes

persist-tun
persist-key

route-up "/usr/local/bin/iptables restart"
script-security 3 system

verb 0

Remote OpenVPN config:

server 192.168.12.0 255.255.255.0
client-to-client
dev tun0
proto udp
port 1194

tls-server
ca ca.crt
cert server.crt
key server.key
dh dh2048.pem
crl-verify crl.pem

cipher AES-128-CBC
auth SHA256
reneg-sec 60

user nobody
group nogroup

comp-lzo no
mtu-disc yes

ifconfig-push 192.168.12.2 192.168.12.1

keepalive 10 60
persist-tun
persist-key

route-up "/etc/init.d/iptables restart"

verb 0

Last edited by tazmanian (2014-05-15 15:30:50)

Offline

#2 2014-04-23 06:08:22

roentgen
Member
Registered: 2011-03-15
Posts: 91

Re: [solved] TCP outbound bandwidth issues w/ OpenVPN

No real solution here but I'd start by commenting 'mtu-disc' on the server side and put 'mtu-test' on the client side and check the logs (maybe increase verb).

Offline

#3 2014-04-23 06:15:21

train_wreck
Member
Registered: 2011-10-22
Posts: 97

Re: [solved] TCP outbound bandwidth issues w/ OpenVPN

i'll go ahead and weigh in here - i experienced mostly the exact same problem, sending data from the openvpn server to any client was abysmally slow.

what kernel version are you running? i found that this problem started occurring upon upgrade to kernel 3.14, downgrading back to kernel 3.13.6 fixed this issue entirely for me.

EDIT nvm you said you're on 3.14, try downgrading. something seemed borked with the kernel, possibly with how it handles the tunneling adapters (tunX)

Last edited by train_wreck (2014-04-23 06:16:41)

Offline

#4 2014-04-23 06:28:26

tazmanian
Member
Registered: 2009-10-25
Posts: 39

Re: [solved] TCP outbound bandwidth issues w/ OpenVPN

Thanks for the suggestions and the fast responses!

roentgen: I doubt MTU is the issue here. MTU was one of the first things I played with, but things didn't improve. I was running tshark when I tested UDP throughput. When I set the UDP packet sizes too high, I would see the UDP packets locally, but they would fail to show up remotely. For the TCP stream, this would mean dropped packets and retransmits, which isn't what the trace in my original post showed.

train_wreck: I figured it might be a kernel issue. I will try downgrading as you suggest, and report back.

Offline

#5 2014-04-23 06:45:21

tazmanian
Member
Registered: 2009-10-25
Posts: 39

Re: [solved] TCP outbound bandwidth issues w/ OpenVPN

Yep, kernel 3.14 has issues. Downgrading to 3.13.6-2 fixed this. Thanks for the tip, train_wreck!

EDIT: Looks like this is a known issue: https://bugzilla.kernel.org/show_bug.cgi?id=74051
EDIT: linux-3.13.8-1 is fine too. Just don't update to 3.14.

Last edited by tazmanian (2014-04-23 07:02:29)

Offline

#6 2014-04-23 07:17:36

train_wreck
Member
Registered: 2011-10-22
Posts: 97

Re: [solved] TCP outbound bandwidth issues w/ OpenVPN

no prob! smile glad to see i wasn't going crazy about this. hopefully it will be fixed soon.

Offline

#7 2014-04-24 16:49:41

train_wreck
Member
Registered: 2011-10-22
Posts: 97

Re: [solved] TCP outbound bandwidth issues w/ OpenVPN

Looks like it got patched yesterday

https://bugzilla.kernel.org/show_bug.cgi?id=74051#c3

Offline

#8 2014-04-24 19:43:29

Layus
Member
Registered: 2012-04-30
Posts: 12

Re: [solved] TCP outbound bandwidth issues w/ OpenVPN

I just wanted to say thank you to all of you.
I got stuck with this for some time, but I had really no time to investigate.

Downgrading seems to be the only sensible solution for now.
The fix will be included in 3.15.0-rc2 and eventualy in 3.15.0, but we may hope that it will be backported to 3.14.X.
Let's wait and see.

Offline

#9 2014-04-25 12:02:04

lev
Member
Registered: 2010-05-05
Posts: 7

Re: [solved] TCP outbound bandwidth issues w/ OpenVPN

This performance regression renders openvpn with a tun adapter unusable if client and server use kernel 3.14 .
Thus I created a bug report: https://bugs.archlinux.org/task/40089

Offline

#10 2014-04-25 15:10:38

train_wreck
Member
Registered: 2011-10-22
Posts: 97

Re: [solved] TCP outbound bandwidth issues w/ OpenVPN

lev wrote:

This performance regression renders openvpn with a tun adapter unusable if client and server use kernel 3.14 .
Thus I created a bug report: https://bugs.archlinux.org/task/40089

i actually noticed it to be an "either-or" type of thing; my Windows clients were seeing the same thing coming off a 3.14 openvpn server.

yeah, weird issue. like i noticed spurts of even-powers-of-2 sized packets

------------------------------------------------------------
Client connecting to 10.10.10.6, TCP port 5001
TCP window size:  416 KByte
------------------------------------------------------------
[  3] local 10.10.10.1 port 40643 connected with 10.10.10.6 port 5001
[ ID] Interval       Transfer     Bandwidth
[  3]  0.0- 2.0 sec   512 KBytes  2.10 Mbits/sec
[  3]  2.0- 4.0 sec  0.00 Bytes  0.00 bits/sec
[  3]  4.0- 6.0 sec  0.00 Bytes  0.00 bits/sec
[  3]  6.0- 8.0 sec  0.00 Bytes  0.00 bits/sec
[  3]  8.0-10.0 sec   128 KBytes   524 Kbits/sec
[  3] 10.0-12.0 sec   128 KBytes   524 Kbits/sec
[  3] 12.0-14.0 sec   512 KBytes  2.10 Mbits/sec
[  3] 14.0-16.0 sec   128 KBytes   524 Kbits/sec
[  3] 16.0-18.0 sec   512 KBytes  2.10 Mbits/sec
[  3] 18.0-20.0 sec   128 KBytes   524 Kbits/sec
[  3] 20.0-22.0 sec   384 KBytes  1.57 Mbits/sec
[  3] 22.0-24.0 sec   256 KBytes  1.05 Mbits/sec
[  3] 24.0-26.0 sec   512 KBytes  2.10 Mbits/sec
[  3] 26.0-28.0 sec   384 KBytes  1.57 Mbits/sec
[  3] 28.0-30.0 sec   256 KBytes  1.05 Mbits/sec
[  3] 30.0-32.0 sec   128 KBytes   524 Kbits/sec
[  3] 32.0-34.0 sec   640 KBytes  2.62 Mbits/sec
[  3] 34.0-36.0 sec   384 KBytes  1.57 Mbits/sec
[  3] 36.0-38.0 sec   384 KBytes  1.57 Mbits/sec
[  3] 38.0-40.0 sec   384 KBytes  1.57 Mbits/sec
[  3] 40.0-42.0 sec   128 KBytes   524 Kbits/sec
.
.
.

Offline

#11 2014-04-27 05:01:38

lev
Member
Registered: 2010-05-05
Posts: 7

Re: [solved] TCP outbound bandwidth issues w/ OpenVPN

The mentioned commit 53d6471cef17262d3ad1c7ce8982a234244f68ec wasn't included into 3.14.2 ...
Please vote for the bug so the patch is included into the package.

Offline

#12 2014-05-01 22:58:57

tazmanian
Member
Registered: 2009-10-25
Posts: 39

Re: [solved] TCP outbound bandwidth issues w/ OpenVPN

Thanks for filing that bug report, lev. Looks like the patch made it into 3.14.2-1.

Offline

#13 2014-05-04 21:23:00

train_wreck
Member
Registered: 2011-10-22
Posts: 97

Re: [solved] TCP outbound bandwidth issues w/ OpenVPN

can confirm that it is indeed fixed in 3.14.2-1

Offline

#14 2014-05-10 11:37:28

AndyLee
Member
Registered: 2014-05-09
Posts: 2

Re: [solved] TCP outbound bandwidth issues w/ OpenVPN

I just registered to say I had this exact problem yesterday on Fedora's kernel-3.14.2-200.fc20.x86_64, so I'm not so sure the patch has been applied, or it has regressed again since...

I run a mysql multiple master system, replicating between my remote webserver and home network over openvpn for the last few years, and yesterday noticed that replication had inexplicably stalled in *one* direction.
The webserver slave (3.13.9-100 fc19) would connect to my local master (3.14.2-200 fc20), exec a few queries then stall. Mysql's processlist on the master would say "writing to net", then disappear after a minute or so.
Issuing another "stop slave; start slave;" would reconnect, do a few more queries then stall again. Not good!

I also transparently proxy mail from selected countries using iptables over the vpn to relieve the webserver of processing 10,000 spam messages a day. With short tcp sessions, that seemed to be ok. I would not have noticed if larger messages were timing out.
Suspecting a tcp problem, I tried doing some multimegabyte scp transfers over the vpn to the webserver. They stalled after receiving about 32k of data if initiated from the webserver, and stall after about a megabyte if initiated locally.

Playing with openvpn's  tun-mtu, fragment, and mssfix options had no effect, but I worked around replication temporarily on real ip using an ssh tunnel with port forwarding, but still had half a vpn!

After losing a lot of hair packet sniffing and overhauling iptables, I finally rebooted into the previous kernel 3.13.10-200, and all was sweet again.

Before upgrading the kernel again, I'd like to know for sure if this has been patched!

Offline

#15 2014-05-10 12:57:01

brebs
Member
Registered: 2007-04-03
Posts: 3,742

Re: [solved] TCP outbound bandwidth issues w/ OpenVPN

AndyLee wrote:

Fedora's kernel

You'll have to check Fedora's kernel patches.

Offline

#16 2014-05-17 13:41:54

AndyLee
Member
Registered: 2014-05-09
Posts: 2

Re: [solved] TCP outbound bandwidth issues w/ OpenVPN

brebs wrote:
AndyLee wrote:

Fedora's kernel

You'll have to check Fedora's kernel patches.

Thanks, appears to be fixed and working in kernel 3.14.3-200.

Cheers.

Offline

#17 2014-06-13 15:14:19

speculatrix
Member
Registered: 2014-06-10
Posts: 2

Re: [solved] TCP outbound bandwidth issues w/ OpenVPN

I got hit with this problem with the kernel 3.14 in the opensuse non-core repository. Took me quite a while to realise it wasn't my users' computers behind the linux firewall, but the firewall which I'd recently updated from 3.11. I booted back to the standard kernel and sanity was restored.

Offline

Pages: 1

Board footer

Powered by FluxBB