[SOLVED] Need advice for troubleshooting TCP performance problem

Markus.N2 · 2019-03-16 19:28:38

I have two PC's running ArchLinux, one is the "multimedia / communication central" (MCC) running 24/7 and the other one is the "work PC" (WPC). Mostly, I copy data from the MCC to the WPC (e.g. recorded TV shows), but recently I copied greater amounts of data in the other direction and then I noticed a network performance issue.

I did some performance testing with a laptop borrowed from a friend to check which PC is causing the issue. The testing was done using netio. All computers as well as the router have gigabit ethernet connections. These are the test results:

TCP WPC -> MCC: 13 MByte/s
TCP MCC -> WPC: 112 MByte/s

TCP Laptop -> MCC: 13 MByte/s
TCP MCC -> Laptop: 112 MByte/s

TCP Laptop -> WPC: 112 MByte/s
TCP WPC -> Laptop: 112 MByte/s

UDP in any direction: 112 MByte/s

The data rate was independent of client/server role and varied only very little with the packet size.

Conclusion so far: The (almost) constant UDP speed rules out a hardware problem and is a strong indication that it's not the router's fault.

But now I'm stuck. I have no idea what to check on the MCC.

Any ideas ?

Last edited by Markus.N2 (2019-03-22 22:07:57)

Bevan · 2019-03-16 19:43:19

One thing you may look at are the different TCP offloading features. Sometimes these don't work as expected. You can view and control them using ethtool and try if enabling / disabling certain offloading features helps:
https://access.redhat.com/documentation … c-offloads

seth · 2019-03-16 22:27:16

Is ECN still enabled on the "MCC"?

sysctl net.ipv4.tcp_ecn

Maniaxx · 2019-03-17 01:05:01

13MB/s looks suspiciously close to 100Mbit/s. Did you check if its really connected with 1Gbit/s? Maybe its an negotiation/fallback issue. Can you put a different NIC into MCC for testing purpose? A short Wireshark capture could help as well. Maybe something useful shows up (heavy fragmentation, dropped packets or something).

Arch had TCP BBR congestion enabled by default for some time (later revoked and set back to cubic). Maybe its still enabled somewhere. It gave me a hardcap as well under several conditions.

$ sysctl net.ipv4.tcp_congestion_control
net.ipv4.tcp_congestion_control = cubic

Last edited by Maniaxx (2019-03-17 01:11:15)

Markus.N2 · 2019-03-20 20:31:51

Hi folks,
thanx for the replies so far.

@Bevan
I've followed the link you provided and I'm still trying to learn what all of these options mean. Comparing the output of ethtool -k on both of my PC's, there are lots of differences. I'm currently working through them.

@seth
The setting is the same on both of my PC's:

$ sysctl net.ipv4.tcp_ecn
net.ipv4.tcp_ecn = 2

@Maniaxx
The setting is the same on both of my PC's:

$ sysctl net.ipv4.tcp_congestion_control
net.ipv4.tcp_congestion_control = cubic

Maniaxx · 2019-03-21 04:17:40

Please provide from MCC:

$ cat /proc/cpuinfo

What exactly is the MCC? SBC (single board computer), embedded, atx-pc...?
Check cpu workload with 'top' or similar on MCC during transfer.

Markus.N2 · 2019-03-21 07:08:06

MCC is an energy-saving PC with a 10 Watt CPU:
ASUS J3455M-E (mATX) with a Celeron J3455.

It's running 24/7, so the low-Watt CPU is important.

Edit:
In fact, most of the differences shown by ethtool are marked as fixed.

Last edited by Markus.N2 (2019-03-21 07:10:07)

seth · 2019-03-21 08:03:08

I would have assumed that netio does not rely on any disk IO, but maybe this is relevant: can you "wget -O /dev/null" into the MCC at full speed, but "wget -O /path/to/normal/storage" drops to the known limits?
(You could setup https://www.archlinux.org/packages/extr … /lighttpd/ on the WPC to feed it)

Markus.N2 · 2019-03-21 12:18:01

I will try this later ... but it's not very likely that it makes a difference: Both PC's have their root and /home filesystems on SSDs. However, there are additional HDDs (2.5" on the MCC and 3.5" on the WPC) that have different speed of course.
The MCC only holds videos recorded from TV (via the VDR software) on the additional HDD, the WPC holds the videos after cutting out the commercials on the HDD.

Maniaxx · 2019-03-21 15:06:19

MMC:
Cpu workload ok during transfer? PCIe cards? What NIC exactly?

Markus.N2 · 2019-03-21 15:15:24

Yes, CPU load is OK. NIC is Realtek RTL8111H.

Please remember that the issue is asymetric: it occurs only when sending to the MCC, not when receiving from the MCC.

seth · 2019-03-21 15:25:14

r8169 or r8168?
And does it make a difference?

Markus.N2 · 2019-03-21 15:45:31

Well ... I think the easiest and most reliable way to provide complete and correct infos is a link to the manufacturer's page for this board:
https://www.asus.com/Motherboards/J3455 … fications/

seth · 2019-03-21 15:51:10

Those are different kernel modules, both will likely support your chip (lspci is better than some flyer, btw)

r8169 is part of the kernel, r8168 is here

I assume this means (lspci -v) r8169 and would suggest to try r8168 (esp. if lspci has that as 2nd part of the device ID)

Maniaxx · 2019-03-21 16:58:27

If cpu doesn't come close to 25% it should be ok (also check sending side).

Take a quick look at the data flow with wireshark. Look out for massive re-transmissions and errors. Also from the sending side.

Last edited by Maniaxx (2019-03-21 16:59:24)

Markus.N2 · 2019-03-21 19:50:12

@seth
lspci says:

Ethernet controller: Realtek Semiconductor Co., Ltd. RTL8111/8168/8411 PCI Express Gigabit Ethernet Controller (rev 15)

@Maniaxx
CPU load during test is below 10%

Last edited by Markus.N2 (2019-03-21 19:51:08)

seth · 2019-03-21 21:00:52

So please try the r8168 kernel module.

dmartins · 2019-03-21 22:07:43

Try to rule out any physical issues: try a different ethernet cable between your switch/router and MCC. Connect MCC to a different port on the switch/router.

As Maniaxx has suggested, analysis of packet captures run on both client and server during a file transfer or netio test might identify issues with tcp retransmissions. I'm used to working on remote servers so I tend to use tcpdump for capturing network traffic, and wireshark to analyze the captures. Wireshark can do the capture part as well though. Here's how you'd do it with tcpdump:

(replace eth0 with your network interface name)
On the server (MCC)

Start netio in server mode
# ./linux-amd64 -s
In another window, start tcpdump
# tcpdump -i eth0 -v -w /tmp/server.pcap port 18767

On the client (WPC)

Start tcpdump
# tcpdump -i eth0 -v -w /tmp/client.pcap port 18767
In another window, start netio
# ./linux-amd64 -t -b1k <server IP>

These pcap files can be viewed in wireshark. Feel free to compress them and share them online if you aren't sure what you're looking for. (https://send.firefox.com/ is good for this)

This redhat document has a good explanation of how the kernel receives packets, how to identify the bottlenecks, and the tuning options available. It is a few years old but looks like it is still relevant.
Start with checking ethtool -S and netstat -s for anything related to Rx errors on MCC.

Markus.N2 · 2019-03-21 22:33:10

Whow, that helped.
Now I have the same speed in TCP and UDP, tx and rx.

Thanx a lot.

CarbonChauvinist · 2019-03-22 01:31:13

Markus.N2 wrote:

Whow, that helped.

What helped?

GrimBandito · 2019-03-23 22:12:24

On your MCC pc, check the network connection settings... is it using gigabit or 100megabit or 10 megabit? Is it full duplex or half duplex. Whats the mtu? These settings can easily be misconfigured in network manager.

seth · 2019-03-23 22:34:37

@GrimBandito, though we don't exactly know "what" ultimately helped, the problem was asymmetric and limited to TCP.

Maniaxx · 2019-03-24 01:08:26

Probably the r8168<>r8169 driver switch. There are so many hw revisions of this chips. Reports are all mixed up on the web.

seth · 2019-03-24 06:40:04

I guess so as well - but that's not knowing ;-)

Arch Linux

#1 2019-03-16 19:28:38

[SOLVED] Need advice for troubleshooting TCP performance problem

#2 2019-03-16 19:43:19

Re: [SOLVED] Need advice for troubleshooting TCP performance problem

#3 2019-03-16 22:27:16

Re: [SOLVED] Need advice for troubleshooting TCP performance problem

#4 2019-03-17 01:05:01

Re: [SOLVED] Need advice for troubleshooting TCP performance problem

#5 2019-03-20 20:31:51

Re: [SOLVED] Need advice for troubleshooting TCP performance problem

#6 2019-03-21 04:17:40

Re: [SOLVED] Need advice for troubleshooting TCP performance problem

#7 2019-03-21 07:08:06

Re: [SOLVED] Need advice for troubleshooting TCP performance problem

#8 2019-03-21 08:03:08

Re: [SOLVED] Need advice for troubleshooting TCP performance problem

#9 2019-03-21 12:18:01

Re: [SOLVED] Need advice for troubleshooting TCP performance problem

#10 2019-03-21 15:06:19

Re: [SOLVED] Need advice for troubleshooting TCP performance problem

#11 2019-03-21 15:15:24

Re: [SOLVED] Need advice for troubleshooting TCP performance problem

#12 2019-03-21 15:25:14

Re: [SOLVED] Need advice for troubleshooting TCP performance problem

#13 2019-03-21 15:45:31

Re: [SOLVED] Need advice for troubleshooting TCP performance problem

#14 2019-03-21 15:51:10

Re: [SOLVED] Need advice for troubleshooting TCP performance problem

#15 2019-03-21 16:58:27

Re: [SOLVED] Need advice for troubleshooting TCP performance problem

#16 2019-03-21 19:50:12

Re: [SOLVED] Need advice for troubleshooting TCP performance problem

#17 2019-03-21 21:00:52

Re: [SOLVED] Need advice for troubleshooting TCP performance problem

#18 2019-03-21 22:07:43

Re: [SOLVED] Need advice for troubleshooting TCP performance problem

#19 2019-03-21 22:33:10

Re: [SOLVED] Need advice for troubleshooting TCP performance problem

#20 2019-03-22 01:31:13

Re: [SOLVED] Need advice for troubleshooting TCP performance problem

#21 2019-03-23 22:12:24

Re: [SOLVED] Need advice for troubleshooting TCP performance problem

#22 2019-03-23 22:34:37

Re: [SOLVED] Need advice for troubleshooting TCP performance problem

#23 2019-03-24 01:08:26

Re: [SOLVED] Need advice for troubleshooting TCP performance problem

#24 2019-03-24 06:40:04

Re: [SOLVED] Need advice for troubleshooting TCP performance problem

Board footer