[SOLVED] Slow network problem

mahmutyildiz · 2017-03-31 12:08:18

Hi,

I am using archlinux for a few years and i am very satisfied. But i have very slow network problem in a few weeks. In my company this problem is only present in archlinux machines. There is no problem in windows machines and in an ubuntu machine.

When i telnet our proxy machine (10.251.75.170:8080), generally first few requests connect the proxy immediately. But in the next attempt, console remains "Trying 10.251.75.170..." and sometimes in 10 seconds sometimes after 1 minute the connection is established. When i applied the same test on other machines (windows and ubuntu), i didnt see any waiting. All machines are in the same network.

I watched the network traffic for ~5 seconds with tcpdump command when a "waiting .." state, but i didnt see anythink about the problem. I added the output of tcpdump command below.

I searched the internet about this problem, but i could not find anythink. What happened a few weeks ago and we started living this problem, i have no idea. I will be very happy if you can help.

tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on enp0s31f6, link-type EN10MB (Ethernet), capture size 262144 bytes
14:27:44.889058 IP 10.251.74.168.51275 > 239.255.255.250.ssdp: UDP, length 174
14:27:44.891710 IP my-archlinux.39713 > abcsrv01.mycompany.domain: 37314+ PTR? 250.255.255.239.in-addr.arpa. (46)
14:27:44.971213 IP abcsrv01.mycompany.domain > my-archlinux.39713: 37314 NXDomain 0/1/0 (103)
14:27:44.973472 IP my-archlinux.50286 > abcsrv01.mycompany.domain: 32219+ PTR? 168.74.251.10.in-addr.arpa. (44)
14:27:44.974049 IP abcsrv01.mycompany.domain > my-archlinux.50286: 32219 NXDomain 0/0/0 (44)
14:27:44.976266 IP my-archlinux.57385 > abcsrv01.mycompany.domain: 8501+ PTR? 1.75.251.10.in-addr.arpa. (42)
14:27:44.976843 IP abcsrv01.mycompany.domain > my-archlinux.57385: 8501* 1/0/0 PTR abcsrv01.ggm.bim. (72)
14:27:45.033584 IP 10.251.74.124.mdns > 224.0.0.251.mdns: 0 PTR (QM)? _googlecast._tcp.local. (40)
14:27:45.033632 IP6 fe80::921b:eff:fe8f:73d8.mdns > ff02::fb.mdns: 0 PTR (QM)? _googlecast._tcp.local. (40)
14:27:45.035490 IP my-archlinux.58058 > abcsrv01.mycompany.domain: 1837+ PTR? 251.0.0.224.in-addr.arpa. (42)
14:27:45.035869 IP 10.251.74.124.49122 > 239.255.255.250.ssdp: UDP, length 172
14:27:45.115290 IP abcsrv01.mycompany.domain > my-archlinux.58058: 1837 NXDomain 0/1/0 (99)
14:27:45.117599 IP my-archlinux.42244 > abcsrv01.mycompany.domain: 16162+ PTR? 124.74.251.10.in-addr.arpa. (44)
14:27:45.146785 ARP, Request who-has 10.251.74.4 tell 10.251.74.113, length 46
14:27:45.187806 IP my-archlinux.57787 > abcsrv01.mycompany.domain: 6115+ PTR? b.f.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.2.0.f.f.ip6.arpa. (90)
14:27:45.265118 IP my-archlinux.57123 > abcsrv01.mycompany.domain: 14669+ PTR? 4.74.251.10.in-addr.arpa. (42)
14:27:45.336631 IP abcsrv01.mycompany.domain > my-archlinux.57123: 14669 NXDomain 0/0/0 (42)
14:27:45.338976 IP my-archlinux.58783 > abcsrv01.mycompany.domain: 54621+ PTR? 113.74.251.10.in-addr.arpa. (44)
14:27:45.415623 IP6 fe80::c84e:66a6:1d31:e96d.dhcpv6-client > ff02::1:2.dhcpv6-server: dhcp6 solicit
14:27:45.424226 IP abcsrv01.mycompany.domain > my-archlinux.58783: 54621 NXDomain 0/0/0 (44)
14:27:45.425381 IP my-archlinux.42433 > abcsrv01.mycompany.domain: 23361+ PTR? 2.0.0.0.1.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.0.2.0.f.f.ip6.arpa. (90)
14:27:45.504795 IP abcsrv01.mycompany.domain > my-archlinux.42433: 23361 NXDomain 0/1/0 (154)
14:27:45.506100 IP my-archlinux.56593 > abcsrv01.mycompany.domain: 19259+ PTR? d.6.9.e.1.3.d.1.6.a.6.6.e.4.8.c.0.0.0.0.0.0.0.0.0.0.0.0.0.8.e.f.ip6.arpa. (90)
14:27:45.610500 IP abcsrv01.mycompany.domain > my-archlinux.56593: 19259 NXDomain 0/1/0 (154)
14:27:45.721646 IP6 fe80::70a6:37a6:6665:effb.dhcpv6-client > ff02::1:2.dhcpv6-server: dhcp6 solicit
14:27:45.723506 IP my-archlinux.58872 > abcsrv01.mycompany.domain: 35644+ PTR? b.f.f.e.5.6.6.6.6.a.7.3.6.a.0.7.0.0.0.0.0.0.0.0.0.0.0.0.0.8.e.f.ip6.arpa. (90)
14:27:45.828712 IP abcsrv01.mycompany.domain > my-archlinux.58872: 35644 NXDomain 0/1/0 (154)
14:27:46.038096 STP 802.1d, Config, Flags [none], bridge-id c017.34:6f:90:03:43:00.800c, length 43
14:27:46.197930 ARP, Request who-has 10.6.50.254 tell 10.6.50.62, length 46
14:27:46.199778 IP my-archlinux.41612 > abcsrv01.mycompany.domain: 20350+ PTR? 254.50.6.10.in-addr.arpa. (42)
14:27:46.261196 IP my-archlinux.36326 > 10.251.75.170.http-alt: Flags [S], seq 4248248199, win 29200, options [mss 1460,sackOK,TS val 3207389997 ecr 0,nop,wscale 7], length 0
14:27:46.271205 IP abcsrv01.mycompany.domain > my-archlinux.41612: 20350 NXDomain 0/0/0 (42)
14:27:46.273549 IP my-archlinux.38858 > abcsrv01.mycompany.domain: 6573+ PTR? 62.50.6.10.in-addr.arpa. (41)
14:27:46.341579 IP abcsrv01.mycompany.domain > my-archlinux.38858: 6573 NXDomain 0/0/0 (41)
14:27:46.343971 IP my-archlinux.42827 > abcsrv01.mycompany.domain: 60750+ PTR? 170.75.251.10.in-addr.arpa. (44)
14:27:46.344492 IP abcsrv01.mycompany.domain > my-archlinux.42827: 60750 NXDomain* 0/1/0 (129)
14:27:46.554240 IP 10.251.74.152.netbios-ns > 10.251.74.255.netbios-ns: NBT UDP PACKET(137): QUERY; REQUEST; BROADCAST
14:27:46.556151 IP my-archlinux.50549 > abcsrv01.mycompany.domain: 40208+ PTR? 255.74.251.10.in-addr.arpa. (44)
14:27:46.634725 IP abcsrv01.mycompany.domain > my-archlinux.50549: 40208 NXDomain 0/0/0 (44)
14:27:46.637098 IP my-archlinux.35266 > abcsrv01.mycompany.domain: 2031+ PTR? 152.74.251.10.in-addr.arpa. (44)
14:27:46.710064 IP abcsrv01.mycompany.domain > my-archlinux.35266: 2031 NXDomain 0/0/0 (44)
14:27:46.713157 IP6 fe80::7d33:855a:116c:e6ff.dhcpv6-client > ff02::1:2.dhcpv6-server: dhcp6 solicit
14:27:46.715063 IP my-archlinux.42830 > abcsrv01.mycompany.domain: 64266+ PTR? f.f.6.e.c.6.1.1.a.5.5.8.3.3.d.7.0.0.0.0.0.0.0.0.0.0.0.0.0.8.e.f.ip6.arpa. (90)
14:27:46.818625 IP abcsrv01.mycompany.domain > my-archlinux.42830: 64266 NXDomain 0/1/0 (154)
14:27:46.842492 IP6 fe80::3430:7c6b:70f4:ca7c.dhcpv6-client > ff02::1:2.dhcpv6-server: dhcp6 solicit
14:27:46.844289 IP my-archlinux.54739 > abcsrv01.mycompany.domain: 3993+ PTR? c.7.a.c.4.f.0.7.b.6.c.7.0.3.4.3.0.0.0.0.0.0.0.0.0.0.0.0.0.8.e.f.ip6.arpa. (90)
14:27:46.950647 IP abcsrv01.mycompany.domain > my-archlinux.54739: 3993 NXDomain 0/1/0 (154)
14:27:46.963101 IP 10.251.74.116.netbios-ns > 10.251.74.255.netbios-ns: NBT UDP PACKET(137): QUERY; REQUEST; BROADCAST
14:27:46.964996 IP my-archlinux.51545 > abcsrv01.mycompany.domain: 39423+ PTR? 116.74.251.10.in-addr.arpa. (44)
14:27:46.965458 IP abcsrv01.mycompany.domain > my-archlinux.51545: 39423 NXDomain 0/0/0 (44)
14:27:47.265584 IP my-archlinux.36326 > 10.251.75.170.http-alt: Flags [S], seq 4248248199, win 29200, options [mss 1460,sackOK,TS val 3207390299 ecr 0,nop,wscale 7], length 0
14:27:47.303411 IP 10.251.74.152.netbios-ns > 10.251.74.255.netbios-ns: NBT UDP PACKET(137): QUERY; REQUEST; BROADCAST
14:27:47.323991 IP6 fe80::20af:246:f049:d5fe.dhcpv6-client > ff02::1:2.dhcpv6-server: dhcp6 solicit
14:27:47.325931 IP my-archlinux.54143 > abcsrv01.mycompany.domain: 38715+ PTR? e.f.5.d.9.4.0.f.6.4.2.0.f.a.0.2.0.0.0.0.0.0.0.0.0.0.0.0.0.8.e.f.ip6.arpa. (90)
14:27:47.438428 IP abcsrv01.mycompany.domain > my-archlinux.54143: 38715 NXDomain 0/1/0 (154)
14:27:47.712750 IP 10.251.74.116.netbios-ns > 10.251.74.255.netbios-ns: NBT UDP PACKET(137): QUERY; REQUEST; BROADCAST
14:27:47.721763 IP6 fe80::70a6:37a6:6665:effb.dhcpv6-client > ff02::1:2.dhcpv6-server: dhcp6 solicit
14:27:48.039498 STP 802.1d, Config, Flags [none], bridge-id c017.34:6f:90:03:43:00.800c, length 43
14:27:48.125659 IP 10.251.74.126.55849 > 239.255.255.250.ssdp: UDP, length 174
14:27:48.127541 IP my-archlinux.60983 > abcsrv01.mycompany.domain: 44323+ PTR? 126.74.251.10.in-addr.arpa. (44)
14:27:48.128053 IP abcsrv01.mycompany.domain > my-archlinux.60983: 44323 NXDomain 0/0/0 (44)
14:27:48.178495 IP 10.251.74.152.netbios-ns > 10.251.74.255.netbios-ns: NBT UDP PACKET(137): QUERY; REQUEST; BROADCAST
14:27:48.462870 IP 10.251.74.116.netbios-ns > 10.251.74.255.netbios-ns: NBT UDP PACKET(137): QUERY; REQUEST; BROADCAST
14:27:48.634033 IP 10.251.74.211.netbios-ns > 10.251.74.255.netbios-ns: NBT UDP PACKET(137): QUERY; REQUEST; BROADCAST
14:27:48.636029 IP my-archlinux.39868 > abcsrv01.mycompany.domain: 16267+ PTR? 211.74.251.10.in-addr.arpa. (44)
14:27:48.711745 IP abcsrv01.mycompany.domain > my-archlinux.39868: 16267 NXDomain 0/0/0 (44)
14:27:48.836654 ARP, Request who-has 10.251.74.159 tell 10.251.74.143, length 46
14:27:48.838563 IP my-archlinux.45268 > abcsrv01.mycompany.domain: 47042+ PTR? 159.74.251.10.in-addr.arpa. (44)
14:27:48.848945 ARP, Request who-has 10.251.74.143 tell 10.251.74.159, length 46
14:27:48.904846 IP abcsrv01.mycompany.domain > my-archlinux.45268: 47042 NXDomain 0/0/0 (44)
14:27:48.907214 IP my-archlinux.51807 > abcsrv01.mycompany.domain: 56312+ PTR? 143.74.251.10.in-addr.arpa. (44)
14:27:48.907712 IP abcsrv01.mycompany.domain > my-archlinux.51807: 56312 NXDomain 0/0/0 (44)
14:27:49.126512 IP 10.251.74.126.55849 > 239.255.255.250.ssdp: UDP, length 174
14:27:49.393179 IP 10.251.74.211.netbios-ns > 10.251.74.255.netbios-ns: NBT UDP PACKET(137): QUERY; REQUEST; BROADCAST
14:27:49.478955 IP my-archlinux.36326 > 10.251.75.170.http-alt: Flags [S], seq 4248248199, win 29200, options [mss 1460,sackOK,TS val 3207390963 ecr 0,nop,wscale 7], length 0
14:27:50.039135 STP 802.1d, Config, Flags [none], bridge-id c017.34:6f:90:03:43:00.800c, length 43
14:27:50.126548 IP 10.251.74.126.55849 > 239.255.255.250.ssdp: UDP, length 174
14:27:50.143748 IP 10.251.74.211.netbios-ns > 10.251.74.255.netbios-ns: NBT UDP PACKET(137): QUERY; REQUEST; BROADCAST
14:27:50.200850 ARP, Request who-has 10.6.50.254 tell 10.6.50.62, length 46
14:27:50.254959 IP6 fe80::8460:4e83:7cfa:d1fd.dhcpv6-client > ff02::1:2.dhcpv6-server: dhcp6 solicit
14:27:50.256860 IP my-archlinux.59291 > abcsrv01.mycompany.domain: 16176+ PTR? d.f.1.d.a.f.c.7.3.8.e.4.0.6.4.8.0.0.0.0.0.0.0.0.0.0.0.0.0.8.e.f.ip6.arpa. (90)
14:27:50.369187 IP abcsrv01.mycompany.domain > my-archlinux.59291: 16176 NXDomain 0/1/0 (154)
14:27:50.686306 IP 10.251.74.203.61232 > 239.203.13.64.sonardata: UDP, length 31
14:27:50.687822 ARP, Request who-has 10.251.74.203 tell 10.251.74.210, length 46
14:27:50.688144 IP my-archlinux.54541 > abcsrv01.mycompany.domain: 8080+ PTR? 64.13.203.239.in-addr.arpa. (44)
14:27:50.688224 ARP, Request who-has 10.251.74.203 tell 10.251.74.182, length 46
14:27:50.688539 ARP, Request who-has 10.251.74.203 tell 10.251.74.206, length 46
14:27:50.689485 ARP, Request who-has 10.251.74.203 tell 10.251.74.138, length 46
14:27:50.699759 ARP, Request who-has 10.251.74.230 tell 10.251.74.203, length 46
14:27:50.868048 IP my-archlinux.39812 > abcsrv01.mycompany.domain: 44884+ PTR? 210.74.251.10.in-addr.arpa. (44)
14:27:50.948735 IP my-archlinux.53709 > abcsrv01.mycompany.domain: 3518+ PTR? 182.74.251.10.in-addr.arpa. (44)
14:27:50.949309 IP abcsrv01.mycompany.domain > my-archlinux.53709: 3518 NXDomain 0/0/0 (44)
14:27:50.951624 IP my-archlinux.39559 > abcsrv01.mycompany.domain: 22871+ PTR? 206.74.251.10.in-addr.arpa. (44)
14:27:50.954228 IP my-archlinux.49127 > abcsrv01.mycompany.domain: 60908+ PTR? 138.74.251.10.in-addr.arpa. (44)
14:27:51.029795 IP my-archlinux.56966 > abcsrv01.mycompany.domain: 4491+ PTR? 230.74.251.10.in-addr.arpa. (44)
14:27:51.126568 IP 10.251.74.126.55849 > 239.255.255.250.ssdp: UDP, length 174
14:27:51.722002 IP6 fe80::70a6:37a6:6665:effb.dhcpv6-client > ff02::1:2.dhcpv6-server: dhcp6 solicit
14:27:52.065036 STP 802.1d, Config, Flags [none], bridge-id c017.34:6f:90:03:43:00.800c, length 43
14:27:52.275011 IP 10.251.74.117.39826 > 239.255.255.250.ssdp: UDP, length 171
14:27:52.276816 IP my-archlinux.51561 > abcsrv01.mycompany.domain: 39355+ PTR? 117.74.251.10.in-addr.arpa. (44)
14:27:52.277389 IP abcsrv01.mycompany.domain > my-archlinux.51561: 39355 NXDomain 0/0/0 (44)
14:27:52.478732 IP6 fe80::9d7e:b334:8bcb:4350.dhcpv6-client > ff02::1:2.dhcpv6-server: dhcp6 solicit
14:27:52.480688 IP my-archlinux.36418 > abcsrv01.mycompany.domain: 53134+ PTR? 0.5.3.4.b.c.b.8.4.3.3.b.e.7.d.9.0.0.0.0.0.0.0.0.0.0.0.0.0.8.e.f.ip6.arpa. (90)
14:27:52.595242 IP abcsrv01.mycompany.domain > my-archlinux.36418: 53134 NXDomain 0/1/0 (154)
14:27:52.885433 ARP, Request who-has 10.251.74.184 tell 10.251.74.220, length 46
14:27:52.887355 IP my-archlinux.53818 > abcsrv01.mycompany.domain: 22000+ PTR? 184.74.251.10.in-addr.arpa. (44)
14:27:52.887438 ARP, Request who-has 10.251.74.198 tell 10.251.74.220, length 46
14:27:52.887864 IP abcsrv01.mycompany.domain > my-archlinux.53818: 22000 NXDomain 0/0/0 (44)
14:27:52.889881 ARP, Request who-has 10.251.74.15 tell 10.251.74.220, length 46
14:27:52.889883 IP my-archlinux.60684 > abcsrv01.mycompany.domain: 47957+ PTR? 220.74.251.10.in-addr.arpa. (44)
14:27:52.892080 ARP, Request who-has 10.251.74.172 tell 10.251.74.220, length 46
14:27:52.965424 IP my-archlinux.56340 > abcsrv01.mycompany.domain: 56696+ PTR? 198.74.251.10.in-addr.arpa. (44)
14:27:53.041304 IP abcsrv01.mycompany.domain > my-archlinux.56340: 56696 NXDomain 0/0/0 (44)
14:27:53.043761 IP my-archlinux.39143 > abcsrv01.mycompany.domain: 43860+ PTR? 15.74.251.10.in-addr.arpa. (43)
14:27:53.112957 IP abcsrv01.mycompany.domain > my-archlinux.39143: 43860 NXDomain 0/0/0 (43)
14:27:53.115330 IP my-archlinux.45670 > abcsrv01.mycompany.domain: 23004+ PTR? 172.74.251.10.in-addr.arpa. (44)
14:27:53.189027 IP abcsrv01.mycompany.domain > my-archlinux.45670: 23004 NXDomain 0/0/0 (44)
14:27:53.275872 IP 10.251.74.117.39826 > 239.255.255.250.ssdp: UDP, length 171
14:27:53.555295 IP 10.251.75.171.http-alt > my-archlinux.41316: Flags [F.], seq 420646072, ack 1187298283, win 139, options [nop,nop,TS val 336562385 ecr 3698919810], length 0
14:27:53.557191 IP my-archlinux.36376 > abcsrv01.mycompany.domain: 58855+ PTR? 171.75.251.10.in-addr.arpa. (44)
14:27:53.557700 IP abcsrv01.mycompany.domain > my-archlinux.36376: 58855 NXDomain* 0/1/0 (129)
14:27:53.595510 IP my-archlinux.41316 > 10.251.75.171.http-alt: Flags [.], ack 1, win 301, options [nop,nop,TS val 3698955920 ecr 336562385], length 0
^C
122 packets captured
180 packets received by filter
58 packets dropped by kernel

Last edited by mahmutyildiz (2017-04-04 13:24:57)

ewaller · 2017-03-31 15:02:42

Welcome to the Arch Linux Forums.

Wired or Wireless?
What are the output of ip addr and ip route on the Arch machines and on the Ubuntu machine?

mahmutyildiz · 2017-04-03 06:43:20

Thank you ewaller.

My friend who uses archlinux downgraded his arch kernel to 4.9.11-1-ARCH . And now he has no problem. "uname -r" output is 4.10.6-1-ARCH in my arch.

My ip addr output:

1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
2: enp0s31f6: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UP group default qlen 1000
    link/ether 90:1b:0e:8f:73:f5 brd ff:ff:ff:ff:ff:ff
    inet 10.251.74.188/24 brd 10.251.74.255 scope global enp0s31f6
       valid_lft forever preferred_lft forever
3: docker0: <BROADCAST,MULTICAST> mtu 1500 qdisc noqueue state DOWN group default 
    link/ether 02:42:17:58:c7:35 brd ff:ff:ff:ff:ff:ff
    inet 172.17.0.1/16 scope global docker0
       valid_lft forever preferred_lft forever
4: br-caddc1748097: <BROADCAST,MULTICAST> mtu 1500 qdisc noqueue state DOWN group default 
    link/ether 02:42:d3:e5:68:91 brd ff:ff:ff:ff:ff:ff
    inet 172.18.0.1/16 scope global br-caddc1748097
       valid_lft forever preferred_lft forever

My ip route output:

default via 10.251.74.254 dev enp0s31f6 src 10.251.74.188 metric 202 
10.251.74.0/24 dev enp0s31f6 proto kernel scope link src 10.251.74.188 metric 202

My friend (4.9.11-1-ARCH)'s ipp addr output:

1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host 
       valid_lft forever preferred_lft forever
2: enp0s31f6: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UP group default qlen 1000
    link/ether 90:1b:0e:8f:73:d8 brd ff:ff:ff:ff:ff:ff
    inet 10.251.74.124/24 brd 10.251.74.255 scope global enp0s31f6
       valid_lft forever preferred_lft forever
    inet6 fe80::921b:eff:fe8f:73d8/64 scope link 
       valid_lft forever preferred_lft forever

My friend (4.9.11-1-ARCH)'s ipp addr output:

default via 10.251.74.254 dev enp0s31f6 proto static metric 202 
10.251.74.0/24 dev enp0s31f6 proto kernel scope link src 10.251.74.124 metric 202

ewaller · 2017-04-03 14:21:40

That all looks rational. My suggestions would be to try to see if downgrading your kernel helps. You might also try it without a docker instance.

mahmutyildiz · 2017-04-04 13:19:38

One of my friends finally found a solution. After following command our problem is solved.

sysctl -w net.ipv4.tcp_timestamps=0

But we are still curious about the cause of the problem.

Arch Linux

#1 2017-03-31 12:08:18

[SOLVED] Slow network problem

#2 2017-03-31 15:02:42

Re: [SOLVED] Slow network problem

#3 2017-04-03 06:43:20

Re: [SOLVED] Slow network problem

#4 2017-04-03 14:21:40

Re: [SOLVED] Slow network problem

#5 2017-04-04 13:19:38

Re: [SOLVED] Slow network problem

Board footer