You are not logged in.
Hello all,
I have two Arch Linux systems, a home server and my private laptop. I just programmed a small program to automate my Pioneer/Onkyo entertainment system.
The program opens a TCP socket connection to my radio, sends commands and receives status updates.
Strangely, the program works fine on a Windows laptop and on my Arch laptop, but fails fast when it runs on my home server: The socket is opened and I receive data for a few seconds, before there is no more data. When I try to send a new command, the program fails because of a broken pipe. When I used strace, I saw timeouts everywhere.
As the program works fine on my one Arch machine but not on the other, it looks to me like a system configuration problem. If it was a firewall problem, I would expect that the connection is prevented but not dropped.
What are good places to look for wrong configurations?
Thank you very much in advance!
Last edited by grisu (2020-12-09 13:00:18)
Offline
I don't have an immediate answers, but I do have some questions that may give more information and therefore a better chance to solve the issue.
1) Between your Arch systems and the radio, are they on wifi or ethernet?
2) Are all the systems on the same subnet?
3) What language is your program written in? Can you include any log information?
"Oh, they have the internet on computers now."
Offline
1) The device with the problems is connected via ethernet cable, so is the radio. Both notebooks that don't have connection problems work on both ethernet cable and wi-fi.
2) All devices are in the same subnet.
3) The original program was written in C# and running on .NET Core 3.1. I just rewrote a python script that reproduces the problem as well.
#!/usr/bin/env python3
import socket
with socket.socket(socket.AF_INET, socket.SOCK_STREAM) as s:
s.connect(('192.168.1.59', 60128))
while True:
data = s.recv(1024)
if not data:
break
print('Received', repr(data))Running strace on this script:
socket(AF_INET, SOCK_STREAM|SOCK_CLOEXEC, IPPROTO_IP) = 3
connect(3, {sa_family=AF_INET, sin_port=htons(60128), sin_addr=inet_addr("192.168.1.59")}, 16) = 0
recvfrom(3, "ISCP\0\0\0\20\0\0\0+\1\0\0\0!1NJA2-http://19"..., 1024, 0, NULL, NULL) = 59
write(1, "Received b'ISCP\\x00\\x00\\x00\\x10\\"..., 110Received b'ISCP\x00\x00\x00\x10\x00\x00\x00+\x01\x00\x00\x00!1NJA2-http://192.168.1.59/album_art.cgi\x1a\r\n'
) = 110
recvfrom(3, "ISCP\0\0\0\20\0\0\0\n\1\0\0\0!1MVL09\32\r\n", 1024, 0, NULL, NULL) = 26
write(1, "Received b'ISCP\\x00\\x00\\x00\\x10\\"..., 78Received b'ISCP\x00\x00\x00\x10\x00\x00\x00\n\x01\x00\x00\x00!1MVL09\x1a\r\n'
) = 78
recvfrom(3, "ISCP\0\0\0\20\0\0\0$\1\0\0\0!1FLD566F6C756D6"..., 1024, 0, NULL, NULL) = 52
write(1, "Received b'ISCP\\x00\\x00\\x00\\x10\\"..., 103Received b'ISCP\x00\x00\x00\x10\x00\x00\x00$\x01\x00\x00\x00!1FLD566F6C756D652020202020202039\x1a\r\n'
) = 103
recvfrom(3, "ISCP\0\0\0\20\0\0\0\n\1\0\0\0!1MVL0A\32\r\n", 1024, 0, NULL, NULL) = 26
write(1, "Received b'ISCP\\x00\\x00\\x00\\x10\\"..., 78Received b'ISCP\x00\x00\x00\x10\x00\x00\x00\n\x01\x00\x00\x00!1MVL0A\x1a\r\n'
) = 78
recvfrom(3, "ISCP\0\0\0\20\0\0\0$\1\0\0\0!1FLD566F6C756D6"..., 1024, 0, NULL, NULL) = 52
write(1, "Received b'ISCP\\x00\\x00\\x00\\x10\\"..., 103Received b'ISCP\x00\x00\x00\x10\x00\x00\x00$\x01\x00\x00\x00!1FLD566F6C756D652020202020203130\x1a\r\n'
) = 103
recvfrom(3, "ISCP\0\0\0\20\0\0\0\n\1\0\0\0!1LMD0A\32\r\n", 1024, 0, NULL, NULL) = 26
write(1, "Received b'ISCP\\x00\\x00\\x00\\x10\\"..., 78Received b'ISCP\x00\x00\x00\x10\x00\x00\x00\n\x01\x00\x00\x00!1LMD0A\x1a\r\n'
) = 78
brk(0x5586077df000) = 0x5586077df000
recvfrom(3, "ISCP\0\0\0\20\0\0\0\n\1\0\0\0!1MOT00\32\r\n", 1024, 0, NULL, NULL) = 26
write(1, "Received b'ISCP\\x00\\x00\\x00\\x10\\"..., 78Received b'ISCP\x00\x00\x00\x10\x00\x00\x00\n\x01\x00\x00\x00!1MOT00\x1a\r\n'
) = 78
recvfrom(3, "ISCP\0\0\0\20\0\0\0\n\1\0\0\0!1RAS00\32\r\n", 1024, 0, NULL, NULL) = 26
write(1, "Received b'ISCP\\x00\\x00\\x00\\x10\\"..., 78Received b'ISCP\x00\x00\x00\x10\x00\x00\x00\n\x01\x00\x00\x00!1RAS00\x1a\r\n'
) = 78
recvfrom(3, "", 1024, 0, NULL, NULL) = 0
close(3) = 0
rt_sigaction(SIGINT, {sa_handler=SIG_DFL, sa_mask=[], sa_flags=SA_RESTORER, sa_restorer=0x7f397c3bc6a0}, {sa_handler=0x7f397c61c84a, sa_mask=[], sa_flags=SA_RESTORER, sa_restorer=0x7f397c3bc6a0}, 8) = 0
exit_group(0) = ?
+++ exited with 0 +++If there is a better way to produce helpful logs, I'm happy to provide those.
Offline
~ ·> sysctl net.ipv4.tcp_retries1
~ ·> sysctl net.ipv4.tcp_retries2
~ ·> sysctl net.ipv4.tcp_keepalive_time
~ ·> sysctl net.ipv4.tcp_keepalive_intvl
~ ·> sysctl net.ipv4.tcp_keepalive_probes
~ ·> sysctl net.ipv4.tcp_syn_retriesLast edited by seth (2020-12-08 08:25:26)
Offline
net.ipv4.tcp_retries1 = 3
net.ipv4.tcp_retries2 = 15
net.ipv4.tcp_keepalive_time = 7200
net.ipv4.tcp_keepalive_intvl = 75
net.ipv4.tcp_keepalive_probes = 9
net.ipv4.tcp_syn_retries = 6Those values are the same on both systems, working and not working.
These are the values that are different or don't exist on the working system:
net.bridge.bridge-nf-call-arptables = 1
net.bridge.bridge-nf-call-ip6tables = 1
net.bridge.bridge-nf-call-iptables = 1
net.bridge.bridge-nf-filter-pppoe-tagged = 0
net.bridge.bridge-nf-filter-vlan-tagged = 0
net.bridge.bridge-nf-pass-vlan-input-dev = 0
net.ipv4.conf.all.accept_redirects = 0
net.ipv4.conf.all.forwarding = 1
net.ipv4.conf.all.proxy_arp = 1
net.ipv4.conf.all.send_redirects = 0
net.ipv4.conf.default.forwarding = 1
net.ipv4.ip_forward = 1
net.ipv4.tcp_max_orphans = 32768
net.ipv4.tcp_max_syn_backlog = 512
net.ipv4.tcp_max_tw_buckets = 32768
net.ipv4.tcp_mem = 92967 123956 185934
net.ipv4.udp_mem = 185934 247912 371868
net.netfilter.nf_conntrack_acct = 0
net.netfilter.nf_conntrack_buckets = 65536
net.netfilter.nf_conntrack_checksum = 1
net.netfilter.nf_conntrack_count = 696
net.netfilter.nf_conntrack_dccp_loose = 1
net.netfilter.nf_conntrack_dccp_timeout_closereq = 64
net.netfilter.nf_conntrack_dccp_timeout_closing = 64
net.netfilter.nf_conntrack_dccp_timeout_open = 43200
net.netfilter.nf_conntrack_dccp_timeout_partopen = 480
net.netfilter.nf_conntrack_dccp_timeout_request = 240
net.netfilter.nf_conntrack_dccp_timeout_respond = 480
net.netfilter.nf_conntrack_dccp_timeout_timewait = 240
net.netfilter.nf_conntrack_events = 1
net.netfilter.nf_conntrack_expect_max = 1024
net.netfilter.nf_conntrack_frag6_high_thresh = 4194304
net.netfilter.nf_conntrack_frag6_low_thresh = 3145728
net.netfilter.nf_conntrack_frag6_timeout = 60
net.netfilter.nf_conntrack_generic_timeout = 600
net.netfilter.nf_conntrack_gre_timeout = 30
net.netfilter.nf_conntrack_gre_timeout_stream = 180
net.netfilter.nf_conntrack_helper = 0
net.netfilter.nf_conntrack_icmp_timeout = 30
net.netfilter.nf_conntrack_icmpv6_timeout = 30
net.netfilter.nf_conntrack_log_invalid = 0
net.netfilter.nf_conntrack_max = 262144
net.netfilter.nf_conntrack_sctp_timeout_closed = 10
net.netfilter.nf_conntrack_sctp_timeout_cookie_echoed = 3
net.netfilter.nf_conntrack_sctp_timeout_cookie_wait = 3
net.netfilter.nf_conntrack_sctp_timeout_established = 432000
net.netfilter.nf_conntrack_sctp_timeout_heartbeat_acked = 210
net.netfilter.nf_conntrack_sctp_timeout_heartbeat_sent = 30
net.netfilter.nf_conntrack_sctp_timeout_shutdown_ack_sent = 3
net.netfilter.nf_conntrack_sctp_timeout_shutdown_recd = 0
net.netfilter.nf_conntrack_sctp_timeout_shutdown_sent = 0
net.netfilter.nf_conntrack_tcp_be_liberal = 0
net.netfilter.nf_conntrack_tcp_loose = 1
net.netfilter.nf_conntrack_tcp_max_retrans = 3
net.netfilter.nf_conntrack_tcp_timeout_close = 10
net.netfilter.nf_conntrack_tcp_timeout_close_wait = 60
net.netfilter.nf_conntrack_tcp_timeout_established = 432000
net.netfilter.nf_conntrack_tcp_timeout_fin_wait = 120
net.netfilter.nf_conntrack_tcp_timeout_last_ack = 30
net.netfilter.nf_conntrack_tcp_timeout_max_retrans = 300
net.netfilter.nf_conntrack_tcp_timeout_syn_recv = 60
net.netfilter.nf_conntrack_tcp_timeout_syn_sent = 120
net.netfilter.nf_conntrack_tcp_timeout_time_wait = 120
net.netfilter.nf_conntrack_tcp_timeout_unacknowledged = 300
net.netfilter.nf_conntrack_timestamp = 0
net.netfilter.nf_conntrack_udp_timeout = 30
net.netfilter.nf_conntrack_udp_timeout_stream = 120
net.nf_conntrack_max = 262144Offline
I think there is only one connection allowed at a time. Did you close the other clients?
A wireshark trace might be helpful, especially a comparison between working and non working client.
Offline
As far as I can see, the number of clients doesn't matter: I can connect with two clients at once (e. g. smartphone and notebook) whereas the home server as a single client still fails.
I'm not an expert with Wireshark, but my router allows the recording of traffic. I really hope this helps.
While the radio was playing, I connected the working client first (192.168.1.39). I stopped the program after several seconds. Then I connected the failing client (192.168.1.10) which stopped by itself.
Last edited by grisu (2020-12-15 07:01:21)
Offline
There's no communication on port 60128 in that log.
The netfilter settings are probably irrelevant, can you please run a strace with timestamps for your demo on the good and the bad system and post both?
Offline
strace of the working client and of the failing client.
Offline
That doesn't look like a timeout.
What if you don't quit on an empty package but just continue reading?
Offline
I'd really love to wait for new data, but my programs won't receive anything anymore. When there are several clients connected to the radio, only the client on this particular system stops receiving.
The python implementation given above is just to show what I'm trying to do. My original client written in C# waits forever, but as soon as it tries to send through the dead connection, the program crashes.
Offline
Did you check dmesg/journalctl for coincident errors arond the NIC?
And really wireshark the connection on the failing system - https://wiki.archlinux.org/index.php/Wireshark
It's not very complicated, you run wireshark, maybe select the NIC to watch, press record and run your program.
Offline
Here we have the Wireshark output on both systems. Failing:
115 1.413846151 192.168.1.10 → 192.168.1.59 TCP 74 47668 → 60128 [SYN] Seq=0 Win=64240 Len=0 MSS=1460 SACK_PERM=1 TSval=275374517 TSecr=0 WS=128
116 1.416811054 192.168.1.59 → 192.168.1.10 TCP 74 60128 → 47668 [SYN, ACK] Seq=0 Ack=1 Win=28960 Len=0 MSS=1460 SACK_PERM=1 TSval=1185937526 TSecr=275374517 WS=32
117 1.416854466 192.168.1.10 → 192.168.1.59 TCP 66 47668 → 60128 [ACK] Seq=1 Ack=1 Win=64256 Len=0 TSval=275374520 TSecr=1185937526
118 1.422134800 192.168.1.59 → 192.168.1.10 TCP 66 60128 → 47024 [FIN, ACK] Seq=1 Ack=1 Win=905 Len=0 TSval=1185937527 TSecr=275345614
119 1.422366229 192.168.1.10 → 192.168.1.59 TCP 66 47024 → 60128 [FIN, ACK] Seq=1 Ack=2 Win=501 Len=0 TSval=275374526 TSecr=1185937527
120 1.424142574 192.168.1.59 → 192.168.1.10 TCP 66 60128 → 47024 [ACK] Seq=2 Ack=2 Win=905 Len=0 TSval=1185937527 TSecr=275374526
121 1.451311550 192.168.1.59 → 192.168.1.10 TCP 125 60128 → 47668 [PSH, ACK] Seq=1 Ack=1 Win=28960 Len=59 TSval=1185937530 TSecr=275374520
122 1.451345521 192.168.1.10 → 192.168.1.59 TCP 66 47668 → 60128 [ACK] Seq=1 Ack=60 Win=64256 Len=0 TSval=275374555 TSecr=1185937530
139 1.880293494 192.168.1.59 → 192.168.1.255 UDP 1050 10102 → 10102 Len=1008
282 4.399966262 192.168.1.59 → 192.168.1.255 UDP 1050 10102 → 10102 Len=1008
357 6.919003155 192.168.1.59 → 192.168.1.255 UDP 1050 10102 → 10102 Len=1008
475 9.309210626 192.168.1.10 → 192.168.1.59 TCP 74 47722 → 60128 [SYN] Seq=0 Win=64240 Len=0 MSS=1460 SACK_PERM=1 TSval=275382412 TSecr=0 WS=128
477 9.311868096 192.168.1.59 → 192.168.1.10 TCP 74 60128 → 47722 [SYN, ACK] Seq=0 Ack=1 Win=28960 Len=0 MSS=1460 SACK_PERM=1 TSval=1185938316 TSecr=275382412 WS=32
478 9.311901634 192.168.1.10 → 192.168.1.59 TCP 66 47722 → 60128 [ACK] Seq=1 Ack=1 Win=64256 Len=0 TSval=275382415 TSecr=1185938316
479 9.316267784 192.168.1.59 → 192.168.1.10 TCP 66 60128 → 47668 [FIN, ACK] Seq=60 Ack=1 Win=28960 Len=0 TSval=1185938316 TSecr=275374555
480 9.316363270 192.168.1.10 → 192.168.1.59 TCP 66 47668 → 60128 [FIN, ACK] Seq=1 Ack=61 Win=64256 Len=0 TSval=275382420 TSecr=1185938316
481 9.319407091 192.168.1.59 → 192.168.1.10 TCP 66 60128 → 47668 [ACK] Seq=61 Ack=2 Win=28960 Len=0 TSval=1185938317 TSecr=275382420
489 9.356930857 192.168.1.59 → 192.168.1.10 TCP 125 60128 → 47722 [PSH, ACK] Seq=1 Ack=1 Win=28960 Len=59 TSval=1185938320 TSecr=275382415
490 9.356955223 192.168.1.10 → 192.168.1.59 TCP 66 47722 → 60128 [ACK] Seq=1 Ack=60 Win=64256 Len=0 TSval=275382460 TSecr=1185938320
507 9.371007896 192.168.1.10 → 192.168.1.59 TCP 93 47722 → 60128 [PSH, ACK] Seq=1 Ack=60 Win=64256 Len=27 TSval=275382474 TSecr=1185938320
508 9.371597035 192.168.1.59 → 192.168.1.10 TCP 66 60128 → 47722 [ACK] Seq=60 Ack=28 Win=28960 Len=0 TSval=1185938322 TSecr=275382474
512 9.372228488 192.168.1.10 → 192.168.1.59 TCP 93 47722 → 60128 [PSH, ACK] Seq=28 Ack=60 Win=64256 Len=27 TSval=275382476 TSecr=1185938322
514 9.372668064 192.168.1.59 → 192.168.1.10 TCP 66 60128 → 47722 [ACK] Seq=60 Ack=55 Win=28960 Len=0 TSval=1185938322 TSecr=275382476
516 9.373386579 192.168.1.10 → 192.168.1.59 TCP 93 47722 → 60128 [PSH, ACK] Seq=55 Ack=60 Win=64256 Len=27 TSval=275382477 TSecr=1185938322
517 9.373850571 192.168.1.59 → 192.168.1.10 TCP 66 60128 → 47722 [ACK] Seq=60 Ack=82 Win=28960 Len=0 TSval=1185938322 TSecr=275382477
518 9.374623136 192.168.1.10 → 192.168.1.59 TCP 93 47722 → 60128 [PSH, ACK] Seq=82 Ack=60 Win=64256 Len=27 TSval=275382478 TSecr=1185938322
521 9.375022263 192.168.1.59 → 192.168.1.10 TCP 66 60128 → 47722 [ACK] Seq=60 Ack=109 Win=28960 Len=0 TSval=1185938322 TSecr=275382478
522 9.375759917 192.168.1.10 → 192.168.1.59 TCP 93 47722 → 60128 [PSH, ACK] Seq=109 Ack=60 Win=64256 Len=27 TSval=275382479 TSecr=1185938322
523 9.376203422 192.168.1.59 → 192.168.1.10 TCP 66 60128 → 47722 [ACK] Seq=60 Ack=136 Win=28960 Len=0 TSval=1185938322 TSecr=275382479
527 9.376874297 192.168.1.10 → 192.168.1.59 TCP 93 47722 → 60128 [PSH, ACK] Seq=136 Ack=60 Win=64256 Len=27 TSval=275382480 TSecr=1185938322
528 9.377376177 192.168.1.59 → 192.168.1.10 TCP 66 60128 → 47722 [ACK] Seq=60 Ack=163 Win=28960 Len=0 TSval=1185938323 TSecr=275382480
529 9.378098837 192.168.1.10 → 192.168.1.59 TCP 93 47722 → 60128 [PSH, ACK] Seq=163 Ack=60 Win=64256 Len=27 TSval=275382481 TSecr=1185938323
530 9.378600901 192.168.1.59 → 192.168.1.10 TCP 66 60128 → 47722 [ACK] Seq=60 Ack=190 Win=28960 Len=0 TSval=1185938323 TSecr=275382481
531 9.379284584 192.168.1.10 → 192.168.1.59 TCP 93 47722 → 60128 [PSH, ACK] Seq=190 Ack=60 Win=64256 Len=27 TSval=275382483 TSecr=1185938323
532 9.379728257 192.168.1.59 → 192.168.1.10 TCP 66 60128 → 47722 [ACK] Seq=60 Ack=217 Win=28960 Len=0 TSval=1185938323 TSecr=275382483
538 9.380429555 192.168.1.10 → 192.168.1.59 TCP 93 47722 → 60128 [PSH, ACK] Seq=217 Ack=60 Win=64256 Len=27 TSval=275382484 TSecr=1185938323
539 9.380876895 192.168.1.59 → 192.168.1.10 TCP 66 60128 → 47722 [ACK] Seq=60 Ack=244 Win=28960 Len=0 TSval=1185938323 TSecr=275382484
540 9.381565797 192.168.1.10 → 192.168.1.59 TCP 93 47722 → 60128 [PSH, ACK] Seq=244 Ack=60 Win=64256 Len=27 TSval=275382485 TSecr=1185938323
541 9.381956976 192.168.1.59 → 192.168.1.10 TCP 66 60128 → 47722 [ACK] Seq=60 Ack=271 Win=28960 Len=0 TSval=1185938323 TSecr=275382485
542 9.382715697 192.168.1.10 → 192.168.1.59 TCP 93 47722 → 60128 [PSH, ACK] Seq=271 Ack=60 Win=64256 Len=27 TSval=275382486 TSecr=1185938323
547 9.383106299 192.168.1.59 → 192.168.1.10 TCP 66 60128 → 47722 [ACK] Seq=60 Ack=298 Win=28960 Len=0 TSval=1185938323 TSecr=275382486
549 9.383949840 192.168.1.10 → 192.168.1.59 TCP 93 47722 → 60128 [PSH, ACK] Seq=298 Ack=60 Win=64256 Len=27 TSval=275382487 TSecr=1185938323
550 9.384408963 192.168.1.59 → 192.168.1.10 TCP 66 60128 → 47722 [ACK] Seq=60 Ack=325 Win=28960 Len=0 TSval=1185938323 TSecr=275382487
551 9.385175626 192.168.1.10 → 192.168.1.59 TCP 93 47722 → 60128 [PSH, ACK] Seq=325 Ack=60 Win=64256 Len=27 TSval=275382488 TSecr=1185938323
552 9.385593616 192.168.1.59 → 192.168.1.10 TCP 66 60128 → 47722 [ACK] Seq=60 Ack=352 Win=28960 Len=0 TSval=1185938323 TSecr=275382488
579 9.421419875 192.168.1.59 → 192.168.1.255 UDP 1050 10102 → 10102 Len=1008
582 9.449223807 192.168.1.59 → 192.168.1.10 TCP 92 60128 → 47722 [PSH, ACK] Seq=60 Ack=352 Win=28960 Len=26 TSval=1185938330 TSecr=275382488
583 9.449280339 192.168.1.10 → 192.168.1.59 TCP 66 47722 → 60128 [ACK] Seq=352 Ack=86 Win=64256 Len=0 TSval=275382553 TSecr=1185938330
586 9.481876566 192.168.1.59 → 192.168.1.10 TCP 92 60128 → 47722 [PSH, ACK] Seq=86 Ack=352 Win=28960 Len=26 TSval=1185938333 TSecr=275382553
587 9.481924073 192.168.1.10 → 192.168.1.59 TCP 66 47722 → 60128 [ACK] Seq=352 Ack=112 Win=64256 Len=0 TSval=275382585 TSecr=1185938333
588 9.499095586 192.168.1.59 → 192.168.1.10 TCP 92 60128 → 47722 [PSH, ACK] Seq=112 Ack=352 Win=28960 Len=26 TSval=1185938335 TSecr=275382585
589 9.499388693 192.168.1.10 → 192.168.1.59 TCP 66 47722 → 60128 [ACK] Seq=352 Ack=138 Win=64256 Len=0 TSval=275382603 TSecr=1185938335
590 9.515230089 192.168.1.59 → 192.168.1.10 TCP 92 60128 → 47722 [PSH, ACK] Seq=138 Ack=352 Win=28960 Len=26 TSval=1185938336 TSecr=275382603
591 9.515604592 192.168.1.10 → 192.168.1.59 TCP 66 47722 → 60128 [ACK] Seq=352 Ack=164 Win=64256 Len=0 TSval=275382619 TSecr=1185938336
592 9.532143812 192.168.1.59 → 192.168.1.10 TCP 93 60128 → 47722 [PSH, ACK] Seq=164 Ack=352 Win=28960 Len=27 TSval=1185938338 TSecr=275382619
593 9.532270501 192.168.1.10 → 192.168.1.59 TCP 66 47722 → 60128 [ACK] Seq=352 Ack=191 Win=64256 Len=0 TSval=275382636 TSecr=1185938338
594 9.548685052 192.168.1.59 → 192.168.1.10 TCP 141 60128 → 47722 [PSH, ACK] Seq=191 Ack=352 Win=28960 Len=75 TSval=1185938340 TSecr=275382636
595 9.548811076 192.168.1.10 → 192.168.1.59 TCP 66 47722 → 60128 [ACK] Seq=352 Ack=266 Win=64256 Len=0 TSval=275382652 TSecr=1185938340
596 9.564400042 192.168.1.59 → 192.168.1.10 TCP 131 60128 → 47722 [PSH, ACK] Seq=266 Ack=352 Win=28960 Len=65 TSval=1185938341 TSecr=275382652
597 9.564553658 192.168.1.10 → 192.168.1.59 TCP 66 47722 → 60128 [ACK] Seq=352 Ack=331 Win=64256 Len=0 TSval=275382668 TSecr=1185938341
599 9.580697120 192.168.1.59 → 192.168.1.10 TCP 92 60128 → 47722 [PSH, ACK] Seq=331 Ack=352 Win=28960 Len=26 TSval=1185938343 TSecr=275382668
600 9.580833616 192.168.1.10 → 192.168.1.59 TCP 66 47722 → 60128 [ACK] Seq=352 Ack=357 Win=64256 Len=0 TSval=275382684 TSecr=1185938343
601 9.648973418 192.168.1.59 → 192.168.1.10 TCP 93 60128 → 47722 [PSH, ACK] Seq=357 Ack=352 Win=28960 Len=27 TSval=1185938350 TSecr=275382684
602 9.649100710 192.168.1.10 → 192.168.1.59 TCP 66 47722 → 60128 [ACK] Seq=352 Ack=384 Win=64256 Len=0 TSval=275382752 TSecr=1185938350
658 10.388002669 192.168.1.10 → 192.168.1.59 TCP 93 47722 → 60128 [PSH, ACK] Seq=352 Ack=384 Win=64256 Len=27 TSval=275383491 TSecr=1185938350
659 10.388609838 192.168.1.59 → 192.168.1.10 TCP 66 60128 → 47722 [ACK] Seq=384 Ack=379 Win=28960 Len=0 TSval=1185938424 TSecr=275383491
663 10.460312246 192.168.1.59 → 192.168.1.10 TCP 1514 60128 → 47722 [ACK] Seq=384 Ack=379 Win=28960 Len=1448 TSval=1185938431 TSecr=275383491
664 10.460335479 192.168.1.10 → 192.168.1.59 TCP 66 47722 → 60128 [ACK] Seq=379 Ack=1832 Win=64128 Len=0 TSval=275383564 TSecr=1185938431
665 10.460417226 192.168.1.59 → 192.168.1.10 TCP 1514 60128 → 47722 [ACK] Seq=1832 Ack=379 Win=28960 Len=1448 TSval=1185938431 TSecr=275383491
666 10.460428377 192.168.1.10 → 192.168.1.59 TCP 66 47722 → 60128 [ACK] Seq=379 Ack=3280 Win=63360 Len=0 TSval=275383564 TSecr=1185938431
667 10.460529916 192.168.1.59 → 192.168.1.10 TCP 1514 60128 → 47722 [ACK] Seq=3280 Ack=379 Win=28960 Len=1448 TSval=1185938431 TSecr=275383491
668 10.460539446 192.168.1.10 → 192.168.1.59 TCP 66 47722 → 60128 [ACK] Seq=379 Ack=4728 Win=64128 Len=0 TSval=275383564 TSecr=1185938431
669 10.460633350 192.168.1.59 → 192.168.1.10 TCP 1514 60128 → 47722 [ACK] Seq=4728 Ack=379 Win=28960 Len=1448 TSval=1185938431 TSecr=275383491
670 10.460638678 192.168.1.10 → 192.168.1.59 TCP 66 47722 → 60128 [ACK] Seq=379 Ack=6176 Win=63360 Len=0 TSval=275383564 TSecr=1185938431
671 10.460739240 192.168.1.59 → 192.168.1.10 TCP 1514 60128 → 47722 [ACK] Seq=6176 Ack=379 Win=28960 Len=1448 TSval=1185938431 TSecr=275383491
672 10.460744051 192.168.1.10 → 192.168.1.59 TCP 66 47722 → 60128 [ACK] Seq=379 Ack=7624 Win=64128 Len=0 TSval=275383564 TSecr=1185938431
673 10.460791874 192.168.1.59 → 192.168.1.10 TCP 791 60128 → 47722 [PSH, ACK] Seq=7624 Ack=379 Win=28960 Len=725 TSval=1185938431 TSecr=275383491
674 10.460795088 192.168.1.10 → 192.168.1.59 TCP 66 47722 → 60128 [ACK] Seq=379 Ack=8349 Win=63488 Len=0 TSval=275383564 TSecr=1185938431
675 10.469184795 192.168.1.10 → 192.168.1.59 TCP 93 47722 → 60128 [PSH, ACK] Seq=379 Ack=8349 Win=64128 Len=27 TSval=275383572 TSecr=1185938431
678 10.469684036 192.168.1.59 → 192.168.1.10 TCP 66 60128 → 47722 [ACK] Seq=8349 Ack=406 Win=28960 Len=0 TSval=1185938432 TSecr=275383572
681 10.508679854 192.168.1.59 → 192.168.1.10 TCP 92 60128 → 47722 [PSH, ACK] Seq=8349 Ack=406 Win=28960 Len=26 TSval=1185938436 TSecr=275383572
682 10.508704531 192.168.1.10 → 192.168.1.59 TCP 66 47722 → 60128 [ACK] Seq=406 Ack=8375 Win=64128 Len=0 TSval=275383612 TSecr=1185938436
780 11.928797323 192.168.1.59 → 192.168.1.255 UDP 1050 10102 → 10102 Len=1008Working:
37 1.558648129 192.168.1.39 → 192.168.1.59 TCP 74 46970 → 60128 [SYN] Seq=0 Win=64240 Len=0 MSS=1460 SACK_PERM=1 TSval=3589391864 TSecr=0 WS=128
38 1.597895602 192.168.1.59 → 192.168.1.255 UDP 1050 10102 → 10102 Len=1008
39 2.366543202 192.168.1.59 → 192.168.1.39 TCP 74 60128 → 46970 [SYN, ACK] Seq=0 Ack=1 Win=28960 Len=0 MSS=1460 SACK_PERM=1 TSval=1185934553 TSecr=3589391864 WS=32
40 2.366631028 192.168.1.39 → 192.168.1.59 TCP 66 46970 → 60128 [ACK] Seq=1 Ack=1 Win=64256 Len=0 TSval=3589392672 TSecr=1185934553
43 2.389682168 192.168.1.59 → 192.168.1.39 TCP 125 60128 → 46970 [PSH, ACK] Seq=1 Ack=1 Win=28960 Len=59 TSval=1185934636 TSecr=3589392672
44 2.389746327 192.168.1.39 → 192.168.1.59 TCP 66 46970 → 60128 [ACK] Seq=1 Ack=60 Win=64256 Len=0 TSval=3589392695 TSecr=1185934636
59 4.157855263 192.168.1.59 → 192.168.1.255 UDP 1050 10102 → 10102 Len=1008
172 6.616031551 192.168.1.59 → 192.168.1.255 UDP 1050 10102 → 10102 Len=1008
345 11.633506189 192.168.1.59 → 192.168.1.255 UDP 1050 10102 → 10102 Len=1008What I find really interesting is that there are lots of packages before the program stops on the failing machine, although I sill can't identify the problem.
Neither dmesg nor journalctl show an error at that point of time.
Offline
fyi, wireshark can save those logs and you (me) can then open them in a different instance for (much easier ;-) reading.
475 9.309210626 192.168.1.10 → 192.168.1.59 TCP 74 47722 → 60128 [SYN] Seq=0 Win=64240 Len=0 MSS=1460 SACK_PERM=1 TSval=275382412 TSecr=0 WS=128
477 9.311868096 192.168.1.59 → 192.168.1.10 TCP 74 60128 → 47722 [SYN, ACK] Seq=0 Ack=1 Win=28960 Len=0 MSS=1460 SACK_PERM=1 TSval=1185938316 TSecr=275382412 WS=32
478 9.311901634 192.168.1.10 → 192.168.1.59 TCP 66 47722 → 60128 [ACK] Seq=1 Ack=1 Win=64256 Len=0 TSval=275382415 TSecr=1185938316
479 9.316267784 192.168.1.59 → 192.168.1.10 TCP 66 60128 → 47668 [FIN, ACK] Seq=60 Ack=1 Win=28960 Len=0 TSval=1185938316 TSecr=275374555
480 9.316363270 192.168.1.10 → 192.168.1.59 TCP 66 47668 → 60128 [FIN, ACK] Seq=1 Ack=61 Win=64256 Len=0 TSval=275382420 TSecr=1185938316
481 9.319407091 192.168.1.59 → 192.168.1.10 TCP 66 60128 → 47668 [ACK] Seq=61 Ack=2 Win=28960 Len=0 TSval=1185938317 TSecr=275382420
489 9.356930857 192.168.1.59 → 192.168.1.10 TCP 125 60128 → 47722 [PSH, ACK] Seq=1 Ack=1 Win=28960 Len=59 TSval=1185938320 TSecr=275382415
…192.168.1.59 (the radio?!) is sending a FIN which get ACK from 192.168.1.10 (the client?!) - at this point the communication is over.
https://en.wikipedia.org/wiki/Transmiss … ermination
This does not seem to happen in the working connection?
Comparing the package payload (wireshark stores those as well) might shed light on why the radio feels like terminating the connection - or both engage in a push communication on the zombie connection afterwards…
lspci
ip a
ip r
ethtool <dev>Offline
This somewhat sounds like an MTU issue. Using the following command what's the MTU value on all machines?
$ ip addr | grep mtuMore info on an MTU mismatch:
https://packetu.com/2014/07/24/recogniz … tu-issues/
Last edited by murffatksig (2020-12-09 04:29:39)
"Oh, they have the internet on computers now."
Offline
As the home server is an old notebook, I connected the device again via wifi and disconnected the ethernet cable. The result is still the same.
Hopefully things get clearer when you can read the PCAP file.
Before I executed the following commands I plugged the ethernet cable back in.
lspci
0:00.0 Host bridge: Intel Corporation Haswell-ULT DRAM Controller (rev 0b)
00:02.0 VGA compatible controller: Intel Corporation Haswell-ULT Integrated Graphics Controller (rev 0b)
00:03.0 Audio device: Intel Corporation Haswell-ULT HD Audio Controller (rev 0b)
00:14.0 USB controller: Intel Corporation 8 Series USB xHCI HC (rev 04)
00:16.0 Communication controller: Intel Corporation 8 Series HECI #0 (rev 04)
00:16.3 Serial controller: Intel Corporation 8 Series HECI KT (rev 04)
00:19.0 Ethernet controller: Intel Corporation Ethernet Connection I218-LM (rev 04)
00:1b.0 Audio device: Intel Corporation 8 Series HD Audio Controller (rev 04)
00:1c.0 PCI bridge: Intel Corporation 8 Series PCI Express Root Port 1 (rev e4)
00:1c.3 PCI bridge: Intel Corporation 8 Series PCI Express Root Port 4 (rev e4)
00:1c.4 PCI bridge: Intel Corporation 8 Series PCI Express Root Port 5 (rev e4)
00:1d.0 USB controller: Intel Corporation 8 Series USB EHCI #1 (rev 04)
00:1f.0 ISA bridge: Intel Corporation 8 Series LPC Controller (rev 04)
00:1f.2 SATA controller: Intel Corporation 8 Series SATA Controller 1 [AHCI mode] (rev 04)
00:1f.3 SMBus: Intel Corporation 8 Series SMBus Controller (rev 04)
02:00.0 Network controller: Intel Corporation Wireless 7260 (rev 73)
03:00.0 SD Host controller: O2 Micro, Inc. SD/MMC Card Reader Controller (rev 01)ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
inet 127.0.0.1/8 scope host lo
valid_lft forever preferred_lft forever
inet6 ::1/128 scope host
valid_lft forever preferred_lft forever
2: eno1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UP group default qlen 1000
link/ether ec:f4:bb:14:ea:9e brd ff:ff:ff:ff:ff:ff
altname enp0s25
inet 192.168.1.10/24 brd 192.168.1.255 scope global dynamic noprefixroute eno1
valid_lft 863456sec preferred_lft 863456sec
inet6 fe80::4d7:29cf:844f:bfdf/64 scope link noprefixroute
valid_lft forever preferred_lft forever
3: wlp2s0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
link/ether 80:86:f2:6e:e4:11 brd ff:ff:ff:ff:ff:ff
inet 192.168.1.50/24 brd 192.168.1.255 scope global dynamic noprefixroute wlp2s0
valid_lft 863178sec preferred_lft 863178sec
inet6 fe80::3a5f:b1f4:ac28:8560/64 scope link noprefixroute
valid_lft forever preferred_lft forever
4: wg0: <POINTOPOINT,NOARP,UP,LOWER_UP> mtu 1420 qdisc noqueue state UNKNOWN group default qlen 1000
link/none
inet 192.168.42.1/32 scope global wg0
valid_lft forever preferred_lft forever
5: docker0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default
link/ether 02:42:3f:d4:ab:c1 brd ff:ff:ff:ff:ff:ff
inet 172.17.0.1/16 brd 172.17.255.255 scope global docker0
valid_lft forever preferred_lft forever
inet6 fe80::42:3fff:fed4:abc1/64 scope link
valid_lft forever preferred_lft forever
6: br-12bace3c068f: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default
link/ether 02:42:3a:07:4d:43 brd ff:ff:ff:ff:ff:ff
inet 172.23.0.1/16 brd 172.23.255.255 scope global br-12bace3c068f
valid_lft forever preferred_lft forever
inet6 fe80::42:3aff:fe07:4d43/64 scope link
valid_lft forever preferred_lft forever
7: br-564adedc6209: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default
link/ether 02:42:b1:ef:4a:98 brd ff:ff:ff:ff:ff:ff
inet 172.22.0.1/16 brd 172.22.255.255 scope global br-564adedc6209
valid_lft forever preferred_lft forever
inet6 fe80::42:b1ff:feef:4a98/64 scope link
valid_lft forever preferred_lft forever
9: veth202d47d@if8: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master br-564adedc6209 state UP group default
link/ether ae:c5:00:7f:c8:0d brd ff:ff:ff:ff:ff:ff link-netnsid 1
inet6 fe80::acc5:ff:fe7f:c80d/64 scope link
valid_lft forever preferred_lft forever
11: vethfd7202c@if10: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master br-12bace3c068f state UP group default
link/ether ea:01:5e:06:2b:fe brd ff:ff:ff:ff:ff:ff link-netnsid 3
inet6 fe80::e801:5eff:fe06:2bfe/64 scope link
valid_lft forever preferred_lft forever
13: vethcaff0f5@if12: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master docker0 state UP group default
link/ether 56:1b:f3:84:11:a0 brd ff:ff:ff:ff:ff:ff link-netnsid 2
inet6 fe80::541b:f3ff:fe84:11a0/64 scope link
valid_lft forever preferred_lft forever
15: vetha8ad245@if14: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master br-564adedc6209 state UP group default
link/ether 3e:b3:1d:37:2e:b6 brd ff:ff:ff:ff:ff:ff link-netnsid 0
inet6 fe80::3cb3:1dff:fe37:2eb6/64 scope link
valid_lft forever preferred_lft forever
17: veth80bdfad@if16: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master br-564adedc6209 state UP group default
link/ether 8a:c0:cb:ba:7f:92 brd ff:ff:ff:ff:ff:ff link-netnsid 4
inet6 fe80::88c0:cbff:feba:7f92/64 scope link
valid_lft forever preferred_lft forever
19: veth7b6e8a8@if18: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master br-12bace3c068f state UP group default
link/ether 92:d7:40:65:86:a5 brd ff:ff:ff:ff:ff:ff link-netnsid 5
inet6 fe80::90d7:40ff:fe65:86a5/64 scope link
valid_lft forever preferred_lft forever
21: veth38085e4@if20: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master br-12bace3c068f state UP group default
link/ether 0e:aa:0b:a6:f6:e2 brd ff:ff:ff:ff:ff:ff link-netnsid 4
inet6 fe80::caa:bff:fea6:f6e2/64 scope link
valid_lft forever preferred_lft forever
23: veth7575be9@if22: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master br-564adedc6209 state UP group default
link/ether fe:ac:a3:a5:e7:62 brd ff:ff:ff:ff:ff:ff link-netnsid 3
inet6 fe80::fcac:a3ff:fea5:e762/64 scope link
valid_lft forever preferred_lft forever
25: vethb7db1fe@if24: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master br-564adedc6209 state UP group default
link/ether 5a:64:1c:2a:22:81 brd ff:ff:ff:ff:ff:ff link-netnsid 5
inet6 fe80::5864:1cff:fe2a:2281/64 scope link
valid_lft forever preferred_lft foreverip r
default via 192.168.1.1 dev eno1 proto dhcp metric 100
default via 192.168.1.1 dev wlp2s0 proto dhcp metric 600
172.17.0.0/16 dev docker0 proto kernel scope link src 172.17.0.1
172.22.0.0/16 dev br-564adedc6209 proto kernel scope link src 172.22.0.1
172.23.0.0/16 dev br-12bace3c068f proto kernel scope link src 172.23.0.1
192.168.1.0/24 dev eno1 proto kernel scope link src 192.168.1.10 metric 100
192.168.1.0/24 dev wlp2s0 proto kernel scope link src 192.168.1.50 metric 600
192.168.42.2 dev wg0 scope link
192.168.42.3 dev wg0 scope link
192.168.42.4 dev wg0 scope link
192.168.42.5 dev wg0 scope link
192.168.42.6 dev wg0 scope link ethtool eno1
Settings for eno1:
Supported ports: [ TP ]
Supported link modes: 10baseT/Half 10baseT/Full
100baseT/Half 100baseT/Full
1000baseT/Full
Supported pause frame use: No
Supports auto-negotiation: Yes
Supported FEC modes: Not reported
Advertised link modes: 10baseT/Half 10baseT/Full
100baseT/Half 100baseT/Full
1000baseT/Full
Advertised pause frame use: No
Advertised auto-negotiation: Yes
Advertised FEC modes: Not reported
Speed: 1000Mb/s
Duplex: Full
Auto-negotiation: on
Port: Twisted Pair
PHYAD: 1
Transceiver: internal
MDI-X: off (auto)
Supports Wake-on: pumbg
Wake-on: g
Current message level: 0x00000007 (7)
drv probe link
Link detected: yesThis somewhat sounds like an MTU issue. Using the following command what's the MTU value on all machines?
$ ip addr | grep mtu
The devices on the failing machine:
2: eno1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UP group default qlen 1000
3: wlp2s0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000and on the working one
2: enp57s0f1: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc fq_codel state DOWN group default qlen 1000I can't see a difference here.
Last edited by grisu (2020-12-15 07:00:51)
Offline
Lose docker, try again.
Offline
Lose docker, try again.
What exactly do you mean by "lose"? I stopped all the containers and the docker services when I tried to identify the problem. No change.
Now I uninstalled docker and stopped wireguard, rebooted, and only eno1, lo and wlp2s0 are left. The problem persists.
Offline
What exactly do you mean by "lose"?
Whatever you had to do to get rid of bridge and virtual devices.
But it's really just that after an equal hello, the radio tells your server to gtfo before starting the same push communication as with your notebook.
The suspicious part is that it does so for both NICs on the server, there's no delay in the communication, no missing ACK… does the radio keep a log?
Offline
I finally found the reason why the radio stops the connection and my problem is solved:
I already had a smart home system (FHEM) that I wanted to replace by my own software to control the radio. The FHEM software is running on the same system.
So apparently my radio does not "like" if one device opens two connections and terminates it after a few seconds of communication.
Honestly, I'm feeling a bit dumb not having thought of the FHEM software which I'm now going to delete forever.
I'm really thankful for your support. Really. Please keep up the great work!
Offline
That probably explains the various sorce ports (rather than just multiple attempts) and on a closer look the communication on 58566 actually doesn't get finished in the shared segments.
Please always remember to mark resolved threads by editing your initial posts subject - so others will know that there's no task left, but maybe a solution to find.
Thanks.
Offline