You are not logged in.

#1 2016-01-25 18:54:20

budric
Member
Registered: 2016-01-14
Posts: 11

[S]Please help troubleshooting network connectivity between 2 subnets.

Hi,
I have 2 subnets 192.168.10.0/24 and 192.168.20.0/24 where connection between the two is "spotty" (most of the time it doesn't work especially between 2 particular hosts 192.168.10.139 and 192.168.20.107). 

The packets make it as far as the interface on the other hosts and routing doesn't appear to be an issue.  However the packets don't reach the application, instead the host resets the connection.  But there's no firewall or any setting I can see that would cause this.

In an attempt to troubleshoot I ran ncat to listen, along with tcpdump to show packets on 192.168.10.139, and then attempt to send data from 192.168.20.107 using ncat.
On the "server"

root@192.168.10.139 ~ # ncat -vvvvv -l 3333                                                                                                                                                              
Ncat: Version 7.01 ( https://nmap.org/ncat )
NCAT DEBUG: Initialized fdlist with 103 maxfds
Ncat: Listening on :::3333
NCAT DEBUG: Added fd 3 to list, nfds 1, maxfd 3
Ncat: Listening on 0.0.0.0:3333
NCAT DEBUG: Added fd 4 to list, nfds 2, maxfd 4
NCAT DEBUG: Added fd 0 to list, nfds 3, maxfd 4
NCAT DEBUG: Initialized fdlist with 100 maxfds
NCAT DEBUG: selecting, fdmax 4
root@192.168.10.139 ~ # tcpdump -i any -n -A -vv src port 3333 or dst port 3333
tcpdump: listening on any, link-type LINUX_SLL (Linux cooked), capture size 262144 bytes
13:32:31.234613 IP (tos 0x0, ttl 63, id 40715, offset 0, flags [DF], proto TCP (6), length 60)
    192.168.20.107.32878 > 192.168.10.139.3333: Flags [S], cksum 0xfde4 (correct), seq 2212447212, win 29200, options [mss 1460,sackOK,TS val 94231783 ecr 0,nop,wscale 7], length 0
E..<..@.?..i...k..
..n....C.......r............
............
13:32:31.234690 IP (tos 0x0, ttl 64, id 0, offset 0, flags [DF], proto TCP (6), length 60)
    192.168.10.139.3333 > 192.168.20.107.32878: Flags [S.], cksum 0x8613 (correct), seq 1547931860, ack 2212447213, win 28960, options [mss 1460,sackOK,TS val 64718781 ecr 94231783,nop,wscale 7], length 0
E..<..@.@..u..
....k...n\C....C...q ...........
............
13:32:31.234828 IP (tos 0x0, ttl 63, id 40716, offset 0, flags [DF], proto TCP (6), length 52)
    192.168.20.107.32878 > 192.168.10.139.3333: Flags [.], cksum 0x251b (correct), seq 1, ack 1, win 229, options [nop,nop,TS val 94231783 ecr 64718781], length 0
E..4..@.?..p...k..
..n....C.\C......%......
........
13:32:31.234869 IP (tos 0x0, ttl 64, id 24887, offset 0, flags [DF], proto TCP (6), length 40)
    192.168.10.139.3333 > 192.168.20.107.32878: Flags [R], cksum 0x950d (correct), seq 1547931861, win 0, length 0
E..(a7@.@.9R..
....k...n\C......P.......
^C
4 packets captured
4 packets received by filter
0 packets dropped by kernel

On the "client" attempting to send data

root@192.168.20.107 ~ # echo "Hello" | ncat -vvvvvv 192.168.10.139 3333                                                      :(
Ncat: Version 7.01 ( https://nmap.org/ncat )
libnsock nsock_set_loglevel(): Set log level to DEBUG
NCAT DEBUG: Using system default trusted CA certificates and those in /usr/share/ncat/ca-bundle.crt.
libnsock nsock_iod_new2(): nsock_iod_new (IOD #1)
libnsock event_new(): event_new (IOD #1) (EID #8)
libnsock nsock_connect_tcp(): TCP connection requested to 192.168.10.139:3333 (IOD #1) EID 8
libnsock nsock_pool_add_event(): NSE #8: Adding event (timeout in 10000ms)
libnsock nsock_loop(): nsock_loop() started (no timeout). 1 events pending
libnsock nsock_trace_handler_callback(): Callback: CONNECT SUCCESS for EID 8 [192.168.10.139:3333]
Ncat: Connected to 192.168.10.139:3333.
libnsock nsock_iod_new2(): nsock_iod_new (IOD #2)
libnsock event_new(): event_new (IOD #1) (EID #18)
libnsock nsock_read(): Read request from IOD #1 [192.168.10.139:3333] (timeout: -1ms) EID 18
libnsock nsock_pool_add_event(): NSE #18: Adding event (timeout in -1453746751236ms)
libnsock event_new(): event_new (IOD #2) (EID #26)
libnsock nsock_readbytes(): Read request for 0 bytes from IOD #2 [peer unspecified] EID 26
libnsock nsock_pool_add_event(): NSE #26: Adding event (timeout in -1453746751237ms)
libnsock event_delete(): event_delete (IOD #1) (EID #8)
libnsock nsock_trace_handler_callback(): Callback: READ ERROR [Connection reset by peer (104)] for EID 18 [192.168.10.139:3333]
Ncat: Connection reset by peer.

What I read from this is the packets are arriving at 192.168.10.139 a tcp connection is being established but then the kernel resets the connection.  The packets never reach the application.

My assumptions/things I have tested
- I can ping back and forth between the two hosts therefore routing appears to work.
- tcpdump shows received traffic also leads me to believe routing works.
- I doubt it's the switch/cables because I see no tx/rx errors (ran ip -s link errors/dropped are at 0 on both hosts).
- iptables module is installed on 192.168.10.139 but no filtering is defined

root@192.168.10.139 ~ # iptables -t filter -S
-P INPUT ACCEPT
-P FORWARD ACCEPT
-P OUTPUT ACCEPT
root@192.168.10.139 ~ # iptables -t nat -S
-P PREROUTING ACCEPT
-P INPUT ACCEPT
-P OUTPUT ACCEPT
-P POSTROUTING ACCEPT
root@192.168.10.139 ~ # iptables -t raw -S
-P PREROUTING ACCEPT
-P OUTPUT ACCEPT
root@192.168.10.139 ~ # iptables -t mangle -S
-P PREROUTING ACCEPT
-P INPUT ACCEPT
-P FORWARD ACCEPT
-P OUTPUT ACCEPT
-P POSTROUTING ACCEPT

- there is no /etc/deny.hosts and as far as I can tell it's deprecated on arch
- rp_filter is disabled during the test on all interfaces

root@192.168.10.139 ~ #  sysctl net.ipv4.conf.all.rp_filter &&  sysctl net.ipv4.conf.enp1s0.rp_filter
net.ipv4.conf.all.rp_filter = 0
net.ipv4.conf.enp1s0.rp_filter = 0

- I don't see any glaring errors from dmesg or journalctl -xe.

For reference my routing tables are below, and ip rule list is empty.

root@192.168.10.139 ~ # ip route show
default via 192.168.10.1 dev enp1s0  src 192.168.10.139  metric 202
192.168.10.0/24 dev enp1s0  proto kernel  scope link  src 192.168.10.139  metric 202
...
root@192.168.20.107 ~ # ip route show                                                                                        :(
default via 192.168.20.1 dev eth0
192.168.20.0/24 dev eth0  proto kernel  scope link  src 192.168.20.107


My kernels are 4.4.0-4-ARCH and 4.3.3-3-ARCH

I'm at the end of my wits as to what the problem is.  Would appreciate any help.

Thanks.

Last edited by budric (2016-02-10 15:27:57)

Offline

#2 2016-01-25 20:04:32

alphaniner
Member
From: Ancapistan
Registered: 2010-07-12
Posts: 2,810

Re: [S]Please help troubleshooting network connectivity between 2 subnets.

Forgive my noobishness, but with that netmask how is any communication at all possible?


But whether the Constitution really be one thing, or another, this much is certain - that it has either authorized such a government as we have had, or has been powerless to prevent it. In either case, it is unfit to exist.
-Lysander Spooner

Offline

#3 2016-01-25 20:21:41

budric
Member
Registered: 2016-01-14
Posts: 11

Re: [S]Please help troubleshooting network connectivity between 2 subnets.

alphaniner wrote:

Forgive my noobishness, but with that netmask how is any communication at all possible?

Sorry I'm not sure I see what netmask you're referring to as being the problem?
192.168.20.107 is host ip, netmask should be 255.255.255.0, so network portion is 192.168.20.0 and host portion is the 107.  Using CIDR notation to express the same thing it's 192.168.20.0/24.

I'm by no means an expert.  I could have screwed up the configuration...

Offline

#4 2016-01-25 20:42:06

alphaniner
Member
From: Ancapistan
Registered: 2010-07-12
Posts: 2,810

Re: [S]Please help troubleshooting network connectivity between 2 subnets.

It's my understanding that for 192.168.10.X to communicate with 192.168.20.X, both must have a subnet no greater than* 255.255.224.0 . In CIDR that's <IP>/19 . But I have only cursory understanding of networking so I may be wrong.

* I don't know if that's the correct terminology; eg. 255.255.240.0 is "too high" whereas 255.255.192.0 would also allow them to communicate.


But whether the Constitution really be one thing, or another, this much is certain - that it has either authorized such a government as we have had, or has been powerless to prevent it. In either case, it is unfit to exist.
-Lysander Spooner

Offline

#5 2016-01-25 22:54:37

branch
Member
Registered: 2014-03-16
Posts: 209

Re: [S]Please help troubleshooting network connectivity between 2 subnets.

@alphaniner: He has a router between the two /24 subnets.

Try adding ICMP packets to/from 192.168.20.107 to the tcpdump filter to see if any errors are being signaled.

Offline

#6 2016-01-26 01:23:10

budric
Member
Registered: 2016-01-14
Posts: 11

Re: [S]Please help troubleshooting network connectivity between 2 subnets.

@branch good suggestion
Here is ICMP and ARP capture.  There's an additional ICMP redirect in the sequence.

tcpdump -i any -n -A -vv src port 3333 or dst port 3333 or icmp or arp
tcpdump: listening on any, link-type LINUX_SLL (Linux cooked), capture size 262144 bytes
20:10:23.944561 IP (tos 0x0, ttl 63, id 26080, offset 0, flags [DF], proto TCP (6), length 60)
    192.168.20.107.47278 > 192.168.10.139.3333: Flags [S], cksum 0x6fee (correct), seq 2481144156, win 29200, options [mss 1460,sackOK,TS val 101393596 ecr 0,nop,wscale 7], length 0
E..<e.@.?.5....k..
.......A\......r.o..........
..$.........
20:10:23.944675 IP (tos 0x0, ttl 64, id 0, offset 0, flags [DF], proto TCP (6), length 60)
    192.168.10.139.3333 > 192.168.20.107.47278: Flags [S.], cksum 0xa50e (correct), seq 430104128, ack 2481144157, win 28960, options [mss 1460,sackOK,TS val 71880595 ecr 101393596,nop,wscale 7], length 0
E..<..@.@..u..
....k.......@..A]..q ...........
.H....$.....
20:10:23.944826 IP (tos 0xc0, ttl 64, id 56571, offset 0, flags [none], proto ICMP (1), length 88)
    192.168.10.1 > 192.168.10.139: ICMP redirect 192.168.20.107 to host 192.168.20.107, length 68
	IP (tos 0x0, ttl 63, id 0, offset 0, flags [DF], proto TCP (6), length 60)
    192.168.10.139.3333 > 192.168.20.107.47278: Flags [S.], cksum 0xa50e (correct), seq 430104128, ack 2481144157, win 28960, options [mss 1460,sackOK,TS val 71880595 ecr 101393596,nop,wscale 7], length 0
E..X....@.....
...
....`...kE..<..@.?..u..
....k.......@..A]..q ...........
.H....$.....
20:10:23.944877 IP (tos 0x0, ttl 63, id 26081, offset 0, flags [DF], proto TCP (6), length 52)
    192.168.20.107.47278 > 192.168.10.139.3333: Flags [.], cksum 0x4415 (correct), seq 1, ack 1, win 229, options [nop,nop,TS val 101393597 ecr 71880595], length 0
E..4e.@.?.5....k..
.......A]...A....D......
..$..H..
20:10:23.944902 IP (tos 0x0, ttl 64, id 49885, offset 0, flags [DF], proto TCP (6), length 40)
    192.168.10.139.3333 > 192.168.20.107.47278: Flags [R], cksum 0x5202 (correct), seq 430104129, win 0, length 0
E..(..@.@.....
....k.......A....P...R...
20:10:28.947921 ARP, Ethernet (len 6), IPv4 (len 4), Request who-has 192.168.10.1 tell 192.168.10.139, length 28
.........=~.G...
.........
.
20:10:28.948152 ARP, Ethernet (len 6), IPv4 (len 4), Reply 192.168.10.1 is-at <redacted>, length 46
........>.......
..=~.G...
...................

They are connected to the same switch, so I can see why the redirect.  However I don't see ARP requests during that sequence for where 192.168.20.107 is, it just sends reset packet immediately.

I ran the same test on my laptop on 192.168.10.0/24 (connected via wireless AP) and it works fine and receives the message.  The laptop doesn't receive icmp redirect.

Last edited by budric (2016-01-26 01:31:11)

Offline

#7 2016-01-26 03:01:50

branch
Member
Registered: 2014-03-16
Posts: 209

Re: [S]Please help troubleshooting network connectivity between 2 subnets.

Wait... they are on the same switch?

My apologies to alphaniner, i saw talk of routing and assumed there was a normal router.

So you are relying on the router to route packets back onto the same network segment they came in on to reach the other "subnet"? Hence the redirect telling the 10.139 to send packets directly to the 20.107? This is a *strange* configuration, in that the logical topology does not match the physical topology and yet no formal abstraction mechanism such as vlan's is in use. I suspect the problem has to do with the route the 10.139 has been redirected to use not being in its routing table.

The best way to fix it of course is to make the logical topology match the physical, so if you only have one physical network segment then configure it as a single subnet. If there is some reason you cannot do this then you are left with trying cludgy workarounds. Maybe try  preventing redirects from being accepted on the 10.139 through sysctl net.ipv4.conf.all.accept_redirects, or you could try adding the 20.107 directly to the routing table on the 10.139.

Offline

#8 2016-01-26 03:38:08

budric
Member
Registered: 2016-01-14
Posts: 11

Re: [S]Please help troubleshooting network connectivity between 2 subnets.

There is indeed a router that is supposed to route between the 2 networks.  I want to treat them as separate networks.  I just don't have a switch that supports vlans...or another switch for that matter.  I thought separate networks would be my "poor mans version of vlans".  I could put guests and other stuff on the different network, then through firewalls I could easily restrict traffic.

I understand it's not as secure as vlans, but it's good enough to start for me.  For example many of my guests aren't going to be actively hacking the network, but if they connect to the "guest network" they don't get access to my shares.

Your suggestion led me to find the sysctl setting http://www.tldp.org/LDP/solrhe/Securing … sec57.html that I'll give a try in a bit.  I'm also looking for a way to disable this on the router - i.e. prevent the router from sending icmp redirect.

Offline

#9 2016-01-26 04:19:33

branch
Member
Registered: 2014-03-16
Posts: 209

Re: [S]Please help troubleshooting network connectivity between 2 subnets.

Another option may be to configure the network as a single /19 as alphaniner suggested, keep the current address assignments, and restrict access to services based on address range via either the configuration of the actual services or firewall rules on the servers.

You may be able to get it to work in the current configuration, but be warned that if you want to configure IPv6 the same way it will be tedious at best.

Offline

#10 2016-01-26 05:12:05

budric
Member
Registered: 2016-01-14
Posts: 11

Re: [S]Please help troubleshooting network connectivity between 2 subnets.

I ended up disabling ICMP redirect on my router by defining net.ipv4.conf.all.send_redirects=0 and net.ipv4.conf.default.send_redirect=0 (For some reason I needed both http://unix.stackexchange.com/questions … -redirect).  Now I can send data.

I'm not sure why that host wasn't making use of redirect because on 192.168.10.139 net.ipv4.conf.enp1s0.accept_redirects = 1 for the interface in question, although net.ipv4.conf.all.accept_redirects was set to 0, maybe that's why.

Thanks for the suggestion about single network, with different mentally separated segments.  That certainly works as well, and is more efficient as it all doesn't have to be routed.  But this problem was driving me crazy because it should have worked the way I set it up.  Now I learned a little more.

Offline

Board footer

Powered by FluxBB