You are not logged in.

#1 2020-06-30 17:41:49

EdeWolf
Member
Registered: 2016-01-06
Posts: 79

Weird bridge / routing problem

Ok, it is getting mighty complicated, but hope dies last.

So I have a bridge (1), that connects to both the lan and some lxc containers. Another, dual homed, host is connected to that lan, with the not connected interface hosting a bridge (2) as well, that hosts VMs.

Now the problem is, from inside a VM I cannot reach the lxc containers. Strangely however, from inside any container, I can reach any of the VM. So it is a one way issue.
And in addition, that problem applies only to ip4, with ip6 I do not experience these issues.

To give some numbers: The lan bridge (1) has the ip 172.16.32.1 and lets give one container the 172.16.32.100.


The other host has the 172.16.32.5 connected to the lan, it's bridge (2) has the 192.168.17.1, with one VM having the 192.168.17.10.

The lan bridge (1), ip: 172.16.32.1 has configured a route to reach the 192.168.17.0/24 via 172.16.32.5, the ip of the VM host.

So from the VM, 192.168.17.10, I can reach the bridge (1), 172.16.32.1, but not the attached container, 172.16.32.100. However, from inside the container 172.16.32.100 I can ping the VM, 192.168.17.10.


Now the interesting part:
Running tcpdump on the host with the 172.16.32.1 does show an echo reply from the container:


19:00:42.610210 IP 192.168.17.10 > 172.16.32.100: ICMP echo request, id 9, seq 488, length 64
19:00:42.610468 IP 172.16.32.100 > 192.168.17.10: ICMP echo reply, id 9, seq 488, length 64
19:00:43.634277 IP 192.168.17.10 > 172.16.32.100: ICMP echo request, id 9, seq 489, length 64
19:00:43.634572 IP 172.16.32.100 > 192.168.17.10: ICMP echo reply, id 9, seq 489, length 64

However, on the VM host (172.16.32.5), these do not arrive:

19:01:57.371755 IP 192.168.17.10 > 172.16.32.100: ICMP echo request, id 9, seq 561, length 64
19:01:58.395691 IP 192.168.17.10 > 172.16.32.100: ICMP echo request, id 9, seq 562, length 64
19:01:59.419799 IP 192.168.17.10 > 172.16.32.100: ICMP echo request, id 9, seq 563, length 64


Pinging the lxc host (172.16.32.1) however from the VM, and everything is fine;

19:03:11.716703 IP 192.168.17.10 > 172.16.32.1: ICMP echo request, id 10, seq 1, length 64
19:03:11.717474 IP 172.16.32.1 > 192.168.17.10: ICMP echo reply, id 10, seq 1, length 64

Just the containers attached to the bridge do not respond.

Doing a traceroute from a container (172.16.32.100) to the VM:
traceroute to 192.168.17.10 (192.168.17.10), 30 hops max, 46 byte packets
1  172.16.32.1  0.025 ms  0.018 ms  0.020 ms
2  172.16.37.5  0.237 ms  0.204 ms  0.225 ms
3  192.168.17.10  0.670 ms !A  0.610 ms !A  0.466 ms !A


The other way round however:
# tracepath -n 172.16.32.100
1?: [LOCALHOST]                      pmtu 1500
1:  192.168.17.1                                          0.281ms
1:  192.168.17.1                                          0.372ms
2:  no reply

Doing the traceroute to another physical host in the lan, however, works:

# tracepath -n 172.16.32.1
1?: [LOCALHOST]                      pmtu 1500
1:  192.168.17.1                                          0.398ms
1:  192.168.17.1                                          0.238ms
2:  172.16.32.7                                           0.974ms reached
     Resume: pmtu 1500 hops 2 back 2

What makes me wonder here, why don't I see the VM host IP (172.16.32.5) as a hop?  This I believe is the key to my problem. How can the not connected VM bridge ip go straight to the 172.16.32.0 lan without using the local, lan connected interface?


The corresponding tcpdump on 172.16.32.1:
19:07:04.678954 IP 192.168.17.10.40249 > 172.16.32.100.44446: UDP, length 1472
19:07:04.679183 IP 172.16.32.100 > 192.168.17.10: ICMP 172.16.32.101 udp port 44446 unreachable, length 556
19:07:05.680256 IP 192.168.17.10.40249 > 172.16.32.100.44447: UDP, length 1472
19:07:05.680530 IP 172.16.32.100 > 192.168.17.10: ICMP 172.16.32.101 udp port 44447 unreachable, length 556
19:07:06.681176 IP 192.168.17.10.40249 > 172.16.32.100.44448: UDP, length 1472


Why is the IP 192.168.17.10 unreachable? The route is existent? Maybe with that amateur ascii it becomes clearer: the 192.168.17.10 cannot reach the 172.16.32.100, but everything else and any other other client in the lan. The 172.16.32.100 however can reach the 192.168.17.10. Makes no sense whatsoever to me:

            +-+-+-+-+-+-+ Box 2 +-+-+-+-+-+-+
            |                               |
            |    +-+-+-+-+-+-+-+-+-+-+-+    |
            |    |     Bridge (VM)     |    |
            |    |  +-+-+-+-+-+-+-+-+-+|    |
            |    |  | vm 192.168.17.10 |    |
            |    |  +-+-+-+-+-+-+-+-+-+|    |
            |    |  |eth 1|            |    |
            |    |  +-+-+-+            |    |
            |    |   192.168.17.1/24   |    |
            |    +-+-+-+-+-+-+-+-+-+-+-+    |
            |                               |
            |    +-+-+-+-+-+-+-+-+-+-+-+    |
            |    |      LAN HOST       |    |
            |    |       +-+-+-+       |    |
 ||======== |    |       |eth0 |       |    |
 ||         |    |       +-+-+-+       |    |
 ||         |    |   172.16.32.5/24    |    |
 ||         |    +-+-+-+-+-+-+-+-+-+-+-+    |
 ||         |                               |
 ||         +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
 ||
 ||
 ||         +-+-+-+-+-+-+ Box 1 +-+-+-+-+-+-+
 ||         |                               |
 ||         |    +-+-+-+-+-+-+-+-+-+-+-+    |
 ||         |    |   Bridge (LAN)      |    |
 ||         |    | +-+-+-+-+-+-+-+-+-+ |    |
 ||======== |    | |lxc 172.16.32.100  |    |
            |    | +-+-+-+-+-+-+-+-+-+ |    |
            |    | |eth0 |             |    |
            |    | +-+-+-+             |    | 
            |    |  172.16.32.1/24     |    |
            |    +-+-+-+-+-+-+-+-+-+-+-+    |
            +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
   
  

Last edited by EdeWolf (2020-07-04 08:58:21)

Offline

#2 2020-07-01 13:50:03

Lone_Wolf
Member
From: Netherlands, Europe
Registered: 2005-10-04
Posts: 11,868

Re: Weird bridge / routing problem

Network connections follow the OSI model, https://en.wikipedia.org/wiki/OSI_model
IPv4 and IPv6 function on layer 4 while routing functions on layer 3.
connections between neighbouring network devices however  are  on layer 2.

The addresses you use are in separate private networks . IPv4 and IPv6 handle those differently , https://en.wikipedia.org/wiki/Private_network


Accessing different network requires routing , accessing one private network from another private network is disabled by default and requires specfic setup to work .

Your diagram indicates that the two devices on box1 are in the same lan and so is box2 eth0 .
These 3 devices can make connections on layer 4 and below.

On box2 2 separate lans are present. On the IP layer 4 they would need a router to communicate and none appears to be configured, so no layer 4 / IP communication .

Your results do indicate there's some communication though.
My best guess is that box2 eth0 & box2 eth1 are seeing eachother and communicating on layer 2.
No idea which mechanism is responsible for 172.16.32.7 being reachable from the VM .

The above is simplification that doesn't take into account that VM setups usually include their own routing support and linux bridges function differently then the bridge devices defined in the osi model (can't prove that last part, it may be a misunderstanding ).
Add ipv4 private networks and things get really messy.

In order to get this setup to function correctly on ipv4 , I'd probably ditch the bridges and maybe the vm routing stuff .

The linux bridges would be replaced by network cards so every device that needs network connections has its own network device.
Add switches to connect all devices in the same lan and routers to connect devices in different lans.

Using physical devices would be easiest, but if virtual devices are needed I would only use them when they are compliant with osi model and network protocols .

I have had good results in the past with VDE2 and VMWARE workstation network tools. TUN/TAP can also be useful but tend to be harder to setup.

The network parts of libvirt-manager , virtualbox , linux bridging , docker are things I try to stay away from.

Simple solution : ditch ipv4 and only use ipv6 .

If that's not possible : provide info which things you want to be able to connect and details about the bridge setups & vm network setup .

Last edited by Lone_Wolf (2020-07-01 13:50:46)


Disliking systemd intensely, but not satisfied with alternatives so focusing on taming systemd.


(A works at time B)  && (time C > time B ) ≠  (A works at time C)

Offline

#3 2020-07-03 10:49:48

EdeWolf
Member
Registered: 2016-01-06
Posts: 79

Re: Weird bridge / routing problem

Thanks very much for your long and detailed answer. Truth is, I did not really expect any answer at all. Highly appreciated. Hopefully, at least I do believe to having some basic networking knowledge. No expert though by any stretch.

In short, my problem ist, that all physical hosts on one another subnet are reachable from inside a VM, without those, that are being attached to a bridge - even though being in the same subnet. 

There is no routing going on inside the containers/VMs, they alle just have the default gw, that is the "host interface" of the bridge.   Or rather the bridge IP, to probably be more correct.

So, in the lan, we have 3 addresses: 172.16.32.1, 172.16.32.100, 172.16.32.7. The middle one is a container connected via a bridge, with the bridge IP being the 172.16.32.1.

So:

192.168.17.10 -> 172.16.32.1 OK! ( bridge IP)
192.168.17.10 -> 172.16.32.100 not OK (container, connected to above bride)
192.168.17.10 -> 172.16.32.7 OK! (phsysical network of another host connected to the lan)

As to be seen, all are within the same subnet, that is within the same broadcast domain. And that general routing is working, is proven by:

172.16.32.100 -> 192.168.17.10 OK!

So pinging:
192.168.17.10 -> 172.16.32.100 not OK
but
172.16.32.100 -> 192.168.17.10 OK!


The actual riddle is the traceroute from inside a virtual machine:

# tracepath -n 172.16.32.7
1?: [LOCALHOST]                      pmtu 1500
1:  192.168.17.1                                          0.398ms
1:  192.168.17.1                                          0.238ms
2:  172.16.32.7                                           0.974ms reached
     Resume: pmtu 1500 hops 2 back 2

I would have suspected it to be:

# tracepath -n 172.16.32.7
1?: [LOCALHOST]                      pmtu 1500
1:  192.168.17.1                                          0.398ms
1:  192.168.17.1                                          0.238ms
1: 172.16.23.5 <---- !MISSING!
2:  172.16.32.7                                           0.974ms reached
     Resume: pmtu 1500 hops 2 back 2

Desipte this trace is working, why does is not take the local host interface connected to the lan as hop? 


By the way, it is not VM specific, I just chose those terms to better differentiate between the two hosts, the problem persists with virtualbox, as well as with qemu as well as with lxc. And on more than one host.

Edit: For these tests VM=virtualbox, I am not using VMWare. This may have been misleading.

Last edited by EdeWolf (2020-07-03 11:02:42)

Offline

#4 2020-07-03 11:29:18

EdeWolf
Member
Registered: 2016-01-06
Posts: 79

Re: Weird bridge / routing problem

Trying to summarize my bad ascii art:

Dual Homed Host (VM)
VM Client: 192.168.17.10
VM Client GW: 192.168.17.1/24 (Bridge IP br.vbox / eth1)
Host Interface Lan: 172.16.32.5/24 (eth.lan0)

Routes
$ ip route show
default via 172.16.32.1 dev eth.lan0 proto static
172.16.32.0/24 dev eth.lan0 proto kernel scope link src 172.16.32.5
192.168.17.0/24 dev br.vbox proto kernel scope link src 192.168.17.1

LXC Host:
LXC Guest: 172.16.32.100/24
LXC GW: 172.16.32.1/24 (Bridge IP br.lan / eth0)

Routes:
# ip route show
default dev ppp0 scope link
172.16.32.0/24 dev br.lan proto kernel scope link src 172.16.32.1
192.168.17.0/24 via 172.16.32.5 dev br.lan proto static

ipv4/ip_forward is set to 1 on all machines.

Last edited by EdeWolf (2020-07-03 11:31:31)

Offline

#5 2020-07-03 12:16:37

seth
Member
Registered: 2012-09-03
Posts: 49,992

Re: Weird bridge / routing problem

How exactly is the dual homed server configured?
Typically there would be a firewall (iptables/netfilter) to keep the subnets apart, so incoming traffic from 192.168.17.* for 172.16.32.* would be dropped and that seems to be the case here.

(The next question would be: if you want common network segment w/o separated subnets, why is this a dual homed server itfp?)

Offline

#6 2020-07-03 12:57:48

EdeWolf
Member
Registered: 2016-01-06
Posts: 79

Re: Weird bridge / routing problem

Thanks very much for joining. There is no firewall involved here at all.
The default settings, I have neither iptables nor nftables rules:

# iptables -L
Chain INPUT (policy ACCEPT)
target     prot opt source               destination         

Chain FORWARD (policy ACCEPT)
target     prot opt source               destination         

Chain OUTPUT (policy ACCEPT)
target     prot opt source               destination         

The problem seems to be, that the ping reply from the container:

19:00:42.610210 IP 192.168.17.10 > 172.16.32.100: ICMP echo request, id 9, seq 488, length 64
19:00:42.610468 IP 172.16.32.100 > 192.168.17.10: ICMP echo reply, id 9, seq 488, length 64

Does for some reason and despite a matching route, not arrive on the dual homed host:

19:01:57.371755 IP 192.168.17.10 > 172.16.32.100: ICMP echo request, id 9, seq 561, length 64
19:01:58.395691 IP 192.168.17.10 > 172.16.32.100: ICMP echo request, id 9, seq 562, length 64

Those are the dumps taken from the hosts, not inside any VM/container. The upper one from 172.16.32.1, the lower one from the dual homed host.

The reply is missing. However, as written before, when the container pings itself (instead of being pinged), the ping request and reply arrive.

What configuration information precicely are you looking for? I'll happily provide those.

Offline

#7 2020-07-04 08:47:38

EdeWolf
Member
Registered: 2016-01-06
Posts: 79

Re: Weird bridge / routing problem

The VM host:

$ ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host 
       valid_lft forever preferred_lft forever
2: eth.lan1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel master br.vbox state UP group default qlen 1000
    link/ether d0:50:99:72:3f:41 brd ff:ff:ff:ff:ff:ff
3: eth.lan0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UP group default qlen 1000
    link/ether d0:50:99:72:3f:43 brd ff:ff:ff:ff:ff:ff
    inet 172.16.32.5/24 brd 172.16.39.255 scope global eth.lan0
       valid_lft forever preferred_lft forever
    inet6 fd8a:94b1:fb84:1:d250:99ff:fe72:3f43/64 scope global dynamic mngtmpaddr noprefixroute 
       valid_lft 86398sec preferred_lft 14398sec
    inet6 fe80::4078:a461:fdb3:22d5/64 scope link stable-privacy 
       valid_lft forever preferred_lft forever
4: br.vbox: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
    link/ether 16:60:44:55:66:10 brd ff:ff:ff:ff:ff:ff
    inet 192.168.17.1/24 brd 192.168.17.255 scope global br.vbox
       valid_lft forever preferred_lft forever
    inet6 fdc6:2452:11ef:17::1/64 scope global 
       valid_lft forever preferred_lft forever
    inet6 fe80::c10e:7ee5:bfef:587c/64 scope link stable-privacy 
       valid_lft forever preferred_lft forever
$  ip route show
default via 172.16.32.250 dev eth.lan0 proto static 
172.16.32.0/24 dev eth.lan0 proto kernel scope link src 172.16.32.5 
192.168.17.0/24 dev br.vbox proto kernel scope link src 192.168.17.1 
$  bridge link show
2: eth.lan1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 master br.vbox state forwarding priority 32 cost 4 

And the(one) VM on this host:

# ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host 
       valid_lft forever preferred_lft forever
2: enp0s3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UP group default qlen 1000
    link/ether 16:60:44:55:66:17 brd ff:ff:ff:ff:ff:ff
    inet 192.168.17.7/24 brd 192.168.17.255 scope global enp0s3
       valid_lft forever preferred_lft forever
    inet6 fdc6:2452:11ef:17:1460:44ff:fe55:6617/64 scope global dynamic mngtmpaddr noprefixroute 
       valid_lft 86395sec preferred_lft 14395sec
    inet6 fe80::1460:44ff:fe55:6617/64 scope link 
       valid_lft forever preferred_lft forever
# ip route show
default via 192.168.17.1 dev enp0s3 proto static 
192.168.17.0/24 dev enp0s3 proto kernel scope link src 192.168.17.7 

Last edited by EdeWolf (2020-07-04 08:51:40)

Offline

Board footer

Powered by FluxBB