You are not logged in.

#1 2020-06-30 17:41:49

EdeWolf
Member
Registered: 2016-01-06
Posts: 46

Weird bridge / routing problem

Ok, it is getting mighty complicated, but hope dies last.

So I have a bridge (1), that connects to both the lan and some lxc containers. Another, dual homed, host is connected to that lan, with the not connected interface hosting a bridge (2) as well, that hosts VMs.

Now the problem is, from inside a VM I cannot reach the lxc containers. Strangely however, from inside any container, I can reach any of the VM. So it is a one way issue.
And in addition, that problem applies only to ip4, with ip6 I do not experience these issues.

To give some numbers: The lan bridge (1) has the ip 172.16.32.1 and lets give one container the 172.16.32.100.


The other host has the 172.16.32.5 connected to the lan, it's bridge (2) has the 192.168.17.1, with one VM having the 192.168.17.10.

The lan bridge (1), ip: 172.16.32.1 has configured a route to reach the 192.168.17.0/24 via 172.16.32.5, the ip of the VM host.

So from the VM, 192.168.17.10, I can reach the bridge (1), 172.16.32.1, but not the attached container, 172.16.32.100. However, from inside the container 172.16.32.100 I can ping the VM, 192.168.17.10.


Now the interesting part:
Running tcpdump on the host with the 172.16.32.1 does show an echo reply from the container:


19:00:42.610210 IP 192.168.17.10 > 172.16.32.100: ICMP echo request, id 9, seq 488, length 64
19:00:42.610468 IP 172.16.32.100 > 192.168.17.10: ICMP echo reply, id 9, seq 488, length 64
19:00:43.634277 IP 192.168.17.10 > 172.16.32.100: ICMP echo request, id 9, seq 489, length 64
19:00:43.634572 IP 172.16.32.100 > 192.168.17.10: ICMP echo reply, id 9, seq 489, length 64

However, on the VM host (172.16.32.5), these do not arrive:

19:01:57.371755 IP 192.168.17.10 > 172.16.32.100: ICMP echo request, id 9, seq 561, length 64
19:01:58.395691 IP 192.168.17.10 > 172.16.32.100: ICMP echo request, id 9, seq 562, length 64
19:01:59.419799 IP 192.168.17.10 > 172.16.32.100: ICMP echo request, id 9, seq 563, length 64


Pinging the lxc host (172.16.32.1) however from the VM, and everything is fine;

19:03:11.716703 IP 192.168.17.10 > 172.16.32.1: ICMP echo request, id 10, seq 1, length 64
19:03:11.717474 IP 172.16.32.1 > 192.168.17.10: ICMP echo reply, id 10, seq 1, length 64

Just the containers attached to the bridge do not respond.

Doing a traceroute from a container (172.16.32.100) to the VM:
traceroute to 192.168.17.10 (192.168.17.10), 30 hops max, 46 byte packets
1  172.16.32.1  0.025 ms  0.018 ms  0.020 ms
2  172.16.37.5  0.237 ms  0.204 ms  0.225 ms
3  192.168.17.10  0.670 ms !A  0.610 ms !A  0.466 ms !A


The other way round however:
# tracepath -n 172.16.32.100
1?: [LOCALHOST]                      pmtu 1500
1:  192.168.17.1                                          0.281ms
1:  192.168.17.1                                          0.372ms
2:  no reply

Doing the traceroute to another physical host in the lan, however, works:

# tracepath -n 172.16.32.1
1?: [LOCALHOST]                      pmtu 1500
1:  192.168.17.1                                          0.398ms
1:  192.168.17.1                                          0.238ms
2:  172.16.32.7                                           0.974ms reached
     Resume: pmtu 1500 hops 2 back 2

What makes me wonder here, why don't I see the VM host IP (172.16.32.5) as a hop?  This I believe is the key to my problem. How can the not connected VM bridge ip go straight to the 172.16.32.0 lan without using the local, lan connected interface?


The corresponding tcpdump on 172.16.32.1:
19:07:04.678954 IP 192.168.17.10.40249 > 172.16.32.100.44446: UDP, length 1472
19:07:04.679183 IP 172.16.32.100 > 192.168.17.10: ICMP 172.16.32.101 udp port 44446 unreachable, length 556
19:07:05.680256 IP 192.168.17.10.40249 > 172.16.32.100.44447: UDP, length 1472
19:07:05.680530 IP 172.16.32.100 > 192.168.17.10: ICMP 172.16.32.101 udp port 44447 unreachable, length 556
19:07:06.681176 IP 192.168.17.10.40249 > 172.16.32.100.44448: UDP, length 1472


Why is the IP 192.168.17.10 unreachable? The route is existent? Maybe with that amateur ascii it becomes clearer: the 192.168.17.10 cannot reach the 172.16.32.100, but everything else and any other other client in the lan. The 172.16.32.100 however can reach the 192.168.17.10. Makes no sense whatsoever to me:

            +-+-+-+-+-+-+ Box 2 +-+-+-+-+-+-+
            |                               |
            |    +-+-+-+-+-+-+-+-+-+-+-+    |
            |    |     Bridge (VM)     |    |
            |    |  +-+-+-+-+-+-+-+-+-+|    |
            |    |  | vm 192.168.17.10 |    |
            |    |  +-+-+-+-+-+-+-+-+-+|    |
            |    |  |eth 1|            |    |
            |    |  +-+-+-+            |    |
            |    |   192.168.17.1/24   |    |
            |    +-+-+-+-+-+-+-+-+-+-+-+    |
            |                               |
            |    +-+-+-+-+-+-+-+-+-+-+-+    |
            |    |      LAN HOST       |    |
            |    |       +-+-+-+       |    |
 ||======== |    |       |eth0 |       |    |
 ||         |    |       +-+-+-+       |    |
 ||         |    |   172.16.37.5/21    |    |
 ||         |    +-+-+-+-+-+-+-+-+-+-+-+    |
 ||         |                               |
 ||         +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
 ||
 ||
 ||         +-+-+-+-+-+-+ Box 1 +-+-+-+-+-+-+
 ||         |                               |
 ||         |    +-+-+-+-+-+-+-+-+-+-+-+    |
 ||         |    |   Bridge (LAN)      |    |
 ||         |    | +-+-+-+-+-+-+-+-+-+ |    |
 ||======== |    | |lxc 172.16.32.100  |    |
            |    | +-+-+-+-+-+-+-+-+-+ |    |
            |    | |eth0 |             |    |
            |    | +-+-+-+             |    | 
            |    |  172.16.32.1/21     |    |
            |    +-+-+-+-+-+-+-+-+-+-+-+    |
            +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
   
  

Offline

#2 Yesterday 13:50:03

Lone_Wolf
Member
From: Netherlands, Europe
Registered: 2005-10-04
Posts: 8,189

Re: Weird bridge / routing problem

Network connections follow the OSI model, https://en.wikipedia.org/wiki/OSI_model
IPv4 and IPv6 function on layer 4 while routing functions on layer 3.
connections between neighbouring network devices however  are  on layer 2.

The addresses you use are in separate private networks . IPv4 and IPv6 handle those differently , https://en.wikipedia.org/wiki/Private_network


Accessing different network requires routing , accessing one private network from another private network is disabled by default and requires specfic setup to work .

Your diagram indicates that the two devices on box1 are in the same lan and so is box2 eth0 .
These 3 devices can make connections on layer 4 and below.

On box2 2 separate lans are present. On the IP layer 4 they would need a router to communicate and none appears to be configured, so no layer 4 / IP communication .

Your results do indicate there's some communication though.
My best guess is that box2 eth0 & box2 eth1 are seeing eachother and communicating on layer 2.
No idea which mechanism is responsible for 172.16.32.7 being reachable from the VM .

The above is simplification that doesn't take into account that VM setups usually include their own routing support and linux bridges function differently then the bridge devices defined in the osi model (can't prove that last part, it may be a misunderstanding ).
Add ipv4 private networks and things get really messy.

In order to get this setup to function correctly on ipv4 , I'd probably ditch the bridges and maybe the vm routing stuff .

The linux bridges would be replaced by network cards so every device that needs network connections has its own network device.
Add switches to connect all devices in the same lan and routers to connect devices in different lans.

Using physical devices would be easiest, but if virtual devices are needed I would only use them when they are compliant with osi model and network protocols .

I have had good results in the past with VDE2 and VMWARE workstation network tools. TUN/TAP can also be useful but tend to be harder to setup.

The network parts of libvirt-manager , virtualbox , linux bridging , docker are things I try to stay away from.

Simple solution : ditch ipv4 and only use ipv6 .

If that's not possible : provide info which things you want to be able to connect and details about the bridge setups & vm network setup .

Last edited by Lone_Wolf (Yesterday 13:50:46)


Multi-init booting with apg Openrc and systemd coexisting
Automounting : not needed, i prefer pmount
Aur helpers : makepkg + my own local repo === rarely need them

Offline

Board footer

Powered by FluxBB