You are not logged in.
Hello,
A bit complicated, but worth a try and I give my very best.
I am having troubles bringing ipvlan l3s with nspawn to work. Basically, I can reach every ip that is somehow physically related to the ipvlan, but none outside. Be it on the same host or, naturally then, on any other.
So, from the following diagram, from inside any container, I of course am able to reach the other container, the lokal ipvlan interface (10.10.30.1/24) and the IP address of eth1 (172.16.42.1/24).
However, I can not reach the ip of eth0 or any host attached thereto.
From the outside host (192.168.17.1), I can reach eth1 (172.16.42.1/24) as well as the local ipvlan interface (10.10.30.1/24). But not any container. Needless to say, there is no paketfilter is place (all policies are accept and I just log icmp) and ipforwarding is globally enabled for both ip4 and ip6:
+---------------------------------------------------------------------+
| |
| |
| +----------------+ +----------------+ |
| | ipvlan-1 nspawn| | ipvlan-2 nspawn| |
| | 10.10.10.1/24/ | | 10.10.20.1/24 | |
| | | | | |
| | default via | | default via | |
| | 10.10.10.1 | | 10.10.20.1 | |
| +-------+--------+ +--------+-------+ |
| | | |
| | | |
| | +---------------+ | |
| +--+ ipvlan +--+ |
| | 10.10.30.1/24 | |
| default via 192.168.17.1 +---------------+ |
| +----------------+ +---------------+ |
| | eth0 | | eth1 | |
| | 192.168.17.21 | | 172.16.42.1/24| |
| | /24 | | | |
+-------+-------+--------+----------+-------+-------+-----------------+
| |
| | route add 10.10.30.1 dev ipvlan
| +------ route add 10.10.20.1 dev ipvlan
| route add 10.10.10.1 dev ipvlan
+------+-------+--------+-----------------+
| | eth0 | |
| | 192.168.17.1 | |
| | /24 | |
| +----------------+ |
| |
| 10.10.0.0/16 via 192.168.17.21 |
| 172.16.42.0/24 via 192.168.17.21 |
| |
+-----------------------------------------+
The surprising part seems to be, that the communication between the two physical interfaces is broken, not to the ipvlan itself.
Pinging from the outside host:
Aug 02 09:34:32 arch-21 kernel: nftables: INPUTv4 IN=eth0 OUT=ipvlan MAC=16:60:44:55:66:31:16:60:44:55:66:10:08:00 SRC=192.168.17.1 DST=10.10.20.1 LEN=84 TOS=0x00 PREC=0x00 TTL=63 ID=44986 DF PROTO=ICMP TYPE=8 CODE=0 ID=12 SEQ=1291
Aug 02 09:34:32 arch-21 kernel: nftables: INPUTv4 IN= OUT=eth1 SRC=10.10.20.1 DST=192.168.17.1 LEN=84 TOS=0x00 PREC=0x00 TTL=64 ID=32428 PROTO=ICMP TYPE=0 CODE=0 ID=12 SEQ=1291 and a tcpdump shows pretty much the same:
# tcpdump -nvi any icmp
tcpdump: data link type LINUX_SLL2
tcpdump: listening on any, link-type LINUX_SLL2 (Linux cooked v2), snapshot length 262144 bytes
09:58:09.294204 eth0 In IP (tos 0x0, ttl 64, id 55025, offset 0, flags [DF], proto ICMP (1), length 84)
192.168.17.1 > 10.10.20.1: ICMP echo request, id 18, seq 1, length 64
09:58:09.294249 ipvlan Out IP (tos 0x0, ttl 63, id 55025, offset 0, flags [DF], proto ICMP (1), length 84)
192.168.17.1 > 10.10.20.1: ICMP echo request, id 18, seq 1, length 64
09:58:09.294269 eth1 Out IP (tos 0x0, ttl 64, id 34127, offset 0, flags [none], proto ICMP (1), length 84)
10.10.20.1 > 192.168.17.1: ICMP echo reply, id 18, seq 1, length 64Basically, the ping actually seems to get through to the container, who creates a reply, but when the packet hits the physical eth1 interface, it does not now how to proceed to eth0. Which is strage, as both interfaces are in the local namespace and known to the kernel and I am not sure, how to create the obvious missing route - something that should not be necessary in the first place.
This is true for the reverse as well, now trying to ping the outside host (192.168.17.1) from inside a container:
Aug 02 10:01:37 arch-21 kernel: nftables: INPUTv4 IN= OUT=eth1 SRC=10.10.20.1 DST=192.168.17.1 LEN=84 TOS=0x00 PREC=0x00 TTL=64 ID=1014 DF PROTO=ICMP TYPE=8 CODE=0 ID=46770 SEQ=1 # tcpdump -nvi any icmp
tcpdump: data link type LINUX_SLL2
tcpdump: listening on any, link-type LINUX_SLL2 (Linux cooked v2), snapshot length 262144 bytes
10:02:24.443408 eth1 Out IP (tos 0x0, ttl 64, id 9925, offset 0, flags [DF], proto ICMP (1), length 84)
10.10.20.1 > 192.168.17.1: ICMP echo request, id 17210, seq 1, length 64Jep, it is just that one stage.
And again, the paket reaches eth1, but does not get forwarded to eth0. Or eth0 does not know how to reply, or wants to take a different path to get to the container address (10.10.20.1). But I would not know how to figure out as tracepath is no help. It stops at the eth0 ip.
Finally, the same picture when trying to ping eth0 (192.168.17.21), instead of the outside host from inside a container:
Aug 02 10:04:51 arch-21 kernel: nftables: INPUTv4 IN= OUT=eth1 SRC=10.10.20.1 DST=192.168.17.21 LEN=84 TOS=0x00 PREC=0x00 TTL=64 ID=22100 DF PROTO=ICMP TYPE=8 CODE=0 ID=57791 SEQ=10 # tcpdump -nvi any icmp
tcpdump: data link type LINUX_SLL2
tcpdump: listening on any, link-type LINUX_SLL2 (Linux cooked v2), snapshot length 262144 bytes
10:05:56.759114 eth1 Out IP (tos 0x0, ttl 64, id 32409, offset 0, flags [DF], proto ICMP (1), length 84)
10.10.20.1 > 192.168.17.21: ICMP echo request, id 30783, seq 1, length 64Again, the pakets do not get beyond eth1.
The routing table of the host:
# ip r
default via 192.168.17.1 dev eth0 proto static
10.10.10.0/24 dev ipvlan proto static scope link
10.10.20.0/24 dev ipvlan proto static scope link
10.10.30.0/24 dev ipvlan proto kernel scope link src 10.10.30.1
172.16.42.0/24 dev eth1 proto kernel scope link src 172.16.42.1
192.168.17.0/24 dev eth0 proto kernel scope link src 192.168.17.21 # cat /proc/sys/net/ipv4/conf/all/forwarding
1
# cat /proc/sys/net/ipv4/conf/eth0/forwarding
1
# cat /proc/sys/net/ipv4/conf/eth1/forwarding
1Clearly, the kernel sees all interfaces, why, since it already has left the ipvlan (as eth1 is not really part of it, just the host interface) don't the pakets get forwarded?
Thanks for any ideas or input
Ede
Edit: Typo in ifname
Last edited by EdeWolf (2024-08-02 10:34:19)
Offline
Just an interesting side note: It seems to be an ipv4 problem only, ipv6 has no such issues. Using ip6 I can happily ping around in all directions and all targets do work. For trying to fix ip4 I've played around with manually trying to add neighbours, but so far no success at all.
This is how an ip6 ping from a container to the outside host (subnet: 1a17) looks:
Aug 02 13:48:27 arch-21 kernel: nftables: INPUTv6 IN= OUT=eth0 SRC=fde7:dead:beef:1020:0000:0000:0000:0001 DST=fde7:dead:beef:1a17:0000:0000:0000:0001 LEN=104 TC=0 HOPLIMIT=64 FLOWLBL=228688 PROTO=ICMPv6 TYPE=128 CODE=0 ID=58082 SEQ=22
Aug 02 13:48:27 arch-21 kernel: nftables: INPUTv6 IN=eth0 OUT=ipvlan MAC=16:60:44:55:66:31:16:60:44:55:66:10:86:dd SRC=fde7:dead:beef:1a17:0000:0000:0000:0001 DST=fde7:dead:beef:1020:0000:0000:0000:0001 LEN=104 TC=0 HOPLIMIT=63 FLOWLBL=1034024 PROTO=ICMPv6 TYPE=129 CODE=0 ID=58082 SEQ=2213:48:54.542345 eth0 Out IP6 (flowlabel 0x37d50, hlim 64, next-header ICMPv6 (58) payload length: 64) fde7:dead:beef:1020::1 > fde7:dead:beef:1a17::1: [icmp6 sum ok] ICMP6, echo request, id 58082, seq 49
13:48:54.542431 eth0 In IP6 (flowlabel 0xfc728, hlim 64, next-header ICMPv6 (58) payload length: 64) fde7:dead:beef:1a17::1 > fde7:dead:beef:1020::1: [icmp6 sum ok] ICMP6, echo reply, id 58082, seq 49
13:48:54.542441 ipvlan Out IP6 (flowlabel 0xfc728, hlim 63, next-header ICMPv6 (58) payload length: 64) fde7:dead:beef:1a17::1 > fde7:dead:beef:1020::1: [icmp6 sum ok] ICMP6, echo reply, id 58082, seq 49and the return path, pinging a container from the outside host:
Aug 02 13:44:20 arch-21 kernel: nftables: INPUTv6 IN=eth0 OUT=ipvlan MAC=16:60:44:55:66:31:16:60:44:55:66:10:86:dd SRC=fde7:dead:beef:1a17:0000:0000:0000:0001 DST=fde7:dead:beef:1010:0000:0000:0000:0001 LEN=104 TC=0 HOPLIMIT=63 FLOWLBL=877727 PROTO=ICMPv6 TYPE=128 CODE=0 ID=42 SEQ=142
Aug 02 13:44:20 arch-21 kernel: nftables: INPUTv6 IN= OUT=eth0 SRC=fde7:dead:beef:1010:0000:0000:0000:0001 DST=fde7:dead:beef:1a17:0000:0000:0000:0001 LEN=104 TC=0 HOPLIMIT=64 FLOWLBL=819854 PROTO=ICMPv6 TYPE=129 CODE=0 ID=42 SEQ=142 13:42:00.468017 eth0 In IP6 (flowlabel 0xd649f, hlim 64, next-header ICMPv6 (58) payload length: 64) fde7:dead:beef:1a17::1 > fde7:dead:beef:1010::1: [icmp6 sum ok] ICMP6, echo request, id 42, seq 4
13:42:00.468047 ipvlan Out IP6 (flowlabel 0xd649f, hlim 63, next-header ICMPv6 (58) payload length: 64) fde7:dead:beef:1a17::1 > fde7:dead:beef:1010::1: [icmp6 sum ok] ICMP6, echo request, id 42, seq 4
13:42:00.468065 eth0 Out IP6 (flowlabel 0xc828e, hlim 64, next-header ICMPv6 (58) payload length: 64) fde7:dead:beef:1010::1 > fde7:dead:beef:1a17::1: [icmp6 sum ok] ICMP6, echo reply, id 42, seq 4So, if against all odds, anyone has an idea, what may go on here, or rather, what may be the differences, that would be really helpful, as I am running out of ideas. I do realise, eth1 is not used in this case, at least not mentioned. It is just not clear, why. The ipv6 routing looks pretty much the same.
# ip -6 r
fde7:dead:beef:421::/64 dev eth1 proto kernel metric 256 pref medium
fde7:dead:beef:1010::/64 dev ipvlan proto static metric 1024 pref medium
fde7:dead:beef:1020::/64 dev ipvlan proto static metric 1024 pref medium
fde7:dead:beef:1030::1 dev ipvlan proto kernel metric 256 pref medium
fde7:dead:beef:1a17::/64 dev eth0 proto ra metric 2048 expires 86176sec pref low
fe80::/64 dev eth0 proto kernel metric 256 pref medium
fe80::/64 dev eth1 proto kernel metric 256 pref medium
default via fe80::1a17:1 dev eth0 proto ra metric 2048 expires 5176sec pref low
# ip r
default via 192.168.17.1 dev eth0 proto static
10.10.10.0/24 dev ipvlan proto static scope link
10.10.20.0/24 dev ipvlan proto static scope link
10.10.30.0/24 dev ipvlan proto kernel scope link src 10.10.30.1
172.16.42.0/24 dev eth1 proto kernel scope link src 172.16.42.1
192.168.17.0/24 dev eth0 proto kernel scope link src 192.168.17.21 Edit: added logs
Last edited by EdeWolf (2024-08-02 14:02:28)
Offline
Just sorting the routes from above for better comparisson of ip4 and ip6, to figure out, why ip4 uses eth1 as an output interface, while it seemingly does not with ip6:
fde7:dead:beef:421::/64 dev eth1 proto kernel metric 256 pref medium
172.16.42.0/24 dev eth1 proto kernel scope link src 172.16.42.1
-
fde7:dead:beef:1010::/64 dev ipvlan proto static metric 1024 pref medium
10.10.10.0/24 dev ipvlan proto static scope link
-
fde7:dead:beef:1020::/64 dev ipvlan proto static metric 1024 pref medium
10.10.20.0/24 dev ipvlan proto static scope link
-
fde7:dead:beef:1030::1 dev ipvlan proto kernel metric 256 pref medium
10.10.30.0/24 dev ipvlan proto kernel scope link src 10.10.30.1
-
fde7:dead:beef:1a17::/64 dev eth0 proto ra metric 2048 expires 86176sec pref low
192.168.17.0/24 dev eth0 proto kernel scope link src 192.168.17.21
-
default via fe80::1a17:1 dev eth0 proto ra metric 2048 expires 5176sec pref low
default via 192.168.17.1 dev eth0 proto static
---
fe80::/64 dev eth0 proto kernel metric 256 pref medium
fe80::/64 dev eth1 proto kernel metric 256 pref mediumOffline
Just another rather strange oddity:
With ip6 I do not even need an ip address on the local ipvlan interface. It still works, even from outside.
For ip4 that is required (at least to being able to ping the container from within the local host. ip6 still works from everywhere). Having an ip address on the local host interface (eth1) is optional in both cases - with the caveat, that the container currently cannot talk to the outside world using ipv4. But they cannot both ways.
Offline