You are not logged in.
Hi,
I upgraded systemd from 246.6 to 247.1 and it has broken the networking for all my (unprivileged) containers.
This post (https://forum.proxmox.com/threads/solve … lxc.80115/) says the cause that since version 247, systemd-networkd is launched in a separate namespace and that the solution is enable namespace nesting.
That post concerns LXC containers. Any idea how to enable namespace nesting with systemd-nspawnd (or another way to solve the problem)?
Barius
Last edited by barius (2020-12-26 21:58:44)
Offline
More information...
Firstly, I have struck out part of the original post because I now question its relevance: the issue in that proxmox forum post happens when the archlinux container upgraded to systemd-247, the host being debian-based with a slightly older systemd.
I have the problem when the host runs systemd-247.1 or 247.2 and it is resolved if I downgrade the host to systemd-246. It does not matter if the container runs systemd 246 or 247.
The nested namespace trail did not lead me anywhere (my lack of understanding?) and adding --capability=all to the container did not help either.
Secondly, I have determined that the problem specifically affects bridged networking. (Post subject edited accordingly.)
A virtual ethernet link seems to work. I.e. if I spin up a test container like this, everything works as expected. I can ping the container from the host and vice-versa:
sudo systemd-nspawn --boot --network-veth --private-users=pick --machine=test1However, if I spin it up like this, ping in both directions answers with "Destination Host Unreachable" although all the interfaces are up:
sudo systemd-nspawn --boot --network-bridge=vbr0 --private-users=pick --machine=test1I am using bridged networking and static IPv4 addresses because the nspawnd containers run services which must be reachable on the LAN.
The host is set up like this:
::::::::::::::
/etc/systemd/network/10-vbr0.netdev
::::::::::::::
[Match]
Virtualization=no
[NetDev]
Description=Virtual ethernet bridge
Name=vbr0
Kind=bridge
[Bridge]
ForwardDelaySec=0::::::::::::::
/etc/systemd/network/20-eno1.network
::::::::::::::
[Match]
Name=eno1
[Network]
DHCP=no
LinkLocalAddressing=no
Bridge=vbr0::::::::::::::
/etc/systemd/network/50-vbr0.network
::::::::::::::
[Match]
Name=vbr0
[Network]
DHCP=no
LinkLocalAddressing=no
Address=192.168.69.2/26
Gateway=192.168.69.1::::::::::::::
/etc/systemd/network/60-vb.network
::::::::::::::
[Match]
Name=vb-*
Driver=veth
[Network]
Description=Host nic connected to bridge and container
DHCP=no
LinkLocalAddressing=no$ ip address
[cut]
2: eno1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel master vbr0 state UP group default qlen 1000
link/ether b8:ca:3a:b8:a6:ec brd ff:ff:ff:ff:ff:ff
altname enp0s25
3: vbr0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
link/ether 52:67:88:8c:e0:20 brd ff:ff:ff:ff:ff:ff
inet 192.168.69.2/26 brd 192.168.69.63 scope global vbr0
valid_lft forever preferred_lft forever
[cut]
14: vb-test1@if2: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master vbr0 state UP group default qlen 1000
link/ether 7a:9a:68:bf:c2:b9 brd ff:ff:ff:ff:ff:ff link-netnsid 3Inside the container (I disabled this when testing virtual ethernet link without bridge to have the out-of-the-box DHCP address):
::::::::::::::
/etc/systemd/network/60-container.network
::::::::::::::
[Match]
Virtualization=container
Name=host*
[Network]
Description=Container nic connected to host
DHCP=no
LinkLocalAddressing=no
Address=192.168.69.14/26
Gateway=192.168.69.1# ip address
[cut]
2: host0@if14: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
link/ether 92:0a:f6:15:cb:b8 brd ff:ff:ff:ff:ff:ff link-netnsid 0
inet 192.168.69.14/26 brd 192.168.69.63 scope global host0
valid_lft forever preferred_lft foreverThe host is aware of the container's IP address but cannot reach it.
$ machinectl list
MACHINE CLASS SERVICE OS VERSION ADDRESSES
[cut]
test1 container systemd-nspawn arch - 192.168.69.14$ ping -nc2 192.168.69.14
PING 192.168.69.14 (192.168.69.14) 56(84) bytes of data.
From 192.168.69.2 icmp_seq=1 Destination Host Unreachable
From 192.168.69.2 icmp_seq=2 Destination Host Unreachable
--- 192.168.69.14 ping statistics ---
2 packets transmitted, 0 received, +2 errors, 100% packet loss, time 1017msIf I downgrade systemd on the host
sudo pacman -U --asdeps /var/cache/pacman/pkg/systemd-246.6-1-x86_64.pkg.tar.zstthen restart systemd-networkd and the container, then everything works again:
$ ping -nc2 192.168.69.14
PING 192.168.69.14 (192.168.69.14) 56(84) bytes of data.
64 bytes from 192.168.69.14: icmp_seq=1 ttl=64 time=0.031 ms
64 bytes from 192.168.69.14: icmp_seq=2 ttl=64 time=0.032 ms
--- 192.168.69.14 ping statistics ---
2 packets transmitted, 2 received, 0% packet loss, time 1009ms
rtt min/avg/max/mdev = 0.031/0.031/0.032/0.000 msI am out of ideas. Any tips what to try next? It is really not a long-term option to hold systemd back to version 246 forever...
Offline
So, it turns out that the problem lies with the /etc/systemd/network/60-vb.network above. Not so much that there's anything wrong with the content of the file but rather, it's mere presence...
The option --network-bridge=vbr0 works very much like the --network-veth, but with two differences:
The virtual ethernet interface on the host has a name prefixed with "vb-" instead of "ve-".
The vb-container_name is also connected to the virtual bridge vbr0 on the host.
That is the documented behaviour.
As long as you is no systemd.network(5) file matching that vb-* interface, that is how it works. However, if there is a matching *.network file, then the behaviour changes starting with systemd-247:
Up until systemd-246.6, the settings in the *.network file are honoured and the vb-* interface is connected to the bridge named by the --network-bridge= option.
This is also the behaviour I would expect from reading the documentation: systemd-nspawn(1).
Starting from systemd-247.1, the connection to the named bridge is no longer made if there exists a *.network file matching the vb-* interface. If such a file exists, it must also (redundantly) contain the Bridge= setting in the [Network] section.
The file I mentioned previously can thus be "fixed"
::::::::::::::
/etc/systemd/network/60-vb.network
::::::::::::::
[Match]
Name=vb-*
Driver=veth
[Network]
Description=Host nic connected to bridge and container
DHCP=no
Bridge=vbr0IMHO this is a regression in systemd because it contradicts the documentation of the --network-bridge= option in systemd-nspawn(1), namely:
"Adds the host side of the Ethernet link [...] to the specified Ethernet bridge interface."
Any informed thoughts on this last point?
Offline