You are not logged in.

#1 2023-06-04 05:22:14

SJ
Member
Registered: 2017-12-25
Posts: 17

[Solved]Provide different gateways for qemu and/or container instances

On my host with a single physial interface I want to run multiple instances of qemu. In my particular use case, it's multiple instances of the Android Studio emulator. It would probably be a good idea to separate those instances even further in multiple docker or podman containers. Hence the subject.
Unfortunately, there are not a lot of configuration options for their network settings and how these connect to the host.

There are two scenarios I was hoping two support:
- Let a dedicated firewall route those instances to the correct gateways. In this case, all instances need their own respective IP address to write the rules for. I also need the instances to use the correct VLAN.
- Configure everything on the host. The instances need to be able to run in parallel and gateway configuration need to possible individually for all instances.

Note: If there is an even better approach to my overall goal, please let me know.

I cannt test the first approach right now, so I describe the setup for the second approach:
My host has a single physical interface. Two VLAN interfaces are configured. Please see a schematic of the setup. More detailed configuration below.

            ---------
            | vlan2 |
            ---------
---------       |
| eno1  | ______|
---------       |
            ---------
            | vlan3 |
            --------- 

$ cat /etc/systemd/network/eno1.network
[Match]
Name=eno1

[Network]
DHCP=no
VLAN=vlan.2
VLAN=vlan.3

$ cat /etc/systemd/network/vlan.2.netdev
[NetDev]
Name=vlan.2
Kind=vlan

[VLAN]
Id=2

$ cat /etc/systemd/network/vlan.3.netdev
[NetDev]
Name=vlan.3
Kind=vlan

[VLAN]
Id=3

$ cat /etc/systemd/network/vlan.2.network
[Match]
Name=vlan.2

[Network]
Address=192.168.2.10/24
Gateway=192.168.2.200/24
DNS=8.8.8.8

$ cat /etc/systemd/network/vlan.3.network
[Match]
Name=vlan.3

[Network]
Address=192.168.3.10/24

In the 192.168.2.0/24 subnet, multiple gateways are available and different qemu instances should be configured to use different gateways.
With this setup, all emulator traffic is forwared to vlan.2. I don't fully understand how. So for me, it's more of a lucky case that the "correct" VLAN interface is used. vlan.3 is just an internal management subnet which the emulator instances should not use.
Now even with a single emulator instance on the host - how can I force it to use, for example, gateway 192.168.2.201?

This is how far I got:

            ---------      -------      --------
            | vlan2 |______| br0 |______| tap0 |
            ---------      -------      --------
---------       |
| eno1  | ______|
---------       |
            ---------
            | vlan3 |
            --------- 

$ cat /etc/systemd/network/eno1.network
[Match]
Name=eno1

[Network]
DHCP=no
VLAN=vlan.2
VLAN=vlan.3

$ cat /etc/systemd/network/vlan.2.netdev
[NetDev]
Name=vlan.2
Kind=vlan

[VLAN]
Id=2

$ cat /etc/systemd/network/vlan.3.netdev
[NetDev]
Name=vlan.3
Kind=vlan

[VLAN]
Id=3

$ cat /etc/systemd/network/vlan.2.network
[Match]
Name=vlan.2

[Network]
DHCP=no
Bridge=br0

$ cat /etc/systemd/network/vlan.3.network
[Match]
Name=vlan.3

[Network]
Address=192.168.3.10/24

$ cat /etc/systemd/network/br0.netdev

[NetDev]
Name=br0
Kind=bridge

$ cat /etc/systemd/network.br0.network
[Match]
Name=br0

[Network]
Address=192.168.2.10/24
Gateway=192.168.2.200/24
DNS=8.8.8.8

$ cat /etc/systemd/network/tap0.netdev
[NetDev]
Name=tap0
Kind=tap

[Tap]
User=myuser

$ cat /etc/systemd/network/tap0.network
[Match]
Name=tap0

[Network]
Bridge=bro

The idea here was that I can connect qemu to a tap network device. In theory, it should be possible to configure IP address and gateway on this device. However, that alone is not enough. I have to bridge the tap device to the physical interface (as tried above). Now I'm facing the problem that bridge ports are not allowed to have an IP configurarion.

At this point, I am out of ideas. But maybe you can help me out?

Last edited by SJ (2023-06-11 15:19:18)

Offline

#2 2023-06-04 07:41:26

-thc
Member
Registered: 2017-03-15
Posts: 482

Re: [Solved]Provide different gateways for qemu and/or container instances

Because your VMs need access to your LAN the most simplistic approach (IMHO) would be to give your VMs that access (https://wiki.archlinux.org/title/QEMU#T … _with_QEMU) and configure the TCP/IP stack on the VMs accordingly. The VMs would be as visible as every other device on your LAN.

If you want to isolate the VMs network traffic it would be very convenient if the VMs gateway would support tagged VLANs. Configuring such a VLAN on the VMs gateway and the VMs itself would do the trick.

If you need a more elaborate network segmentation or your gateway doesn't support VLANs - another approach may be better.

Offline

#3 2023-06-04 09:25:17

SJ
Member
Registered: 2017-12-25
Posts: 17

Re: [Solved]Provide different gateways for qemu and/or container instances

So if I understand you correctly, the approach I've tried in my first post should work. But I'm still unsure how to configure the tap interface. It should have its own IP address, subnet and gateway. But if I bridge the tap interface to my physical interface, I'm not allowed to set those (only on the bridge, but there I configure the host networking already). Does that mean I need a new bridge for every single tap interface for every new VM instance?
It would be really great if you could walk me through the host networking setup, preferrably the systemd-networkd way.

Offline

#4 2023-06-04 10:55:39

Lone_Wolf
Member
From: Netherlands, Europe
Registered: 2005-10-04
Posts: 11,848

Re: [Solved]Provide different gateways for qemu and/or container instances

preferrably the systemd-networkd way

The method -thc describes will work with any qemu installation on any platform supported by qemu (example : it will work on artix linux, void linux, devuan, bsd* and even systemd using linuxes) .

If you want to limit yourself to hosts that can run systemd-networkd , you need to use systemd-netdev files to create interfaces for the VMs . see https://wiki.archlinux.org/title/System … tdev_files

Last edited by Lone_Wolf (2023-06-04 10:57:03)


Disliking systemd intensely, but not satisfied with alternatives so focusing on taming systemd.


(A works at time B)  && (time C > time B ) ≠  (A works at time C)

Offline

#5 2023-06-04 11:31:46

-thc
Member
Registered: 2017-03-15
Posts: 482

Re: [Solved]Provide different gateways for qemu and/or container instances

SJ wrote:

So if I understand you correctly, the approach I've tried in my first post should work.

That would be possible - I use a virtual firewall to separate Windows VMs from my home LAN - but it's more work. The benefit is you don't need VLANs for the separation - it's done by the virtual firewall.

SJ wrote:

But I'm still unsure how to configure the tap interface. It should have its own IP address, subnet and gateway. But if I bridge the tap interface to my physical interface, I'm not allowed to set those (only on the bridge, but there I configure the host networking already). Does that mean I need a new bridge for every single tap interface for every new VM instance?

When you bridge physical or virtual interfaces they lose their "identity" as individual adapters - try to visualize a bridge as a single physical network adapter which has been "welded together" from all bridge members. This new adapter can have  an IP address - but will even work (as a bridge) without one. The bridge members are no longer separate adapters.

Depending on your network planning you may not need a bridge at all.

SJ wrote:

It would be really great if you could walk me through the host networking setup, preferrably the systemd-networkd way.

We're not there yet. We need to know - besides the gateway "junction" - what you want to separate (the VMs and host should be able to reach A,B,C) and why VLANs are necessary.

Offline

#6 2023-06-04 13:15:48

SJ
Member
Registered: 2017-12-25
Posts: 17

Re: [Solved]Provide different gateways for qemu and/or container instances

The "must haves" are:
- The host needs to be able to see and reach the whole LAN.
- The VM instances need to be able to reach (but not neccessarily see) a predefined gateway.
- The host needs to be able to communicate with each VM instance individually.

And the "should haves" are:
- The VM instances should not see the LAN, nor each other.
- Other machines, except the host, should not be able to communicate with the VM instances.

The VLAN interfaces in my first post are neccessary, because the host is connected to a trunk port.
Putting all the VMs on the host in an additional, separate VLAN, can be a task for the future.

I was hoping to solve this with virtual interfaces instead of custom routes. The reasoning here is that if the interface is not present, startup of the VM will fail. If a route is not present, the VM might see and reach parts of the network it wasn't supposed to.

Offline

#7 2023-06-04 14:23:39

-thc
Member
Registered: 2017-03-15
Posts: 482

Re: [Solved]Provide different gateways for qemu and/or container instances

SJ wrote:

The "must haves" are:
- The host needs to be able to see and reach the whole LAN.
- The VM instances need to be able to reach (but not neccessarily see) a predefined gateway.
- The host needs to be able to communicate with each VM instance individually.

And the "should haves" are:
- The VM instances should not see the LAN, nor each other.
- Other machines, except the host, should not be able to communicate with the VM instances.

All of this could be managed through a virtual firewall - with the minor exception that the VMs will "see" their assigned gateway.

SJ wrote:

The VLAN interfaces in my first post are neccessary, because the host is connected to a trunk port.

What kind of "trunk" do you mean? LACP? VLAN trunk? If you mean the latter - what VLANs does the port (your host connects to) accept? 0 (untagged), 2 and 3? If so, what's the design idea behind that? Does your host have VLAN interfaces ("eno1.2" and "eno1.3") with separate IP spaces?

Last edited by -thc (2023-06-04 14:25:42)

Offline

#8 2023-06-04 14:52:55

SJ
Member
Registered: 2017-12-25
Posts: 17

Re: [Solved]Provide different gateways for qemu and/or container instances

-thc wrote:

All of this could be managed through a virtual firewall - with the minor exception that the VMs will "see" their assigned gateway.

That would not be an issue. I just don't know how to implement the virtual firewall. Specifically, the VMs are instances of the Android Studio emulator, which is a customized qemu. So my options in configuring the guest are very limited.

-thc wrote:

What kind of "trunk" do you mean? LACP? VLAN trunk? If you mean the latter - what VLANs does the port (your host connects to) accept? 0 (untagged), 2 and 3? If so, what's the design idea behind that? Does your host have VLAN interfaces ("eno1.2" and "eno1.3") with separate IP spaces?

It's a VLAN trunk that only accepts the VLAN IDs 2 and 3. As shown above, the host configures two VLAN interfaces, each with a static IP in different ranges. The main design decision behind this is separation of different "services" into their own network segments. Surely there are alternatives, some maybe even better. However, as of now, this design is a fact and I am not able to reconfigure VLANs.

Offline

#9 2023-06-04 17:39:30

-thc
Member
Registered: 2017-03-15
Posts: 482

Re: [Solved]Provide different gateways for qemu and/or container instances

SJ wrote:

Specifically, the VMs are instances of the Android Studio emulator, which is a customized qemu. So my options in configuring the guest are very limited.

As long as those VMs get their network configuration via DHCP this should not be a problem - the router/firewall VM is an additional VM.

SJ wrote:

It's a VLAN trunk that only accepts the VLAN IDs 2 and 3. As shown above, the host configures two VLAN interfaces, each with a static IP in different ranges. The main design decision behind this is separation of different "services" into their own network segments. Surely there are alternatives, some maybe even better. However, as of now, this design is a fact and I am not able to reconfigure VLANs.

O.K.

So the roadmap I would suggest looks like this:

1. Create an additional QEMU firewall VM with two network adapters: One bridged to VLAN 2 and one only for "internal" communication.

2. Set up the firewall VM to support IP forwarding, masquerading, DNS forwarding (all three from "internal" to VLAN 2) and DHCP serving ("internal"). The firewall ruleset reflects forwarding to different gateways (source based routing) and all packet flow restrictions.

3. Attach the Android emulator VMs to the "internal" network.

I cannot say if QEMU networking supports this approach - I've implemented firewall/router VMs only via VIrtualBox, VMWare ESXi and VMWare Workstation Player.

Offline

#10 2023-06-05 09:38:00

Lone_Wolf
Member
From: Netherlands, Europe
Registered: 2005-10-04
Posts: 11,848

Re: [Solved]Provide different gateways for qemu and/or container instances

VMWare products, virtualbox, libvirt, docker etc all have their own tooling to setup/manage  virtual networks.

Qemu provides basic networking functionailty. For complicated setups with qemu 'real' network devices can be used (like those from cisco , juniper and their ilk) .
Incase those devices are not available, vde 2 can emulate them.

An advantage of vde2 is that it provides general networking functionality which is RFC standards compliant and just like physical devices can be connect everything that has a network interface .


Disliking systemd intensely, but not satisfied with alternatives so focusing on taming systemd.


(A works at time B)  && (time C > time B ) ≠  (A works at time C)

Offline

#11 2023-06-11 15:18:53

SJ
Member
Registered: 2017-12-25
Posts: 17

Re: [Solved]Provide different gateways for qemu and/or container instances

Instead of configuring an additional QEMU VM, I configured forwarding, routing and snat on the host. The steps I took are very similar to the suggestions of -thc.

1) Create the tap device and assign it an IP address.
2) Run a DHCP Server on the tap interface. Use this to provide a DNS server address to the VM as well as a dynamic IP. The gateway option of the DHCP server should point to the static IP that was assigned to the tap interface.
3) Configure nftables to forward traffic from the tap interface to the physical interface and do snat.
4) Add a custom route so that traffic coming from the tap interface goes out on the respective gateway address.
5) Pass the tap interface to the QEMU instance (the Android emulator has a special flag for that)

Offline

Board footer

Powered by FluxBB