Arch Linux

david784

Member

Registered: 2020-01-30

Posts: 1

Step by step, setting up custom bridged network for libvirt

It took me quite a while to figure all this out from multiple sources, so I thought I'd compile it all in one place.

Mission: disable the default networking in libvirtd, and set up a custom bridged network.

Preliminary setup

There are a few environment variables I configured when first starting on this script. Since the script is split up throughout this post I'm including all the env vars here to start:

BRIDGE=br0
SUBN=192.168.122
NETWORK=$SUBN.0
NETMASK=24
GATEWAY=$SUBN.1
DHCPRANGE=$SUBN.2,$SUBN.254

Also most of this needs to be done as root. You can add sudo to the commands, or you can just use a root shell.

First step: set up a bridged adapter.

The first guide I found for this showed how to do this with brctl. But after reading ArchWiki Network bridge (The use of brctl is deprecated and is considered obsolete), I switched. At first I set this up using iproute2 from that wiki.

ip link add name "$BRIDGE" type bridge
ip link set "$BRIDGE" up
ip addr add dev "$BRIDGE" "$GATEWAY"/"$NETMASK"

But to make it persistent across reboot, I finally set it up with NetworkManager:

cat >//etc/NetworkManager/system-connections/br0.connection <<EOF
[connection]
id=$BRIDGE
uuid=f82d3c39-74e6-4f7c-bafe-3368d96eab1e
type=bridge
interface-name=$BRIDGE
permissions=

[bridge]

[ipv4]
address1=$GATEWAY/$NETMASK
dns-search=
method=manual

[ipv6]
addr-gen-mode=stable-privacy
dns-search=
method=auto

[proxy]
EOF

nmcli connection reload

Step 2: turn on ip forwarding

Most guides showed how to do this temporarily:

echo 1 | dd of=/proc/sys/net/ipv4/ip_forward > /dev/null

But to make it persist after reboot:

cat >/etc/tmpfiles.d/ip-forward.conf <<EOF
w- /proc/sys/net/ipv4/ip_forward - - - - 1
EOF

Step 3: iptables forwarding

This part is pretty vanilla, and mostly copied straight from the qemu networking guide. I tweaked it a little, specifically with the more generic ACCEPT rules for lo and the $BRIDGE.

cat >/etc/iptables/iptables.rules <<EOF
*nat
:PREROUTING ACCEPT [61:9671]
:POSTROUTING ACCEPT [121:7499]
:OUTPUT ACCEPT [132:8691]
-A POSTROUTING -s $NETWORK/$NETMASK -j MASQUERADE
COMMIT
*filter
:INPUT ACCEPT [1453:976046]
:FORWARD ACCEPT [0:0]
:OUTPUT ACCEPT [1605:194911]
-A INPUT -i lo -j ACCEPT
-A INPUT -i $BRIDGE -j ACCEPT
-A FORWARD -i $BRIDGE -o $BRIDGE -j ACCEPT
-A FORWARD -s $NETWORK/$NETMASK -i $BRIDGE -j ACCEPT
-A FORWARD -d $NETWORK/$NETMASK -o $BRIDGE -m state --state RELATED,ESTABLISHED -j ACCEPT
-A FORWARD -o $BRIDGE -j REJECT --reject-with icmp-port-unreachable
-A FORWARD -i $BRIDGE -j REJECT --reject-with icmp-port-unreachable
COMMIT
EOF

# restore using new configuration
iptables-restore /etc/iptables/iptables.rules

Step 4: dnsmasq configuration

First thing I did was set up the dnsmasq.conf to load additional configs from /etc/dnsmasq.d:

echo "conf-dir=/etc/dnsmasq.d" >> /etc/dnsmasq.conf
mkdir /etc/dnsmasq.d/

Then I set up a few separate config files. First file sets up DNS

cat >/etc/dnsmasq.d/default.conf <<EOF
domain-needed
bogus-priv
filterwin2k
no-resolv  # /etc/resolv.conf points to the dnsmasq server. Don't want a circular reference!
server=/private.dn/192.168.30.1 # used for VPN private domain
server=1.1.1.1
server=8.8.8.8
server=8.8.4.4
EOF

Second file is so I can easily turn on/off debug logging in syslog:

echo "#log-queries" >/etc/dnsmasq.d/log.conf

Final file is to configure the DHCP server, which the qemu virtual machine will use. I've got a line in here for no-dhcp-interface on the loopback, which I've got commented out. I had initially added it (along with a bunch of other ones) when I was trying to get things working...but I commented it out at some point, and once I got things working I just left it.

cat >/etc/dnsmasq.d/libvirt.conf <<EOF
strict-order
interface=$BRIDGE
interface=lo
#no-dhcp-interface=lo
bind-interfaces
dhcp-range=$DHCPRANGE
pid-file=/var/run/qemu-dnsmasq-$BRIDGE.pid
dhcp-leasefile=/var/run/qemu-dnsmasq-$BRIDGE.leases
EOF

Next, disable the default libvirt networking (if it's been started), and start/enable dnsmasq.

virsh net-destroy default
virsh net-autostart default --disable 
systemctl start dnsmasq
systemctl enable dnsmasq

Finally I reconfigured the NetworkManager connections for my wifi etc to use the dnsmasq server for its DNS. I did this through the GUI, by setting the IPv4 to DHCP addresses only, and setting 127.0.0.1 as my DNS server.

Step 5: libvirt VM configuration, etc

The NIC for the VM needs to be reconfigured to use bridge device, instead of the default "Virtual network" NAT. I did this using the graphical interface, but here is a snippet of the relevant lines from the XML:

<interface type="bridge">
  <source bridge="br0"/>

And then...enjoy having more confidence and control over your network configuration!

Last edited by david784 (2021-12-17 01:01:14)