VDE2 Bridge leads to spurious MAC on some boots, not on others

mostlyharmless · 2015-12-27 22:43:09

Goal: to have a static IP address on my host and virtual machines, all of which can access the router/gateway and each other:
Problem: some boots, everything works perfectly. Other boots the host gets a random MAC address and IP from the router and the VMs get nothing. The router also has an IP reservation to assign specific IPs to specific MAC addresses.
History: Used to work all of the time; haven't changed the network setup, but of course, I have updated Arch regularly, with unknown effects on qemu, systemd and netctl.

Background:
I followed the instructions on https://wiki.archlinux.org/index.php/QEMU.
You can skip the wall of code tags and text and assume I have followed directions.

Question: WTH is going on?

Begin wall of code tags:

First I made a VDE service

/etc/systemd/scripts/qemu-network-env

#!/bin/sh
# QEMU/VDE network environment preparation script

# The IP configuration for the tap device that will be used for
# the virtual machine network:

TAP_DEV=tap0
TAP_IP=192.168.100.254
TAP_MASK=24
TAP_NETWORK=192.168.100.0

# Host interface
NIC=eno1

case "$1" in
  start)
        echo -n "Starting VDE network for QEMU: "

        # If you want tun kernel module to be loaded by script uncomment here
	#modprobe tun 2>/dev/null
	## Wait for the module to be loaded
 	#while ! lsmod | grep -q "^tun"; do echo "Waiting for tun device"; sleep 1; done

        # Start tap switch
        vde_switch -tap "$TAP_DEV" -daemon -mod 660 -group users

        # Bring tap interface up
        ip address add "$TAP_IP"/"$TAP_MASK" dev "$TAP_DEV"
        ip link set "$TAP_DEV" up

        # Start IP Forwarding
        echo "1" > /proc/sys/net/ipv4/ip_forward
        iptables -t nat -A POSTROUTING -s "$TAP_NETWORK"/"$TAP_MASK" -o "$NIC" -j MASQUERADE
        ;;
  stop)
        echo -n "Stopping VDE network for QEMU: "
        # Delete the NAT rules
        iptables -t nat -D POSTROUTING "$TAP_NETWORK"/"$TAP_MASK" -o "$NIC" -j MASQUERADE

        # Bring tap interface down
        ip link set "$TAP_DEV" down

        # Kill VDE switch
        pgrep -f vde_switch | xargs kill -TERM
        ;;
  restart|reload)
        $0 stop
        sleep 1
        $0 start
        ;;
  *)
        echo "Usage: $0 {start|stop|restart|reload}"
        exit 1
esac
exit 0

Then the systemd service:

/etc/systemd/system/qemu-network-env.service

[Unit]
Description=Manage VDE Switch

[Service]
Type=oneshot
ExecStart=/etc/systemd/scripts/qemu-network-env start
ExecStop=/etc/systemd/scripts/qemu-network-env stop
RemainAfterExit=yes

[Install]
WantedBy=multi-user.target

Made it executable chmod u+x /etc/systemd/scripts/qemu-network-env, and started and enabled it

Then I made a bridge

#brctl addbr br0
# brctl addif br0 eno1
# brctl addif br0 tap0
# dhcpcd br0

and used the startup scripts

/etc/netctl/ethernet-noip

Description='A more versatile static Ethernet connection'
Interface=eno1
Connection=ethernet
IP=no

and

/etc/systemd/system/vde2@.service

[Unit]
Description=Network Connectivity for %i
Wants=network.target
Before=network.target

[Service]
Type=oneshot
RemainAfterExit=yes
ExecStart=/usr/bin/vde_switch -tap %i -daemon -mod 660 -group users
ExecStart=/usr/bin/ip link set dev %i up
ExecStop=/usr/bin/ip addr flush dev %i
ExecStop=/usr/bin/ip link set dev %i down

[Install]
WantedBy=multi-user.target

Enabled that as well. Then I followed to the suggested link to bridge with netctl: https://wiki.archlinux.org/index.php/Bridge_with_netctl and made a static bridge

 
/etc/netctl/bridge

Description="Example Bridge connection"
Interface=br0
Connection=bridge
BindsToInterfaces=(eno1,tap0)
IP=static
Address='192.168.1.9/24'
Gateway='192.168.1.1'
## Ignore (R)STP and immediately activate the bridge
SkipForwardingDelay=yes

The netctl profiles bridge and ethernet-noip are enabled and started.

End wall of Code tags.

Here's the result on a non-working boot:

1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default 
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host 
       valid_lft forever preferred_lft forever
2: eno1: <BROADCAST,MULTICAST,PROMISC,UP,LOWER_UP> mtu 1500 qdisc fq_codel master br0 state UP group default qlen 1000
    link/ether 74:d0:2b:2a:XX:XX brd ff:ff:ff:ff:ff:ff
    inet6 fe80::573c:afae:XXXX:XXXX/64 scope link 
       valid_lft forever preferred_lft forever
4: tap0: <BROADCAST,MULTICAST,PROMISC,UP,LOWER_UP> mtu 1500 qdisc fq_codel master br0 state UNKNOWN group default qlen 500
    link/ether 2a:91:c4:55:XX:XX brd ff:ff:ff:ff:ff:ff
    inet6 fe80::2891:c4ff:XXXX:XXXX/64 scope link 
       valid_lft forever preferred_lft forever
5: br0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default 
    link/ether 2a:91:c4:55:XX:XX brd ff:ff:ff:ff:ff:ff
    inet 192.168.1.43/24 brd 192.168.1.255 scope global br0
       valid_lft forever preferred_lft forever
    inet6 fe80::2891:c4ff:XXXX:XXXX/64 scope link 
       valid_lft forever preferred_lft forever
8: vnet0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel master br0 state UNKNOWN group default qlen 500
    link/ether fe:54:00:XX:XX:XX brd ff:ff:ff:ff:ff:ff
    inet6 fe80::b33d:2c88:XXXX:XXXX/64 scope link 
       valid_lft forever preferred_lft forever

Note that br0 was given a random MAC address (the MAC address of the tap0 device) starting with 2a: and an IP of 192.168.1.43 by the router. If I stop the bridge with sudo netctl stop bridge and restart it with sudo netctl start bridge, I now get:

9: br0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default 
    link/ether 2a:91:c4:55:XX:XX brd ff:ff:ff:ff:ff:ff
    inet 192.168.1.9/24 brd 192.168.1.255 scope global br0
       valid_lft forever preferred_lft forever
    inet 192.168.1.43/24 brd 192.168.1.255 scope global secondary br0
       valid_lft forever preferred_lft forever
    inet6 fe80::4d3b:87ae:XXXX:XXXX/64 scope link 
       valid_lft forever preferred_lft forever

and the VMs still don't get a network. Other times when I boot, the bridge gets the MAC address belonging to the eno1 ethernet card, beginning with 74:, and gets the IP 192.168.1.9 and everything works perfectly. Here's an example:

ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default 
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host 
       valid_lft forever preferred_lft forever
2: eno1: <BROADCAST,MULTICAST,PROMISC,UP,LOWER_UP> mtu 1500 qdisc fq_codel master br0 state UP group default qlen 1000
    link/ether 74:d0:2b:2a:XX:XX brd ff:ff:ff:ff:ff:ff
    inet6 fe80::573c:afae:XXXX:XXXX/64 scope link 
       valid_lft forever preferred_lft forever
4: tap0: <BROADCAST,MULTICAST,PROMISC,UP,LOWER_UP> mtu 1500 qdisc fq_codel master br0 state UNKNOWN group default qlen 500
    link/ether aa:4b:c2:62:XX:XX brd ff:ff:ff:ff:ff:ff
    inet6 fe80::a84b:c2ff:XXXX:XXXX/64 scope link 
       valid_lft forever preferred_lft forever
5: br0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default 
    link/ether 74:d0:2b:2a:XX:XX brd ff:ff:ff:ff:ff:ff
    inet 192.168.1.9/24 brd 192.168.1.255 scope global br0
       valid_lft forever preferred_lft forever
    inet6 fe80::76d0:2bff:XXXX:XXXX/64 scope link 
       valid_lft forever preferred_lft forever
8: vnet0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel master br0 state UNKNOWN group default qlen 500
    link/ether fe:54:00:87:XX:XX brd ff:ff:ff:ff:ff:ff
    inet6 fe80::b33d:2c88:XXXX:XXXX/64 scope link 
       valid_lft forever preferred_lft forever

So apparently the bridge gets the tap0 device's MAC half the time & a random IP address from the router, leaving the VM network dead. The other half of the time it gets the ethernet MAC and everything works. There is no MAC spoofing set up.

EDIT:

Well, let's mark this as SOLVED. Weird how these things work, somethimes you just have to post them and the problem you've been staring at solves itself.. I tried the solution from this
https://github.com/joukewitteveen/netctl/issues/111, essentially forcing the MAC address of the bridge to the ethernet address.

Interface=br0
Connection=bridge
BindsToInterfaces=(eno1 tap0)
IP=no
ExecUpPost="ip link set dev br0 address $(cat /sys/class/net/eno1/address); IP=dhcp; ip_set"
ExecDownPre="IP=dhcp"

## Ignore (R)STP and immediately activate the bridge
SkipForwardingDelay=yes

There should probably be a correction on the wiki: ok did that.

Last edited by mostlyharmless (2015-12-27 23:27:30)

Arch Linux

#1 2015-12-27 22:43:09

VDE2 Bridge leads to spurious MAC on some boots, not on others

Board footer