Bringing up tap0 and vde2 at boot for use with virtual machines.

troutwine · 2011-06-12 21:52:12

I'm attempting to configure vde2 and to bring up tap0 at boot time; ultimately I want to run several local, networked qemu-kvm machines. Here's where I am so far:

[root@doritos ~]# /etc/rc.d/vde start
:: Starting vde_switch vde-config                                                 [DONE] 
:: Starting slirpvde for vde-config                                               [DONE]

I have the tun module loaded:

[root@doritos ~]# lsmod | grep tun
tun                    12784  0

This is my /etc/conf.d/vde:

# Set options for a vde daemon script
VDE_CONFIG="vde-config"
VDE_CONNECTION="vde-connection"

And /etc/vde/{vde-config,vde-connection}:

# Set options vde_switch program
# To activate this vde profile add it to /etc/conf.d/vde

# Global options
VDE_NUMPORTS="32"  # Number of ports (default 32)
VDE_HUB="no"       # [yes|no] Make the switch act as a hub
VDE_FSTP="no"      # [yes|no] Activate the fast spanning tree protocol
VDE_MAC=""         # Set the Switch MAC address
VDE_PRIOTIRY=""    # Set the priority for FST (MAC extension)
VDE_HASHSIZE=""    # Hash table size

# Options from datasock module
VDE_SOCK="/var/run/vde/sample.sock"  # control directory pathname
VDE_SOCK_MODE="660"                  # Standard access mode for comm sockets (octal)
VDE_SOCK_GROUP="root"                # Group owner for comm sockets

# Options from consmgmt module
VDE_MANAGEMENT_SOCK="/var/run/vde/sample.mgmt"  # path of the management UNIX socket
VDE_MANAGEMENT_SOCK_MODE="660"                  # management UNIX socket access mode (octal)

# Other options to parse to vde_switch
VDE_OPTIONS=""

# VDE with internet support
# You can use either tuntap method or slirpvde method.

# Options from tuntap module
VDE_TAP="tap0"  # Enable routing through TAP tap interface (comma separated array eg.:"tap0,tap1").

# slirpvde daemon support
SLIRP="yes"         # [yes|no] enable/disable SLIRP daemon support
SLIRP_DHCP="yes"    # [yes|no] turn on the DHCP server for the network autoconfiguration of all the units connected to the VDE
SLIRP_NETWORK="" # specify the network address (default 10.0.2.0)
# other options to parse to slirpvde
SLIRP_OPTIONS=""

and

# You can add vde switch connections below
# just without the # at the beginning:
# vde_plug /var/run/vde/sample.sock = vde_plug /var/run/vde/sample2.sock
# or if hosts are different use this syntax:
# vde_plug /var/run/vde/sample.sock = ssh host2 vde_plug /var/run/vde/sample.sock

The interface tap0 does not yet exist. The article here shows the use of ifconfig to bring tap0 up. This bug suggests that it is now possible to configure netcfg to bring tap0 up at boot; I could find no documentation on the matter.

Additionally, even when I do bring up tap0 manually and ensure that vde has been correctly been started I find that:

$ qemu-kvm -hda ~/virtual_machines/debian -m 512 -cdrom Downloads/torrents/debian-6.0.0-i386-netinst.iso -boot d -vga std  -net nic -net vde
qemu-kvm: -net vde: Device 'vde' could not be initialized

The wiki is kind of a grab-bag of information; where do I go from here?

Last edited by troutwine (2011-06-12 22:15:09)

falconindy · 2011-06-12 22:26:24

/usr/bin/vde_switch -tap tap0 -mod 660 -group kvm
/sbin/ip addr add 10.0.2.1/24 dev tap0
/sbin/ip link set dev tap0 up

That's all it takes (assuming you added 'tun' to your MODULES array in /etc/rc.conf). You could create your own daemon with this or add it to /etc/rc.local. Note that 10.0.2.1 is somewhat arbitrary. My real lan is 10.0.1.0/24, so I just added 1 and made the tap0 device .1 on the new /24.

I assume the problem you're encountering when launching qemu-kvm is that the control socket created by vde_switch is owned by root, not kvm.

edit: I suggest using virtio block devices. They're a lot faster, with the only downside being that AIF doesn't support them (no idea about Debian). Instead of using something like -hda file.img, use:

-drive file=path/to/img.qcow,if=virtio

You'll need virtio modprobed on the host, and virtio_blk and virtio_pci modprobed in the guest.

Last edited by falconindy (2011-06-12 22:29:59)

troutwine · 2011-06-13 01:15:32

falconindy wrote:

/usr/bin/vde_switch -tap tap0 -mod 660 -group kvm
/sbin/ip addr add 10.0.2.1/24 dev tap0
/sbin/ip link set dev tap0 up
That's all it takes (assuming you added 'tun' to your MODULES array in /etc/rc.conf). You could create your own daemon with this or add it to /etc/rc.local. Note that 10.0.2.1 is somewhat arbitrary. My real lan is 10.0.1.0/24, so I just added 1 and made the tap0 device .1 on the new /24.

Thanks; I added all but the first command to rc.local. I've also changed group ownership in vde-config to "kvm" from "root". For reference, here is my /etc/vde/vde-config:

# Set options vde_switch program
# To activate this vde profile add it to /etc/conf.d/vde

# Global options
VDE_NUMPORTS="32"  # Number of ports (default 32)
VDE_HUB="no"       # [yes|no] Make the switch act as a hub
VDE_FSTP="no"      # [yes|no] Activate the fast spanning tree protocol
VDE_MAC=""         # Set the Switch MAC address
VDE_PRIOTIRY=""    # Set the priority for FST (MAC extension)
VDE_HASHSIZE=""    # Hash table size

# Options from datasock module
VDE_SOCK="/var/run/vde/sample.sock"  # control directory pathname
VDE_SOCK_MODE="660"                  # Standard access mode for comm sockets (octal)
vde_sock_group="kvm"                # Group owner for comm sockets

# Options from consmgmt module
VDE_MANAGEMENT_SOCK="/var/run/vde/sample.mgmt"  # path of the management UNIX socket
VDE_MANAGEMENT_SOCK_MODE="660"                  # management UNIX socket access mode (octal)

# Other options to parse to vde_switch
VDE_OPTIONS=""

# VDE with internet support
# You can use either tuntap method or slirpvde method.

# Options from tuntap module
VDE_TAP="tap0"  # Enable routing through TAP tap interface (comma separated array eg.:"tap0,tap1").

# slirpvde daemon support
SLIRP="yes"         # [yes|no] enable/disable SLIRP daemon support
SLIRP_DHCP="yes"    # [yes|no] turn on the DHCP server for the network autoconfiguration of all the units connected to the VDE
SLIRP_NETWORK="" # specify the network address (default 10.0.2.0)
# other options to parse to slirpvde
SLIRP_OPTIONS=""

Now, confusingly, here's the output of me starting the vde_switch as a part of /etc/rc.d/vde:

[root@doritos ~]# /etc/rc.d/vde start
:: Starting vde_switch vde-config                                                                                                                       [FAIL] 
:: Starting slirpvde for vde-config                                                                                                                     [DONE]

A bit of inspection shows that the $OPTIONS passed to vde_switch in this script read "-n", becoming this after the lines:

# get options from profile                                                                                                                               
[ -n "$VDE_NUMPORTS" ] && OPTIONS="-n $VDE_NUMPORTS"

However, I _do_ have VDE_NUMPORTS defined, as you can see above. What's the deal?

Additionally, if I choose to start vde_switch manually I am able to start my qemu-kvm machine, but am unable to gather up a DHCP address with the following dnsmasq configuration:

interface=tap0
listen-address=127.0.0.1
dhcp-range=10.0.2.100,10.0.2.150,12h

I use the same command invocation for qemu-kvm as before.

EDIT:

It seems that the following in /etc/rc.local does not bring up tap0:

#!/bin/bash
#
# /etc/rc.local: Local multi-user startup script.
#
/sbin/ip addr add 10.0.2.1/24 dev tap0
/sbin/ip link set dev tap0 up

sysctl -w net.ipv4.ip_forward=1

Last edited by troutwine (2011-06-13 02:12:13)

potatoe · 2011-06-13 03:23:43

falconindy wrote:

I suggest using virtio block devices. They're a lot faster, with the only downside being that AIF doesn't support them (no idea about Debian). Instead of using something like -hda file.img, use:
-drive file=path/to/img.qcow,if=virtio
You'll need virtio modprobed on the host, and virtio_blk and virtio_pci modprobed in the guest.

As an aside, I'd recommend also just starting out from the beginning with the new -device and -netdev qemu option format, rather than the deprecated -net and -drive ones. I don't know about the virtio block driver, but for networking kvm is much slower if you use '-net nic,model=virtio','-net tap' options instead of the new '-netdev tap,id=mynet','-device virtio-net-pci,netdev=mynet' format. I get 600-650Mb/s on a simple iperf to a remote host with the first format, 800-850Mb/s with the second plus lower CPU usage too (vs. 900-950Mb/s on the native host), due to the non-deprecated format enabling full virtio offloading. I think the newer command-line format is also necessary if you want to use a 'vhost=on' option for the netdev (to make use of the kernel-level virtio 'vhost-net' module on the host for even lower CPU usage, and possibly a slight boost in latency and speed).

I imagine it's a similar improvement for vde instead of tap+bridge networking, and to be safe I'd also go ahead and use the new '-drive file=/tmp/blah.img,if=none,id=mydrive','-device virtio-blk-pci,drive=mydrive' format for disk too.

EDIT: BTW, you shouldn't need to modprobe virtio on the host to use virtio. My understanding is that module is for guests, on the host Virtio is all user-space inside qemu. vhost-net is a kernel-space implementation of the virtio network driver, but it just requires modprobing vhost-net, not virtio, and it's optional.

Last edited by potatoe (2011-06-13 05:09:33)

falconindy · 2011-06-13 03:56:33

Great, except how do you use tap networking as non-root? The initial setup seems like it requires bridging with something like uml_utilities which is a pain in the ass. VDE is fast enough for me if it's going to be a pain in the butt to setup this newfangled method...

Last edited by falconindy (2011-06-13 03:57:38)

potatoe · 2011-06-13 05:55:17

falconindy wrote:

Great, except how do you use tap networking as non-root? The initial setup seems like it requires bridging with something like uml_utilities which is a pain in the ass. VDE is fast enough for me if it's going to be a pain in the butt to setup this newfangled method...

Shouldn't have used 'tap' in the example, sorry. It was just a comment solely on command-line syntax.

I just mean, you should define the network with '-netdev vde,id=mynetname' and '-device virtio-net-pci,netdev=mynetname' instead of with '-net vde' and '-net nic,model=virtio'. It just looks like a pointless syntax change, but using qemu/qemu-kvm's new syntax really does make a difference. Virtio won't ever be fully enabled with the old syntax, only with the new syntax.

It makes a huge difference in speed with tap interfaces, using the new syntax. Unfortunately, I just tested it with vde and kvm doesn't seem to ever enable full virtio support when running with vde, under either syntax form (ethtool -k eth0 shows most features as off both ways, and I only get about about 450Mb/s both ways). On the plus side, the new syntax didn't show any decrease in performance, but no gain either.

However, the new syntax is still a good habit to get into, as it will result in a larger performance gain if you ever switch to tap, and going forward it will probably be required for future virtio improvements, including ones targeting vde and future improvements to the block (file) virtio driver.

If you want the speed boost of tap networking and full virtio but without root, I don't know. A suid qemu-ifup script to do the 'ip link set tapX up' and 'brctl addif br0 tapX', I guess, but that's ugly. VDE is obviously the easiest and most versatile choice if it's fast enough for your use, also if you go with bridges they have annoying delays for all traffic whenever any single tap interface (e.g. any VMs) are added/removed.

falconindy · 2011-06-13 12:51:07

Yeah that's what I suspected. No worries, I've migrated to the new (undocumented in qemu-kvm) syntax and kept VDE. The "need" for an SUID script is more than to scare me away.

@troutwine: If the /etc/rc.local script doesn't work, are you sure the tun module is loaded prior to that (i.e., in your MODULES array). I created a systemd unit to handle all this and it just works™ as long as the module is loaded.

troutwine · 2011-06-13 15:29:42

If I stuff everything into rc.local there are no problems; I take this as an indication that Arch's default scripts for vde are rather broken. The wiki, judging by potatoe's comments, is also horribly out of date.

Arch Linux

#1 2011-06-12 21:52:12

Bringing up tap0 and vde2 at boot for use with virtual machines.

#2 2011-06-12 22:26:24

Re: Bringing up tap0 and vde2 at boot for use with virtual machines.

#3 2011-06-13 01:15:32

Re: Bringing up tap0 and vde2 at boot for use with virtual machines.

#4 2011-06-13 03:23:43

Re: Bringing up tap0 and vde2 at boot for use with virtual machines.

#5 2011-06-13 03:56:33

Re: Bringing up tap0 and vde2 at boot for use with virtual machines.

#6 2011-06-13 05:55:17

Re: Bringing up tap0 and vde2 at boot for use with virtual machines.

#7 2011-06-13 12:51:07

Re: Bringing up tap0 and vde2 at boot for use with virtual machines.

#8 2011-06-13 15:29:42

Re: Bringing up tap0 and vde2 at boot for use with virtual machines.

Board footer