HowTo QEMU Networking with Wireless & QEMU host GNS3 networking

hunterthomson · 2010-06-22 02:31:43

###########################

Read Post #10 for the HowTo

###########################

Aloha,

I have setup QEMU-KVM with bridged-NAT networking. I can run it as normal user and they connect to the Internet and I can ping the br0 but the VM's can't talk to each-other and my host OS can not talk to the VM's. I am thinking I should be able to just set some iptables rules to allow the VM-to-VM-to-Host communication but I am having trouble figuring it out.

I have added this to my /etc/sudoers file

Cmnd_Alias QEMU=/sbin/ifconfig,/sbin/modprobe,/usr/sbin/brctl,/usr/bin/tunctl,/usr/bin/qemu_iptables_rules
%kvm ALL=NOPASSWD: QEMU

Then I edited this script to septate the iptables rules so I can run them as root.
http://wiki.qemu.org/Documentation/Networking/NAT

Here are the two scripts. I have qemu-ifup in /etc and qemu_iptables_rules in /usr/bin

qemu-ifup

#!/bin/sh
#
# Copyright IBM, Corp. 2010  
#
# Authors:
#  Anthony Liguori <aliguori@us.ibm.com>
#  --EDITED for Archlinux--
#
# This work is licensed under the terms of the GNU GPL, version 2.  See
# the COPYING file in the top-level directory.

# Set to the name of your bridge
BRIDGE=br0

# Network information
NETWORK=192.168.53.0
NETMASK=255.255.255.0
GATEWAY=192.168.53.1
DHCPRANGE=192.168.53.2,192.168.53.254

# Optionally parameters to enable PXE support
TFTPROOT=
BOOTP=

do_brctl() {
    sudo brctl "$@"
}

do_ifconfig() {
    sudo ifconfig "$@"
}

check_bridge() {
    if do_brctl show | grep "^$1" > /dev/null 2> /dev/null; then
    return 1
    else
    return 0
    fi
}

create_bridge() {
    do_brctl addbr "$1"
    do_brctl stp "$1" off
    do_brctl setfd "$1" 0
    do_ifconfig "$1" "$GATEWAY" netmask "$NETMASK" up
}

setup_bridge_nat() {
    if check_bridge "$1" ; then
    create_bridge "$1"
    sudo qemu_iptables_rules "$1"
    fi
}

setup_bridge_vlan() {
    if check_bridge "$1" ; then
    create_bridge "$1"
    #start_dnsmasq "$1"
    fi
}

setup_bridge_nat "$BRIDGE"

if test "$1" ; then
    do_ifconfig "$1" 0.0.0.0 up
    do_brctl addif "$BRIDGE" "$1"
fi

qemu_iptables_rules

#!/bin/sh
#
# This sets up the iptables rules for NAT networking for qemu
#

# Set to the name of your bridge
BRIDGE=br0

# Network information
NETWORK=192.168.53.0
NETMASK=255.255.255.0
GATEWAY=192.168.53.1
DHCPRANGE=192.168.53.2,192.168.53.254

do_dd() {
    dd "$@"
}

do_iptables_restore() {
    iptables-restore "$@"
}

enable_ip_forward() {
    echo 1 | do_dd of=/proc/sys/net/ipv4/ip_forward > /dev/null
}


add_filter_rules() {
do_iptables_restore <<EOF
# Generated by iptables-save v1.3.6 on Fri Aug 24 15:20:25 2007
*nat
:PREROUTING ACCEPT [61:9671]
:POSTROUTING ACCEPT [121:7499]
:OUTPUT ACCEPT [132:8691]
-A POSTROUTING -s $NETWORK/$NETMASK -j MASQUERADE 
COMMIT
# Completed on Fri Aug 24 15:20:25 2007
# Generated by iptables-save v1.3.6 on Fri Aug 24 15:20:25 2007
*filter
:INPUT ACCEPT [1453:976046]
:FORWARD ACCEPT [0:0]
:OUTPUT ACCEPT [1605:194911]
-A FORWARD -i $1 -o $1 -j ACCEPT 
-A FORWARD -s $NETWORK/$NETMASK -i $BRIDGE -j ACCEPT 
-A FORWARD -d $NETWORK/$NETMASK -o $BRIDGE -m state --state RELATED,ESTABLISHED -j ACCEPT 
-A FORWARD -o $BRIDGE -j REJECT --reject-with icmp-port-unreachable 
-A FORWARD -i $BRIDGE -j REJECT --reject-with icmp-port-unreachable 
COMMIT
# Completed on Fri Aug 24 15:20:25 2007
EOF
}


enable_ip_forward
add_filter_rules

Last edited by hunterthomson (2010-06-23 23:22:20)

fukawi2 · 2010-06-22 08:41:49

Ick, that's messy... I would just add your physical interface (I presume eth0) to the bridge (br0). That way all your VM's are on the same Layer 2 network as your Host machine, leaving you with a single bridge, and no NAT.

To answer your question though, as best I can see, it should be working due to this line:

-A FORWARD -i $1 -o $1 -j ACCEPT

Anything coming in and going out the same interface (bounce routing) is ACCEPT'ed.

1) How are you trying to access each VM from the other VM's?
2) What is the output of `iptables -nvL` while the VM's are running and after you've tried to access between VM's?

Last edited by fukawi2 (2010-06-22 08:42:41)

hunterthomson · 2010-06-22 09:11:15

Well, form what I have read bridging doesn't work with Wireless drivers.

Really it is a fairly good way to go. Basically, I start a br0 connected to nothing. Then start a tap0 and bridge it to br0. Then I use iptables to forward traffic out to the Internet.

EDIT: My mistake see next reply....

I was thinking I could set-up a second bridge br1 and then give the VM's a second tap and then bridge the second tap interfaces to the second br1..... Then use iptables to connect br0 and br1....

I don't know I need a brake. My head is swimming now.

. . . . . Hum, now that I think about it this sounds like a routing problem.

Last edited by hunterthomson (2010-06-22 11:41:51)

hunterthomson · 2010-06-22 09:35:52

Owe... my mistake here is the output of iptables -nvL

sudo iptables -nvL
Chain INPUT (policy ACCEPT 238 packets, 43966 bytes)
 pkts bytes target     prot opt in     out     source               destination         

Chain FORWARD (policy ACCEPT 0 packets, 0 bytes)
 pkts bytes target     prot opt in     out     source               destination         
   79  5234 ACCEPT     all  --  br0    br0     0.0.0.0/0            0.0.0.0/0           
   38  2136 ACCEPT     all  --  br0    *       192.168.53.0/24      0.0.0.0/0           
   32  4115 ACCEPT     all  --  *      br0     0.0.0.0/0            192.168.53.0/24     state RELATED,ESTABLISHED 
    0     0 REJECT     all  --  *      br0     0.0.0.0/0            0.0.0.0/0           reject-with icmp-port-unreachable 
    0     0 REJECT     all  --  br0    *       0.0.0.0/0            0.0.0.0/0           reject-with icmp-port-unreachable 

Chain OUTPUT (policy ACCEPT 130 packets, 19240 bytes)
 pkts bytes target     prot opt in     out     source               destination

fukawi2 · 2010-06-22 10:59:18

hunterthomson wrote:

. . . . . Hum, now that I think about it this sounds like a routing problem.

I don't see how it can be a routing problem when everything is on the same Layer 2 and in the same subnet....

How are you trying to access across VM's? ping? ssh?

hunterthomson · 2010-06-22 11:40:23

Well,... I am just pinging.

Even better would be to get the VM's to talk to each other in GNS3, but I can't get that working ether.

I can set-up a cloud to the tap0. Then setup a c2697 to ping google.com through it.

But if I boot a VM of Windows 2008 and connect it to a c2697 with NM-16ESW Switch and the Switch conneced to the tap0 Cloud. Then VM can't ping the br0 or google.com or another Windows 2008 VM.

VM -> NM-16ESW -> VM ..... No go
|
tap0 Could

Last edited by hunterthomson (2010-06-22 13:06:25)

fukawi2 · 2010-06-22 23:00:03

Well now you've lost me... c2697? NM-16ESW?

hunterthomson · 2010-06-22 23:19:47

I am guessing now that iptables is just not capable of routing packets form the VM to VM. in this configuration.

Maybe if I set up

br0192.168.53.1, --- br1 192.168.53.2, --- br2 192.168.53.3

(tap0 - br0), (tap1 - br1), (tap2 - br2)

(VM0 - tap0 - br0), (VM1 - tap1 - br1), (VM2 - tap2 - br2)

Then routed traffic with iptables between the bridge interfaces br0, br1, br2
But that is kind of crazy.

Ya, I think I'll go the VDE way of doing it.

----------------------------

Ya GNS3 is off topic. GNS3 is a GUI for "Dynamips" router visualization software. With it I can boot Cisco IOS of Cisco routers in the c1700, c2600, c2691, c3600, c3700, c7200 models and PIX and ASA and IDS/IPS'es, Frame Relay, ....... Then I can connect them with virtual Ethernet or Serial or other types of cables. I can also boot QEMU guest VM's in the framework and connect them to the Cisco routers. I can also connect tap interfaces form my Host OS to the routers and VM's.

Last edited by hunterthomson (2010-06-22 23:32:44)

hunterthomson · 2010-06-23 00:40:31

Hum, well I setup the VDE, and they were not responding to pings. So then I grew a brain and installed wireshark on one of the VM's. (I should have done that ages ago....) Sure enough the VM's are simply not responding to the pings. But they are receiving them. I install telnet and sure enough I can telnet between the VM's..... stupid me.

I bet it was working all along with the setup I had before...... Yep, I was working all along... VM -VM - Host

What threw me was that this was originally a VirtualBox VM that I converted to a .qcow2 ; In Virtual Box they responded to the pings.... from that I remember...

QEMU/KVM is CRAZY FAST compared to VirtualBox. I am super impressed. To top it off it can use one "Compressed 16GB to 6.6GB" .qcow and then use small overlay images and boot like 6 VM off of it without taking up 100's of GB's. It is super cool. I also use the ksm too.

Any way problem solved. The problem being the one between the keyboard and the chair.

Last edited by hunterthomson (2010-06-23 01:11:10)

hunterthomson · 2010-06-23 04:17:36

Okay this took me a few days of sold work to figure out so I'll write up how to do this so anyone ells who whats to do this doesn't have the same problems.... and for personal reference.

All the VM's and stuff will have to be set with Static IP's you can run a DHCP server. But your better off not doing that.

Just when your setting the IP's on the VM's make sure you set them with the same Netmask and in the same network like 192.168.53.1-255 ....

Like in Linux Guest VM's.....

# ifconfig eth0 192.168.53.2 netmask 255.255.255.0

Also you may need to set the defult gateway. The defult gateway in these setups will be the IP address of the Bridge br0 interface. In my how to it will be 192.168.53.1
So set it like this in Linux Guest VM's

# route add default gw 192.168.53.1 eth0

- How To connect QEMU to Internet and how to Connect Qemu host to GNS3 -

- How to connect qemu networking "The tap way"-

# pacman -S bridge-utils uml_utilities sudo

1. Add bridge and tun to your MODULES line in /etc/rc.conf:

MODULES=( ... bridge tun)

2. Use visudo to add the following to your sudoers file:

 Cmnd_Alias      QEMU=/sbin/ifconfig,/sbin/modprobe,/usr/sbin/brctl,/usr/bin/tunctl,/usr/bin/qemu_iptables_rules
 %kvm     ALL=NOPASSWD: QEMU

3. Make your user a member of the kvm group

gpasswd -a USERNAME kvm

4. Make this scrip which you launch qemu with. Name it run-qemu:
put it in /usr/bin
Make it executable.... chmod 755 /usr/bin/run-qemu
Make it owned by root:kvm..... chown root:kvm /usr/bin/run-qemu

 #!/bin/bash
 USERID=`whoami`
 IFACE=$(sudo tunctl -b -u $USERID)
 
 qemu-kvm -net nic -net tap,ifname="$IFACE" $*
 
 sudo tunctl -d $IFACE &> /dev/null

5. Copy this script to /etc/qemu-ifup Make it owned by root:kvm and set permissions to 750

#!/bin/sh
#
# Copyright IBM, Corp. 2010  
#
# Authors:
#  Anthony Liguori <aliguori@us.ibm.com>
#  --EDITED by user-hunterthomson for Archlinux--
#
# This work is licensed under the terms of the GNU GPL, version 2.  See
# the COPYING file in the top-level directory.

# Set to the name of your bridge
BRIDGE=br0

# Network information
NETWORK=192.168.53.0
NETMASK=255.255.255.0
GATEWAY=192.168.53.1
DHCPRANGE=192.168.53.2,192.168.53.254

# Optionally parameters to enable PXE support
TFTPROOT=
BOOTP=

do_brctl() {
    sudo brctl "$@"
}

do_ifconfig() {
    sudo ifconfig "$@"
}

check_bridge() {
    if do_brctl show | grep "^$1" > /dev/null 2> /dev/null; then
    return 1
    else
    return 0
    fi
}

create_bridge() {
    do_brctl addbr "$1"
    do_brctl stp "$1" off
    do_brctl setfd "$1" 0
    do_ifconfig "$1" "$GATEWAY" netmask "$NETMASK" up
}

setup_bridge_nat() {
    if check_bridge "$1" ; then
    create_bridge "$1"
    sudo qemu_iptables_rules "$1"
    fi
}

setup_bridge_vlan() {
    if check_bridge "$1" ; then
    create_bridge "$1"
    #start_dnsmasq "$1"
    fi
}

setup_bridge_nat "$BRIDGE"

if test "$1" ; then
    do_ifconfig "$1" 0.0.0.0 up
    do_brctl addif "$BRIDGE" "$1"
fi

6. Make a this scrip to set the iptables rules to connect the br0 interface with the internet.
Name it /usr/bin/qemu_iptables_rules
Make it owned by root:kvm and set permissions to 750

#!/bin/sh
#
# This sets up the iptables rules for NAT networking for qemu
#
#
# Copyright IBM, Corp. 2010  
#
# Authors:
#  Anthony Liguori <aliguori@us.ibm.com>
#  --EDITED by user-hunterthomson for Archlinux--
#
# This work is licensed under the terms of the GNU GPL, version 2.  See
# the COPYING file in the top-level directory.


# Set to the name of your bridge
BRIDGE=br0

# Network information
NETWORK=192.168.53.0
NETMASK=255.255.255.0
GATEWAY=192.168.53.1
DHCPRANGE=192.168.53.2,192.168.53.254

do_dd() {
    dd "$@"
}

do_iptables_restore() {
    iptables-restore "$@"
}

enable_ip_forward() {
    echo 1 | do_dd of=/proc/sys/net/ipv4/ip_forward > /dev/null
}


add_filter_rules() {
do_iptables_restore <<EOF
# Generated by iptables-save v1.3.6 on Fri Aug 24 15:20:25 2007
*nat
:PREROUTING ACCEPT [61:9671]
:POSTROUTING ACCEPT [121:7499]
:OUTPUT ACCEPT [132:8691]
-A POSTROUTING -s $NETWORK/$NETMASK -j MASQUERADE 
COMMIT
# Completed on Fri Aug 24 15:20:25 2007
# Generated by iptables-save v1.3.6 on Fri Aug 24 15:20:25 2007
*filter
:INPUT ACCEPT [1453:976046]
:FORWARD ACCEPT [0:0]
:OUTPUT ACCEPT [1605:194911]
-A FORWARD -i $1 -o $1 -j ACCEPT 
-A FORWARD -s $NETWORK/$NETMASK -i $BRIDGE -j ACCEPT 
-A FORWARD -d $NETWORK/$NETMASK -o $BRIDGE -m state --state RELATED,ESTABLISHED -j ACCEPT 
-A FORWARD -o $BRIDGE -j REJECT --reject-with icmp-port-unreachable 
-A FORWARD -i $BRIDGE -j REJECT --reject-with icmp-port-unreachable 
COMMIT
# Completed on Fri Aug 24 15:20:25 2007
EOF
}


enable_ip_forward
add_filter_rules

7. Then to launch a VM, do something like this

run-qemu -hda myvm.img -m 512

- How To setup QEMU Networking with VDE -

With this way you still need the /usr/bin/qemu_iptables_rules script and all the permissions and /etc/sudoers file set. Read above for that....

1. install VDE

pacman -S vde

2. Create a hub for VDE. The '-d' option make the vde_switch run in daemon mode.

$ vde_switch --numports 4 --hub --mod 777 --group users --tap tap0 -d

3. Make a bridge interface

# brctl addbr br0
# brctl stp br0 off
# brctl setfd br0 0
# ifconfig br0 192.168.53.1 netmask 255.255.255.0 up

4. Make a tap interface and bridge it to br0

# tunctl -b -u YOUUSERNAME
# ifconfig tap0 0.0.0.0 up
# brctl addif br0 tap0

5. Run the /usr/bin/qemu_iptables_rules to setup NAT networking between the br0 and the internet

# qemu_iptables_rules

6. Now to start a qemu VM connected to the VDE you do something like this.

$ qemu-kvm -net vde -net nic,vlan=0,macaddr=52:54:00:00:EE:03 vmqemuimegdisk.qcow -m 512

- How to connect QEMU Host to GNS3 and How to connect GNS3 to Cloud Internet connection -

Okay, "Supposedly" you should be able to just run a qemu host in GNS3 and connect it to the other routers and stuff. However, I have not been able to get this to work. It behaves as if the Ethernet cables are not connected.

So the best way I found is to set-up you QEMU Hosts or VM's.... on a VDE. Then connect the VDE to a Cloud in GNS3. This is also cool because this way the VM's are running independent of GNS3. So if you want to kill GNS3 you don't have to kill the VM's or disconnect them form the VDE switch (or hub it you set it up with that flag)

1. Set up the VDE and boot the QEMU host connected to it. You will need to do steps 1-5 of the above HowTo for qemu and VDE.

2.Then in GNS3 make a Cloud. In the Cloud's configuration click on the NIO VDE tab. The Control file and Local file will depend on how you set-up VDE. However, on my system it just defaulted to /tmp/vde.ctl You can just do a locate or what have you to find it. I'll assume it is in /tmp/vde.ctl

Type this for Control file: /tmp/vde.ctl/ctl

( 4 in this is the port that the Cloud will connect to on the VDE)

Type this for Local file: /tmp/vde.ctl/4

3. Then hit Apply and OK. Now you can connect routers and stuff to it.

- Alternate way to connect QEMU Hosts to GNS3 -

You could also set-up the QEMU VM's on the VDE but don't connect the VDE to the tap0.
That way the QEMU VM's will not have Internet connection, and then make a second Cloud connected to the tap0 This way you can have fun routing traffic form the VM's to the Internet.

1. Set up the VDE and boot the QEMU VM's connected to it.

$ vde_switch --numports 4 --hub --mod 777 --group users --tap tap0 -d
$ qemu-kvm -net vde -net nic,vlan=0,macaddr=52:54:00:00:EE:03 vmqemuimegdisk.qcow -m 512

2. Setup a bridge adapter

# brctl addbr br0
# brctl stp br0 off
# brctl setfd br0 0
# ifconfig br0 192.168.53.1 netmask 255.255.255.0 up

3. Setup a tap adapter and bridge it to the bridge adapter

# tunctl -b -u YOUUSERNAME
# ifconfig tap0 0.0.0.0 up
# brctl addif br0 tap0

4. Run the /usr/bin/qemu_iptables_rules script I show you how to make in the fist section of this post.

# qemu_iptables_rules

5. Connect the VDE to a Cloud in GNS3 like I showed you in the fist HowTo for GNS3 connection above

6. Now connect the tap0 interface to a Cloud in GNS3
In the Cloud configuration select the NIO TAP tab and enter in the name of the tap interface you created like.... tap0

7. Then Hit Add and Apply and OK.... Now you can Connect to the VDE switch/or/hub with the QEMU VM's on it in the one cloud and connect to the Internet through the other tap0 cloud.

Last edited by hunterthomson (2010-06-25 11:18:08)

Arch Linux

#1 2010-06-22 02:31:43

HowTo QEMU Networking with Wireless & QEMU host GNS3 networking

#2 2010-06-22 08:41:49

Re: HowTo QEMU Networking with Wireless & QEMU host GNS3 networking

#3 2010-06-22 09:11:15

Re: HowTo QEMU Networking with Wireless & QEMU host GNS3 networking

#4 2010-06-22 09:35:52

Re: HowTo QEMU Networking with Wireless & QEMU host GNS3 networking

#5 2010-06-22 10:59:18

Re: HowTo QEMU Networking with Wireless & QEMU host GNS3 networking

#6 2010-06-22 11:40:23

Re: HowTo QEMU Networking with Wireless & QEMU host GNS3 networking

#7 2010-06-22 23:00:03

Re: HowTo QEMU Networking with Wireless & QEMU host GNS3 networking

#8 2010-06-22 23:19:47

Re: HowTo QEMU Networking with Wireless & QEMU host GNS3 networking

#9 2010-06-23 00:40:31

Re: HowTo QEMU Networking with Wireless & QEMU host GNS3 networking

#10 2010-06-23 04:17:36

Re: HowTo QEMU Networking with Wireless & QEMU host GNS3 networking

Board footer