You are not logged in.

#1 2010-05-09 10:41:40

xintron
Member
From: Sweden
Registered: 2008-09-25
Posts: 32
Website

Public and private bridge

I've got a server running a few virtual servers. The problem is that my ISP is only giving me 5 IP-addresses and therefor I would like to use the host-machines IP for some virtual machines and some should have their own.

I have a public bridge setup (br0) which bridges eth0 on the host. Virtual machines then connect using a tap interface directly to br0 and get their own IP-address. Now to the problem. I want to have some virtual machines only use an "internal" IP, those using the host-machines external IP.

Example: My host gets the ip 8.8.8.8. Now my first virtual machine connects to the br0 interface and get 9.9.9.9. My second machine should use an ip and connect to the internet using the host ip (8.8.8.8). I've tried setting up yet another bridge (br1) and connect the virtual machine to that bridge. It works so that the host and the virtual machine can reach each other on 10.0.0.x but the guest OS can't make use of the internet and I can't forward any traffic to the guest either.

An attempt to a graphical version:

Internet
     |
   br0 <--------------> tap0 -> virtual machine #1
     |
   eth0                       
                                br1 <----> tap1 virtual Machine #2

So, how would you guys set this up, getting it to work as described?

Offline

#2 2010-05-09 11:48:05

Sin.citadel
Member
Registered: 2008-01-22
Posts: 267

Re: Public and private bridge

configure your vm guest for default gateway to your host bridge (10.0.0.1), then use iptables to NAT the connections to the external interface br0

iptables -t nat -A POSTROUTING -s 10.0.0.0/24 -o br0 -p all -j MASQUERADE # use SNAT if u get a static ip

make sure to enable ip forwarding, as well as setting a default route on your host.

Offline

#3 2010-05-20 16:27:36

xintron
Member
From: Sweden
Registered: 2008-09-25
Posts: 32
Website

Re: Public and private bridge

I've got the guest to use the host as gateway but for some reason they can't ping each other anymore. The iptables-step is configured and ip_forwarding is enabled. The guest can't reach out to the internet and I've been trying a few things with routes but can't seem to get it to work. Here's my `route` output:

Kernel IP routing table
Destination     Gateway         Genmask         Flags Metric Ref    Use Iface
10.0.0.0        *               255.255.255.0   U     0      0        0 br1
x.x.x.x         *               255.255.255.0   U     208    0        0 br0
default         x.x.x.x         0.0.0.0         UG    208    0        0 br0

Tried to set 10.0.0.0 to use the same gateway as my external connection on br0 (masked the IP) but without luck sad

Offline

#4 2010-05-21 08:08:35

Sin.citadel
Member
Registered: 2008-01-22
Posts: 267

Re: Public and private bridge

the above route output is for your host, so internet should be working on it correctly. With br1, you can ping your guest vm and vice-versa, did u set the guest to use the default gateway for host with the ip of br1? show the guest vm route output. after setting the default route in the vm guest, are you still able to ping the br1 ip address?

Last edited by Sin.citadel (2010-05-21 08:08:48)

Offline

#5 2010-05-21 08:21:40

xintron
Member
From: Sweden
Registered: 2008-09-25
Posts: 32
Website

Re: Public and private bridge

Host:

Kernel IP routing table
Destination     Gateway         Genmask         Flags Metric Ref    Use Iface
10.0.0.0        *               255.255.255.0   U     0      0        0 br1
x.x.x.x         *               255.255.255.0   U     208    0        0 br0
default         x.x.x.x         0.0.0.0         UG    208    0        0 br0

Guest:

Kernel IP routing table
Destination     Gateway         Genmask         Flags Metric Ref    Use Iface
10.0.0.0        *               255.255.255.0   U     0      0        0 eth0
default         10.0.0.1        0.0.0.0         UG    0      0        0 eth0

Internet is working just fine on the host and the host and guest can ping each other (host 10.0.0.1 and guest 10.0.0.2). The only problem is that traffic from guest isn't forwarded correctly on the host out to the internet.

Offline

#6 2010-05-21 08:59:51

Sin.citadel
Member
Registered: 2008-01-22
Posts: 267

Re: Public and private bridge

you should enable ip forwarding on the host by editing /etc/sysctl.conf and setting

net.ipv4.ip_forward=1

also, since the above iptables rules i posted will work correctly, you should set the dns server addresses in your guest vm to reflect those provided by your isp or use openDNS 208.67.222.222

see if your internet works or not, if it doesnt, see if you can ping 66.211.214.131 (archlinux.org ip) from the guest

Offline

#7 2010-05-21 09:17:30

xintron
Member
From: Sweden
Registered: 2008-09-25
Posts: 32
Website

Re: Public and private bridge

Everything you've posted is set correctly but my guest still can't ping anything except the host os (get package lost).

Here's my iptables output:

iptables -vL
Chain INPUT (policy DROP 0 packets, 0 bytes)
 pkts bytes target     prot opt in     out     source               destination         
   29  2794 ACCEPT     icmp --  any    any     anywhere             anywhere            
17365 1100K ACCEPT     tcp  --  any    any     anywhere             anywhere            tcp dpt:ssh 
 5253   37M ACCEPT     all  --  any    any     anywhere             anywhere            state RELATED,ESTABLISHED 
    4   240 ACCEPT     all  --  lo     any     anywhere             anywhere            
   88  4056 REJECT     tcp  --  any    any     anywhere             anywhere            reject-with tcp-reset 
  943  223K REJECT     udp  --  any    any     anywhere             anywhere            reject-with icmp-port-unreachable 
  677 18956 REJECT     all  --  any    any     anywhere             anywhere            reject-with icmp-proto-unreachable 

Chain FORWARD (policy DROP 0 packets, 0 bytes)
 pkts bytes target     prot opt in     out     source               destination         
 630K  239M ACCEPT     all  --  br0    any     anywhere             anywhere            

Chain OUTPUT (policy ACCEPT 13636 packets, 75M bytes)
 pkts bytes target     prot opt in     out     source               destination         


iptables -t nat -vL
Chain PREROUTING (policy ACCEPT 419 packets, 36571 bytes)
 pkts bytes target     prot opt in     out     source               destination         

Chain POSTROUTING (policy ACCEPT 410 packets, 35695 bytes)
 pkts bytes target     prot opt in     out     source               destination         
    0     0 MASQUERADE  all  --  any    br0     10.0.0.0/24          anywhere            
    1    84 MASQUERADE  all  --  any    eth0    anywhere             anywhere            

Chain OUTPUT (policy ACCEPT 7 packets, 2184 bytes)
 pkts bytes target     prot opt in     out     source               destination

Offline

#8 2010-05-21 09:56:54

Sin.citadel
Member
Registered: 2008-01-22
Posts: 267

Re: Public and private bridge

the problem is your FORWARD chain, set its default policy to ACCEPT (or at least to test) and then create a firewall rule in it to access packetts coming from br1 to travel through to br0

Offline

#9 2010-05-21 13:49:17

xintron
Member
From: Sweden
Registered: 2008-09-25
Posts: 32
Website

Re: Public and private bridge

Now we're getting somewhere smile There are still a few problems. I got the connection to the NAT:ed machine to work but the problem was that the connection to my bridged machine stopped working. Now I got the following iptables-setup which allows me to connect to the "public" (own IP from ISP) VPS and my private (NAT:ed) VPS:

iptables -vL
Chain INPUT (policy DROP 0 packets, 0 bytes)
 pkts bytes target     prot opt in     out     source               destination         
    0     0 ACCEPT     icmp --  any    any     anywhere             anywhere            
   38  2840 ACCEPT     tcp  --  any    any     anywhere             anywhere            tcp dpt:ssh 
    0     0 ACCEPT     tcp  --  any    any     anywhere             anywhere            tcp dpt:2222 
    0     0 ACCEPT     all  --  any    any     anywhere             anywhere            state RELATED,ESTABLISHED 
    0     0 ACCEPT     all  --  lo     any     anywhere             anywhere            
    0     0 REJECT     tcp  --  any    any     anywhere             anywhere            reject-with tcp-reset 
  143 15972 REJECT     udp  --  any    any     anywhere             anywhere            reject-with icmp-port-unreachable 
    6   168 REJECT     all  --  any    any     anywhere             anywhere            reject-with icmp-proto-unreachable 

Chain FORWARD (policy DROP 0 packets, 0 bytes)
 pkts bytes target     prot opt in     out     source               destination         
   78  7597 ACCEPT     tcp  --  br0    br1     anywhere             10.0.0.2            tcp dpt:2222 
 4802 1798K ACCEPT     all  --  br0    any     anywhere             anywhere            state RELATED,ESTABLISHED 
 3668  298K ACCEPT     all  --  any    br0     anywhere             anywhere            

Chain OUTPUT (policy ACCEPT 22 packets, 4760 bytes)
 pkts bytes target     prot opt in     out     source               destination         


iptables -t nat -vL
Chain PREROUTING (policy ACCEPT 407 packets, 35096 bytes)
 pkts bytes target     prot opt in     out     source               destination         
    1    60 DNAT       tcp  --  br0    any     anywhere             anywhere            tcp dpt:2222 to:10.0.0.2:2222 

Chain POSTROUTING (policy ACCEPT 409 packets, 35223 bytes)
 pkts bytes target     prot opt in     out     source               destination         
    0     0 MASQUERADE  all  --  any    eth0    10.0.0.0/24          anywhere            

Chain OUTPUT (policy ACCEPT 1 packets, 67 bytes)
 pkts bytes target     prot opt in     out     source               destination

The problem is that when connecting to my host IP on port 2222 it takes about 30s before I'm prompted with a password from SSH on my guest machine. To me it seems like it first tries to use the port on the host and when that fails it forwards to the guest os. I've tried a few setups but none have worked. How do I get the host to just forward the traffic on port 2222 directly to my guest (10.0.0.2) and at the same time allow my guest (10.0.0.2) to connect to the internet (currently I can't ping from my guest)?

Offline

#10 2010-05-21 14:31:40

Sin.citadel
Member
Registered: 2008-01-22
Posts: 267

Re: Public and private bridge

since port 2222 needs to be forwarded, and nothing is running on port 2222 forhost, you should remove its rule from iptables INPUT chain. also, if the problem is with iptables, try setting both FORWARD/INPUT chains to ACCEPT, and check if the setup is working as it should, this will tell u if the problem is with the firewall settings or something else.

also, in your POSTROUTING rule, set the output interface to br0, not eth0 (eth0 doesnt have an ip, only br0 does)

Offline

#11 2010-05-21 14:44:05

xintron
Member
From: Sweden
Registered: 2008-09-25
Posts: 32
Website

Re: Public and private bridge

Sin.citadel wrote:

since port 2222 needs to be forwarded, and nothing is running on port 2222 forhost, you should remove its rule from iptables INPUT chain. also, if the problem is with iptables, try setting both FORWARD/INPUT chains to ACCEPT, and check if the setup is working as it should, this will tell u if the problem is with the firewall settings or something else.

also, in your POSTROUTING rule, set the output interface to br0, not eth0 (eth0 doesnt have an ip, only br0 does)

Thank you very much. Didn't notice that INPUT rule, stupid me tongue Now I got the hang of it and it works just great. So thanks for taking your time to help me through this and making me see more clearly when it comes to iptables smile

Offline

Board footer

Powered by FluxBB