[solved] Problems forwarding host port to libvirt/qemu guest

TehVulpes · 2018-01-10 21:49:21

Hello, forum. Not 100% sure where to post this, but this board looks fit.

I've been trying to forward a port on my host machine to a libvirt/qemu guest on a virtual NIC. The host can access the guest through virbr0 (guest IP is 192.168.122.28), and the packets coming in from the outside network arrive through enp3s0. The guest has 2 NICs: ens3, which has access to the outside world and was created from running "virt-install ... --network user ...", and ens7, which is on the same net as the host's virbr0 and is only accessible to other guests and the host.

To forward the port, so far I've tried a number of things. Following the directions here, I've tried running the following both as a qemu hook and by manually running the following before starting a guest:

# iptables -I FORWARD -o virbr0 -d 192.168.122.28 -j ACCEPT
# iptables -t nat -I PREROUTING -p tcp --dport 80 -j DNAT --to 192.168.122.28:80

This is enough for the host to forward port 80 to the guest, but the guest filters those packets out because reverse path filtering is enabled. I've tried disabling the rp_filter, and that solves it being filtered before reaching the application. However, once the application tries to respond, it doesn't appear to ever reach the remote machine; I looked for something coming through in wireshark. I can access the guest service perfectly fine from the qemu host machine (e.g. lynx 192.168.122.28), but trying to access it from any remote machine fails.

So far I've tried many solutions that others have posted online but for some reason I can't figure out, none are working. As far as I can tell, either I'm fundamentally misunderstanding something on my own setup or I'm forwarding the port wrong. Any help would be greatly appreciated.

Last edited by TehVulpes (2018-01-15 01:27:56)

Lone_Wolf · 2018-01-11 12:12:13

output of route on the host and the 192.168.122.29 guest ?

How is virbr0 configured ?

what are ip-address & netmask of ens 3 & ens7 ?

TehVulpes · 2018-01-11 23:53:08

Hey, thanks for responding. Here comes a page full of config details...

Just running "route" alone didn't work, I'm guessing you meant "ip route"?

host # ip route
default via 10.30.43.254 dev enp3s0 proto dhcp metric 100 
10.30.42.0/23 dev enp3s0 proto kernel scope link src 10.30.42.127 metric 100 
192.168.122.0/24 dev virbr0 proto kernel scope link src 192.168.122.1

guest # ip route
default via 10.0.2.2 dev ens3 proto dhcp metric 100 
default via 192.168.122.1 dev ens7 proto dhcp metric 20100 
10.0.2.0/24 dev ens3 proto kernel scope link src 10.0.2.15 metric 100 
192.168.122.0/24 dev ens7 proto kernel scope link src 192.168.122.28 metric 100

Here's info on the guest ens3 / ens7...
ens3 IP/netmask: 10.0.2.15/24
ens7 IP/netmask: 192.168.122.28/24

Something to note is that the host can't access the guest through ens3, only through ens7.

The host's virbr0 is managed through libvirt.

host # virsh net-dumpxml internal
<network connections='1'>
  <name>internal</name>
  <uuid>c0268c53-0922-4d21-9f48-dc9a43a9130a</uuid>
  <forward mode='nat'>
    <nat>
      <port start='1024' end='65535'/>
    </nat>
  </forward>
  <bridge name='virbr0' stp='on' delay='0'/>
  <mac address='52:54:00:66:2f:49'/>
  <ip address='192.168.122.1' netmask='255.255.255.0'>
    <dhcp>
      <range start='192.168.122.2' end='192.168.122.254'/>
    </dhcp>
  </ip>
</network>

I don't know if it's worth mentioning or not, but while I posted trying to forward port 80, I'm having the exact same behavior with higher ports too. I tried port 2345 just to be sure it wasn't an issue with using a port < 1024.

And finally, here's how the adapters are defined in the guest.

host # virsh edit guest
<domain type='kvm'>
  ...
  <devices>
    ...
    <interface type='user'>
      <mac address='52:54:00:fd:3f:f0'/>
      <model type='rtl8139'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x0'/>
    </interface>
    <interface type='network'>
      <mac address='52:54:00:00:00:02'/>
      <source network='internal'/>
      <model type='rtl8139'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x07' function='0x0'/>
    </interface>
    ...
  </devices>
</domain>

Lone_Wolf · 2018-01-12 12:10:14

There's a route command in [core]/net-tools [1] that gets info directly from kernel, but ip route output is also usuable.

So we're dealing with 3 networks :

A 10.30.42.0/23
B 192.168.122.0/24
C 10.0.2.0/24

The host is in A and B and can access machines in both, while the guest is in B and C .
atm there's no routing between A & B , B & C , A & C .
Given that all 3 are private networks connecting them will require adding static routes, which may not be what you want.

In what network are the machines that are allowed to contact the guest located ?
A, C , internet (or maybe all 3 ) ?

[1]
net-tools has some commands that can be very handy for low level troubleshooting like mii-tool & netstat .

Last edited by Lone_Wolf (2018-01-12 12:26:23)

TehVulpes · 2018-01-13 04:10:15

Network A has the other machines that will be accessing services on the guests. I'm not worried about getting to it from the internet since I can join network A with a VPN. My original intent was to allow other machines on A to access the guests through port forwarding. Network C is the one that I've recently figured out is redundant, and I might remove it since I now realize I most likely don't need it.

In the meantime, I've just set up a spare machine with pfSense, and I've been able to achieve my initial goal with that. It's not as clean since it means another machine, but it's easier for me to understand. I'm definitely no network expert!

If you think that's a good solution, I can just stick with that. I'd still like to try getting it all done on just the one host if that's doable and you think it's a good idea.

Lone_Wolf · 2018-01-13 13:48:28

wrong thread, stay tuned for real answer

Last edited by Lone_Wolf (2018-01-13 13:49:27)

Lone_Wolf · 2018-01-13 14:45:37

Functionally speaking, your setup is a common scenario that includes a client , a host and a guest .
The desire to control what kind of connections they can use leads to having multiple networks with router/firewall devices keeping the networks separate.
Still soeaking functionally it only matters whether the router/firewall devices do their job following RFCs, not how they are implemented .
Using a pfsense router/firewall device is a good solution.

If at some point in time you prefer to reduce the number of physical devices, look into setting up vde2 on your host.
iptables/nftables can take care of the firewall part, but they are not great as a router imo.

Arch Linux

#1 2018-01-10 21:49:21

[solved] Problems forwarding host port to libvirt/qemu guest

#2 2018-01-11 12:12:13

Re: [solved] Problems forwarding host port to libvirt/qemu guest

#3 2018-01-11 23:53:08

Re: [solved] Problems forwarding host port to libvirt/qemu guest

#4 2018-01-12 12:10:14

Re: [solved] Problems forwarding host port to libvirt/qemu guest

#5 2018-01-13 04:10:15

Re: [solved] Problems forwarding host port to libvirt/qemu guest

#6 2018-01-13 13:48:28

Re: [solved] Problems forwarding host port to libvirt/qemu guest

#7 2018-01-13 14:45:37

Re: [solved] Problems forwarding host port to libvirt/qemu guest

Board footer