VirtualBox and QEMU/KVM - What to do with two arch-nemesis

lapsio · 2017-12-16 05:36:14

I'm not sure if there's forum section for workstations but I guess there's no such thing so here I go:

I'm using lots of virtualization. Like insane amounts of virtualization. I'm working in IT security and also still studying IT at uni so I usually need to simulate a lot of different environments and even full networks. Thus I'm using a lot of VBox and VMWare machines. It typically works really well. For additional security I'm using VM for internet (host is isolated), router VM for other VMs, VM for RAID array management which has more recent kernel for btrfs, VMs with Windows Server for work when I need it, VMs with various distros, notably CentOS as many commercial tools are only supported on RedHat, VMs for specific projects and so on. All in all around 60 VMs with 3-6 running simultaneously on average.

I'm using Virtualbox for simple ad-hoc VMs because it has nice solution for isolated networks and it's just quick to set up while VMWare is used for some high-performance heavier VMs or ones that are already prepackaged for VMWare. I'm also using VMWare for my gaming Windows VM. And I don't have any troubles with that setup because VMWare and VBox VMs go along really well. In fact whole networking for my VMWare machines is managed by VBox VM.

The problem is that I want to finally hit VGA passthrough. And here is the issue - it seems that the only virtualization platform realistically used with VGA passthrough is QEMU/KVM. Unfortunately - it collides with VirtualBox. As in - you can't use QEMU with KVM when any VBox VM using KVM is running. Even more unfortunately it doesn't strictly mean "using KVM" but in fact any form of using physical virtualization features. So my choice is either to use VBox full software (not really appealing) or QEMU without KVM and I'm not even sure if you can do vga passthrough in such case.

My entire ecosystem is strongly dependent on those VBox VMs. Getting rid of them would probably mean switching exclusively to VMWare and it doesn't sound like an easy task to migrate all those VMs. Plus I don't want to be that much dependent on commercial, quite expensive product. I'm also not that much for QEMU migration because guest addons integration on graphical system level (auto scaling, fullscreen, auto mouse capture) is really useful. And I never heard of QEMU equivalent.

Is there any way to:

a) VGA passthrough to VBox VM (I know it should be possible in theory but never seen any success report)
b) VGA passthrough to VMWare VM
c) run VBox and QEMU/KVM simultaneously
d) easily migrate VBox VMs to either VMWare or QEMU (with config, not just disk)
e) any other way of achieving VGA passthrough in such ecosystem?

I know It's quite broad question. I'm basically looking for options because I have no idea which of them are even possible.

Last edited by lapsio (2017-12-16 05:57:34)

Lone_Wolf · 2017-12-16 12:22:24

In fact whole networking for my VMWare machines is managed by VBox VM.

Funny, i've always viewed virtualbox networking as the 2nd weakest part they provide (X support for linux guests is the weakest one).

-------------------------------------------------------------------------------------

VMWare ESXi does support pCI Passthrough, but i think you're usiing vmplayer or workstation ?

Although most people seem to use it with qemu, libvirt is designed to work with multiple virtualization platforms .

For networking i suggest vde2 , it's supposed to be as close to using physical network devices as you can get.

lapsio · 2017-12-16 14:55:49

Lone_Wolf wrote:

Funny, i've always viewed virtualbox networking as the 2nd weakest part they provide (X support for linux guests is the weakest one).

Idk, for me with KDE guest at least random parts of taskbar don't randomly blink or disappear unlike they do with WS2016 guests so X support might not be the worst part . Yeah VirtualBox networking is bad IF you're using more than 1 vCPU (which is kinda funny dependency) I've been bringing up this anomaly for quite a long time already as well as many other users do eg.: https://www.virtualbox.org/ticket/10157 With single core VMs (and router VM doesn't really need dozen of computing power) it's on the same level as VMWare networking, which has latency just as bas as average consumer physical network card (while being all in all loopback networking so should be much faster). But it's "okay".

Lone_Wolf wrote:

VMWare ESXi does support pCI Passthrough, but i think you're usiing vmplayer or workstation ?

Yes I'm using Workstation 12.5. And I'm almost certain VMWare WS doesn't support any kind of pci passthrough, while VBox does have XML entries allowing it in theory at least:https://www.virtualbox.org/manual/ch09.html though as I said I've only seen failure reports around the web.

Lone_Wolf wrote:

Although most people seem to use it with qemu, libvirt is designed to work with multiple virtualization platforms .

Well this is getting interesting. I thought libvirt is only kind of manager lib, not responsible for actual virtualization. How is that related to PCI passthrough in VBox or concurrence of VBox and QEMU? I'm getting this kind of error precisely: https://askubuntu.com/questions/413511/ … each-other Though I don't need to remove modules. Simply closing VBox VM or QEMU/KVM enables me to run another hypervisor VMs again without any problem, they just can't work at once.

I really wonder how VMWare achieves compatibility with VBox and QEMU/KVM. I even tried forcing VT-x use in VMWare and it still works with no visible performance degradation comparing to when VBox is not running. It sounds like VBox or QEMU are poorly handling VT-x support...

Last edited by lapsio (2017-12-16 15:02:30)

pr0dukter · 2018-02-08 16:46:07

@lapsio lapsio

hope that paged you :D

where did you end up on your vmware adventure?

im starting on mine i picked up a rtl-sdr software defined radio for a smooth $9.00 so im installing skywave linux since its all radio distro, only downside is i think its ubuntu based.... hopefully its atleast debian. i know arch has tools but im not familiar with any of them so im gonna learn a bit with this distro but I digress...

did you find a graphics sollution? run across any amazing guides or PKGBUILDS on vga passthru? i know theres like a million of them i read them sometimes and dream :D

P.S. this isnt a necro is it mods? its almost 2 months but its not closed....

lapsio · 2018-02-08 17:00:46

Well technically I didn't find solution that satisfies me yet. Afaik vga passthrough with VMWare is possible only with ESXI which is type 1 hypervisor unlike Workstation which is pure type 2 (KVM and VBox are kind of hybrid pokemons - KVM being type 1 with some elements of type 2 and VBox being type 2 with some elements characteristic for type 1)

However for now I'm not seeking for solutions actively. I'm trying to slowly migrate from VBox to VMWare with my machines one by one starting with vdi to vmdk conversion. So in the end I'll probably end up with KVM + VMWare Workstation combo, removing VBox from this setup completely.

ewaller · 2018-02-08 17:04:39

pr0dukter wrote:

P.S. this isnt a necro is it mods? its almost 2 months but its not closed....

We expect that members are mature enough to show restraint rather than automatically closing threads. I note that, in this case, you did elicit a response from the OP, so I would not call in necro.

damjan · 2018-02-08 19:36:09

I didn't catch why you (@lapsio) don't just use qemu? gnome-boxes is a niceish simple frontend, more features but a bit more complex is libvirt + virt-manager

lapsio · 2018-02-08 23:31:22

damjan wrote:

I didn't catch why you (@lapsio) don't just use qemu? gnome-boxes is a niceish simple frontend, more features but a bit more complex is libvirt + virt-manager

VMWare isn't really that much part of question as I need it anyways but I think you mean why I have problem with VBox and KVM whereas I could just use KVM exclusively. Well - mainly because of VBox simplicity. What kind of buys me is:
- limited but at least some kind of gpu 3d acceleration both with windows and linux guests (without passthrough). For example now I'm writing from Vivaldi in my read-only internet VM and YouTube likes GPU acceleration.
- quick internal networking (I don't end up with dozen of ad-hoc bridges unlike with vmware)
- quick shared folder WITHOUT guest-host network connectivity so I can have network level isolation between guest and host yet still have shared storage
- guest addons in general shared clipboard, nice multi monitor support etc.
- special disk write modes (immutable, passthrough)
- lack of significant disadvantages (apart from lack of vga passthrough)

All of those are also fulfilled by VMWare Workstation (with 2 major differences - no virtio networking, a bit more clumsy internal networking setup but better 3d acceleration in vmware), however I'm not really sure if KVM can also provide those. I'm not really talking about workarounds. Because those always exist for example I can just set up regular samba - but then I need network connectivity between host and guest and it's just not that convenient. I don't have new Intel HD GPU to use GVT-d or whatever was the name of this shared gpu passthrough tech either so GPU acceleration is also not that certain I guess.

Yeah I think in the end I'll phase out to VMWare + KVM duet and get rid of VBox completely but It's process that requires me to remake my VMs one-by-one to meet new environment and It's process that takes some time as it's not really automatic, seamless migration.

Last edited by lapsio (2018-02-08 23:35:13)

damjan · 2018-02-09 17:50:31

lapsio wrote:

- limited but at least some kind of gpu 3d acceleration both with windows and linux guests (without passthrough). For example now I'm writing from Vivaldi in my read-only internet VM and YouTube likes GPU acceleration.

afaik qemu does support this virtio + virgl - I haven't personally put much testing into this though.

lapsio wrote:

- quick internal networking (I don't end up with dozen of ad-hoc bridges unlike with vmware)

thats the default in plain qemu, so yes.

lapsio wrote:

- quick shared folder WITHOUT guest-host network connectivity so I can have network level isolation between guest and host yet still have shared storage

see: qemu built-in smb server https://wiki.archlinux.org/index.php/QE … SMB_server

lapsio wrote:

- guest addons in general shared clipboard, nice multi monitor support etc.

never tried multi monitor, shared clipboard works with spice (which is the default in boxes, virt-manager)

lapsio wrote:

- special disk write modes (immutable, passthrough)

yes, and much more. for ex. single read-only base image and multiple writable snapshots - can work in parallel too.

also, afaik neither vbox nor vmware use the kernel /dev/kvm, and need at least some kernel (out of tree, proprietary) drivers on the host - which is very annoying.

lapsio · 2018-02-09 18:51:01

damjan wrote:

afaik qemu does support this virtio + virgl - I haven't personally put much testing into this though.
...
thats the default in plain qemu, so yes.
...
see: qemu built-in smb server https://wiki.archlinux.org/index.php/QE … SMB_server
...
never tried multi monitor, shared clipboard works with spice (which is the default in boxes, virt-manager)
...
also, afaik neither vbox nor vmware use the kernel /dev/kvm, and need at least some kernel (out of tree, proprietary) drivers on the host - which is very annoying.

I see. virgl has been introduced this summer, I didn't know about it.

QEMU by default uses NAT networking. By internal networks I meant named virtual switches without creating bridge interfaces on host. So that you can create eg. 6 separate virtual networks with 4 VMs attached to each and few virtual routers connected to multiple networks, filtering traffic between them quite easily. Without need to create permanent host level bridges. I'm not using NAT at all because host is isolated from internet access. Internet to VMs is provided by connecting router VM to physical, dedicated, unadressed interface on host.

Built in Samba works only for NAT networking and requires networking in the first place so it's quite useless. According to arch wiki it just starts regular samba server with auto-generated config

Shared clipboard in Spice seems to be bi-directional only. I'm using uni-directional shared clipboard and drag'n'drop only from guest to host in order to avoid sensitive data leakage from host to guest.

vbox requires proprietary blobs for extensions that provide certain functionality, though it's not necessary but it works in general. vmware is fully proprietary and yeah I know it's annoying especially that it actually never works with bleeding edge kernels and requires manual source patching for each kernel update so LTS is a must.

In general I'm not using VMs as "standalone" machines that much. In most cases I'm creating full virtualized infrastructures with routers and multiple VMs responsible for various taks, simulating various environments. For now there's over 60 specialized VMs organized into multiple laboratory networks interconnected in varying degree. VMWare and VBox VMs are interconnected using host level bridges but VBox exclusive networks don't use host bridge interfaces and don't spam host network config.

Of course VBox is not perfect. It's really far from being perfect but stuff it does is done pretty fine. I'd dare to say it's even more convenient than VMWare. For example ACPI support seems to be quite poor on vmware. When I click "Shutdown" button in hypervisor VMWare usually simply hard powers off VM while VBox seems to handle ACPI shutdown signal much better. It makes it MUCH simpler to gently down multiple VMs at once with VBox

Last edited by lapsio (2018-02-09 19:00:20)

rubenvb · 2018-02-15 19:41:29

Just want to chime in that I've fully switched to Qemu/libvirt/virt-manager and have not regretted it one second.

I mainly run a Windows 10 VM, and after installing the Spice Guest Tools I have everything related to autoscaling and copy-paste.

Shared folders are easily set up by running a samba server on the host, which is available on 10.0.2.2 by default. Note this setup for shared folders is by far the fastest I've seen (comparing with VirtualBox and VMWare) and the most powerful (wrt correct time reporting, executability on the network drive etc.).

Also: I can easily resume my laptop when the VM is running, where VirtualBox and VMWare prevented hibernation (I shut the lid, so this is kind of important). It does take longer because well, the RAM is fuller. Virt-manager's UI also has easy ways to pause the VM, and even save it like VirtualBox. You can also run libvirtd as a system-controlled service and e.g. boot VM's when the host boots etc. All nicely integrated in the Linux ecosystem.

Setting up your VM's might take a tad more time and is definitely not so user-friendly as the commercial solutions, but the end result is a lot more robust and flexible than the others (well, barring the fact that VMWare is capable of OpenGL 3.0 without GPU passthrough).

In short, I'd strongly suggest giving virt-manager+Qemu a decent test run. Fight through the initial setup process, and go the full mile to set it up correctly before you judge it.

lapsio · 2018-02-15 20:14:03

rubenvb wrote:

Also: I can easily resume my laptop when the VM is running, where VirtualBox and VMWare prevented hibernation (I shut the lid, so this is kind of important). It does take longer because well, the RAM is fuller. Virt-manager's UI also has easy ways to pause the VM, and even save it like VirtualBox.

It works since Workstation 14.

QEMU/KVM even with virgl doesn't support DirectX does it?...

Point of network drive without network connectivity with host is security. Not performance. If you're detonating malware in virtual machines, the last thing you want to do is to give guest any significant access to host. That includes clipboard from host to guest (while guest to host still would be nice to have), write access to shared storage, network connectivity with host, any stuff that isn't fully bulletproof like eg. USB controllers.

However thing I found really nice in QEMU is support for wav soundcard which allows you to basically save audio out to pipe and then play it, completely isolating soundcard as is from guest, yet allowing audio playback. With VMWare/VBox there seems to always be danger of mic hijack if malicious VM has sound support enabled.

Last edited by lapsio (2018-02-15 20:59:18)

Arch Linux

#1 2017-12-16 05:36:14

VirtualBox and QEMU/KVM - What to do with two arch-nemesis

#2 2017-12-16 12:22:24

Re: VirtualBox and QEMU/KVM - What to do with two arch-nemesis

#3 2017-12-16 14:55:49

Re: VirtualBox and QEMU/KVM - What to do with two arch-nemesis

#4 2018-02-08 16:46:07

Re: VirtualBox and QEMU/KVM - What to do with two arch-nemesis

#5 2018-02-08 17:00:46

Re: VirtualBox and QEMU/KVM - What to do with two arch-nemesis

#6 2018-02-08 17:04:39

Re: VirtualBox and QEMU/KVM - What to do with two arch-nemesis

#7 2018-02-08 19:36:09

Re: VirtualBox and QEMU/KVM - What to do with two arch-nemesis

#8 2018-02-08 23:31:22

Re: VirtualBox and QEMU/KVM - What to do with two arch-nemesis

#9 2018-02-09 17:50:31

Re: VirtualBox and QEMU/KVM - What to do with two arch-nemesis

#10 2018-02-09 18:51:01

Re: VirtualBox and QEMU/KVM - What to do with two arch-nemesis

#11 2018-02-15 19:41:29

Re: VirtualBox and QEMU/KVM - What to do with two arch-nemesis

#12 2018-02-15 20:14:03

Re: VirtualBox and QEMU/KVM - What to do with two arch-nemesis

Board footer