Arch Linux

Duelist

Member

Registered: 2014-09-22

Posts: 358

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

I'm feeling like my GPU's vendor dumped me.
Some months ago i've tried to update my card's VBIOS on UEFI, it downloaded the image but failed to apply it due to lack of drivers or some other stuff - it wasn't my pc, so i didn't want to mess with owner's windows system like installing drivers.

But i got the ROM. For one card only, but that's fine.
Tried passing it wth romfile and OVMF just hanged, aw suggested to test it on real hardware to see if it's just poorly written and broken. So i gave up for some time.

Now, i've installed windows system on a bare metal system using UEFI+GPT and even secureboot was working. But ASUS UEFI Update tools now says that... my GPU doesn't support UEFI. Considering the fact i have the actual ROM... It seems strange.

So, it's time to begin my journey on writing messages to their support team.

---

The forum rules prohibit requesting support for distributions other than arch.
I gave up. It was too late.
What I was trying to do.
The reference about VFIO and KVM VGA passthrough.

Ansa89

Member

Registered: 2014-08-30

Posts: 20

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

check whatever you use for your vm's disk. Be it filesystem's(you use file as a storage) health or actual hardware disk.
It seems to me like a kernel's problem, not a VM one.

I'm using a dedicated physical disk through lvm (I created a virtual disk using the whole physical disk and passed it to qemu).
It's a kernel oops for sure, but the process that generated it was qemu (for what I can understand from logs).

Also, if you're sure that your host disks are okay, try preparing your windows to migrating on virtio-blk-pci instead of scsi-way. That involves creating a dummy-drive on virtio-blk-pci device and feeding windows with drivers from virtio.iso and then "reconnecting" the drive.

The disk seems good to me, which qemu options I need to use virtio-blk-pci?

Duelist

Member

Registered: 2014-09-22

Posts: 358

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

The disk seems good to me, which qemu options I need to use virtio-blk-pci?

First - you've got to create a -device entry with virtio-blk-pci and drive=null, where -drive=/dev/null,id=null,if=none,format=raw .
Second - you boot windows with virtio.iso plugged into it, load the drivers for that disk controller.
Third - you change the drive of given virtio-blk-pci device to your windows drive and it should boot. If it gets BSOD 7B during startup - drivers for disk controller aren't installed.

Also, you appear to be using Q35 with it's built-in AHCI(SATA) controller, which is known to have some issues.

Hope that helps.

---

The forum rules prohibit requesting support for distributions other than arch.
I gave up. It was too late.
What I was trying to do.
The reference about VFIO and KVM VGA passthrough.

Ansa89

Member

Registered: 2014-08-30

Posts: 20

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

First - you've got to create a -device entry with virtio-blk-pci and drive=null, where -drive=/dev/null,id=null,if=none,format=raw .
Second - you boot windows with virtio.iso plugged into it, load the drivers for that disk controller.
Third - you change the drive of given virtio-blk-pci device to your windows drive and it should boot. If it gets BSOD 7B during startup - drivers for disk controller aren't installed.

Thanks, I'll try ASAP.
What are the benefits of using virtio-blk instead of virtio-scsi?

Also, you appear to be using Q35 with it's built-in AHCI(SATA) controller, which is known to have some issues.

I don't think so: as you can see from the script I'm using to launch qemu (posted previously), I don't specify any "-m" option and looking at the output of "qemu-system-x86_64 -machine help" I found this:

...
pc                   Standard PC (i440FX + PIIX, 1996) (alias of pc-i440fx-2.2)
pc-i440fx-2.2        Standard PC (i440FX + PIIX, 1996) (default)
...

What is that makes you think I'm using q35?

aw

Member

Registered: 2013-10-04

Posts: 921

Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

check whatever you use for your vm's disk. Be it filesystem's(you use file as a storage) health or actual hardware disk.
It seems to me like a kernel's problem, not a VM one.

I'm using a dedicated physical disk through lvm (I created a virtual disk using the whole physical disk and passed it to qemu).

Maybe it's a collision of the host and guest LVM since you're using a whole disk and it will be visible to both.  Is the guest volume group active in the host?

---

http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Cubex

Member

Registered: 2014-04-06

Posts: 24

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

It's posible to use the intended GPU to passthrough in first slot and tell the host to use the second slot GPU? In most motherboards the first slot has x16 and the second one has x8...

Duelist

Member

Registered: 2014-09-22

Posts: 358

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

It's posible to use the intended GPU to passthrough in first slot and tell the host to use the second slot GPU? In most motherboards the first slot has x16 and the second one has x8...

Yes.
Moreover, i have nvidia GT610 plugged in into pci-e x1 slot. Bus-Device-Function is 04:00.0
Passing through two HD7750 in x16 and x8(somehow runs as x4, but wired as x8, may be a driver's problem).
It works.
BUT!
1. You have to plunge the GPUs to pci-stub BEFORE ANYTHING.
2. Nvidia driver fails to detect the card without BusID option.

Section "Device"
    Identifier     "Device0"
    Driver         "nvidia"
    VendorName     "NVIDIA Corporation"
    BoardName      "GeForce GT 610"
    BusID          "PCI:04:00:00"
EndSection

EDIT:
Oh, the only downside is - there will be no emergency console available on ctrl-alt-F[1-12] since there is no VGA output.
Grub will show you your custom fonts, but things like plymouth, using kernel mode setting, won't work.
There is a bright side in it - usually VFIO-bind service is loaded after X, so if X fails to load due to bad drivers(like after a kernel rebuild, which should be worked around using package manager, but i'm lazy and sort of know what i'm doing) - primary GPU won't be hooked to pci-stub and VGA will work and you'll get to login in the console.

Also, gigabyte motherboards may offer a switch, which changes the primary GPU, solving everything. My ASUS doesn't have that feature.

Last edited by Duelist (2014-12-26 16:19:07)

---

The forum rules prohibit requesting support for distributions other than arch.
I gave up. It was too late.
What I was trying to do.
The reference about VFIO and KVM VGA passthrough.

Cubex

Member

Registered: 2014-04-06

Posts: 24

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

The console thing maybe can be solved by using fbcon, for example adding fbcon=map:111111 in bootline
and thanks for confirming the posibility of first slot

Last edited by Cubex (2014-12-26 17:22:36)

Myranti

Member

Registered: 2010-03-04

Posts: 20

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

I though that I should post what I've done so far that has enabled me to get my passthrough working and the issues I've run into along the way. Note: I'm still having at least 1 issue, which is an issue with some flickering black lines on the host side of the passthrough.

Specifications:
Host OS: Arch Linux - I've manually patched the current kernel (from ABS) with the ACS patch. This was an interesting experience required some modification to patch the correct lines.
Guest OS: Windows 8.1 Pro
Motherboard: Asrock C226-WS (Not the C226-WS+)
CPU: Intel Xeon E3-1241v3
Host GPU: Radeon R5 230
Guest GPU: Radeon R9 270X
(I'm using the open source radeon driver, not catalyst)
Note: I'm also using OVMF (which is actually in the Arch repos in Extra), but apart from manually patching the kernel, all other core software involved (qemu, libvirt, virt-manager) is from Arch repos, not from the AUR.

Edit: I've replaced the original motherboard (Asus P9D WS) with an Asrock C226-WS which allows me to put my passthrough VGA card in the top (closest to CPU) slot, since the video seems to initialise as per slot number since the full size slots are PCIe Slot 1, 3 and 6, with 6 being the top slot. This has given me a little extra performance (going from PCIe 3 x4 to x8) as well, bringing me closer to around 98% performance (somewhere around 98.5%), up from around 95%/96%, which is good enough for me.

The devices that I am passing through to the guest include the USB 3 card, the onboard audio (using a PCIe sound card on the host), one of the onboard LAN ports (Dual LAN, both Intel I210) and the R9 270X of course. At this point, I will mention that the original USB 3 card I was using ended up causing issues as the device wouldn't reset properly once it was released from the VM the first time leading to error's along the line of:

vfio-pci 0000:01:00.0: Refused to change power state, currently in D3
vfio_pci_disable: Failed to reset device 0000:01:00.0 (-22)

This also meant I could only pass through the device once from a fresh boot and then have to reboot to do it again. It would also cause irq issues like the following conflict with the sound card (I did remove the sound card, thinking that was the issue but it conflicted with other devices):

genirq: Flags mismatch irq 16. 00000000 (vfio-intx(0000:01:00.0)) vs. 00000080 (snd_oxygen_lib)

I had other cards on hand and replaced the USB 3 card with one that only partially worked (over-current conditions, leaving 2 ports not working) and then finally with one that would reset properly when released by the VM. I mention all this so that it might provide some insight for those that may be having these issues. The other main issue was with IOMMU groups (libvirt/qemu couldn't bind the devices being passed through without binding other devices in the IOMMU group) and required the ACS patch and the addition of pcie_acs_override=downstream to the kernel parameters in GRUB.

Kernel Parameters (through GRUB):

intel_iommu=on pcie_acs_override=downstream pci-stub.ids=1002:6810,1002:aab0,1106:3432,8086:8c20

I will also note here that the id for the dual LAN port hasn't been bound to pci-stub here as both devices share the same id, but it appears to work fine, perhaps due to it being an Intel NIC.

Kernel Modules (in initramfs via /etc/mkinicpio.conf) - except ahci, which was already in my modules line before this:

MODULES="ahci pci-stub vfio-pci vfio-iommu-type1"

I also have /etc/modprobe.d/kvm-qemu.conf

options kvm ignore_msrs=1

And /etc/modprobe.d/vfio-iommu.conf

options vfio_iommu_type1 allow_unsafe_interrupts=1

Edit: I was using the vfio-bind script, but I was able to just assign devices to pci-stub and let libvirt manage this (managed='yes' hostdev entries). The only issue was that both LAN ports on this dual LAN board are the same vendor:model in lspci and my network scripts were adding any en* devices to the network bridge and so my host box wouldn't connect to the network. This has since been fixed but only assigning the one I want to a network and leaving the other disconnected.

At this point, I diverged slightly from others and used libvirt along with virt-manager to configure setup and configure VM and the passthrough.

I used virt-manager to create a new virtual machine, specifying Windows 8.1 as the guest OS, created a temporary hdd image and completed the setup without customizing the install, leaving it as an i440fx system (after watching Alex Williamson's KVM Forum presentation, seems like there's still potential issues with q35 and i440fx works fine). I then went through and removed some of the extra devices I didn't need, leaving the IDE, PCI and USB controllers in place and ended up just using VNC and cirrus video to get the guest installed, rather than having all the spice stuff in place. I modified a few of the settings, told it to copy the hosts cpu config, adjusted the cpu topology and then used virsh to edit the xml file of the VM. I specifically did not pass through the HDMI for the GPU being passed through in the end, as it causes the host screen to go fuzzy and it doesn't release/reset properly once the guest VM shuts down.

Yet another note: I didn't pass the GPU through until I had installed the OS using the VNC/cirrus video and had ensured that my other devices were working correctly and that the VM would startup and shutdown repeatedly without issue without the GPU bieng passed through.

Of the changes via virsh, one of them was the addition of the OVMF loader, specifying a whole physical hard drive, rather than an image and modifying some of the hostdev entries with the vfio driver (I'll point these out further down).

Edit: I modified the VM after it being noted that I wasn't using Virtio for the disk (virtio is definitely faster for me), so I went through the process to get the driver installed then switched the disk to virtio initially, then to scsi (using virtio-scsi controller) and added the discard="unmap" to the driver line for the disk since I'm using an SSD.

This is how the updated xml file for my VM looks:

<domain type='kvm'>
  <name>windows</name>
  <uuid>b49fc4cc-149e-4904-8a5c-e74fb5a4c417</uuid>
  <memory unit='KiB'>12582912</memory>
  <currentMemory unit='KiB'>12582912</currentMemory>
  <memoryBacking>
    <hugepages/>
    <nosharepages/>
  </memoryBacking>
  <vcpu placement='static'>6</vcpu>
  <os>
    <type arch='x86_64' machine='pc-i440fx-2.1'>hvm</type>
    <loader type='pflash'>/var/lib/libvirt/images/ovmf_x64.bin</loader>
    <bootmenu enable='no'/>
  </os>
  <features>
    <acpi/>
    <apic/>
    <pae/>
    <hyperv>
      <relaxed state='on'/>
      <vapic state='on'/>
      <spinlocks state='on' retries='8191'/>
    </hyperv>
  </features>
  <cpu mode='custom' match='exact'>
    <model fallback='allow'>SandyBridge</model>
    <vendor>Intel</vendor>
    <topology sockets='1' cores='6' threads='1'/>
    <feature policy='require' name='vme'/>
    <feature policy='require' name='dtes64'/>
    <feature policy='require' name='invpcid'/>
    <feature policy='require' name='vmx'/>
    <feature policy='require' name='erms'/>
    <feature policy='require' name='xtpr'/>
    <feature policy='require' name='smep'/>
    <feature policy='require' name='pbe'/>
    <feature policy='require' name='est'/>
    <feature policy='require' name='monitor'/>
    <feature policy='require' name='smx'/>
    <feature policy='require' name='abm'/>
    <feature policy='require' name='tm'/>
    <feature policy='require' name='acpi'/>
    <feature policy='require' name='fma'/>
    <feature policy='require' name='osxsave'/>
    <feature policy='require' name='ht'/>
    <feature policy='require' name='pdcm'/>
    <feature policy='require' name='pdpe1gb'/>
    <feature policy='require' name='fsgsbase'/>
    <feature policy='require' name='f16c'/>
    <feature policy='require' name='ds'/>
    <feature policy='require' name='invtsc'/>
    <feature policy='require' name='tm2'/>
    <feature policy='require' name='avx2'/>
    <feature policy='require' name='ss'/>
    <feature policy='require' name='bmi1'/>
    <feature policy='require' name='bmi2'/>
    <feature policy='require' name='pcid'/>
    <feature policy='require' name='ds_cpl'/>
    <feature policy='require' name='movbe'/>
    <feature policy='require' name='rdrand'/>
  </cpu>
  <clock offset='localtime'>
    <timer name='rtc' tickpolicy='catchup'/>
    <timer name='pit' tickpolicy='delay'/>
    <timer name='hpet' present='no'/>
    <timer name='hypervclock' present='yes'/>
  </clock>
  <on_poweroff>destroy</on_poweroff>
  <on_reboot>restart</on_reboot>
  <on_crash>restart</on_crash>
  <pm>
    <suspend-to-mem enabled='no'/>
    <suspend-to-disk enabled='no'/>
  </pm>
  <devices>
    <emulator>/usr/sbin/qemu-system-x86_64</emulator>
    <disk type='block' device='disk'>
      <driver name='qemu' type='raw' cache='none' io='native' discard='unmap'/>
      <source dev='/dev/sdd'/>
      <target dev='sda' bus='scsi'/>
      <boot order='2'/>
      <address type='drive' controller='0' bus='0' target='0' unit='0'/>
    </disk>
    <disk type='block' device='disk'>
      <driver name='qemu' type='raw' cache='none' io='native'/>
      <source dev='/dev/sde'/>
      <target dev='sdb' bus='scsi'/>
      <address type='drive' controller='0' bus='0' target='0' unit='1'/>
    </disk>
    <disk type='block' device='cdrom'>
      <driver name='qemu' type='raw'/>
      <target dev='hdc' bus='ide'/>
      <readonly/>
      <boot order='1'/>
      <address type='drive' controller='0' bus='0' target='0' unit='1'/>
    </disk>
    <controller type='pci' index='0' model='pci-root'/>
    <controller type='ide' index='0'>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x1'/>
    </controller>
    <controller type='usb' index='0' model='nec-xhci'>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x05' function='0x0'/>
    </controller>
    <controller type='scsi' index='0' model='virtio-scsi'>
      <driver queues='6'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x09' function='0x0'/>
    </controller>
    <hostdev mode='subsystem' type='pci' managed='yes'>
      <driver name='vfio'/>
      <source>
        <address domain='0x0000' bus='0x05' slot='0x00' function='0x0'/>
      </source>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x04' function='0x0'/>
    </hostdev>
    <hostdev mode='subsystem' type='pci' managed='yes'>
      <driver name='vfio'/>
      <source>
        <address domain='0x0000' bus='0x00' slot='0x1b' function='0x0'/>
      </source>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x06' function='0x0'/>
    </hostdev>
    <hostdev mode='subsystem' type='pci' managed='yes'>
      <driver name='vfio'/>
      <source>
        <address domain='0x0000' bus='0x01' slot='0x00' function='0x0'/>
      </source>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x0'/>
    </hostdev>
    <hostdev mode='subsystem' type='pci' managed='yes'>
      <source>
        <address domain='0x0000' bus='0x0f' slot='0x00' function='0x0'/>
      </source>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x0'/>
    </hostdev>
    <memballoon model='virtio'>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x07' function='0x0'/>
    </memballoon>
  </devices>
</domain>

The relevant extra changes I made are:
The OVMF loader, which I copied to my libvirt images directory (which is a symlink anyway) as it complained that it didn't have permissions and I was too lazy to mess around with it.

    <loader type='pflash'>/var/lib/libvirt/images/ovmf_x64.bin</loader>

The disk type for the system drive (SSD), which I specified as block and specified the source drive (/dev/sdd in this case). As noted earlier, this was changed as I'm using virtio-scsi and unmap to pass through trim/discard commands from the guest OS to the drive:

    <disk type='block' device='disk'>
      <driver name='qemu' type='raw' cache='none' io='native' discard='unmap'/>
      <source dev='/dev/sdd'/>
      <target dev='sda' bus='scsi'/>
      <boot order='2'/>
      <address type='drive' controller='0' bus='0' target='0' unit='0'/>
    </disk>

And the addition of the vfio device driver name for the host devices being passed through (just added between the hostdev lines):

      <driver name='vfio'/>

There was a bit of fiddling amongst all this obviously (as per some of my notes above for issues I ran into) but ultimately it is, for the most part, working. I haven't yet tested the system on bare metal to get an indication of the performance when it's not in the VM, but comparing some limited benchmarking to what I can find equivalents of online (Unigine Valley benchmark), I think I'm within about 5% of bare metal performance. It might be more like within 10%, but I have yet to do more testing to determine how it compares to relatively equivalent hardware.

Hopefully some of this is useful to others with their attempts and I think I've covered most of the work I've done to get it to this point.

Edit: I removed the seperate pci-stub conf and softdep on drm and put my pci-stub line as a kernel parameter. I also added a second drive for storage (steam backups, etc). I've also enabled hugepages (using the Arch wiki guide), though this required a restart to work properly without throwing errors from libvirt.

Last edited by Myranti (2015-01-20 11:43:49)

pkim

Member

Registered: 2014-12-27

Posts: 9

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

All,

I wanted to report my successful setup so that others may have another data point trying to do the same. I also need others to see what setup works, because before embarking on trying to do this and sinking down a significant amount of time, I didn't know whether I would succeed or not. I tried Googling my motherboard and the setup with vfio but didn't find any with my setup, so I didn't have 100% confidence that I would be able to get my system running.

Host OS: Fedora 21 (64bit, 3.17.7 kernel)
Guest OS: Windows 7 (64bit)
CPU: i7-4790K (one of the very few K versions that supports Vt-d)
Motherboard: Asus Sabertooth Z97 mark 1 (had to enable virtualization and Vt-d in BIOS)
Host GPU: Geforce 970 GTX (Nvidia's proprietary driver v346.22)
Guest GPU: Geforce 560Ti (Nvidia's driver v347.09)
System RAM: DDR3-1600 4x8GB (Crucial Ballistix Sport)
Emulator/Hypervisor: Running qemu-kvm, which is managed by libvirtd

I believe I ran into every conceivable problem that has been reported by others in this very long thread. Some of the issues and how I resolved them:

1. Windows reported Code 43 - had to hide from Windows (more specifically from Nvidia's driver) that it was running as a VM - info here (modified the -cpu switch that was being sent by libvirt, include kvm=off and eliminate all hv_* options) - I am starting to see why Linus infamously gave Nvidia the finger on one of the videos I found on the internet. Why would Nvidia actively engage in locking out their card from being run in VM? I doubt that would increase the sales of their Quadro line of cards or to decrease their support line load because of guys like me trying to get this to work with their cards. Instead they decided to turn this into a game of cat and mouse hide and seek. I don't understand their decision other than because they have such a dominant position that they think they can get away with it. AMD isn't providing any competition to Nvidia as I am reading that Radeon cards have their own set of issues.
2. Windows reported Code 12 - Initially ran without any video or graphics for the VM in libvirt to bypass this issue, but I couldn't figure out how to pipe the sound out - ended up using "Xen" for video card in libvirt. That allowed the sound the come through and eliminated Code 12 in Windows.
3. qemu reported that it couldn't lock memory - I played with ilimit and editing /etc/security/limits.conf, but none of that worked (makes sense, I was running qemu-kvm as root - no limit) - it turned out that selinux was blocking ipc_lock operation for qemu-kvm. I used the excellent Fedora's selinux debugging tool to enable it and now it works.
4. I had issues with iommu grouping. The two Nvidia cards were being grouped into the same group and qemu refused to work because the iommu grouping was "not a viable grouping." I had initially assumed that the PCIe ACS patch from Alex Williamson patch here has either gone upstream or Fedora team incorporated the patch by now, but no amount of tweaking on the boot command pcie_acs_override=downstream would work. But in few posts, Alex says that this patch will not go upstream due to the nature of the patch and the bug report on fedora bugzilla report here makes it clear that it hadn't made it even as a patch on Fedora. I also downloaded the source RPM and checked out quirks.c file and realized that it did not have the patch (I'm at the latest kernel version 3.17.7 for FC21). I wanted to do this the "right" way and went about creating a custom RPM from the source RPM (plus the ACS patch). I used the custom kernel compiling guide here . The patch also didn't work out of the box, I had to manually patch it in for 3.17.7. I also had to create a link for python-config in /usr/local/bin. The stock installation doesn't seem to have the link and it breaks the kernel compile.
5. I had to put the following options to grub boot up - "intel_iommu=on pci-stub.ids=10de:1200,10de:0e0c pcie_acs_override=downstream"

I've had other issues too, but many of them were caused by not following some of the nicely written guides on the net.

Here's what I used as a base guide here .
I didn't rebuild qemu, as I was already at version 2.1.2. I also didn't have to worry about nosnoop and i915 patch. I installed the stock qemu and kvm from Fedora. I did install the vfio-bind service as the guide indicated - this is important as we need bring the device into the /dev/vfio/ directory. I did update /etc/libvirt/qemu.conf to allow qemu to run as root and included the new cgroup_device_acl options. Instead of creating the libvirt xml from scratch as the guide suggested, I actually use libvirt to create a VM first. I then continued to install the guest OS and the virtio stuff needed to get it working. Then I started to modify the xml using "virsh edit [domain name]" to make updates first to use vfio-pci then any additional modifications as needed. The tricky part of editing the xml file was making sure I had the "xmlns:qemu='http://libvirt.org/schemas/domain/qemu/1.0'" line at the top. Otherwise, I kept losing the <qemu:commandline> options.

Windows 7 indicates that I'm at level 7.0 for "Windows Experience Index"
I didn't run any 3D benchmarks, but did run Portal using Steam - no problems seen there and very good performance as far as I can tell, certainly better than a Intel Core 2 Duo E8500 setup I had previously dedicated for gaming.

Ongoing problems - after shutting down the VM, I can't get it to come up again the second time (or the third time, etc). I used the suggested trick of putting the host in Suspend mode/wake up and launching again - that works. But occasionally, I get an interrupt issue that I can't get out of and the whole machine needs rebooting. Also the audio isn't quite right. I'm using ICH6 to emulate and piping it to the host via the Xen setup, but I'm getting fairly frequent glitches/stuttering in audio from the guest.

This capability is a bit of a game-changer for me. I've always had two machines - one for my mission critical stuff in Linux, and another for gaming in Windows. Steam's effort to bring more gaming to Linux is helping, but we're not quite there yet. I think that this setup may be the first time where I feel that I can actually consolidate two machines into one. A big thank you for everyone who allowed this to happen and many people debugging this.

My libvirt xml file:

<domain type='kvm' xmlns:qemu='http://libvirt.org/schemas/domain/qemu/1.0'>
  <name>win7-gaming</name>
  <uuid>ed501ade-d5f1-447e-87d7-a527ba1847b5</uuid>
  <memory unit='KiB'>6291456</memory>
  <currentMemory unit='KiB'>6291456</currentMemory>
  <vcpu placement='static'>4</vcpu>
  <os>
    <type arch='x86_64' machine='pc-i440fx-2.1'>hvm</type>
    <boot dev='hd'/>
  </os>
  <features>
    <acpi/>
    <apic/>
    <pae/>
    <hyperv>
      <relaxed state='on'/>
      <vapic state='on'/>
      <spinlocks state='on' retries='8191'/>
    </hyperv>
  </features>
  <clock offset='localtime'>
    <timer name='rtc' tickpolicy='catchup'/>
    <timer name='pit' tickpolicy='delay'/>
    <timer name='hpet' present='no'/>
    <timer name='hypervclock' present='yes'/>
  </clock>
  <on_poweroff>destroy</on_poweroff>
  <on_reboot>restart</on_reboot>
  <on_crash>restart</on_crash>
  <pm>
    <suspend-to-mem enabled='no'/>
    <suspend-to-disk enabled='no'/>
  </pm>
  <devices>
    <emulator>/usr/bin/qemu-kvm</emulator>
    <disk type='file' device='disk'>
      <driver name='qemu' type='qcow2'/>
      <source file='/var/lib/libvirt/images/win7.qcow2'/>
      <target dev='vdb' bus='virtio'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x09' function='0x0'/>
    </disk>
    <disk type='file' device='cdrom'>
      <driver name='qemu' type='raw'/>
      <source file='/var/lib/libvirt/images/virtio-win-0.1-94.iso'/>
      <target dev='hdb' bus='ide'/>
      <readonly/>
      <address type='drive' controller='0' bus='0' target='0' unit='1'/>
    </disk>
    <controller type='usb' index='0' model='ich9-ehci1'>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x05' function='0x7'/>
    </controller>
    <controller type='usb' index='0' model='ich9-uhci1'>
      <master startport='0'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x05' function='0x0' multifunction='on'/>
    </controller>
    <controller type='usb' index='0' model='ich9-uhci2'>
      <master startport='2'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x05' function='0x1'/>
    </controller>
    <controller type='usb' index='0' model='ich9-uhci3'>
      <master startport='4'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x05' function='0x2'/>
    </controller>
    <controller type='pci' index='0' model='pci-root'/>
    <controller type='ide' index='0'>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x1'/>
    </controller>
    <controller type='virtio-serial' index='0'>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x06' function='0x0'/>
    </controller>
    <interface type='network'>
      <mac address='52:54:00:cb:a5:3b'/>
      <source network='default'/>
      <model type='virtio'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x0'/>
    </interface>
    <serial type='pty'>
      <target port='0'/>
    </serial>
    <console type='pty'>
      <target type='serial' port='0'/>
    </console>
    <channel type='spicevmc'>
      <target type='virtio' name='com.redhat.spice.0'/>
      <address type='virtio-serial' controller='0' bus='0' port='1'/>
    </channel>
    <input type='tablet' bus='usb'/>
    <input type='mouse' bus='ps2'/>
    <input type='keyboard' bus='ps2'/>
    <graphics type='spice' autoport='yes'/>
    <sound model='ich6'>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x04' function='0x0'/>
    </sound>
    <video>
      <model type='xen' vram='4096' heads='1'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x0'/>
    </video>
    <hostdev mode='subsystem' type='usb' managed='yes'>
      <source>
        <vendor id='0x0424'/>
        <product id='0x2514'/>
        <address bus='5' device='2'/>
      </source>
    </hostdev>
    <hostdev mode='subsystem' type='usb' managed='yes'>
      <source>
        <vendor id='0x06a3'/>
        <product id='0x8021'/>
      </source>
    </hostdev>
    <hostdev mode='subsystem' type='usb' managed='yes'>
      <source>
        <vendor id='0x046d'/>
        <product id='0xc52b'/>
        <address bus='5' device='4'/>
      </source>
    </hostdev>
    <redirdev bus='usb' type='spicevmc'>
    </redirdev>
    <redirdev bus='usb' type='spicevmc'>
    </redirdev>
    <memballoon model='virtio'>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x07' function='0x0'/>
    </memballoon>
  </devices>
  <qemu:commandline>
    <qemu:arg value='-device'/>
    <qemu:arg value='ioh3420,bus=pci.0,addr=1c.0,multifunction=on,port=1,chassis=1,id=root.1'/>
    <qemu:arg value='-device'/>
    <qemu:arg value='vfio-pci,host=02:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on'/>
    <qemu:arg value='-device'/>
    <qemu:arg value='vfio-pci,host=02:00.1,bus=pci.0'/>
    <qemu:arg value='-bios'/>
    <qemu:arg value='/usr/share/qemu/bios.bin'/>
    <qemu:arg value='-cpu'/>
    <qemu:arg value='Haswell,kvm=off'/>
  </qemu:commandline>
</domain>

The patch file for kernel 3.17.7:

diff -rupN kernel-3.17.fc21/linux-3.17.7-300.pkim.fc21.x86_64/Documentation/kernel-parameters.txt kernel-3.17.fc21.new/linux-3.17.7-300.pkim.fc21.x86_64/Documentation/kernel-parameters.txt
--- a/Documentation/kernel-parameters.txt	2014-12-26 00:24:21.607861638 -0800
+++ b/Documentation/kernel-parameters.txt	2014-12-25 23:56:13.554340620 -0800
@@ -2655,6 +2655,16 @@ bytes respectively. Such letter suffixes
 		nomsi	Do not use MSI for native PCIe PME signaling (this makes
 			all PCIe root ports use INTx for all services).
 
+	pcie_acs_override =
+			[PCIE] Override missing PCIe ACS support for:
+		downstream
+			All downstream ports - full ACS capabilties
+		multifunction
+			All multifunction devices - multifunction ACS subset
+		id:nnnn:nnnn
+			Specfic device - full ACS capabilities
+			Specified as vid:did (vendor/device ID) in hex
+
 	pcmv=		[HW,PCMCIA] BadgePAD 4
 
 	pd_ignore_unused
diff -rupN kernel-3.17.fc21/linux-3.17.7-300.pkim.fc21.x86_64/drivers/pci/quirks.c kernel-3.17.fc21.new/linux-3.17.7-300.pkim.fc21.x86_64/drivers/pci/quirks.c
--- a/drivers/pci/quirks.c	2014-12-26 00:24:21.157861498 -0800
+++ b/drivers/pci/quirks.c	2014-12-26 00:02:46.714460917 -0800
@@ -3583,6 +3583,108 @@ struct pci_dev *pci_get_dma_source(struc
 	return pci_dev_get(dev);
 }
 
+static bool acs_on_downstream;
+static bool acs_on_multifunction;
+
+#define NUM_ACS_IDS 16
+struct acs_on_id {
+	unsigned short vendor;
+	unsigned short device;
+};
+static struct acs_on_id acs_on_ids[NUM_ACS_IDS];
+static u8 max_acs_id;
+
+static __init int pcie_acs_override_setup(char *p)
+{
+	if (!p)
+		return -EINVAL;
+
+	while (*p) {
+		if (!strncmp(p, "downstream", 10))
+			acs_on_downstream = true;
+		if (!strncmp(p, "multifunction", 13))
+			acs_on_multifunction = true;
+		if (!strncmp(p, "id:", 3)) {
+			char opt[5];
+			int ret;
+			long val;
+
+			if (max_acs_id >= NUM_ACS_IDS - 1) {
+				pr_warn("Out of PCIe ACS override slots (%d)\n",
+					NUM_ACS_IDS);
+				goto next;
+			}
+
+			p += 3;
+			snprintf(opt, 5, "%s", p);
+			ret = kstrtol(opt, 16, &val);
+			if (ret) {
+				pr_warn("PCIe ACS ID parse error %d\n", ret);
+				goto next;
+			}
+			acs_on_ids[max_acs_id].vendor = val;
+
+			p += strcspn(p, ":");
+			if (*p != ':') {
+				pr_warn("PCIe ACS invalid ID\n");
+				goto next;
+			}
+
+			p++;
+			snprintf(opt, 5, "%s", p);
+			ret = kstrtol(opt, 16, &val);
+			if (ret) {
+				pr_warn("PCIe ACS ID parse error %d\n", ret);
+				goto next;
+			}
+			acs_on_ids[max_acs_id].device = val;
+			max_acs_id++;
+		}
+next:
+		p += strcspn(p, ",");
+		if (*p == ',')
+			p++;
+	}
+
+	if (acs_on_downstream || acs_on_multifunction || max_acs_id)
+		pr_warn("Warning: PCIe ACS overrides enabled; This may allow non-IOMMU protected peer-to-peer DMA\n");
+
+	return 0;
+}
+early_param("pcie_acs_override", pcie_acs_override_setup);
+
+static int pcie_acs_overrides(struct pci_dev *dev, u16 acs_flags)
+{
+	int i;
+
+	/* Never override ACS for legacy devices or devices with ACS caps */
+	if (!pci_is_pcie(dev) ||
+	    pci_find_ext_capability(dev, PCI_EXT_CAP_ID_ACS))
+		return -ENOTTY;
+
+	for (i = 0; i < max_acs_id; i++)
+		if (acs_on_ids[i].vendor == dev->vendor &&
+		    acs_on_ids[i].device == dev->device)
+			return 1;
+
+	switch (pci_pcie_type(dev)) {
+	case PCI_EXP_TYPE_DOWNSTREAM:
+	case PCI_EXP_TYPE_ROOT_PORT:
+		if (acs_on_downstream)
+			return 1;
+		break;
+	case PCI_EXP_TYPE_ENDPOINT:
+	case PCI_EXP_TYPE_UPSTREAM:
+	case PCI_EXP_TYPE_LEG_END:
+	case PCI_EXP_TYPE_RC_END:
+		if (acs_on_multifunction && dev->multifunction)
+			return 1;
+	}
+
+	return -ENOTTY;
+}
+
+
 /*
  * AMD has indicated that the devices below do not support peer-to-peer
  * in any system where they are found in the southbridge with an AMD
@@ -3696,6 +3798,7 @@ static const struct pci_dev_acs_enabled
 	{ PCI_VENDOR_ID_ATI, 0x4384, pci_quirk_amd_sb_acs },
 	{ PCI_VENDOR_ID_ATI, 0x4399, pci_quirk_amd_sb_acs },
 	{ PCI_VENDOR_ID_INTEL, PCI_ANY_ID, pci_quirk_intel_pch_acs },
+	{ PCI_ANY_ID, PCI_ANY_ID, pcie_acs_overrides },
 	{ 0 }
 };

the patch for kernel.spec:

--- kernel.spec.orig	2014-12-25 23:12:35.649583082 -0800
+++ kernel.spec	2014-12-26 01:44:27.301825643 -0800
@@ -42,7 +42,7 @@ Summary: The Linux kernel
 # For non-released -rc kernels, this will be appended after the rcX and
 # gitX tags, so a 3 here would become part of release "0.rcX.gitX.3"
 #
-%global baserelease 300
+%global baserelease 301
 %global fedora_build %{baserelease}
 
 # base_sublevel is the kernel version we're starting with and patching
@@ -648,6 +648,9 @@ Patch26099: deal-with-deadlock-in-d_walk
 # git clone ssh://git.fedorahosted.org/git/kernel-arm64.git, git diff master...devel
 Patch30000: kernel-arm64.patch
 
+# pcie_acs_override
+Patch90000: PCIE-ACS-override.patch
+
 # END OF PATCH DEFINITIONS
 
 %endif
@@ -1412,6 +1415,9 @@ ApplyPatch arm64-fix-xgene_enet_process_
 %endif
 %endif
 
+# pcie_acs_override
+ApplyPatch PCIE-ACS-override.patch
+
 # END OF PATCH APPLICATIONS
 
 %endif

Last edited by pkim (2014-12-27 20:28:55)

4kGamer

Member

Registered: 2014-10-29

Posts: 88

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

thank you guys for posting detailed descriptions!

one question for Myranti: I noticed you aren't using Virtio for your disk. Did you do that on purpose? Is it faster that way? Do you also use a spare SSD for Windows 8.1?

Last edited by 4kGamer (2014-12-27 20:55:52)

kainet

Member

Registered: 2014-12-22

Posts: 9

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

thank you guys for posting detailed descriptions!

one question for Myranti: I noticed you aren't using Virtio for your disk. Did you do that on purpose? Is it faster that way? Do you also use a spare SSD for Windows 8.1?

I had tested virtio and piix4-ide for me Windows shows the same 6.9 points for disk performance. I suppose the most performant way is to passthrough sata controller.

aw

Member

Registered: 2013-10-04

Posts: 921

Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

All,

I wanted to report my successful setup so that others may have another data point trying to do the same. I also need others to see what setup works, because before embarking on trying to do this and sinking down a significant amount of time, I didn't know whether I would succeed or not. I tried Googling my motherboard and the setup with vfio but didn't find any with my setup, so I didn't have 100% confidence that I would be able to get my system running.

Congratulations... unfortunately I'm not even sure where to start correcting your config.

a) Z97 (Lynx Point) is supported by kernel ACS quirks, so all you need to do is move your GPU to a difference slot to fix the IOMMU grouping problem, no custom kernel required.  Put it behind a PCH root port slot.

b) All of your libvirt issues are self induced by using the qemu:arg options.  See posts from me for reference to a wrapper script to add the x-vga=on option to the qemu-kvm invocation by libvirt.  This lets libvirt manage the device, you won't need to run with elevated privileges or use the vfio-bind script (see reference to redhat instructions for how to make the wrapper script work with selinux).

c) Your config uses ioh3420 root ports on 440fx chipset.  Stop it, just stop it.  Move the vfio-pci options to proper hostdev options.

d) The whole xen primary graphics, stop that too.

e) libvirt supports kvm=off, why are you specifying it via qemu:args?!  You've also still got various hyper-v extensions enabled in your xml.  See my blog for how to do all of this properly.

f) You're specifying a bios, stop that too.  Completely unnecessary.

What can you expect to achieve by making all these changes?  You don't need the i915 patch and you don't need the ACS patch, so I expect you can make this work with an unmodified kernel, qemu, and libvirt, having libvirt fully manage your VM (no vfio-bind), with just a qemu-kvm wrapper, your pci-stub.ids line, and an selinux module for the script.

---

http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Myranti

Member

Registered: 2010-03-04

Posts: 20

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

one question for Myranti: I noticed you aren't using Virtio for your disk. Did you do that on purpose? Is it faster that way? Do you also use a spare SSD for Windows 8.1?

Good question! Probably because I didn't think of doing it, but I see there's a guide on on linux-kvm.org, so I'll try it out. It wasn't specifically on purpose, I just hadn't really considered it and now that I've had a look, I realise I'd probably need to do it after the initial install anyway.

I'm not sure if it's faster, but I'll do a test before the change and after to see if it has a noticeable impact. Yes, I am using an SSD, not so much a spare, more of an additional drive, since this system was upgraded/built specifically for the purposes of doing this.

Edit: Yes, it does appear to be faster to use virtio.

Last edited by Myranti (2014-12-28 02:29:40)

pkim

Member

Registered: 2014-12-27

Posts: 9

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Alex,

Thanks for the good feedback -

a) Z97 (Lynx Point) is supported by kernel ACS quirks, so all you need to do is move your GPU to a difference slot to fix the IOMMU grouping problem, no custom kernel required.  Put it behind a PCH root port slot.

This is a consequence of the motherboard size, case design, power supply installed at the bottom and the fact that both Nvidia cards are double width. The motherboard has 3 16-lane PCIe slots and the two cards can only be fitted into two of them. In fact, the first thing I tried was to try to relocate one of the two cards to the 3rd slot based on your advice - not physically possible. So rather than buying another bigger case, I decided to try the ACS patch. (I would venture to guess that my situation is not unique.)

b) All of your libvirt issues are self induced by using the qemu:arg options.  See posts from me for reference to a wrapper script to add the x-vga=on option to the qemu-kvm invocation by libvirt.  This lets libvirt manage the device, you won't need to run with elevated privileges or use the vfio-bind script (see reference to redhat instructions for how to make the wrapper script work with selinux).

I think I got this working using a wrapper script that you provided. But I still don't know how to do it without the vfio-bind script - are you suggesting that I do the binding inside the wrapper as well?

c) Your config uses ioh3420 root ports on 440fx chipset.  Stop it, just stop it.  Move the vfio-pci options to proper hostdev options.

Ok, figured it out. Not too many examples out there. A user named "teekay" talked about trying it out and failing on this thread here - but it seems to work for me.

d) The whole xen primary graphics, stop that too.

One of your posts on your blog mentioned using qxl and disabling it so I can use it at least for the sound here. But no matter how I tried, I just don't know how to "disable" the qxl video and eliminate the Code 12 error. I need this so that I can use the host sound.

e) libvirt supports kvm=off, why are you specifying it via qemu:args?!  You've also still got various hyper-v extensions enabled in your xml.  See my blog for how to do all of this properly.

I actually saw this while I was on the hunt to solve this, I remember seeing that you've submitted it to git for inclusion like 3-months ago - I incorrectly assumed that the feature was not included in qemu 2.1.2. I was wrong. I cleaned up my xml - thanks!

f) You're specifying a bios, stop that too.  Completely unnecessary.

I didn't know that. I deleted it and it works fine without it.

Here's my latest xml, free of any qemu:args (still on xen - couldn't figure out how to use qxl):

<domain type='kvm'>
  <name>win7-gaming</name>
  <uuid>ed501ade-d5f1-447e-87d7-a527ba1847b5</uuid>
  <memory unit='KiB'>6291456</memory>
  <currentMemory unit='KiB'>6291456</currentMemory>
  <vcpu placement='static'>4</vcpu>
  <os>
    <type arch='x86_64' machine='pc-i440fx-2.1'>hvm</type>
    <boot dev='hd'/>
  </os>
  <features>
    <acpi/>
    <apic/>
    <pae/>
    <kvm>
      <hidden state='on'/>
    </kvm>
  </features>
  <cpu mode='custom' match='exact'>
    <model fallback='allow'>Haswell</model>
  </cpu>
  <clock offset='localtime'>
    <timer name='rtc' tickpolicy='catchup'/>
    <timer name='pit' tickpolicy='delay'/>
    <timer name='hpet' present='no'/>
  </clock>
  <on_poweroff>destroy</on_poweroff>
  <on_reboot>restart</on_reboot>
  <on_crash>restart</on_crash>
  <pm>
    <suspend-to-mem enabled='no'/>
    <suspend-to-disk enabled='no'/>
  </pm>
  <devices>
    <emulator>/usr/libexec/qemu-kvm-nvidia-vga</emulator>
    <disk type='file' device='disk'>
      <driver name='qemu' type='qcow2'/>
      <source file='/var/lib/libvirt/images/win7.qcow2'/>
      <target dev='vdb' bus='virtio'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x09' function='0x0'/>
    </disk>
    <disk type='file' device='cdrom'>
      <driver name='qemu' type='raw'/>
      <source file='/var/lib/libvirt/images/virtio-win-0.1-94.iso'/>
      <target dev='hdb' bus='ide'/>
      <readonly/>
      <address type='drive' controller='0' bus='0' target='0' unit='1'/>
    </disk>
    <controller type='usb' index='0' model='ich9-ehci1'>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x05' function='0x7'/>
    </controller>
    <controller type='usb' index='0' model='ich9-uhci1'>
      <master startport='0'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x05' function='0x0' multifunction='on'/>
    </controller>
    <controller type='usb' index='0' model='ich9-uhci2'>
      <master startport='2'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x05' function='0x1'/>
    </controller>
    <controller type='usb' index='0' model='ich9-uhci3'>
      <master startport='4'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x05' function='0x2'/>
    </controller>
    <controller type='pci' index='0' model='pci-root'/>
    <controller type='ide' index='0'>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x1'/>
    </controller>
    <controller type='virtio-serial' index='0'>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x06' function='0x0'/>
    </controller>
    <interface type='network'>
      <mac address='52:54:00:cb:a5:3b'/>
      <source network='default'/>
      <model type='virtio'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x0'/>
    </interface>
    <serial type='pty'>
      <target port='0'/>
    </serial>
    <console type='pty'>
      <target type='serial' port='0'/>
    </console>
    <channel type='spicevmc'>
      <target type='virtio' name='com.redhat.spice.0'/>
      <address type='virtio-serial' controller='0' bus='0' port='1'/>
    </channel>
    <input type='tablet' bus='usb'/>
    <input type='mouse' bus='ps2'/>
    <input type='keyboard' bus='ps2'/>
    <graphics type='spice' autoport='yes'/>
    <sound model='ich6'>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x04' function='0x0'/>
    </sound>
    <video>
      <model type='xen' vram='4096' heads='1'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x0'/>
    </video>
    <hostdev mode='subsystem' type='usb' managed='yes'>
      <source>
        <vendor id='0x0424'/>
        <product id='0x2514'/>
        <address bus='5' device='2'/>
      </source>
    </hostdev>
    <hostdev mode='subsystem' type='usb' managed='yes'>
      <source>
        <vendor id='0x06a3'/>
        <product id='0x8021'/>
      </source>
    </hostdev>
    <hostdev mode='subsystem' type='usb' managed='yes'>
      <source>
        <vendor id='0x046d'/>
        <product id='0xc52b'/>
        <address bus='5' device='4'/>
      </source>
    </hostdev>
    <hostdev mode='subsystem' type='pci' managed='yes'>
      <driver name='vfio'/>
      <source>
        <address domain='0x0000' bus='0x02' slot='0x00' function='0x0'/>
      </source>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x08' function='0x0'/>
    </hostdev>
    <hostdev mode='subsystem' type='pci' managed='yes'>
      <driver name='vfio'/>
      <source>
        <address domain='0x0000' bus='0x02' slot='0x00' function='0x1'/>
      </source>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x0a' function='0x0'/>
    </hostdev>
    <redirdev bus='usb' type='spicevmc'>
    </redirdev>
    <redirdev bus='usb' type='spicevmc'>
    </redirdev>
    <memballoon model='virtio'>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x07' function='0x0'/>
    </memballoon>
  </devices>
</domain>

By the way, I also backed out of the three settings in /etc/libvirt/qemu.conf (commented them and let libvirt use the defaults - the VM is very stable and also the sound works better now):

user = "root"
group = "root"
clear_emulator_capabilities = 0

Last edited by pkim (2014-12-28 08:44:06)

4kGamer

Member

Registered: 2014-10-29

Posts: 88

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

@Myranti. I wasn't so sure myself. Glad if it helped you.

And @kainet: It does make sense what you say. I wonder whether sata performance would surpass virtio. Did someone make the comparison?

Last edited by 4kGamer (2014-12-28 12:17:54)

Myranti

Member

Registered: 2010-03-04

Posts: 20

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

@4kGamer: Yeah, I just hadn't considered it. I ended up going with SCSI and using virito-scsi with discard='unmap' so that any trim/discard is passed through.

If I was using another motherboard with additional sata controllers I could try that to see how it went, but alas!

aw

Member

Registered: 2013-10-04

Posts: 921

Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Alex,

Thanks for the good feedback -

a) Z97 (Lynx Point) is supported by kernel ACS quirks, so all you need to do is move your GPU to a difference slot to fix the IOMMU grouping problem, no custom kernel required.  Put it behind a PCH root port slot.

This is a consequence of the motherboard size, case design, power supply installed at the bottom and the fact that both Nvidia cards are double width. The motherboard has 3 16-lane PCIe slots and the two cards can only be fitted into two of them. In fact, the first thing I tried was to try to relocate one of the two cards to the 3rd slot based on your advice - not physically possible. So rather than buying another bigger case, I decided to try the ACS patch. (I would venture to guess that my situation is not unique.)

Too bad, so close to being able to work unmodified.  I have the same problem on my P8H67-M PRO, but I use integrated graphics for the host and my primary guest uses a dual slot GTX750 from the CPU root ports.  My secondary guest uses an HD8570, single width slot off the PCH root port.  In fact I've managed to make that card fanless by cutting down a CoolVivaZ1 so that it still squeezes in above the P/S.

b) All of your libvirt issues are self induced by using the qemu:arg options.  See posts from me for reference to a wrapper script to add the x-vga=on option to the qemu-kvm invocation by libvirt.  This lets libvirt manage the device, you won't need to run with elevated privileges or use the vfio-bind script (see reference to redhat instructions for how to make the wrapper script work with selinux).

I think I got this working using a wrapper script that you provided. But I still don't know how to do it without the vfio-bind script - are you suggesting that I do the binding inside the wrapper as well?

No, libvirt will do it for you, that's what the managed="yes" option in the xml means.

c) Your config uses ioh3420 root ports on 440fx chipset.  Stop it, just stop it.  Move the vfio-pci options to proper hostdev options.

Ok, figured it out. Not too many examples out there. A user named "teekay" talked about trying it out and failing on this thread here - but it seems to work for me.

ioh3420 is a PCIe root port, so the frankenstein configuration that creates is to have a PCIe root port sourced on a conventional PCI bus.  It's not something you'd ever see on real hardware.

d) The whole xen primary graphics, stop that too.

One of your posts on your blog mentioned using qxl and disabling it so I can use it at least for the sound here. But no matter how I tried, I just don't know how to "disable" the qxl video and eliminate the Code 12 error. I need this so that I can use the host sound.

Right, that qxl config is using OVMF and Win8, which is a much more amenable config for simultaneously supporting both assigned and emulated graphics.  With Win7 and VGA, the VM will stop using the emulated graphics once the Nvidia driver loads due to resource conflicts of the VGA space.  I don't know that that would affect audio over spice though.  Theoretically libvirt can support connecting the emulated audio to your local pulseaudio server independent of the emulated graphics, but I've never figured out how to do this.  I'm also suspicious whether things like Steam In-home-streaming work with anything other than the GPU audio device.

e) libvirt supports kvm=off, why are you specifying it via qemu:args?!  You've also still got various hyper-v extensions enabled in your xml.  See my blog for how to do all of this properly.

I actually saw this while I was on the hunt to solve this, I remember seeing that you've submitted it to git for inclusion like 3-months ago - I incorrectly assumed that the feature was not included in qemu 2.1.2. I was wrong. I cleaned up my xml - thanks!

There's also the virt-preview repo for Fedora if you want more bleeding edge QEMU/libvirt.

f) You're specifying a bios, stop that too.  Completely unnecessary.

I didn't know that. I deleted it and it works fine without it.

Here's my latest xml, free of any qemu:args (still on xen - couldn't figure out how to use qxl):

<domain type='kvm'>
  <name>win7-gaming</name>
  <uuid>ed501ade-d5f1-447e-87d7-a527ba1847b5</uuid>
  <memory unit='KiB'>6291456</memory>
  <currentMemory unit='KiB'>6291456</currentMemory>
  <vcpu placement='static'>4</vcpu>
  <os>
    <type arch='x86_64' machine='pc-i440fx-2.1'>hvm</type>
    <boot dev='hd'/>
  </os>
  <features>
    <acpi/>
    <apic/>
    <pae/>
    <kvm>
      <hidden state='on'/>
    </kvm>
  </features>
  <cpu mode='custom' match='exact'>
    <model fallback='allow'>Haswell</model>
  </cpu>
  <clock offset='localtime'>
    <timer name='rtc' tickpolicy='catchup'/>
    <timer name='pit' tickpolicy='delay'/>
    <timer name='hpet' present='no'/>
  </clock>
  <on_poweroff>destroy</on_poweroff>
  <on_reboot>restart</on_reboot>
  <on_crash>restart</on_crash>
  <pm>
    <suspend-to-mem enabled='no'/>
    <suspend-to-disk enabled='no'/>
  </pm>
  <devices>
    <emulator>/usr/libexec/qemu-kvm-nvidia-vga</emulator>
    <disk type='file' device='disk'>
      <driver name='qemu' type='qcow2'/>
      <source file='/var/lib/libvirt/images/win7.qcow2'/>
      <target dev='vdb' bus='virtio'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x09' function='0x0'/>
    </disk>
    <disk type='file' device='cdrom'>
      <driver name='qemu' type='raw'/>
      <source file='/var/lib/libvirt/images/virtio-win-0.1-94.iso'/>
      <target dev='hdb' bus='ide'/>
      <readonly/>
      <address type='drive' controller='0' bus='0' target='0' unit='1'/>
    </disk>
    <controller type='usb' index='0' model='ich9-ehci1'>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x05' function='0x7'/>
    </controller>
    <controller type='usb' index='0' model='ich9-uhci1'>
      <master startport='0'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x05' function='0x0' multifunction='on'/>
    </controller>
    <controller type='usb' index='0' model='ich9-uhci2'>
      <master startport='2'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x05' function='0x1'/>
    </controller>
    <controller type='usb' index='0' model='ich9-uhci3'>
      <master startport='4'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x05' function='0x2'/>
    </controller>
    <controller type='pci' index='0' model='pci-root'/>
    <controller type='ide' index='0'>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x1'/>
    </controller>
    <controller type='virtio-serial' index='0'>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x06' function='0x0'/>
    </controller>
    <interface type='network'>
      <mac address='52:54:00:cb:a5:3b'/>
      <source network='default'/>
      <model type='virtio'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x0'/>
    </interface>
    <serial type='pty'>
      <target port='0'/>
    </serial>
    <console type='pty'>
      <target type='serial' port='0'/>
    </console>
    <channel type='spicevmc'>
      <target type='virtio' name='com.redhat.spice.0'/>
      <address type='virtio-serial' controller='0' bus='0' port='1'/>
    </channel>
    <input type='tablet' bus='usb'/>
    <input type='mouse' bus='ps2'/>
    <input type='keyboard' bus='ps2'/>
    <graphics type='spice' autoport='yes'/>
    <sound model='ich6'>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x04' function='0x0'/>
    </sound>
    <video>
      <model type='xen' vram='4096' heads='1'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x0'/>
    </video>
    <hostdev mode='subsystem' type='usb' managed='yes'>
      <source>
        <vendor id='0x0424'/>
        <product id='0x2514'/>
        <address bus='5' device='2'/>
      </source>
    </hostdev>
    <hostdev mode='subsystem' type='usb' managed='yes'>
      <source>
        <vendor id='0x06a3'/>
        <product id='0x8021'/>
      </source>
    </hostdev>
    <hostdev mode='subsystem' type='usb' managed='yes'>
      <source>
        <vendor id='0x046d'/>
        <product id='0xc52b'/>
        <address bus='5' device='4'/>
      </source>
    </hostdev>
    <hostdev mode='subsystem' type='pci' managed='yes'>
      <driver name='vfio'/>
      <source>
        <address domain='0x0000' bus='0x02' slot='0x00' function='0x0'/>
      </source>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x08' function='0x0'/>
    </hostdev>
    <hostdev mode='subsystem' type='pci' managed='yes'>
      <driver name='vfio'/>
      <source>
        <address domain='0x0000' bus='0x02' slot='0x00' function='0x1'/>
      </source>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x0a' function='0x0'/>
    </hostdev>
    <redirdev bus='usb' type='spicevmc'>
    </redirdev>
    <redirdev bus='usb' type='spicevmc'>
    </redirdev>
    <memballoon model='virtio'>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x07' function='0x0'/>
    </memballoon>
  </devices>
</domain>

By the way, I also backed out of the three settings in /etc/libvirt/qemu.conf (commented them and let libvirt use the defaults - the VM is very stable and also the sound works better now):

user = "root"
group = "root"
clear_emulator_capabilities = 0

Great, that's a much cleaner config IMHO.  If you wanted you could add multifunction="on" to the GPU hostdev config and then you could move the assigned audio device to function 1 of the same slot.  It makes the card appear to the guest more like it actually is, but I haven't found that it functionally behaves any different.

---

http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

4kGamer

Member

Registered: 2014-10-29

Posts: 88

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Can someone help me? I don't know how to passthrough a sata controller.

this is what lspci | gep SATA outputs:

00:1f.2 SATA controller: Intel Corporation 9 Series Chipset Family SATA Controller [AHCI Mode]
07:00.0 SATA controller: ASMedia Technology Inc. ASM1062 Serial ATA Controller (rev 02)
09:00.0 SATA controller: ASMedia Technology Inc. ASM1062 Serial ATA Controller (rev 02)

I want to install Windows on a SATA Disk. I have to passthrough a SATA controller for that right?

First I bound the controller to vfio. But when I try to start the installation, i get this error:

Error starting domain: Requested operation is not valid: PCI device 0000:07:00.0 is not assignable

This is what my XML looks like

<disk type='block' device='disk'>
      <driver name='qemu' type='raw' cache='none' io='native'/>
      <source dev='/dev/sdb'/>
      <target dev='sda' bus='sata'/>
      <boot order='2'/>
      <address type='drive' controller='0' bus='0' target='0' unit='0'/>
    </disk>

I googled it, but I am lost here. Please help

aw

Member

Registered: 2013-10-04

Posts: 921

Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Can someone help me? I don't know how to passthrough a sata controller.

this is what lspci | gep SATA outputs:

00:1f.2 SATA controller: Intel Corporation 9 Series Chipset Family SATA Controller [AHCI Mode]
07:00.0 SATA controller: ASMedia Technology Inc. ASM1062 Serial ATA Controller (rev 02)
09:00.0 SATA controller: ASMedia Technology Inc. ASM1062 Serial ATA Controller (rev 02)

I want to install Windows on a SATA Disk. I have to passthrough a SATA controller for that right?

First I bound the controller to vfio. But when I try to start the installation, i get this error:

Error starting domain: Requested operation is not valid: PCI device 0000:07:00.0 is not assignable

This is what my XML looks like

<disk type='block' device='disk'>
      <driver name='qemu' type='raw' cache='none' io='native'/>
      <source dev='/dev/sdb'/>
      <target dev='sda' bus='sata'/>
      <boot order='2'/>
      <address type='drive' controller='0' bus='0' target='0' unit='0'/>
    </disk>

I googled it, but I am lost here. Please help

You cannot both assign the device and attach the disk behind it using an emulated controller.  One or the other.  The only case where I consider assigning a SATA controller to be a valid option is if you want to boot the image both on bare metal and a VM, and even there it's questionable whether it's really an advantage.  If you're doing this only for performance reasons then you haven't taken virtio tuning as far as it can go.  There's very, very little performance to gain by doing this.

---

http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

4kGamer

Member

Registered: 2014-10-29

Posts: 88

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

thank you Alex, I won't bother with it, back to Virtio then. You're right. I am still not comfortable with virtio tuning, the documentation is great, but I still haven't had success with it. I will try again and probably have more questions for you

aw

Member

Registered: 2013-10-04

Posts: 921

Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

thank you Alex, I won't bother with it, back to Virtio then. You're right. I am still not comfortable with virtio tuning, the documentation is great, but I still haven't had success with it. I will try again and probably have more questions for you

FWIW, I use this on my Steam VM:

    <disk type='file' device='disk'>
      <driver name='qemu' type='raw' cache='none' io='native'/>
      <source file='/mnt/store/vm/Steam.raw'/>
      <backingStore/>
      <target dev='sda' bus='scsi'/>
      <alias name='scsi0-0-0-0'/>
      <address type='drive' controller='0' bus='0' target='0' unit='0'/>
    </disk>
    <controller type='scsi' index='0' model='virtio-scsi'>
      <driver queues='4'/>
      <alias name='scsi0'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x05' function='0x0'/>
    </controller>

The raw disk image actually lives on a relatively slow spinning disk, but I use bcache to cache it across a small SSD.  This setup doesn't always benchmark well, but for real world performance launching a single game, it does fantastic.

EDIT: queues=4 enables multi-queue virtio-scsi.  I use 4 because that's the number of vCPUs provided to the guest.

Last edited by aw (2014-12-28 22:16:07)

---

http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Chaohong Hu

Member

Registered: 2014-12-29

Posts: 2

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

I have successfully passed through graphics card using vfio.

The problem is vfio need an additional monitor connected to the graphics card to show vm,
but i only have one monitor(and two gfxs of course),so i am searching a solution which can
share the only monitor between host's gfx and vm's gfx. I mean when i launch vm the monitor
start to display output from passthru's card and when i shutdown vm, monitor display output from
host's gfx again.

Any idea about this?

4kGamer

Member

Registered: 2014-10-29

Posts: 88

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

@Alex: thank you, I will try your config!

@Chao: You need some sort of switch or hub. If your GPUs both have HDMI outputs, you can use an HDMI switch. In any case a device with one output and two inputs would work.

I use an AV-R, because I use my setup with a TV. But there are other options.

Like a KVM switch  (the abbreviations are coincidental).

Look here: https://www.youtube.com/watch?v=VGxzHkI8rXo

Last edited by 4kGamer (2014-12-29 08:09:39)

pkim

Member

Registered: 2014-12-27

Posts: 9

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

b) All of your libvirt issues are self induced by using the qemu:arg options.  See posts from me for reference to a wrapper script to add the x-vga=on option to the qemu-kvm invocation by libvirt.  This lets libvirt manage the device, you won't need to run with elevated privileges or use the vfio-bind script (see reference to redhat instructions for how to make the wrapper script work with selinux).

I think I got this working using a wrapper script that you provided. But I still don't know how to do it without the vfio-bind script - are you suggesting that I do the binding inside the wrapper as well?

No, libvirt will do it for you, that's what the managed="yes" option in the xml means.

Confirmed! letting libvirt to do all the managing seems to be much smoother and easier (once you figure out the correct xml configuration).

Great, that's a much cleaner config IMHO.  If you wanted you could add multifunction="on" to the GPU hostdev config and then you could move the assigned audio device to function 1 of the same slot.  It makes the card appear to the guest more like it actually is, but I haven't found that it functionally behaves any different.

Agreed. I'll try out the multifunction="on" later just for the sake of making it cleaner.

So basically what I needed it to work is this:

Host OS: Fedora 21 (64bit, 3.17.7 kernel, CUSTOM compiled with PCIe ACS patch from Alex Williamson)
Guest OS: Windows 7 (64bit)
CPU: i7-4790K (one of the very few K versions that supports Vt-d)
Motherboard: Asus Sabertooth Z97 mark 1 (had to enable virtualization and Vt-d in BIOS)
Host GPU: Geforce 970 GTX (Nvidia's proprietary driver v346.22)
Guest GPU: Geforce 560Ti (Nvidia's driver v347.09)
System RAM: DDR3-1600 4x8GB (Crucial Ballistix Sport)
QEMU-KVM: 2.1.2, stock, no configuration changes.
Libvirt: 1.2.9, stock

And the following xml (it is possible to do everything without qemu:args! - also note that I used a wrapper script to insert the x-vga option):

<domain type='kvm'>
  <name>win7-gaming</name>
  <uuid>ed501ade-d5f1-447e-87d7-a527ba1847b5</uuid>
  <memory unit='KiB'>6291456</memory>
  <currentMemory unit='KiB'>6291456</currentMemory>
  <vcpu placement='static'>4</vcpu>
  <os>
    <type arch='x86_64' machine='pc-i440fx-2.1'>hvm</type>
    <boot dev='hd'/>
  </os>
  <features>
    <acpi/>
    <apic/>
    <pae/>
    <kvm>
      <hidden state='on'/>
    </kvm>
  </features>
  <cpu mode='host-model'>
    <model fallback='allow'/>
  </cpu>
  <clock offset='localtime'>
    <timer name='rtc' tickpolicy='catchup'/>
    <timer name='pit' tickpolicy='delay'/>
    <timer name='hpet' present='no'/>
  </clock>
  <on_poweroff>destroy</on_poweroff>
  <on_reboot>restart</on_reboot>
  <on_crash>restart</on_crash>
  <pm>
    <suspend-to-mem enabled='no'/>
    <suspend-to-disk enabled='no'/>
  </pm>
  <devices>
    <emulator>/usr/libexec/qemu-kvm-nvidia-vga</emulator>
    <disk type='file' device='disk'>
      <driver name='qemu' type='qcow2'/>
      <source file='/var/lib/libvirt/images/win7-gaming.qcow2'/>
      <target dev='vdc' bus='virtio'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x0b' function='0x0'/>
    </disk>
    <disk type='block' device='cdrom'>
      <driver name='qemu' type='raw'/>
      <target dev='hdb' bus='ide'/>
      <readonly/>
      <address type='drive' controller='0' bus='0' target='0' unit='1'/>
    </disk>
    <controller type='usb' index='0' model='ich9-ehci1'>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x05' function='0x7'/>
    </controller>
    <controller type='usb' index='0' model='ich9-uhci1'>
      <master startport='0'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x05' function='0x0' multifunction='on'/>
    </controller>
    <controller type='usb' index='0' model='ich9-uhci2'>
      <master startport='2'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x05' function='0x1'/>
    </controller>
    <controller type='usb' index='0' model='ich9-uhci3'>
      <master startport='4'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x05' function='0x2'/>
    </controller>
    <controller type='pci' index='0' model='pci-root'/>
    <controller type='ide' index='0'>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x1'/>
    </controller>
    <controller type='virtio-serial' index='0'>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x06' function='0x0'/>
    </controller>
    <interface type='network'>
      <mac address='52:54:00:cb:a5:3b'/>
      <source network='default'/>
      <model type='virtio'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x0'/>
    </interface>
    <serial type='pty'>
      <target port='0'/>
    </serial>
    <console type='pty'>
      <target type='serial' port='0'/>
    </console>
    <channel type='spicevmc'>
      <target type='virtio' name='com.redhat.spice.0'/>
      <address type='virtio-serial' controller='0' bus='0' port='1'/>
    </channel>
    <input type='tablet' bus='usb'/>
    <input type='mouse' bus='ps2'/>
    <input type='keyboard' bus='ps2'/>
    <graphics type='spice' autoport='yes'/>
    <sound model='ich6'>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x04' function='0x0'/>
    </sound>
    <video>
      <model type='xen' vram='4096' heads='1'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x0'/>
    </video>
    <hostdev mode='subsystem' type='usb' managed='yes'>
      <source>
        <vendor id='0x06a3'/>
        <product id='0x8021'/>
      </source>
    </hostdev>
    <hostdev mode='subsystem' type='pci' managed='yes'>
      <driver name='vfio'/>
      <source>
        <address domain='0x0000' bus='0x02' slot='0x00' function='0x0'/>
      </source>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x08' function='0x0'/>
    </hostdev>
    <hostdev mode='subsystem' type='pci' managed='yes'>
      <driver name='vfio'/>
      <source>
        <address domain='0x0000' bus='0x02' slot='0x00' function='0x1'/>
      </source>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x0a' function='0x0'/>
    </hostdev>
    <hostdev mode='subsystem' type='usb' managed='yes'>
      <source>
        <vendor id='0x046d'/>
        <product id='0xc066'/>
      </source>
    </hostdev>
    <redirdev bus='usb' type='spicevmc'>
    </redirdev>
    <redirdev bus='usb' type='spicevmc'>
    </redirdev>
    <memballoon model='virtio'>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x07' function='0x0'/>
    </memballoon>
  </devices>
</domain>

/usr/libexec/qemu-kvm-nvidia-vga:

#!/bin/sh
exec /bin/qemu-kvm \
        `echo "$@" | sed 's|02:00.0|02:00.0,x-vga=on|g'`

Remaining problems:
1. Once the first VM shuts down, the subsequent booting doesn't work properly. Work-around: suspend the host OS to reset the GPU. There is an annoying problem with losing the background with the suspend/resume on the Nvidia cards. Alt-F2 then r to reset in Gnome.
2. I can't figure out how to pipe only the sound through spice. Work-around: use Xen for video card type.

Last edited by pkim (2014-12-29 12:26:59)