Arch Linux

paigeadele

Member

Registered: 2014-07-17

Posts: 3

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

I got mine working on gentoo, a friend of mine helped me with some performance issues as well. Here all some links to the scripts I use to start the windows VM:

https://github.com/paigeadele/windowlic … windows.sh

I added the (-cpu host,hv_relaxed,hv_vapic,hv_time,hv_spinlocks=0x1000) options a friend of mine said that they help, but I haven't benchmarked it yet to see if there's any difference. I'm using an ATI Radeon 5770 which I may upgrade now that I've at least seen this work. Initially I had configured the cpu topology to use two sockets, 2 cores each and decided to reduce it to 1 and 6 cores in case using 2 sockets might have been adding any contention...was mostly just curious to see how it worked. I haven't found much good documentation on the hv_spinlocks and threads=1 stuff, I'm curious to see how the impact the responsiveness. I also figured out how to setup macvtap which I currently use for all of my libvirtd instances.

https://github.com/paigeadele/windowlic … 5.5-gentoo

someone posted in this forum about enabling preemption helping (though I thought preemption didn't work on it's own.) I also set the ticks frequency to 1000hz and figured at least that option might make some difference.

Everything seems to be working fine, theres one issue with netflix when the playback menu pops up when you hover over it that makes the video lag a bit... and I'm not sure if thats the card being slow or processor bottleneck....lots to figure out but everything is awesome, thank you for this post!

paigeadele

Member

Registered: 2014-07-17

Posts: 3

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Interesting, removed the threads=1 to see if it would let it spin up however many threads it wants, seem to be a little snappier but hard to tell.

http://imgur.com/6W8Nbnl

Was having some trouble with HD video playback on netflix in firefox. It seems a little slow / choppy,... yeah still same.. I wonder why... silverlight / plugin container seems to eat up quite a bit of cpu in the vm

I guess I could unmask the 9999 qemu build and give it a try.

Last edited by paigeadele (2014-07-17 02:54:04)

aw

Member

Registered: 2013-10-04

Posts: 921

Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Interesting, removed the threads=1 to see if it would let it spin up however many threads it wants, seem to be a little snappier but hard to tell.

http://imgur.com/6W8Nbnl

Was having some trouble with HD video playback on netflix in firefox. It seems a little slow / choppy,... yeah still same.. I wonder why... silverlight / plugin container seems to eat up quite a bit of cpu in the vm

threads is just a modifier of the topology of the vCPU, it exposes processor threads, or hyper threads.  threads=1 is the default, each core has one thread.  Core i5/i7 processors often have 2 threads per core.  The total number of vCPUs exposed to the guest is "sockets x cores x threads".  The guest scheduler can make various optimizations given a topology, like optimizing for power efficiency by only running on a single socket, or optimizing for throughput by using cores within the same socket or only a single thread per core, etc.

---

http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

paigeadele

Member

Registered: 2014-07-17

Posts: 3

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

gotcha, yeah I'm building the qemu 9999 package now I'll see if there's much of a performance change since 2.0

mbroemme

Member

From: Cologne

Registered: 2014-02-05

Posts: 40

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

I've tested them against 3.16-rc5 kernel. They apply clean, I've rebooted and re-tested my Linux + fglrx and Windows + catalyst with my R9 290X. It still suffers from the reset problem. On Linux + fglrx I get oops (inside VM) after reboot of VM and X start. On Windows + catalyst I still get the BSOD PAGE_FAULT_IN_NONPAGED_AREA.

Bummer.  Is the oops a regression or is that something you see already?

It is the same.

DLWood1001

Member

Registered: 2014-07-17

Posts: 4

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

aw,

I tested the reset patch against 3.15.5 mainline. Patches applied with no merge problems, however it didn't reset the card. Using Sapphire R9 290x Tri-X. I'll try building 3.16-rc5 tonight and reply back if there is any improvement.

On a side note, your dma-alias-v4 patch series worked great for me; fixed the Intel 82801 issue I was seeing.

aw

Member

Registered: 2013-10-04

Posts: 921

Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

aw,

I tested the reset patch against 3.15.5 mainline. Patches applied with no merge problems, however it didn't reset the card. Using Sapphire R9 290x Tri-X. I'll try building 3.16-rc5 tonight and reply back if there is any improvement.

Thanks for the test, both series weren't really targeted at fixing this, but I needed to add a device specific reset for my Oland-based HD8570 to get it to reset on release and I thought it worth a test to see if it fixed anything.  If you kill the qemu process with these patches, what happens on your r9 290x?  Does the monitor stay in sync and the framebuffer show the last image, or does the monitor go blank and the GPU fan reset to high speed?

On a side note, your dma-alias-v4 patch series worked great for me; fixed the Intel 82801 issue I was seeing.

Great, these should show up in 3.17

---

http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

shrubuntu

Member

Registered: 2012-12-26

Posts: 19

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Anyone have success with the GTX 580?

noctavian

Member

Registered: 2013-07-11

Posts: 17

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Anyone have success with the GTX 580?

Take a look at this
https://bbs.archlinux.org/viewtopic.php … 8#p1371808

And there is also this
https://docs.google.com/spreadsheet/ccc … _web#gid=0

BitMaster

Member

Registered: 2011-01-15

Posts: 4

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

I'm using this setup:

• Asus CS-B with Q87 chipset (BIOS: 1002)

• Intel i7-4790k (with VT-d support)

• Host GPU: Onboard Intel 4600, i915 driver, primary display adapter

• Guest GPU: Asus/Nvidia GeForce GTX 780

• Booting with UEFI (gummiboot)

First I applied all the OP patches (acs, i915 and vga arbiter) and this worked more or less. I got it booting with display on the Nvidia card, but it breaks the intel driver on the host. X switches to software rendering and glxinfo indicates it no longer uses Intel/Mesa OpenGL but instead VMware (?):

$ glxinfo | grep OpenGL
OpenGL vendor string: VMware, Inc.
OpenGL renderer string: Gallium 0.4 on llvmpipe (LLVM 3.4, 256 bits)
OpenGL version string: 3.0 Mesa 10.2.3
OpenGL shading language version string: 1.30
OpenGL context flags: (none)
OpenGL extensions:

Next i tried to minimize the amount of patches (in an effort to pinpoint the intel host driver issue). These are the remaining patches i need to have a working VGA passthrough: i915-vtd.patch.
If i disable the

i915_disable_vga_mem(dev);

line in 'drivers/gpu/drm/i915/i915_dma.c' X is in normal mode again and Intel/Mesa OpenGL is used, but sadly VGA passthrough is broken. This means the display goes to sleep mode when the virtual machine is started.

I scanned the driver code, but I can't find the line that enables the vga memory again (only on 'deinit/cleanup'?). Is it impossible to use the onboard VGA memory in combination with VGA passthrough?
Any suggestions are very welcome!

----
Oh, by the way i disabled/blacklisted the nouveau module. My kernel parameters are nothing special:

$ cat /proc/cmdline
initrd=\initramfs-linux-vtd2.img root=PARTUUID=1b8f88d4-3bb2-4eae-abe8-180d0300f908 rw nouveau.modeset=0 quiet intel_iommu=on pci-stub.ids=10de:1004,10de:0e1a drm.debug=0x02

I don't need to use pci-stub, as my graphics card is not bound to any driver on boot. Before i start the VM, I bind them to vfio (see OP). I use qemu-git from the AUR.

Last edited by BitMaster (2014-07-19 14:58:29)

DanaGoyette

Member

Registered: 2014-01-03

Posts: 46

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

I've been lurking for a while, since the inability to bus-reset the R9 290 is kind of a show-stopper.  I couldn't find an official bug report to follow for it, and the thread I found on the mailing lists has stopped.

I hope OVMF will get support for the Q35 machine soon.  With a UEFI video card and a legacy-free (Windows 8.1) guest, you don't need the Intel VGA Arbiter.

I've noticed that the instructions in the OP, as well as virt-manager, put all passed-through devices directly behind "root port 0". 
Real boards have one device (which may be a switch) behind each port, and each port (numbered starting at 1, not 0) has a different device ID.
I can't see how multiple devices behind one port has ever worked, and it wouldn't surprise me if drivers don't like it.

When I tried Q35 in non-EFI mode, I used this file to give multiple root ports: https://github.com/qemu/qemu/blob/maste … hipset.cfg

(Interestingly enough, the '-m q35' default AHCI controller is named 'ide'; you attach disks to bus ide.0, ide.1, and so on.)

Now, for a question: do you think it would be reasonable to start factoring device types into the building of IOMMU groups?

For example, given multiple video cards behind the Intel CPU ports, they are highly likely to converse peer-to-peer.
Given an LSI SAS controller and an AMD video card, on the other hand, what are the chances of one trying to talk to the other?

Last edited by DanaGoyette (2014-07-19 22:01:42)

k0olfir3

Member

Registered: 2014-07-20

Posts: 7

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Hi all,

this thread is amazing. However, I have to admit that I didn't read all the 94 pages.

I tried to replicate the OP's procedures. Unfortunately, I have not been very succesfull so far. Whenever I test passing through the Video card my monitor stays blank. I have an Intel i7 4790 CPU on an AsRock Z87 Pro 4 board and a NVIDIA GTX 770 card. I use the OP's mainline kernel and git 2.0 from the standard Arch repositories. Here are some further details on my setup:

uname -a
Linux arch-desk 3.15.1-1-mainline #1 SMP PREEMPT Sun Jul 20 00:50:27 CEST 2014 x86_64 GNU/Linux

cat /proc/cmdline
BOOT_IMAGE=/boot/vmlinuz-linux-mainline root=UUID=xxx quiet iommu=1 intel_iommu=on pci-stub.ids=10de:1184,10de:0e0a

I noticed the following errors in dmesg when I run qemu without passing some romfile option:

[ 5716.098747] [drm:hsw_unclaimed_reg_clear] *ERROR* Unknown unclaimed register before writing to c400c
[ 5716.098802] [drm:intel_uncore_check_errors] *ERROR* Unclaimed register before interrupt
[ 5716.098814] [drm:intel_uncore_check_errors] *ERROR* Unclaimed register before interrupt
[ 5716.098827] [drm:intel_uncore_check_errors] *ERROR* Unclaimed register before interrupt

These errors vanish if I use a vga-bios rom file I downloaded from: http://www.techpowerup.com/vgabios/1454 … 30704.html. Both my screen is staying blank in either case. I tried the OP's suggestions:

vfio_iommu_type1.allow_unsafe_interrupts=1 and
kvm_intel emulate_invalid_guest_state=0

But that did not appear to change very much. Do you guys have any suggestions what I could try next? Am I missing something?

EDIT: I forgot to mention that I want to use the internal Intel GPU on the host system.

Last edited by k0olfir3 (2014-07-20 01:25:21)

AKSN74

Member

From: Taichung, Taiwan

Registered: 2014-02-18

Posts: 52

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Hey folks,

I've got some reset tweaks up for testing that might be useful to a number of you:

https://lkml.org/lkml/2014/7/16/562 (patch 1, patch 2, patch 3)

https://lkml.org/lkml/2014/7/16/567 (patch)

With these, the GPU should be reset regardless of how clean the guest was shutdown.  Note that all devices on the bus need to be bound to vfio-pci for this to work.  That means that if you're using pci-stub to sequester unused devices (perhaps the audio function), bind them to vfio-pci or there won't be any change.  The second set of links is a patch necessary for AMD GPU users only and I'd appreciate feedback on whether it solves and of the other reset issues (lack of reset) found on these GPUs.  Thanks

PS - these are kernel patches against the latest development kernel, 3.16-rc5.  They may or may not apply cleanly to older kernels.

Thanks for aw's patch, I'll try this to solve VM reboot stuck issue in these days. And I'll report.

Also, I got another goods news is, now I can run CUDA perfectly in VM right now.

I found that I need to do this with GUI-based OS, so if I tried to run CUDA when the OS is terminal only (like Ubuntu Server), I need to launch GPU acceleration in text mode (for Linux, it means like "init 3")
So I give up to try Ubuntu Server and change guest OS to Fedora 20, and when I finished the installation and entered into system. I checked host OS's dmesg.
And I found that IRQ is already set up while GUI interface launched.

vfio-pci: 0000:03:00.0: irq 89 for MSI/MSI-X

Then I install NVIDIA driver, CUDA SDK, and compile CUDA sample codes. After all, I run some sample application. It works perfectly! Now I'm not get any strange errors.
Also I checked host dmesg each time I run CUDA application, and VFIO not reported IRQ changed message anymore. Just showed while Fedora boot.

But I still need to turn off GUI to get a better performance, fortunately someone knows how to get GPU acceleration with NVIDIA official driver in text mode.
I'll try it up and report that.

nbhs

Member

From: Montevideo, Uruguay

Registered: 2013-05-02

Posts: 402

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Hi all,

this thread is amazing. However, I have to admit that I didn't read all the 94 pages.

I tried to replicate the OP's procedures. Unfortunately, I have not been very succesfull so far. Whenever I test passing through the Video card my monitor stays blank. I have an Intel i7 4790 CPU on an AsRock Z87 Pro 4 board and a NVIDIA GTX 770 card. I use the OP's mainline kernel and git 2.0 from the standard Arch repositories. Here are some further details on my setup:

uname -a
Linux arch-desk 3.15.1-1-mainline #1 SMP PREEMPT Sun Jul 20 00:50:27 CEST 2014 x86_64 GNU/Linux

cat /proc/cmdline
BOOT_IMAGE=/boot/vmlinuz-linux-mainline root=UUID=xxx quiet iommu=1 intel_iommu=on pci-stub.ids=10de:1184,10de:0e0a

I noticed the following errors in dmesg when I run qemu without passing some romfile option:

[ 5716.098747] [drm:hsw_unclaimed_reg_clear] *ERROR* Unknown unclaimed register before writing to c400c
[ 5716.098802] [drm:intel_uncore_check_errors] *ERROR* Unclaimed register before interrupt
[ 5716.098814] [drm:intel_uncore_check_errors] *ERROR* Unclaimed register before interrupt
[ 5716.098827] [drm:intel_uncore_check_errors] *ERROR* Unclaimed register before interrupt

These errors vanish if I use a vga-bios rom file I downloaded from: http://www.techpowerup.com/vgabios/1454 … 30704.html. Both my screen is staying blank in either case. I tried the OP's suggestions:

vfio_iommu_type1.allow_unsafe_interrupts=1 and
kvm_intel emulate_invalid_guest_state=0

But that did not appear to change very much. Do you guys have any suggestions what I could try next? Am I missing something?

EDIT: I forgot to mention that I want to use the internal Intel GPU on the host system.

You need to boot with i915.enable_hd_vgaarb=1 kernel parameter, this will disable dri on the host though.

_pheinrich_

Member

Registered: 2014-05-26

Posts: 53

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Hi,

all is working like expected now!

Config:

MB: Asus P8P67EVO
CPU: i5 3470
Host GPU: nvidia 6600GT
Guest GPU: nvidia GTX 260

Kernel 3.15.5-2 with acs patch
Nvidia Kernel module with http://bpaste.net/show/109185/

BOOT_IMAGE=/boot/vmlinuz-linux-acs root=UUID=3264d3aa-f452-4866-9397-644efdb107a1 rw intel_iommu=on pcie_acs_override=downstream pci-stub.ids=10de:05e2

I want to thank you all for your support! Keep alive this thread.

k0olfir3

Member

Registered: 2014-07-20

Posts: 7

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

You need to boot with i915.enable_hd_vgaarb=1 kernel parameter, this will disable dri on the host though.

That did the trick.

When I just returned to your original post for the further proceedings, I saw that you you mentioned that option there. Did you just put it there or did I really miss it? That would be kind of embarassing. Anyway, thanks for your help.

redger

Member

Registered: 2014-02-26

Posts: 15

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

USING VIRT-MANAGER

This may help someone - after much investigation, trial & error and general messing around I have managed to get VGA passthroufgh working with virt-manager / libvirt  (on Ubuntu Trusty)

you will need to install the virt-manager, libvirt first.
The process assumes you already have a working qemu command and you have saved it to a file

1) Create the standard qemu command and save it into a text file
      Remove all the line feed and backslash entries first (they seem to cause virsh some grief) ie. one long line of text
      NOTE will probably have to add some backslash and linefeeds back in (needed for cd-rom defs at end in my case)
2) use -->  virsh domxml-from-native qemu-argv qemu_command_temp.txt > my_qemu_start.xml
      THis will create a new file (my_qemu_start.xml) which can be imported
      NOTE you may have to add some backslash and linefeeds back in (I needed them for cd defs at end, but then removed themmanually in final machine definition)
3) use --> virsh define my_qemu_start.xml            which will create the actual machine xml (file name will be the machine name specified, plus .xml suffix)
4) use --> export EDITOR=nano; virsh edit machine.xml   ...
     change the boot source to 'hd'   from 'network'   (why virsh creates this I don't know)
     check the emulator line, may need to specify location of qemu eg. in my case /usr/bin/qemu-system-x86_64
     remove the video lines in the definition (because we want primary passthru - no cirrus)
     remove the sdl line  (video type) cause  we don't want it either
     remove the "tablet" definition - it fails to conenct to USB device, along with keyboard and mouse
     remove the "\" from the cd lines (had to be added in previous step)
5) edit /etc/libvirt/qemu.conf and set

     user = "+0"
     group = "+0" 

     
     add /dev/vfio/xxxxx (in my case 1, 15, 16 17)     entries to cgroup_device_acl section

      cgroup_device_acl = [
      "/dev/null", "/dev/full", "/dev/zero",
      "/dev/random", "/dev/urandom",
      "/dev/ptmx", "/dev/kvm", "/dev/kqemu",
      "/dev/rtc","/dev/hpet", "/dev/vfio/vfio",
      "/dev/vfio/1", "/dev/vfio/15", "/dev/vfio/16", "/dev/vfio/17"
      ]

     security_driver = "none"
     security_default_confined = 0
     hugetlbfs_mount = "/dev/hugepages"
     clear_emulator_capabilities = 0
     relaxed_acs_check = 1 

6) reboot to instantiate the new libvirt parameters
7) start new machine from virt-manager ......

NOTE if "open" the machine (initiate virt-viewer) then some new entries may be added to the machine definition which will stop it working. If this does happen use "virsh edit" and remove the gratuitous video, tablet, keyboard, mouse etc ... and check the updated qemu:arg entries, in my case backspaces were added which stopped the machine starting. It looked as though the originally generated definition was being restored ie. pre the edits to "fix" the generation errors.

I started with this

 qemu-system-x86_64 -name win7 -enable-kvm -M q35 -m 6144  -mem-path /dev/hugepages \
-cpu Haswell,hv-time,hv_relaxed,hv_vapic,hv_spinlocks=0x1000 \
-boot menu=on \
-smp 3,sockets=1,cores=3,threads=1 \
-bios /usr/share/qemu/bios.bin -vga none \
-usbdevice tablet \
-spice port=5902,disable-ticketing \
-device ioh3420,bus=pcie.0,addr=1c.0,multifunction=on,port=1,chassis=1,id=root.1 \
-device ahci,bus=pcie.0,id=ahci \
-net nic,macaddr=00:00:00:aa:bb:cc,model=virtio \
-net user \
-device vfio-pci,host=01:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on \
-device vfio-pci,host=01:00.1,bus=root.1,addr=00.1 \
-device vfio-pci,host=02:00.0,bus=pcie.0 \
-device vfio-pci,host=04:00.0,bus=pcie.0 \
-device vfio-pci,host=06:00.0,bus=pcie.0 \
-device vfio-pci,host=05:00.0,bus=pcie.0 \
-drive file=/dev/virt-test/lvwin7_kvm,id=disk1,format=raw,if=virtio \
-drive file=/home/redger/Downloads/isos/virtio-win-0.1-74.iso,id=isocd2 \
-device ide-cd,bus=ahci.2,drive=isocd2 \
-drive file=/home/redger/Downloads/isos/winfiles.iso,id=isocd3 \
-device ide-cd,bus=ahci.3,drive=isocd3 

I edited it and made it into this

qemu-system-x86_64 -name win7_t2 -enable-kvm -M q35 -m 6144  -mem-path /dev/hugepages -cpu Haswell,hv-time,hv_relaxed,hv_vapic,hv_spinlocks=0x1000 -boot menu=on -smp 3,sockets=1,cores=3,threads=1 -bios /usr/share/qemu/bios.bin -vga none -usbdevice tablet -spice port=5902,disable-ticketing -device ioh3420,bus=pcie.0,addr=1c.0,multifunction=on,port=1,chassis=1,id=root.1 -device ahci,bus=pcie.0,id=ahci -net nic,macaddr=00:00:00:aa:bb:cc,model=virtio -net user -device vfio-pci,host=01:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on -device vfio-pci,host=01:00.1,bus=root.1,addr=00.1 -device vfio-pci,host=02:00.0,bus=pcie.0 -device vfio-pci,host=04:00.0,bus=pcie.0 -device vfio-pci,host=06:00.0,bus=pcie.0 -device vfio-pci,host=05:00.0,bus=pcie.0 -drive file=/dev/virt-test/lvwin7_kvm,id=disk1,format=raw,if=virtio \
-drive file=/home/redger/Downloads/isos/virtio-win-0.1-74.iso,id=isocd2 \
-device ide-cd,bus=ahci.2,drive=isocd2 \
-drive file=/home/redger/Downloads/isos/winfiles.iso,id=isocd3 \
-device ide-cd,bus=ahci.3,drive=isocd3

note that I had to restore some of the backslashes and linefeeds because the machine was giving some strange errors like

Error starting domain: internal error: process exited while connecting to monitor: 2014-07-20T12:09:56.863675Z qemu-system-x86_64: -device vfio-pci,host=05:00.0,bus=pcie.0: could not open disk image \ -drive: Could not open '\ -drive': No such file or directory

Other errors related to permissions and use of hugepages were addressed by the changes to /etc/libvirt/qemu.conf

that became this once virsh domxml-from-native qemu-argv was run 

<domain type='kvm' xmlns:qemu='http://libvirt.org/schemas/domain/qemu/1.0'>
  <name>win7_t2</name>
  <uuid>f778660a-9da7-4bad-ba5b-477feb74f2b9</uuid>
  <memory unit='KiB'>6291456</memory>
  <currentMemory unit='KiB'>6291456</currentMemory>
  <vcpu placement='static'>3</vcpu>
  <os>
    <type arch='x86_64' machine='q35'>hvm</type>
    <loader>/usr/share/qemu/bios.bin</loader>
    <boot dev='network'/>
    <boot dev='network'/>
  </os>
  <features>
    <acpi/>
    <hyperv>
      <relaxed state='on'/>
      <vapic state='on'/>
      <spinlocks state='on' retries='4096'/>
    </hyperv>
  </features>
  <cpu mode='custom' match='exact'>
    <model fallback='allow'>Haswell</model>
    <topology sockets='1' cores='3' threads='1'/>
  </cpu>
  <clock offset='utc'/>
  <on_poweroff>destroy</on_poweroff>
  <on_reboot>restart</on_reboot>
  <on_crash>destroy</on_crash>
  <devices>
    <emulator>/usr/bin/qemu-system-x86_64</emulator>
    <disk type='block' device='disk'>
      <driver name='qemu' type='raw'/>
      <source dev='/dev/virt-test/lvwin7_kvm'/>
      <target dev='vda' bus='virtio'/>
    </disk>
    <controller type='sata' index='0'/>
    <controller type='pci' index='0' model='pcie-root'/>
    <controller type='pci' index='1' model='dmi-to-pci-bridge'/>
    <controller type='pci' index='2' model='pci-bridge'/>
    <interface type='user'>
      <mac address='00:00:00:aa:bb:cc'/>
      <model type='virtio'/>
    </interface>
    <input type='tablet' bus='usb'/>
    <input type='mouse' bus='ps2'/>
    <input type='keyboard' bus='ps2'/>
    <graphics type='sdl'/>
    <video>
      <model type='cirrus' vram='9216' heads='1'/>
    </video>
    <memballoon model='none'/>
  </devices>
  <qemu:commandline>
    <qemu:arg value='-mem-path'/>
    <qemu:arg value='/dev/hugepages'/>
    <qemu:arg value='-spice'/>
    <qemu:arg value='port=5902,disable-ticketing'/>
    <qemu:arg value='-device'/>
    <qemu:arg value='ioh3420,bus=pcie.0,addr=1c.0,multifunction=on,port=1,chassis=1,id=root.1'/>
    <qemu:arg value='-device'/>
    <qemu:arg value='ahci,bus=pcie.0,id=ahci'/>
    <qemu:arg value='-device'/>
    <qemu:arg value='vfio-pci,host=01:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on'/>
    <qemu:arg value='-device'/>
    <qemu:arg value='vfio-pci,host=01:00.1,bus=root.1,addr=00.1'/>
    <qemu:arg value='-device'/>
    <qemu:arg value='vfio-pci,host=02:00.0,bus=pcie.0'/>
    <qemu:arg value='-device'/>
    <qemu:arg value='vfio-pci,host=04:00.0,bus=pcie.0'/>
    <qemu:arg value='-device'/>
    <qemu:arg value='vfio-pci,host=06:00.0,bus=pcie.0'/>
    <qemu:arg value='-device'/>
    <qemu:arg value='vfio-pci,host=05:00.0,bus=pcie.0'/>
    <qemu:arg value='\
-drive'/>
    <qemu:arg value='file=/home/redger/Downloads/isos/virtio-win-0.1-74.iso,id=isocd2'/>
    <qemu:arg value='\
-device'/>
    <qemu:arg value='ide-cd,bus=ahci.2,drive=isocd2'/>
    <qemu:arg value='\
-drive'/>
    <qemu:arg value='file=/home/redger/Downloads/isos/winfiles.iso,id=isocd3'/>
    <qemu:arg value='\
-device'/>
    <qemu:arg value='ide-cd,bus=ahci.3,drive=isocd3'/>
  </qemu:commandline>
</domain>

and then became this after running virsh define my_qemu_start.xml and then using virsh edit to remove the unnecessary lines (and change the boot from network to hd)

<!--
WARNING: THIS IS AN AUTO-GENERATED FILE. CHANGES TO IT ARE LIKELY TO BE
OVERWRITTEN AND LOST. Changes to this xml configuration should be made using:
  virsh edit win7_t2
or other application using the libvirt API.
-->

<domain type='kvm' xmlns:qemu='http://libvirt.org/schemas/domain/qemu/1.0'>
  <name>win7_t2</name>
  <uuid>f778660a-9da7-4bad-ba5b-477feb74f2b9</uuid>
  <memory unit='KiB'>6291456</memory>
  <currentMemory unit='KiB'>6291456</currentMemory>
  <vcpu placement='static'>3</vcpu>
  <os>
    <type arch='x86_64' machine='pc-q35-2.0'>hvm</type>
    <loader>/usr/share/qemu/bios.bin</loader>
    <boot dev='hd'/>
  </os>
  <features>
    <acpi/>
    <hyperv>
      <relaxed state='on'/>
      <vapic state='on'/>
      <spinlocks state='on' retries='4096'/>
    </hyperv>
  </features>
  <cpu mode='custom' match='exact'>
    <model fallback='allow'>Haswell</model>
    <topology sockets='1' cores='3' threads='1'/>
  </cpu>
  <clock offset='utc'/>
  <on_poweroff>destroy</on_poweroff>
  <on_reboot>restart</on_reboot>
  <on_crash>destroy</on_crash>
  <devices>
    <emulator>/usr/bin/qemu-system-x86_64</emulator>
    <disk type='block' device='disk'>
      <driver name='qemu' type='raw'/>
      <source dev='/dev/virt-test/lvwin7_kvm'/>
      <target dev='vda' bus='virtio'/>
      <address type='pci' domain='0x0000' bus='0x02' slot='0x02' function='0x0'/>
    </disk>
    <controller type='sata' index='0'>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x1f' function='0x2'/>
    </controller>
    <controller type='pci' index='0' model='pcie-root'/>
    <controller type='pci' index='1' model='dmi-to-pci-bridge'>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x1e' function='0x0'/>
    </controller>
    <controller type='pci' index='2' model='pci-bridge'>
      <address type='pci' domain='0x0000' bus='0x01' slot='0x01' function='0x0'/>
    </controller>
    <interface type='user'>
      <mac address='00:00:00:aa:bb:cc'/>
      <model type='virtio'/>
      <address type='pci' domain='0x0000' bus='0x02' slot='0x01' function='0x0'/>
    </interface>
    <memballoon model='none'/>
  </devices>
  <qemu:commandline>
    <qemu:arg value='-mem-path'/>
    <qemu:arg value='/dev/hugepages'/>
    <qemu:arg value='-spice'/>
    <qemu:arg value='port=5902,disable-ticketing'/>
    <qemu:arg value='-device'/>
    <qemu:arg value='ioh3420,bus=pcie.0,addr=1c.0,multifunction=on,port=1,chassis=1,id=root.1'/>
    <qemu:arg value='-device'/>
    <qemu:arg value='ahci,bus=pcie.0,id=ahci'/>
    <qemu:arg value='-device'/>
    <qemu:arg value='vfio-pci,host=01:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on'/>
    <qemu:arg value='-device'/>
    <qemu:arg value='vfio-pci,host=01:00.1,bus=root.1,addr=00.1'/>
    <qemu:arg value='-device'/>
    <qemu:arg value='vfio-pci,host=02:00.0,bus=pcie.0'/>
    <qemu:arg value='-device'/>
    <qemu:arg value='vfio-pci,host=04:00.0,bus=pcie.0'/>
    <qemu:arg value='-device'/>
    <qemu:arg value='vfio-pci,host=06:00.0,bus=pcie.0'/>
    <qemu:arg value='-device'/>
    <qemu:arg value='vfio-pci,host=05:00.0,bus=pcie.0'/>
    <qemu:arg value='-drive'/>
    <qemu:arg value='file=/home/redger/Downloads/isos/virtio-win-0.1-74.iso,id=isocd2'/>
    <qemu:arg value='-device'/>
    <qemu:arg value='ide-cd,bus=ahci.2,drive=isocd2'/>
    <qemu:arg value='-drive'/>
    <qemu:arg value='file=/home/redger/Downloads/isos/winfiles.iso,id=isocd3'/>
    <qemu:arg value='-device'/>
    <qemu:arg value='ide-cd,bus=ahci.3,drive=isocd3'/>
  </qemu:commandline>
</domain>

which would run as machine win_t2
it was all quite painfull the first time since I tried to create a definition file from scratch - which was completely unsuccessful even though it looked a lot like the final product.

Comments / improvements welcome

nbhs

Member

From: Montevideo, Uruguay

Registered: 2013-05-02

Posts: 402

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

You need to boot with i915.enable_hd_vgaarb=1 kernel parameter, this will disable dri on the host though.

That did the trick.

When I just returned to your original post for the further proceedings, I saw that you you mentioned that option there. Did you just put it there or did I really miss it? That would be kind of embarassing. Anyway, thanks for your help.

It was a ninja edit

k0olfir3

Member

Registered: 2014-07-20

Posts: 7

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Hi all,

I was now able to pass through my GTX 770 card and to install win 7 on the virtual machine. However, in windows I cannot use the card with the NVIDIA drivers. I get that annoying yellow triangle with Code 43.

My research in this thread indicates that this problem is related to the NoSnoop attribute. At least it appears to be enabled on my system:

lspci -vvvv
01:00.0 VGA compatible controller: NVIDIA Corporation GK104 [GeForce GTX 770] (rev a1) (prog-if 00 [VGA controller])
 ...
        Capabilities: [78] Express (v2) Endpoint, MSI 00
                DevCap: MaxPayload 256 bytes, PhantFunc 0, Latency L0s unlimited, L1 <64us
                        ExtTag+ AttnBtn- AttnInd- PwrInd- RBE+ FLReset-
                DevCtl: Report errors: Correctable- Non-Fatal- Fatal- Unsupported-
                        RlxdOrd- ExtTag+ PhantFunc- AuxPwr- NoSnoop+
...

01:00.1 Audio device: NVIDIA Corporation GK104 HDMI Audio Controller (rev a1)
...
        Capabilities: [78] Express (v2) Endpoint, MSI 00
                DevCap: MaxPayload 256 bytes, PhantFunc 0, Latency L0s unlimited, L1 <64us
                        ExtTag+ AttnBtn- AttnInd- PwrInd- RBE+ FLReset-
                DevCtl: Report errors: Correctable- Non-Fatal- Fatal- Unsupported-
                        RlxdOrd- ExtTag+ PhantFunc- AuxPwr- NoSnoop+
...

In this forum aw posted a patch for qemu to solve this issue. However, if I understand this correctly that patch should already be included in qemu >=2.0.0.

On the web I found this site:
http://www.firewing1.com/howtos/fedora-20/create-gaming-virtual-machine-using-vfio-pci-passthrough-kvm
which is suggesting to the following kernel patches.

http://git.kernel.org/cgit/linux/kernel … 7612b7f235
http://git.kernel.org/cgit/linux/kernel … 860dae4d56
http://git.kernel.org/cgit/linux/kernel … a429732ef4

They appear to older than 3.15.1, should I try them?

If yes, can I just edit nbhs's PKGBUILD such that these patches are apllied to linux-mainline? For example by adding something like this to the prepare section?

patch -p1 -i "${srcdir}/whatever_file_name.patch"

Assuming this works out, will I need to pass any additional options to the kernel to enable the patches (as was the case for the i915 patch)?

Or do you have any other ideas that may solve my problem?

Sorry for posing all those stupid questions. I don't know very much about programming, but I really want that gaming virtual machine.

nbhs

Member

From: Montevideo, Uruguay

Registered: 2013-05-02

Posts: 402

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Hi all,

I was now able to pass through my GTX 770 card and to install win 7 on the virtual machine. However, in windows I cannot use the card with the NVIDIA drivers. I get that annoying yellow triangle with Code 43.

My research in this thread indicates that this problem is related to the NoSnoop attribute. At least it appears to be enabled on my system:

lspci -vvvv
01:00.0 VGA compatible controller: NVIDIA Corporation GK104 [GeForce GTX 770] (rev a1) (prog-if 00 [VGA controller])
 ...
        Capabilities: [78] Express (v2) Endpoint, MSI 00
                DevCap: MaxPayload 256 bytes, PhantFunc 0, Latency L0s unlimited, L1 <64us
                        ExtTag+ AttnBtn- AttnInd- PwrInd- RBE+ FLReset-
                DevCtl: Report errors: Correctable- Non-Fatal- Fatal- Unsupported-
                        RlxdOrd- ExtTag+ PhantFunc- AuxPwr- NoSnoop+
...

01:00.1 Audio device: NVIDIA Corporation GK104 HDMI Audio Controller (rev a1)
...
        Capabilities: [78] Express (v2) Endpoint, MSI 00
                DevCap: MaxPayload 256 bytes, PhantFunc 0, Latency L0s unlimited, L1 <64us
                        ExtTag+ AttnBtn- AttnInd- PwrInd- RBE+ FLReset-
                DevCtl: Report errors: Correctable- Non-Fatal- Fatal- Unsupported-
                        RlxdOrd- ExtTag+ PhantFunc- AuxPwr- NoSnoop+
...

In this forum aw posted a patch for qemu to solve this issue. However, if I understand this correctly that patch should already be included in qemu >=2.0.0.

On the web I found this site:
http://www.firewing1.com/howtos/fedora-20/create-gaming-virtual-machine-using-vfio-pci-passthrough-kvm
which is suggesting to the following kernel patches.

http://git.kernel.org/cgit/linux/kernel … 7612b7f235
http://git.kernel.org/cgit/linux/kernel … 860dae4d56
http://git.kernel.org/cgit/linux/kernel … a429732ef4

They appear to older than 3.15.1, should I try them?

If yes, can I just edit nbhs's PKGBUILD such that these patches are apllied to linux-mainline? For example by adding something like this to the prepare section?

patch -p1 -i "${srcdir}/whatever_file_name.patch"

Assuming this works out, will I need to pass any additional options to the kernel to enable the patches (as was the case for the i915 patch)?

Or do you have any other ideas that may solve my problem?

Sorry for posing all those stupid questions. I don't know very much about programming, but I really want that gaming virtual machine.

I believe there are some problems with the lastest nvidia drivers, if you are using the lastest qemu git, you can also try kvm=off as a cpu option, theres a discussion about this a few pages back

Last edited by nbhs (2014-07-20 16:42:45)

k0olfir3

Member

Registered: 2014-07-20

Posts: 7

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

I believe there are some problems with the lastest nvidia drivers, if you are using the lastest qemu git, you can also try kvm=off as a cpu option, theres a discussion about this a few pages back

Great! You always come up with easy solutions when I think everything is terribly complicated. I installed an earlier version of the NVIDIA drivers (332.21) in the virtual machine and now, the graphics are working. Thanks!

DLWood1001

Member

Registered: 2014-07-17

Posts: 4

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

aw,

I tested the reset patch against 3.15.5 mainline. Patches applied with no merge problems, however it didn't reset the card. Using Sapphire R9 290x Tri-X. I'll try building 3.16-rc5 tonight and reply back if there is any improvement.

Thanks for the test, both series weren't really targeted at fixing this, but I needed to add a device specific reset for my Oland-based HD8570 to get it to reset on release and I thought it worth a test to see if it fixed anything.  If you kill the qemu process with these patches, what happens on your r9 290x?  Does the monitor stay in sync and the framebuffer show the last image, or does the monitor go blank and the GPU fan reset to high speed?

On a side note, your dma-alias-v4 patch series worked great for me; fixed the Intel 82801 issue I was seeing.

Great, these should show up in 3.17

Hi Aw,

Sorry about the delay in my response. I tested with the 3.16 rc5 kernel, reset patch applied and with a R9 290X. Still the same result. As an A/B test, I tried an HD7770 it seems to reset just fine with OSX as the guest, however I'm not sure if it had a problem pre-patch as I had never used the 7770 for VGA-Passthrough. I'm going to try an unpatched kernel with the HD 7770 later to see if it was truly the patch that allows this card to reset.

Lastly, in regards to the monitor output on the R9 290x, the output will go out of sync and I don't see/hear anything noticeable with the GPU fans. I've tried force closing QEMU and shutting down the guest gracefully and get the same result.

Last edited by DLWood1001 (2014-07-21 11:40:54)

AKSN74

Member

From: Taichung, Taiwan

Registered: 2014-02-18

Posts: 52

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

aw,

I tested the reset patch against 3.15.5 mainline. Patches applied with no merge problems, however it didn't reset the card. Using Sapphire R9 290x Tri-X. I'll try building 3.16-rc5 tonight and reply back if there is any improvement.

Thanks for the test, both series weren't really targeted at fixing this, but I needed to add a device specific reset for my Oland-based HD8570 to get it to reset on release and I thought it worth a test to see if it fixed anything.  If you kill the qemu process with these patches, what happens on your r9 290x?  Does the monitor stay in sync and the framebuffer show the last image, or does the monitor go blank and the GPU fan reset to high speed?

On a side note, your dma-alias-v4 patch series worked great for me; fixed the Intel 82801 issue I was seeing.

Great, these should show up in 3.17

Hi, aw
I tested the patch, but unluckly it still got hangs when I reboot guest OS (Fedora 20 /w GTX480).
And I'm sure that I patched with kernel 3.16-rc5.

Last edited by AKSN74 (2014-07-21 03:42:19)

ctwdoecoss

Member

Registered: 2014-07-20

Posts: 3

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Did anyone passthrough gpu successfully on a laptop?

apex8

Member

Registered: 2014-03-29

Posts: 60

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Hi,
I had a working Win7 VM using linux-mainline-3.14.1, along with stable qemu 2.0 as well as seabios.
Yesterday I compiled the mainline 3.15.6 package and since that the VM hangs somewhere before boot begins. No output at the monitor.
However using vga=on -vga std the VM boots up as usual.

Is there something that changed in the meantime and whichs needs to be adjusted?