Arch Linux

erganzi

Member

Registered: 2014-07-09

Posts: 19

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Hi, aw:

I just change the kernel version to 3.18.6 from kernel.org. It now work for me.  qemu version is still 2.0.

I compare the result of the command "lspci -vvv -k -s 04:00.0" between linux-3.12.5 and linux-3.18.6, there are just a little different.
If it can give you some idea.

There's really not much question about what the problem is/was, the kvm-vfio device didn't go in until 3.13.  That's the code that toggles whether kvm emulates or ignores operations that invalidate the cache when devices are allowed to perform operations that can bypass it, for example NoSnoop.  That was an early cause of Code 43 errors.  The NoSnoop patch you were using was an early, and fairly broken, attempt to fix that.  The new code is far more reliable, as you can see.

OK, thanks for reply. I will use the version 3.18.6.

Hi, aw:
I have another question that usb device(mouse and keyboard) cannot passthrough

I direct run: 
qemu-system-x86_64 -usb -usbdevice host:17ef:6018

the error is :
qemu: could not add USB device 'host:17ef:6018'

there is no error in /var/log/message

the secondary question is :
I can successful passthrough AMD FirePro R5000, and it worked well. but if I passthrough the r5000 onece, the integrated graphics on host had no responce and the display of screen is messed.

but the Nvidia GRID K2 in same host is OK.

every test the host just plug one card.

Last edited by erganzi (2015-02-12 09:12:25)

Denso

Member

Registered: 2014-08-30

Posts: 179

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

It seems that Q35 is working just fine with ovmf-svn compiled 4 hours ago . Gaming at 3840 x 2160 resolution is excellent inside the VM .

Len

Member

Registered: 2015-01-06

Posts: 23

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Is there anyone who can help me (preferred life-session on IRC) with configuration this on:
Motherboard: GA-Z97X-Gaming 5
CPU: i5-4690K
Host: Intel HD4600 (Arch)
Quest: Gigabyte GTX 970 (Windows)

I spend on this too many hours, still can't figure out why it's not working (old post), still new, can't handle all things on arch. After getting this working, I'll create post for others...

Last edited by Len (2015-02-13 03:09:19)

erganzi

Member

Registered: 2014-07-09

Posts: 19

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

There's really not much question about what the problem is/was, the kvm-vfio device didn't go in until 3.13.  That's the code that toggles whether kvm emulates or ignores operations that invalidate the cache when devices are allowed to perform operations that can bypass it, for example NoSnoop.  That was an early cause of Code 43 errors.  The NoSnoop patch you were using was an early, and fairly broken, attempt to fix that.  The new code is far more reliable, as you can see.

OK, thanks for reply. I will use the version 3.18.6.

Hi, aw:
I have another question that usb device(mouse and keyboard) cannot passthrough

I direct run: 
qemu-system-x86_64 -usb -usbdevice host:17ef:6018

the error is :
qemu: could not add USB device 'host:17ef:6018'

there is no error in /var/log/message

the secondary question is :
I can successful passthrough AMD FirePro R5000, and it worked well. but if I passthrough the r5000 onece, the integrated graphics on host had no responce and the display of screen is messed.

but the Nvidia GRID K2 in same host is OK.

every test the host just plug one card.

sorry, the usb device can not passthrough is my fault.
the libusb library is too old, when I recompiled the new one.
it can passthrough well.

nbhs

Member

From: Montevideo, Uruguay

Registered: 2013-05-02

Posts: 402

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Is there anyone who can help me (preferred life-session on IRC) with configuration this on:
Motherboard: GA-Z97X-Gaming 5
CPU: i5-4690K
Host: Intel HD4600 (Arch)
Quest: Gigabyte GTX 970 (Windows)

I spend on this too many hours, still can't figure out why it's not working (old post), still new, can't handle all things on arch. After getting this working, I'll create post for others...

You're missing the i915 vga arbiter patch, if you're using arch and the kernel i provided make sure you're actually booting it and not arch's one, alternatively you could use OVMF and skip that problem completely

Last edited by nbhs (2015-02-13 04:21:47)

SchaduwBlink

Member

Registered: 2007-01-07

Posts: 59

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

It seems that Q35 is working just fine with ovmf-svn compiled 4 hours ago . Gaming at 3840 x 2160 resolution is excellent inside the VM .

Where did you get the latest ovmf? It's no longer on Jenkins repo for some reason. Or did you compile it by yourself and if so, could you direct me on how to do that and make it split vars?

Thanks.

Denso

Member

Registered: 2014-08-30

Posts: 179

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Where did you get the latest ovmf? It's no longer on Jenkins repo for some reason. Or did you compile it by yourself and if so, could you direct me on how to do that and make it split vars?

Thanks.

There is a package in AUR if you're using Arch :

yaourt -S ovmf-svn

You must have "yaourt" to use the above command .

This will pull the latest code , compile it and then ask you for a confirmation to install it .

It will be installed into :

/usr/share/ovmf/x64/ovmf_code_x64.bin

This is the main binary

AND :

/usr/share/ovmf/x64/ovmf_vars_x64.bin

This is the vars file .

Good luck !

Last edited by Denso (2015-02-13 10:28:05)

SchaduwBlink

Member

Registered: 2007-01-07

Posts: 59

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Where did you get the latest ovmf? It's no longer on Jenkins repo for some reason. Or did you compile it by yourself and if so, could you direct me on how to do that and make it split vars?

Thanks.

There is a package in AUR if you're using Arch :

yaourt -S ovmf-svn

You must have "yaourt" to use the above command .

This will pull the latest code , compile it and then ask you for a confirmation to install it .

It will be installed into :

/usr/share/ovmf/x64/ovmf_code_x64.bin

This is the main binary

AND :

/usr/share/ovmf/x64/ovmf_vars_x64.bin

This is the vars file .

Good luck !

Cheers for that. Forgot to look there. I've always been pulling the pre-compiled one from Jenkins RPM Repo and extracting it.

Denso

Member

Registered: 2014-08-30

Posts: 179

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Cheers for that. Forgot to look there. I've always been pulling the pre-compiled one from Jenkins RPM Repo and extracting it.

AUR is the main reason why I'm using Arch .

Best of luck to you on your virtualization journey !

Tim_J

Member

Registered: 2015-02-07

Posts: 4

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

When I run the vm now, the screen turns off. I still do not have graphics output. Please advise.

It seems that you have the exact same problem that I do, which I posted a few pages ago. My system behaves the same as yours. In my case, I have one GT 210 and two GTX 580s. When the VM is launched and the screen turns off, it uses maximum CPU on a single core but never starts.

I've tried all sorts of things: unplugging the primary monitor at boot, putting GRUB in text-only mode, changing the boot TTY, quiet mode, and all manner of boot options to both GRUB and the kernel. So far, nothing has helped. It seems that the only recourse for us is to reassign the primary adapter (unsupported in both of our cases) or physically move cards around. Since this requires a system drain and cutting new tubes in my case, I'm postponing this for a while to see if anybody can come up with any ideas.

Len

Member

Registered: 2015-01-06

Posts: 23

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Is there anyone who can help me (preferred life-session on IRC) with configuration this on:
Motherboard: GA-Z97X-Gaming 5
CPU: i5-4690K
Host: Intel HD4600 (Arch)
Quest: Gigabyte GTX 970 (Windows)

I spend on this too many hours, still can't figure out why it's not working (old post), still new, can't handle all things on arch. After getting this working, I'll create post for others...

You're missing the i915 vga arbiter patch, if you're using arch and the kernel i provided make sure you're actually booting it and not arch's one, alternatively you could use OVMF and skip that problem completely

Thank you for OVMF hint, it's now displaying to my second graphic card!
But I still can't get it to work on normal user,

vfio: error opening /dev/vfio/1: Permission denied.

I'm in libvirt and kvm groups, this isn't helping, don't know why...

>libvirt/qemu.conf
user = "root"
group ="kvm"
clear_emultaor_capabilities = 0

>libvirt/libvirtd.conf
unix_sock_group = "libvirt"
unix_sock_rw_perms = "0770"

Also is there needed other way to mount raw disk for VM?
right now in config I'm using

-drive file=/dev/sdc,id=disk,format=raw,if=none -device ide-hd,drive=disk

But OVMF bios is not finding my disk, also can't find working way to support AHCI.

Thanks for help

DanaGoyette

Member

Registered: 2014-01-03

Posts: 46

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Hmm, I'd given up on running Windows on a VM, thanks to that Sea Islands bus reset bug. 
I just had an interesting thought:

For me, the biggest annoyance about the bus-reset failure is not just the inability to reboot the guest.
What's worse is that, if I don't stop it in time, the failed boot will make Windows try to automatically fix things.
The automatic fixing has sometimes even undone the very Windows Updates that triggered the reboot.

Is it possible to force Qemu (via libvirt) to shut down the guest instead of restarting it, when the guest requests a reboot?
Even better, is it possible to make that "restart" (but not shutdown) automatically trigger a host suspend, or a host reboot?

Myranti

Member

Registered: 2010-03-04

Posts: 20

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Is it possible to force Qemu (via libvirt) to shut down the guest instead of restarting it, when the guest requests a reboot?
Even better, is it possible to make that "restart" (but not shutdown) automatically trigger a host suspend, or a host reboot?

Assuming you're using libvirt, I think

<on_reboot>destroy</on_reboot>

might be what you're after.

I believe -no-reboot is the Qemu command line option, although I will admit, I haven't tried this.

gessert

Member

Registered: 2015-02-09

Posts: 5

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

When I run the vm now, the screen turns off. I still do not have graphics output. Please advise.

It seems that you have the exact same problem that I do, which I posted a few pages ago. My system behaves the same as yours. In my case, I have one GT 210 and two GTX 580s. When the VM is launched and the screen turns off, it uses maximum CPU on a single core but never starts.

I've tried all sorts of things: unplugging the primary monitor at boot, putting GRUB in text-only mode, changing the boot TTY, quiet mode, and all manner of boot options to both GRUB and the kernel. So far, nothing has helped. It seems that the only recourse for us is to reassign the primary adapter (unsupported in both of our cases) or physically move cards around. Since this requires a system drain and cutting new tubes in my case, I'm postponing this for a while to see if anybody can come up with any ideas.

Yes, I noticed we had basically the same problem. I have tried a similar approach. I'm currently sending my grub and terminal booting messages through my serial port into nirvana.

/etc/default/grub

GRUB_DEFAULT=0
GRUB_TIMEOUT=5
GRUB_DISTRIBUTOR=`lsb_release -i -s 2> /dev/null || echo Debian`
GRUB_CMDLINE_LINUX_DEFAULT="quiet intel_iommu=on vfio_iommu_type1.allow_unsafe_interrupts=1 console=ttyS0,38400n8 nofb"
GRUB_TERMINAL=serial
GRUB_SERIAL_COMMAND="serial --speed=38400 --unit=0 --word=8 --parity=no"
GRUB_CMDLINE_LINUX=""

run via "update-grub"

When booting, I do not see a grub, or any boot-messages anymore. The system seems to just magically boot into X *grin*. This is an ugly "solution", but untill we figure out something proper it's going to work.

Anyway this didn't solve the issue yet. There is one more thing running on the primary card, which I'm currently looking into changing or disabling. When I switch the runlevel - CTRL + ATL + F1 and so on - the primary graphicsadapter is being used. I'm currently trying to figure out how to configure the gettys.

And I'm in way over my head. Any help would be vastly appreciated.

/proc/iomem (excerpt)

90000000-fbffbfff : PCI Bus 0000:00
  a0000000-b1ffffff : PCI Bus 0000:04
    a0000000-afffffff : 0000:04:00.0
    b0000000-b1ffffff : 0000:04:00.0
      b1000000-b12fffff : BOOTFB

Last edited by gessert (2015-02-14 13:14:45)

Flyser

Member

Registered: 2013-12-19

Posts: 29

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Hey guys,
Since my GTX 660 is successfully running as a Quadro K4000 for some time now, I want to check if it actually yields any performance improvements. Specifically I want to compare the following configurations:
Native Windows 8.1 with GTX 660
Virtual Windows 8.1 with GTX 660 (without hyperv extensions)
Virtual Windows 8.1 with Quadro K4000 (with hyperv extensions)
running on my Intel DH87RL Board with Core i7-4771 and latest nvidia game-ready drivers. My libvirt configuration is: https://gist.github.com/flyser/2c52d9b1dfca1f19e2f9

Which benchmarks would you recommend for such a comparison? Anything you would like to see tested?

slis

Member

Registered: 2014-06-02

Posts: 127

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Can u see if u can enable HW OVERLAY in nvidia control panel 3d settings with quadro driver? (i am to lazy to bloat my windows install )

Flyser

Member

Registered: 2013-12-19

Posts: 29

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Can u see if u can enable HW OVERLAY in nvidia control panel 3d settings with quadro driver? (i am to lazy to bloat my windows install )

I doubt that would work, since I am using the nvidia game ready driver and not the quadro specific one.

tuxGaming

Member

Registered: 2015-02-14

Posts: 1

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

I searced for a long time but did not find an answer to my problem:

i am starting with this:
sudo qemu-system-x86_64 -enable-kvm -M q35 -m 1024 -cpu host -smp 2,sockets=1,cores=2,threads=1 -device ioh3420,bus=pcie.0,addr=1c.0,multifunction=on,port=1,chassis=1,id=root.1 -device vfio-pci,host=01:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on -device vfio-pci,host=01:00.1,bus=root.1,addr=00.1 -drive file=/dev/vg0/windows1-clone,id=osdisk,format=raw,if=none -device ide-hd,bus=ide.0,drive=osdisk -drive file=/home/tuxgaming/images/Windows7x64pro.iso,id=isocd,if=none -device ide-cd,bus=ide.1,drive=isocd -drive file=/home/tuxgaming/images/virtio-win-0.1-100.iso,id=virtiocd,if=none -device ide-cd,bus=ide.2,drive=virtiocd -boot menu=on

And i am getting:
This device cannot find enough free resources that it can use. If you want to use this device, you will need to disable one of the other devices on this system. (Code 12)
I think it has a confict with the Standard VGA.
adding -vga none results in a black screen "QEMU 2.1.2 monitor -type 'help' for more information (qemu)" and a blinking cursor. The monitor attacht to the NVIDIA 660ti stays black.

http://de.tinypic.com/view.php?pic=2lbm … N9yIXWc3MU

Thank you for helping out

gessert

Member

Registered: 2015-02-09

Posts: 5

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

I searced for a long time but did not find an answer to my problem:

i am starting with this:
sudo qemu-system-x86_64 -enable-kvm -M q35 -m 1024 -cpu host -smp 2,sockets=1,cores=2,threads=1 -device ioh3420,bus=pcie.0,addr=1c.0,multifunction=on,port=1,chassis=1,id=root.1 -device vfio-pci,host=01:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on -device vfio-pci,host=01:00.1,bus=root.1,addr=00.1 -drive file=/dev/vg0/windows1-clone,id=osdisk,format=raw,if=none -device ide-hd,bus=ide.0,drive=osdisk -drive file=/home/tuxgaming/images/Windows7x64pro.iso,id=isocd,if=none -device ide-cd,bus=ide.1,drive=isocd -drive file=/home/tuxgaming/images/virtio-win-0.1-100.iso,id=virtiocd,if=none -device ide-cd,bus=ide.2,drive=virtiocd -boot menu=on

And i am getting:
This device cannot find enough free resources that it can use. If you want to use this device, you will need to disable one of the other devices on this system. (Code 12)
I think it has a confict with the Standard VGA.
adding -vga none results in a black screen "QEMU 2.1.2 monitor -type 'help' for more information (qemu)" and a blinking cursor. The monitor attacht to the NVIDIA 660ti stays black.

http://de.tinypic.com/view.php?pic=2lbm … N9yIXWc3MU

Thank you for helping out

Iirc the problem stems from the "ioh3420" device. It limits your options. I remember getting rid of it by not using q35. Remove the addresses from the devices as they use the "ioh3420" device.

It should probably look something like this:

sudo qemu-system-x86_64 -enable-kvm -M pc -m 1024 -cpu host -smp 2,sockets=1,cores=2,threads=1 \
-bios /usr/share/seabios/bios.bin -vga none \
-device vfio-pci,host=01:00.0,multifunction=on,x-vga=on \
-device vfio-pci,host=01:00.1 \
-drive file=/dev/vg0/windows1-clone,format=raw,media=disk \
-drive file=/home/tuxgaming/images/Windows7x64pro.iso,media=cdrom \
-drive file=/home/tuxgaming/images/virtio-win-0.1-100.iso,media=cdrom \
-boot menu=on

I'm not sure if you need the seabios option. Give it a try.

mutiny

Member

Registered: 2015-02-15

Posts: 13

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Firstly, I do apologize if these issues have been addressed previously, this thread is massive to read through. I am currently in the process of attempting to utilize PCI/VGA passthrough to a VM guest, but have some concerns and issues I am already encountering. Currently using 3.19 kernel, and the hardware I am using includes:

Intel Core-i7 4790K cpu
Gigabyte Z97X-UD5H motherboard (Z97 "wildcat" pch chipset)
Nvidia GeForce GTX 650 + HDMI audio PCIe device (host gpu)
AMD Radeon HD7870 + HDMI audio PCIe device (pass to guest)
Intel Lan on motherboard (use for host)
Qualcomm Lan on motherboard (pass to guest instead of using virtio)

However, I guess as to be expected for Core-i7 cpus, the PCIe root ports for the x16/x8 slots are provided by the processor itself (as can be seen in Gigabyte's diagram as well):

As a result, I believe any PCIe cards installed in these slots, including the two GPUs I am attempting to use, will appear in the same IOMMU group, and this setup becomes impossible. I think my only options are then to:
1) Use integrated Intel HD4600 graphics for host instead of PCIe card (and hope they are in separate IOMMU groups?)
2) Use potentially unsafe ACS patch to 'force' isolation

Does this seem correct? Are there other alternatives at this point in time? I really don't want to deal with patches with each kernel update, and I definitely would like to be able to use the dedicated GPU I specifically got for this.

The Qualcomm ethernet device is also in the same IOMMU group as a Marvell SATA controller (not the main Z97 chipset SATA controller), some other device (PCIe <--> PCI bridge possibly IIRC), as well as some other PCIe root ports (probably the x1 slots, and of course the ethernet/SATA devices themselves). Interestingly, based on the diagram, the Intel ethernet device appears in its own group and not in the Marvell SATA/bridge/qualcomm IOMMU group. Everything else appears to be in their own IOMMU groups. I'm not sure if this is an expected result based on the Gigabyte diagram and the PCH ACS quirks in more recent kernels.

Also, is it possible that the ACS quirks implemented for the PCH chipset would some time in the future also be done for processors themselves? Such that the particular use case I am encountering will become possible without any kernel patches or resorting to restricting all processor provided PCIe slots to the VM and none for host? As I understand it (please do correct me if wrong), even though the chipset itself does not expose/support any ACS capabilities, the implemented quirks do some of its own checks to reveal at least some information, such as isolation, that ACS would have provided? Then, can these checks also be done for processor provided root ports?

Thanks for any information!

Last edited by mutiny (2015-02-15 13:55:35)

rantoie

Member

Registered: 2015-02-16

Posts: 2

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Hi all,

I'm trying to get things to work with my GeForce 8800, but I keep getting the dreaded Code 43 in Windows. I must be missing something obvious, but I've been combing this thread all day trying various things.

My setup:

Intel(R) Xeon(R) CPU E3-1241 v3 @ 3.50GHz
05:00.0 VGA compatible controller: NVIDIA Corporation G92 [GeForce 8800 GT] (rev a2)
MB: ASRock H97m Pro 4

# dmesg | grep -e DMAR -e IOMMU
[    0.000000] ACPI: DMAR 0x00000000BE62B7F8 000080 (v01 INTEL  BDW      00000001 INTL 00000001)
[    0.000000] Intel-IOMMU: enabled
[    0.023011] dmar: IOMMU 0: reg_base_addr fed90000 ver 1:0 cap d2008c20660462 ecap f010da
[    0.025279] IOAPIC id 8 under DRHD base  0xfed90000 IOMMU 0
[    0.591229] DMAR: No ATSR found
[    0.591290] IOMMU 0 0xfed90000: using Queued invalidation
[    0.591341] IOMMU: Setting RMRR:
[    0.591395] IOMMU: Setting identity map for device 0000:00:14.0 [0xbeea2000 - 0xbeeb0fff]
[    0.591472] IOMMU: Setting identity map for device 0000:00:1a.0 [0xbeea2000 - 0xbeeb0fff]
[    0.591544] IOMMU: Setting identity map for device 0000:00:1d.0 [0xbeea2000 - 0xbeeb0fff]
[    0.591611] IOMMU: Prepare 0-16MiB unity mapping for LPC
[    0.591666] IOMMU: Setting identity map for device 0000:00:1f.0 [0x0 - 0xffffff]
[    0.623526] AMD IOMMUv2 driver by Joerg Roedel <joerg.roedel@amd.com>
[    0.623605] AMD IOMMUv2 functionality not available on this system

I notice that the IOMMUv2 functionality is not available, in fact there is also this odd thing in the dmesg:

[    0.605969] DMAR: No ATSR found
[    0.606030] IOMMU 0 0xfed90000: using Queued invalidation
[    0.606082] IOMMU: Setting RMRR:
[    0.606139] IOMMU: Setting identity map for device 0000:00:14.0 [0xbeea2000 - 0xbeeb0fff]
[    0.606216] IOMMU: Setting identity map for device 0000:00:1a.0 [0xbeea2000 - 0xbeeb0fff]
[    0.606287] IOMMU: Setting identity map for device 0000:00:1d.0 [0xbeea2000 - 0xbeeb0fff]
[    0.606355] IOMMU: Prepare 0-16MiB unity mapping for LPC
[    0.606410] IOMMU: Setting identity map for device 0000:00:1f.0 [0x0 - 0xffffff]
[    0.606476] PCI-DMA: Intel(R) Virtualization Technology for Directed I/O
[    0.606591] ------------[ cut here ]------------
[    0.606643] WARNING: CPU: 2 PID: 1 at /build/linux-y7bjb0/linux-3.16.7-ckt4/drivers/pci/search.c:131 pci_find_upstream_pcie_bridge+0x73/0x80()
[    0.606702] Modules linked in:
[    0.606793] CPU: 2 PID: 1 Comm: swapper/0 Not tainted 3.16.0-4-amd64 #1 Debian 3.16.7-ckt4-3
[    0.606850] Hardware name: To Be Filled By O.E.M. To Be Filled By O.E.M./H97M Pro4, BIOS P1.20 07/29/2014
[    0.606908]  0000000000000009 ffffffff815096a7 0000000000000000 ffffffff810676f7
[    0.607130]  ffff88042c9bd098 ffff88042c9bd000 ffff88042c9bd098 0000000000000000
[    0.607352]  0000000000000000 ffffffff812e4103 ffffffff813f382b 0003000000000000
[    0.607573] Call Trace:
[    0.607623]  [<ffffffff815096a7>] ? dump_stack+0x41/0x51
[    0.607674]  [<ffffffff810676f7>] ? warn_slowpath_common+0x77/0x90
[    0.607725]  [<ffffffff812e4103>] ? pci_find_upstream_pcie_bridge+0x73/0x80
[    0.607777]  [<ffffffff813f382b>] ? intel_iommu_add_device+0x3b/0x210
[    0.607828]  [<ffffffff813e8900>] ? bus_set_iommu+0x50/0x50
[    0.607879]  [<ffffffff813e8925>] ? add_iommu_group+0x25/0x50
[    0.607931]  [<ffffffff8139fa5b>] ? bus_for_each_dev+0x5b/0x90
[    0.607981]  [<ffffffff813e88f3>] ? bus_set_iommu+0x43/0x50
[    0.608033]  [<ffffffff8194a7e3>] ? intel_iommu_init+0x53f/0x586
[    0.608084]  [<ffffffff8190aade>] ? memblock_find_dma_reserve+0x157/0x157
[    0.608135]  [<ffffffff8190aaec>] ? pci_iommu_init+0xe/0x37
[    0.608186]  [<ffffffff8100213c>] ? do_one_initcall+0xcc/0x200
[    0.608238]  [<ffffffff81085fb8>] ? parse_args+0x238/0x470
[    0.608289]  [<ffffffff819021db>] ? kernel_init_freeable+0x15f/0x1e1
[    0.608339]  [<ffffffff8190195e>] ? initcall_blacklist+0xb2/0xb2
[    0.608391]  [<ffffffff814fcc10>] ? rest_init+0x80/0x80
[    0.608441]  [<ffffffff814fcc1a>] ? kernel_init+0xa/0xf0
[    0.608492]  [<ffffffff8150f6bc>] ? ret_from_fork+0x7c/0xb0
[    0.608542]  [<ffffffff814fcc10>] ? rest_init+0x80/0x80
[    0.608595] ---[ end trace f1670364591019ce ]---

See below for the full dmesg.

05:00.0 0300: 10de:0611 (rev a2) (prog-if 00 [VGA controller])
	Subsystem: 10de:054f
	Flags: bus master, fast devsel, latency 0, IRQ 16
	Memory at ee000000 (32-bit, non-prefetchable) [size=16M]
	Memory at c0000000 (64-bit, prefetchable) [size=256M]
	Memory at ec000000 (64-bit, non-prefetchable) [size=32M]
	I/O ports at d000 [size=128]
	[virtual] Expansion ROM at ef000000 [disabled] [size=128K]
	Capabilities: [60] Power Management version 3
	Capabilities: [68] MSI: Enable- Count=1/1 Maskable- 64bit+
	Capabilities: [78] Express Endpoint, MSI 00
	Capabilities: [100] Virtual Channel
	Capabilities: [128] Power Budgeting <?>
	Capabilities: [600] Vendor Specific Information: ID=0001 Rev=1 Len=024 <?>
	Kernel driver in use: vfio-pci

BOOT_IMAGE=/vmlinuz-3.16.0-4-amd64 root=UUID=0fabcc0e-9578-4e2e-8967-fd22994e8189 ro intel_iommu=on pcie_acs_override=downstream

<domain type='kvm'>
  <name>win72</name>
  <uuid>969e8881-64d6-4381-a1d6-699a41cf1be4</uuid>
  <memory unit='KiB'>8388608</memory>
  <currentMemory unit='KiB'>8388608</currentMemory>
  <vcpu placement='static'>4</vcpu>
  <os>
    <type arch='x86_64' machine='pc-i440fx-2.1'>hvm</type>
    <boot dev='hd'/>
  </os>
  <features>
    <acpi/>
    <apic/>
    <pae/>
    <kvm>
      <hidden state='on'/>
    </kvm>
  </features>
  <cpu>
    <topology sockets='1' cores='3' threads='2'/>
  </cpu>
  <clock offset='localtime'>
    <timer name='hypervclock' present='yes'/>
    <timer name='rtc' tickpolicy='catchup'/>
    <timer name='pit' tickpolicy='delay'/>
    <timer name='hpet' present='no'/>
  </clock>
  <on_poweroff>destroy</on_poweroff>
  <on_reboot>restart</on_reboot>
  <on_crash>restart</on_crash>
  <pm>
    <suspend-to-mem enabled='no'/>
    <suspend-to-disk enabled='no'/>
  </pm>
  <devices>
    <emulator>/usr/bin/kvm</emulator>
    <disk type='file' device='disk'>
      <driver name='qemu' type='qcow2'/>
      <source file='/var/lib/libvirt/images/windows.qcow2'/>
      <target dev='hda' bus='ide'/>
      <address type='drive' controller='0' bus='0' target='0' unit='0'/>
    </disk>
    <controller type='usb' index='0' model='ich9-ehci1'>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x05' function='0x7'/>
    </controller>
    <controller type='usb' index='0' model='ich9-uhci1'>
      <master startport='0'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x05' function='0x0' multifunction='on'/>
    </controller>
    <controller type='usb' index='0' model='ich9-uhci2'>
      <master startport='2'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x05' function='0x1'/>
    </controller>
    <controller type='usb' index='0' model='ich9-uhci3'>
      <master startport='4'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x05' function='0x2'/>
    </controller>
    <controller type='pci' index='0' model='pci-root'/>
    <controller type='ide' index='0'>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x1'/>
    </controller>
    <controller type='virtio-serial' index='0'>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x06' function='0x0'/>
    </controller>
    <interface type='network'>
      <mac address='52:54:00:df:34:1a'/>
      <source network='default'/>
      <model type='rtl8139'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x0'/>
    </interface>
    <serial type='pty'>
      <target port='0'/>
    </serial>
    <console type='pty'>
      <target type='serial' port='0'/>
    </console>
    <channel type='spicevmc'>
      <target type='virtio' name='com.redhat.spice.0'/>
      <address type='virtio-serial' controller='0' bus='0' port='1'/>
    </channel>
    <input type='tablet' bus='usb'/>
    <input type='mouse' bus='ps2'/>
    <input type='keyboard' bus='ps2'/>
    <graphics type='spice' autoport='yes'/>
    <sound model='ich6'>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x04' function='0x0'/>
    </sound>
    <video>
      <model type='qxl' ram='65536' vram='9216' heads='1'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x0'/>
    </video>
    <hostdev mode='subsystem' type='pci' managed='yes'>
      <driver name='vfio'/>
      <source>
        <address domain='0x0000' bus='0x05' slot='0x00' function='0x0'/>
      </source>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x08' function='0x0'/>
    </hostdev>
    <redirdev bus='usb' type='spicevmc'>
    </redirdev>
    <redirdev bus='usb' type='spicevmc'>
    </redirdev>
    <redirdev bus='usb' type='spicevmc'>
    </redirdev>
    <redirdev bus='usb' type='spicevmc'>
    </redirdev>
    <memballoon model='virtio'>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x07' function='0x0'/>
    </memballoon>
  </devices>
</domain>

/etc/modprobe.d/nouveau.conf has:
blacklist nouveau

/etc/modprobe.d/kvm.conf has:
options kvm ignore_msrs=1
options vfio_iommu_type1 allow_unsafe_interrupts=1

Last edited by rantoie (2015-02-16 06:19:43)

slis

Member

Registered: 2014-06-02

Posts: 127

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Everything looks good except

<video>
      <model type='qxl' ram='65536' vram='9216' heads='1'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x0'/>
    </video>

You don't need virtual gpu, it could cause error 43 cause your 8800gt becomes secondary adapter and that is unsupported with geforce cards.

mauorrizze

Member

Registered: 2014-08-22

Posts: 18

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

And if you are using a driver newer than 340.52 I think

<timer name='hypervclock' present='yes'/>

should be set to no.

JohnyPea

Member

Registered: 2014-07-17

Posts: 17

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

just to chime in. I haven't been able to find where I got the DEVID patch but there were changes in the latest qemu source and I have modified the patch so it can be applied. So for anyone who is interested in virtual "Quadrification":

http://pastebin.com/APTu6r4f

usage as previously mentioned:
Target virtual card (K4000)
Device Id:     10DE 11FA
Subsystem Id: 10DE 097C

vfio-pci,host=06:00.0,x-vga=on,x-vid=0x10DE,x-did=0x11FA,x-ss-vid=0x10DE,x-ss-did=0x097C,romfile=/whereever/NVIDIA.QuadroK4000.3072.120813.rom

It has some problems with my 660TI 3GB, probably because there isn't a quadro card with exact same hardware configuration. But generally works as K4000 - same RAM config and also as K5000 - same chip.

Regarding some problems with direct host pulseaudio connection I havent found the exact cause of this, but active QXL graphics diver in guest drasticaly reduces performance and causes sound crackling in widows guest (8.1). The device can be assigned to guest so you can see the boot process, but has to be disabled in windows or not installed at all (not activating the display is not enough). Hope some of this helps.

Last edited by JohnyPea (2015-02-16 12:14:21)

rantoie

Member

Registered: 2015-02-16

Posts: 2

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Everything looks good except

<video>
      <model type='qxl' ram='65536' vram='9216' heads='1'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x0'/>
    </video>

You don't need virtual gpu, it could cause error 43 cause your 8800gt becomes secondary adapter and that is unsupported with geforce cards.

Ok, I tried to remove that with virsh edit, and then saved the xml, it comes back automatically as:

    <video>
      <model type='cirrus' vram='9216' heads='1'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x0'/>
    </video>

Reading the libvirt documentation: "For backwards compatibility, if no video is set but there is a graphics in domain xml, then libvirt will add a default video according to the guest type. "

So I tried to remove the graphics section, and the video block. This time it didn't add it back, but when I start the guest, the video is now blank

And if you are using a driver newer than 340.52 I think

<timer name='hypervclock' present='yes'/>

should be set to no.

Thanks, I've set that to 'no'.

Last edited by rantoie (2015-02-16 14:50:05)