Arch Linux

Flyser

Member

Registered: 2013-12-19

Posts: 29

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

I set the Quadro k4000 pci ids for my GTX660

Can you explain how did you achieve that without soldering iron ?

Thank you .

I applied the patch aw wrote ( http://fpaste.org/149722/72280914/ ) and added x-vid=0x10DE,x-did=0x11FA,x-ss-vid=0x10DE,x-ss-did=0x097C,romfile=/path/to/quadro-k4000-bios.rom to the gpu assignment in the qemu cmdline

4kGamer

Member

Registered: 2014-10-29

Posts: 88

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

hi Alex, thank you, I got every command from your blog. That's what I did and at first it worked. After some reboot it doesn't. I reinstalled Windows 4-5 times today, but I couldn't replicate the success I initially had. I assume it must be something I did, I just don't know what ...

Does it matter WHEN I remove those lines? Or when I add the

    <kvm>
      <hidden state='on'/>
    </kvm>

line?

Last edited by 4kGamer (2014-11-13 21:39:17)

aw

Member

Registered: 2013-10-04

Posts: 921

Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

hi Alex, thank you, I got every command from your blog. That's what I did and at first it worked. After some reboot it doesn't. I reinstalled Windows 4-5 times today, but I couldn't replicate the success I initially had. I assume it must be something I did, I just don't know what ...

Does it matter WHEN I remove those lines? Or when I add the

    <kvm>
      <hidden state='on'/>
    </kvm>

line?

Not that I'm aware of.  I just updated my 440FX/OVMF/8.1/GTX750 VM to 344.65 and it still works, so apparently no new subversion.

---

http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

4kGamer

Member

Registered: 2014-10-29

Posts: 88

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

ok, thank you. I'll keep on trying. Will post as soon as it (hopefully) works.

Denso

Member

Registered: 2014-08-30

Posts: 179

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

I applied the patch aw wrote ( http://fpaste.org/149722/72280914/ ) and added x-vid=0x10DE,x-did=0x11FA,x-ss-vid=0x10DE,x-ss-did=0x097C,romfile=/path/to/quadro-k4000-bios.rom to the gpu assignment in the qemu cmdline

Thanks for sharing ! As I said , I saw no real life difference with Hv enhancements on or off (Maybe those who encode videos alot would) . I'll keep my stable configuration for now

EDIT :

@aw

I recieved this error two times uptil now :

[Thu Nov 13 03:13:21 2014] pcieport 0000:00:03.0: AER: Multiple Corrected error received: id=0018
[Thu Nov 13 03:13:21 2014] pcieport 0000:00:03.0: PCIe Bus Error: severity=Corrected, type=Data Link Layer, id=0018(Receiver ID)
[Thu Nov 13 03:13:21 2014] pcieport 0000:00:03.0:   device [8086:2f08] error status/mask=00000040/00002000
[Thu Nov 13 03:13:21 2014] pcieport 0000:00:03.0:    [ 6] Bad TLP

I didn't notice anything while using the VM or the host though . I'm using 3.18-rc4 + ACS patch .

Last edited by Denso (2014-11-14 05:23:22)

Duelist

Member

Registered: 2014-09-22

Posts: 358

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

[Thu Nov 13 03:13:21 2014] pcieport 0000:00:03.0: AER: Multiple Corrected error received: id=0018
[Thu Nov 13 03:13:21 2014] pcieport 0000:00:03.0: PCIe Bus Error: severity=Corrected, type=Data Link Layer, id=0018(Receiver ID)
[Thu Nov 13 03:13:21 2014] pcieport 0000:00:03.0:   device [8086:2f08] error status/mask=00000040/00002000
[Thu Nov 13 03:13:21 2014] pcieport 0000:00:03.0:    [ 6] Bad TLP

I didn't notice anything while using the VM or the host though . I'm using 3.18-rc4 + ACS patch .

No surprise you didn't notice anything.
There happened an error on the pci-e bus that was corrected. The error was noted by the device ID 0018(00:18.0?), as far as i understood, and it is your cpu's root port.
That maay be related to ACS.
Don't you have the new X99 chipset, where aw haven't got the letter confirming device isolation from intel?

---

The forum rules prohibit requesting support for distributions other than arch.
I gave up. It was too late.
What I was trying to do.
The reference about VFIO and KVM VGA passthrough.

Denso

Member

Registered: 2014-08-30

Posts: 179

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

No surprise you didn't notice anything.
There happened an error on the pci-e bus that was corrected. The error was noted by the device ID 0018(00:18.0?), as far as i understood, and it is your cpu's root port.
That maay be related to ACS.
Don't you have the new X99 chipset, where aw haven't got the letter confirming device isolation from intel?

Well , there is no 00:18.0 device reported by lspci , but the error shows "00:03.0" which is :

00:03.0 PCI bridge: Intel Corporation Xeon E5 v3/Core i7 PCI Express Root Port 3 (rev 02)
00:03.2 PCI bridge: Intel Corporation Xeon E5 v3/Core i7 PCI Express Root Port 3 (rev 02)

And yes , I have the X99 platform , and I'm by no means a coder or a kernel hacker , I just applied the ACS patch and -to me as an end user- it just "works" . lol . Host + VMs are very stable (for around 2 days uptil this moment) .

Also editing the quirks.c file and inserting the X99 device IDs manually does the same thing .

Last edited by Denso (2014-11-14 10:29:16)

cshark

Member

Registered: 2014-11-14

Posts: 1

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

I have done the basic steps described in the article and qemu works as espected when invoked with the test setup. Now I want to use virt-manager with libvirt, so I created the important xml part with

# virsh native-to-domxml qemu-args <file with args>

which gave me some xml file with the device args as qemu args. I was wondering, why the devices were "only" arguments and had no proper xml tag, but I guess it's okay. So I created a new vm with virt-manager and added the important parts to the xml file. My final xml looks like this:

<domain type='kvm' xmlns:qemu='http://libvirt.org/schemas/domain/qemu/1.0'>
  <name>win7</name>
  <uuid>c344a358-33bc-4398-b223-4054d6e61680</uuid>
  <memory unit='KiB'>4194304</memory>
  <currentMemory unit='KiB'>4194304</currentMemory>
  <vcpu placement='static'>3</vcpu>
  <os>
    <type arch='x86_64' machine='pc-q35-2.1'>hvm</type>
    <bootmenu enable='yes'/>
  </os>
  <features>
    <acpi/>
    <apic/>
    <pae/>
    <hyperv>
      <relaxed state='on'/>
      <vapic state='on'/>
      <spinlocks state='on' retries='8191'/>
    </hyperv>
  </features>
  <cpu mode='host-model'>
    <model fallback='allow'/>
  </cpu>
  <clock offset='localtime'>
    <timer name='rtc' tickpolicy='catchup'/>
    <timer name='pit' tickpolicy='delay'/>
    <timer name='hpet' present='no'/>
    <timer name='hypervclock' present='yes'/>
  </clock>
  <on_poweroff>destroy</on_poweroff>
  <on_reboot>destroy</on_reboot>
  <on_crash>destroy</on_crash>
  <pm>
    <suspend-to-mem enabled='no'/>
    <suspend-to-disk enabled='no'/>
  </pm>
  <devices>
    <emulator>/usr/local/bin/kvm</emulator>
    <disk type='block' device='disk'>
      <driver name='qemu' type='raw' cache='none' io='native'/>
      <source dev='/dev/sdb2'/>
      <target dev='vda' bus='virtio'/>
      <boot order='2'/>
      <address type='pci' domain='0x0000' bus='0x02' slot='0x03' function='0x0'/>
    </disk>
    <disk type='block' device='cdrom'>
      <driver name='qemu' type='raw'/>
      <source dev='/dev/sr0'/>
      <target dev='hda' bus='ide'/>
      <readonly/>
      <boot order='1'/>
      <address type='drive' controller='0' bus='0' target='0' unit='0'/>
    </disk>
    <controller type='usb' index='0' model='ich9-ehci1'>
      <address type='pci' domain='0x0000' bus='0x02' slot='0x02' function='0x7'/>
    </controller>
    <controller type='usb' index='0' model='ich9-uhci1'>
      <master startport='0'/>
      <address type='pci' domain='0x0000' bus='0x02' slot='0x02' function='0x0' multifunction='on'/>
    </controller>
    <controller type='usb' index='0' model='ich9-uhci2'>
      <master startport='2'/>
      <address type='pci' domain='0x0000' bus='0x02' slot='0x02' function='0x1'/>
    </controller>
    <controller type='usb' index='0' model='ich9-uhci3'>
      <master startport='4'/>
      <address type='pci' domain='0x0000' bus='0x02' slot='0x02' function='0x2'/>
    </controller>
    <controller type='sata' index='0'>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x1f' function='0x2'/>
    </controller>
    <controller type='pci' index='0' model='pcie-root'/>
    <controller type='pci' index='1' model='dmi-to-pci-bridge'>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x1e' function='0x0'/>
    </controller>
    <controller type='pci' index='2' model='pci-bridge'>
      <address type='pci' domain='0x0000' bus='0x01' slot='0x01' function='0x0'/>
    </controller>
    <controller type='ide' index='0'/>
    <interface type='network'>
      <mac address='52:54:00:36:e7:5a'/>
      <source network='virtnetwork'/>
      <model type='rtl8139'/>
      <address type='pci' domain='0x0000' bus='0x02' slot='0x01' function='0x0'/>
</interface>
    <serial type='pty'>
      <target port='0'/>
    </serial>
    <console type='pty'>
      <target type='serial' port='0'/>
    </console>
    <input type='tablet' bus='usb'/>
    <memballoon model='virtio'>
      <address type='pci' domain='0x0000' bus='0x02' slot='0x06' function='0x0'/>
    </memballoon>
  </devices>
  <qemu:commandline>
    <qemu:arg value='-enable-kvm'/>
    <qemu:arg value='-device ioh3420,bus=pcie.0,addr=1c.0,multifunction=on,port=1,chassis=1,id=root.1'/>
    <qemu:arg value='-device vfio-pci,host=02:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on'/>
    <qemu:arg value='-device vfio-pci,host=02:00.1,bus=root.1,addr=00.1'/>
  </qemu:commandline>
</domain>

But when I try to start the vm, I get the following error (even as root):

Internal error: early end of file from monitor: possible problem:
2014-11-14T12:25:20.491803Z qemu-system-x86_64: -device vfio-pci,host=02:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on: vfio_dma_map(0x7f3e0dbf0c70, 0x0, 0x80000000, 0x7f3ce8000000) = -12 (Cannot allocate memory)
2014-11-14T12:25:20.492686Z qemu-system-x86_64: -device vfio-pci,host=02:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on: vfio_dma_map(0x7f3e0dbf0c70, 0x100000000, 0x80000000, 0x7f3d68000000) = -12 (Cannot allocate memory)
2014-11-14T12:25:20.492767Z qemu-system-x86_64: -device vfio-pci,host=02:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on: vfio: memory listener initialization failed for container
2014-11-14T12:25:20.492835Z qemu-system-x86_64: -device vfio-pci,host=02:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on: vfio: failed to setup container for group 16
2014-11-14T12:25:20.492939Z qemu-system-x86_64: -device vfio-pci,host=02:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on: vfio: failed to get group 16
2014-11-14T12:25

I searched a bit, but found nothing about this error... any ideas? Since it has to be either something with libvirt or the config, because the qemu test setup just works fine.

Last edited by cshark (2014-11-14 12:55:04)

Rommy

Member

Registered: 2014-10-17

Posts: 51

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Will the two patches (acs override patch + i915 vga arbiter fixes) go into the main linux packages soon? I do not want to build one myself.

slis

Member

Registered: 2014-06-02

Posts: 127

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

<qemu:arg value='vfio-pci,host=01:00.0,addr=07.0,multifunction=on,x-vga=on,x-vid=0x10DE,x-did=0x11BA,x-ss-vid=0x10DE,x-ss-did=0x0965,romfile=/root/k5000.rom'/>

I can also confirm it's working, 680gtx as quadro k5000/or GRIDK2 (0x11BF) with 341.05 drivers with hyperv on windows7.

edit: as grid k2 also works with geforce driver 344,65 with hyperv on.

Last edited by slis (2014-11-14 19:46:44)

aw

Member

Registered: 2013-10-04

Posts: 921

Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Will the two patches (acs override patch + i915 vga arbiter fixes) go into the main linux packages soon? I do not want to build one myself.

You'll need to use the OVMF approach and switch hardware then, neither of those are headed upstream.

---

http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

aw

Member

Registered: 2013-10-04

Posts: 921

Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

I have done the basic steps described in the article and qemu works as espected when invoked with the test setup. Now I want to use virt-manager with libvirt, so I created the important xml part with

# virsh native-to-domxml qemu-args <file with args>

which gave me some xml file with the device args as qemu args. I was wondering, why the devices were "only" arguments and had no proper xml tag, but I guess it's okay. So I created a new vm with virt-manager and added the important parts to the xml file. My final xml looks like this:

  <qemu:commandline>
    <qemu:arg value='-enable-kvm'/>
    <qemu:arg value='-device ioh3420,bus=pcie.0,addr=1c.0,multifunction=on,port=1,chassis=1,id=root.1'/>
    <qemu:arg value='-device vfio-pci,host=02:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on'/>
    <qemu:arg value='-device vfio-pci,host=02:00.1,bus=root.1,addr=00.1'/>
  </qemu:commandline>

But when I try to start the vm, I get the following error (even as root):

Internal error: early end of file from monitor: possible problem:
2014-11-14T12:25:20.491803Z qemu-system-x86_64: -device vfio-pci,host=02:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on: vfio_dma_map(0x7f3e0dbf0c70, 0x0, 0x80000000, 0x7f3ce8000000) = -12 (Cannot allocate memory)
2014-11-14T12:25:20.492686Z qemu-system-x86_64: -device vfio-pci,host=02:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on: vfio_dma_map(0x7f3e0dbf0c70, 0x100000000, 0x80000000, 0x7f3d68000000) = -12 (Cannot allocate memory)
2014-11-14T12:25:20.492767Z qemu-system-x86_64: -device vfio-pci,host=02:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on: vfio: memory listener initialization failed for container
2014-11-14T12:25:20.492835Z qemu-system-x86_64: -device vfio-pci,host=02:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on: vfio: failed to setup container for group 16
2014-11-14T12:25:20.492939Z qemu-system-x86_64: -device vfio-pci,host=02:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on: vfio: failed to get group 16
2014-11-14T12:25

I searched a bit, but found nothing about this error... any ideas? Since it has to be either something with libvirt or the config, because the qemu test setup just works fine.

By using <qemu:arg> you're effectively hiding the assigned device from libvirt and not allowing it setup device access or configure locked memory limits.  My suggestion would be to not use q35 and to use a wrapper script around qemu-system-x86_64 to add the x-vga option, something like:

#!/bin/sh
exec qemu-system-x86_64 `echo "\$@" | sed 's|02:00.0|02:00.0,x-vga=on|g'`

Better yet, if the GPU supports UEFI, use OVMF instead and avoid all the hassles of VGA.  Some here claim this works for Windows 7 guests.  See the link in my sig for details.

---

http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Nesousx

Member

Registered: 2012-03-27

Posts: 46

Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Hi all,

Just for information, I successfully passed through an Asus GTX 970 Strix from Arch (configured as mentioned in first post). Guest is running Win 7 Pro 64 bits, and here is my command line:

qemu-system-x86_64 -M q35 -enable-kvm \
-m 16384 \
-cpu host,kvm=off -smp 4,sockets=1,cores=4,threads=1 \
-bios /usr/share/qemu/bios.bin -realtime mlock=on -vga none \
-D /var/log/qemu-out.log \
-device ioh3420,bus=pcie.0,addr=1c.0,multifunction=on,port=1,chassis=1,id=root.1 \
-device vfio-pci,host=01:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on \
-device vfio-pci,host=01:00.1,bus=pcie.0 \
-device virtio-scsi-pci,id=scsi \
-drive file=/dev/guest/Windaube_7,id=sys,format=raw -device scsi-hd,drive=sys \
-drive file=/dev/guest/win7,id=data,format=raw -device scsi-hd,drive=data \
-device vfio-pci,host=00:1d.0,bus=pcie.0 \
-device virtio-scsi-pci,id=scsi2 \
-net nic,model=virtio -net bridge,br=virbr0 \
-device ich9-intel-hda,bus=pcie.0,addr=1b.0,id=sound0 \
-device hda-duplex,id=sound0-codec0,bus=sound0.0,cad=0

I only had to disable hv-time from CPU flag in order to get rid of Code 43 error (thanks Nvidia...).

4kGamer

Member

Registered: 2014-10-29

Posts: 88

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

I got rid of the Error Code 43 

I am not quite sure what it was, but I started from scratch, built the kernel provided by nbhs (linux-mainline-3.17.2.tar.gz)
and threw every command I could find into grub. This is what my command line looks like.

GRUB_CMDLINE_LINUX_DEFAULT="i915.enable_hd_vgaarb=1 intel_iommu=on pcie_acs_override=downstream vfio_iommu_type1.allow_unsafe_interrupts=1 quiet"

BUT (there always seems to be a but...)

I have weird audio issues: There is some scratching in the audio. And performance suffers big time.

I am really relying on you guys to help me, cause I don't know how to fix this.

Btw.: I am not sure if it's relevant, but lspci -v gives me this output:

01:00.0 VGA compatible controller: NVIDIA Corporation GM204 [GeForce GTX 980] (rev a1) (prog-if 00 [VGA controller])
        Subsystem: Micro-Star International Co., Ltd. [MSI] Device 3170
        Flags: fast devsel, IRQ 16
        Memory at f6000000 (32-bit, non-prefetchable) [size=16M]
        Memory at e0000000 (64-bit, prefetchable) [size=256M]
        Memory at f0000000 (64-bit, prefetchable) [size=32M]
        I/O ports at e000 [size=128]
        Expansion ROM at f7000000 [disabled] [size=512K]
        Capabilities: [60] Power Management version 3
        Capabilities: [68] MSI: Enable- Count=1/1 Maskable- 64bit+
        Capabilities: [78] Express Legacy Endpoint, MSI 00
        Capabilities: [100] Virtual Channel
        Capabilities: [250] Latency Tolerance Reporting
        Capabilities: [258] L1 PM Substates
        Capabilities: [128] Power Budgeting <?>
        Capabilities: [600] Vendor Specific Information: ID=0001 Rev=1 Len=024 <?>
        Capabilities: [900] #19
        Kernel driver in use: vfio-pci
        Kernel modules: nouveau

01:00.1 Audio device: NVIDIA Corporation GM204 High Definition Audio Controller (rev a1)
        Subsystem: Micro-Star International Co., Ltd. [MSI] Device 3170
        Flags: fast devsel, IRQ 17
        Memory at f7080000 (32-bit, non-prefetchable) [size=16K]
        Capabilities: [60] Power Management version 3
        Capabilities: [68] MSI: Enable- Count=1/1 Maskable- 64bit+
        Capabilities: [78] Express Endpoint, MSI 00
        Kernel driver in use: vfio-pci
        Kernel modules: snd_hda_intel

I can't remember if the last line looked always like that but it's an Nvidia Card and it says Intel, maybe that's the culprit??

aw

Member

Registered: 2013-10-04

Posts: 921

Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

I got rid of the Error Code 43 

I am not quite sure what it was, but I started from scratch, built the kernel provided by nbhs (linux-mainline-3.17.2.tar.gz)
and threw every command I could find into grub. This is what my command line looks like.

GRUB_CMDLINE_LINUX_DEFAULT="i915.enable_hd_vgaarb=1 intel_iommu=on pcie_acs_override=downstream vfio_iommu_type1.allow_unsafe_interrupts=1 quiet"

BUT (there always seems to be a but...)

I have weird audio issues: There is some scratching in the audio. And performance suffers big time.

I am really relying on you guys to help me, cause I don't know how to fix this.

Btw.: I am not sure if it's relevant, but lspci -v gives me this output:

snd_hda_intel supports audio from various vendors, that's not a problem.

Nvidia's implementation of the audio device is fairly flaky, but it usually works better if you configure the device to use MSI interrupts in the guest.  See my blog for how to do that.

---

http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

4kGamer

Member

Registered: 2014-10-29

Posts: 88

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

ok, will do, thank you very much. I'll try that right away

DanaGoyette

Member

Registered: 2014-01-03

Posts: 46

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

EDIT: Solved! The above described BIOS setting change did the trick. I had to remove the card bios from the configuration and everything works just fine. Summary: it has to explicitly be set in the BIOS which video card is the primary, "auto" might not be enough. Thanks for your kind help!

That reminds me: My SuperMicro board has settings both for "primary display" and for which ROM to load for each slot.

If I have the Intel graphics explicitly set to primary, but also allow loading the Radeon's EFI ROM, I get really weird behavior:

* The Radeon hijacks the EFI framebuffer (completely ignoring the "primary display" option).  I seem to recall the Intel DisplayPort lighting up but being black.
* The Radeon ends up with its framebuffer address screwed up, and I get the Radeon EFI framebuffer as /dev/fb0 and the Intel KMS driver as /dev/fb1.
* Neither the Windows nor Linux Radeon drivers (native boot, even) will load.

If I switch the PCIe slot ROM options to "Legacy" or "Disabled", no such problems occur, and I can still use EFI for both host and guest.

dwe11er

Member

Registered: 2014-03-18

Posts: 73

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

EDIT: Solved! The above described BIOS setting change did the trick. I had to remove the card bios from the configuration and everything works just fine. Summary: it has to explicitly be set in the BIOS which video card is the primary, "auto" might not be enough. Thanks for your kind help!

That reminds me: My SuperMicro board has settings both for "primary display" and for which ROM to load for each slot.

If I have the Intel graphics explicitly set to primary, but also allow loading the Radeon's EFI ROM, I get really weird behavior:

* The Radeon hijacks the EFI framebuffer (completely ignoring the "primary display" option).  I seem to recall the Intel DisplayPort lighting up but being black.
* The Radeon ends up with its framebuffer address screwed up, and I get the Radeon EFI framebuffer as /dev/fb0 and the Intel KMS driver as /dev/fb1.
* Neither the Windows nor Linux Radeon drivers (native boot, even) will load.

If I switch the PCIe slot ROM options to "Legacy" or "Disabled", no such problems occur, and I can still use EFI for both host and guest.

Have you tried claiming Radeon with pci-stub?

Pasting the code somewhere inside /etc/modprobe.d, and rebuilding initrd should do the job:

softdep drm pre: pci-stub
softdep snd_hda_intel pre: pci-stub
options pci-stub ids=1002:6818,1002:aab0 # replace vid:devid accordingly

orcephrye

Member

Registered: 2014-11-15

Posts: 9

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Hello Peps,

I got my setup working thanks to this forum post. However I found two issues and hope to find even more help here if possible.

1. Audio emulation seams to sometimes cut out while gaming. Or the audio sounds weird and phased.

2. I noticed a roughly 30% decrease in performance and sometimes the FPS rate can go very lower. As in 1 to 5 FPS before returning to its normal ~30ish% reduce from where it would be if I was bare metal.

Here is my hardware:
Host Card: Nvidia GTX 750 (Nvidia drivers)
Windows Card: Nvidia GTX 670
CPU: AMD FX 9590
MotherBoard: MSI 990FXA-GD80 v2

I start qemu via the command line like so:

# QEMU_ALSA_DAC_BUFFER_SIZE=512 QEMU_ALSA_DAC_PERIOD_SIZE=170 QEMU_AUDIO_DRV=alsa \
> qemu-system-x86_64 -enable-kvm -M q35 -m 16384 -cpu host,kvm=off \
> -smp cpus=8,sockets=1,cores=8,threads=1 \
> -bios /usr/share/qemu/bios.bin -vga none \
> -device vfio-pci,host=06:00.0,multifunction=on,x-vga=on \
> -device vfio-pci,host=06:00.1 \
> -device virtio-scsi-pci,id=scsi \
> -drive file=/dev/sdb,id=disk0,format=raw -device scsi-hd,drive=disk0 \
> -drive file=/dev/sdc,id=disk1,format=raw -device scsi-hd,drive=disk1 \
> -usb -usbdevice host:1532:0109 -usbdevice host:046d:c52b \
> -net nic,macaddr=00:16:35:AF:94:4B -net user \
> -device ich9-intel-hda,bus=pcie.0,addr=1b.0,id=sound0 \
> -device hda-duplex,id=sound0-codec0,bus=sound0.0,cad=0

I also followed this post: http://forums.guru3d.com/showthread.php?t=378044 and made sure my card was using MSI. I also took the liberty of making the emulated ich9 HD audio use MSI. However that seamed to have no real effect.

I also this:

# cat /etc/modprobe.d/60-kvm-amd.conf 
options kvm-amd npt=0

I guess my real question is: Is this normal? Does most people using KVM/Nvidia have a ~30ish % reduction in rendering performance? And/Or times when there are large drops in performance? If not what else am I missing to help this out?

Thanks in advance. :-)

Denso

Member

Registered: 2014-08-30

Posts: 179

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

CPU: AMD FX 9590
MotherBoard: MSI 990FXA-GD80 v2

Hi . I used to have an AMD 8350 + Gigabyte 990FXA-UD5 (If I remember correctly) and it was SO SLOW in gaming and the sound was playing in SlowMo .

When I switched to an i5-3570 (Again if I remember correctly) , it was butter smooth , yes it wasn't like bare-metal but it was pretty darn close .

Now I switched to an X99 platform + GTX770 and I can play games at 2560x1440 60fps (CoD Advanced Warfare for example) no problem .

Here is my Gaming VM script if you're curious :

qemu-system-x86_64 -name gaming -nographic -mem-path /dev/hugepages \
-enable-kvm -m 8192 -cpu host,kvm=off -smp 4,sockets=1,cores=4,threads=1 \
-vga none \
-drive if=pflash,format=raw,readonly,file=/usr/share/ovmf/x64/ovmf_code_x64.bin \
-drive if=pflash,format=raw,file=/VMs/ovmf_gaming.bin \
-device vfio-pci,host=02:00.0,multifunction=on \
-device vfio-pci,host=02:00.1 \
-device vfio-pci,host=10:00.0 \
-drive file=/VMs/Win_Gaming.qed,cache=writeback,if=none,id=drive0,aio=native \
-device virtio-blk-pci,drive=drive0,ioeventfd=on,bootindex=1 \
-drive file=/VMs/Gaming_External.qed,cache=none,if=none,id=drive1,aio=native \
-device virtio-blk-pci,drive=drive1,ioeventfd=on \
-device virtio-scsi-pci,id=scsi \
-drive file=/VMs/Win8.iso,id=iso_install,if=none \
-device scsi-cd,drive=iso_install \
-cdrom /VMs/virtio.iso \
-net nic,model=virtio,macaddr=64:25:5E:87:0E:5C -net bridge,br=br0 \
-localtime \
-monitor unix:/tmp/vm_gaming,server,nowait &

Note that I'm using OVMF + 440FX + HDMI Audio .

Hope this helps !

Last edited by Denso (2014-11-16 03:10:29)

DanaGoyette

Member

Registered: 2014-01-03

Posts: 46

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Have you tried claiming Radeon with pci-stub?

Pasting the code somewhere inside /etc/modprobe.d, and rebuilding initrd should do the job:

softdep drm pre: pci-stub
softdep snd_hda_intel pre: pci-stub
options pci-stub ids=1002:6818,1002:aab0 # replace vid:devid accordingly

In my EFI-Intel + EFI-Radeon case, the Radeon was claimed by the EFI firmware, preventing both radeon and vfio-pci from accessing it properly.  So, blocking the radeon driver changes nothing.
I already figured out my issue; I was just mentioning it for reference, in case somebody else sees similar behavior.

orcephrye

Member

Registered: 2014-11-15

Posts: 9

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Hello Denso! Thanks for your reply!

Sadly I am not sure how useful that info is. Basically it sounds like I would need to buy new hardware. Sadly I just got this AMD setup and spent a pretty penny. Instead of a whole new set of hardware I could possibly buy a new Radeon card and either set how KVM handles that or switch to using Xen.

It appears that switching to OVMF would not help gaming performance at all.. correct?

Also.. "-monitor unix:/tmp/vm_gaming,server,nowait" !? If you dont mind me asking whats going on there? It looks like your are redirecting the output too a file? Is that an socket that redirects to something?

Denso

Member

Registered: 2014-08-30

Posts: 179

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Hello Denso! Thanks for your reply!

Sadly I am not sure how useful that info is. Basically it sounds like I would need to buy new hardware. Sadly I just got this AMD setup and spent a pretty penny. Instead of a whole new set of hardware I could possibly buy a new Radeon card and either set how KVM handles that or switch to using Xen.

It appears that switching to OVMF would not help gaming performance at all.. correct?

Also.. "-monitor unix:/tmp/vm_gaming,server,nowait" !? If you dont mind me asking whats going on there? It looks like your are redirecting the output too a file? Is that an socket that redirects to something?

Hi

I'm sure that you can tune your setup to gain more performance . For my case , switching to an Intel platform was night and day difference (I remember Metro 2033 game , the i5 just blew FX-8350 out of water) .

umm , I'm directing the monitor output to a file , so that I can send a shutdown command to the VM prior to the host's shutdown/reboot like this :

#!/bin/bash

# This code will send an ACPI Shutdown request to our VMs then wait 10 seconds , then kill all remaining "qemu" processes #

echo "system_powerdown" | socat - UNIX-CONNECT:/tmp/vm_winsrv
echo "system_powerdown" | socat - UNIX-CONNECT:/tmp/vm_gaming
echo "system_powerdown" | socat - UNIX-CONNECT:/tmp/vm_main
sleep 10s
killall -9 qemu-system-x86_64

Works like a charm !

sinny

Member

Registered: 2014-05-24

Posts: 75

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Hello Peps,

I got my setup working thanks to this forum post. However I found two issues and hope to find even more help here if possible.

1. Audio emulation seams to sometimes cut out while gaming. Or the audio sounds weird and phased.

2. I noticed a roughly 30% decrease in performance and sometimes the FPS rate can go very lower. As in 1 to 5 FPS before returning to its normal ~30ish% reduce from where it would be if I was bare metal.

Here is my hardware:
Host Card: Nvidia GTX 750 (Nvidia drivers)
Windows Card: Nvidia GTX 670
CPU: AMD FX 9590
MotherBoard: MSI 990FXA-GD80 v2

I start qemu via the command line like so:

# QEMU_ALSA_DAC_BUFFER_SIZE=512 QEMU_ALSA_DAC_PERIOD_SIZE=170 QEMU_AUDIO_DRV=alsa \
> qemu-system-x86_64 -enable-kvm -M q35 -m 16384 -cpu host,kvm=off \
> -smp cpus=8,sockets=1,cores=8,threads=1 \
> -bios /usr/share/qemu/bios.bin -vga none \
> -device vfio-pci,host=06:00.0,multifunction=on,x-vga=on \
> -device vfio-pci,host=06:00.1 \
> -device virtio-scsi-pci,id=scsi \
> -drive file=/dev/sdb,id=disk0,format=raw -device scsi-hd,drive=disk0 \
> -drive file=/dev/sdc,id=disk1,format=raw -device scsi-hd,drive=disk1 \
> -usb -usbdevice host:1532:0109 -usbdevice host:046d:c52b \
> -net nic,macaddr=00:16:35:AF:94:4B -net user \
> -device ich9-intel-hda,bus=pcie.0,addr=1b.0,id=sound0 \
> -device hda-duplex,id=sound0-codec0,bus=sound0.0,cad=0

I also followed this post: http://forums.guru3d.com/showthread.php?t=378044 and made sure my card was using MSI. I also took the liberty of making the emulated ich9 HD audio use MSI. However that seamed to have no real effect.

I also this:

# cat /etc/modprobe.d/60-kvm-amd.conf 
options kvm-amd npt=0

I guess my real question is: Is this normal? Does most people using KVM/Nvidia have a ~30ish % reduction in rendering performance? And/Or times when there are large drops in performance? If not what else am I missing to help this out?

Thanks in advance. :-)

so, basically you did NO basic tuning, but jumped straight to some advanced sutff (e.g. msi)? smart boy, truly. read more of some random forum's post instead of digging this very thread to get all the info

no cpu pinning, no emulated sound parameter tuning (or using passed through audio card), no advanced memory mode (hugetlbfs?) - are you really expecting it to work like a charm out of the box on near-native performance level? for god's sake...

Duelist

Member

Registered: 2014-09-22

Posts: 358

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

no cpu pinning, no emulated sound parameter tuning (or using passed through audio card), no advanced memory mode (hugetlbfs?) - are you really expecting it to work like a charm out of the box on near-native performance level? for god's sake...

Aw, come on, don't be rude, man!

The only thing about CPU i have is -cpu host and -smp option.
The sound is

-device ich9-intel-hda,bus=root.1,addr=00.0,id=sound0 \
-device hda-duplex,id=sound0-codec0,bus=sound0.0,cad=0 \

and i honestly do not know how the hell this works, thanks to nbhs. But it works almost awesome with my host pulse audio setup(fedora, i don't know how pulse works either) - there's little stuttering occuring when there is too much sound(like, playing a really weird music from youtube).Appears like i've got the sound input working too.
The hugepages option just breaks everything apart for me.

And you know what? Performance decrease is about 10%. The only thing that bottlenecks the whole bunch is the disk image file relying on a 2FAST4U loud WD hard drive with 5 or 7 bad sectors.

The only problem i am having with the CPU performance - it's switching it's frequency too damn often, but that depends on one host UEFI setting and hits the host system too.

---

The forum rules prohibit requesting support for distributions other than arch.
I gave up. It was too late.
What I was trying to do.
The reference about VFIO and KVM VGA passthrough.