You are not logged in.

#1401 2014-03-19 21:39:04

rawuza
Member
Registered: 2013-12-22
Posts: 7

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

ahl wrote:
rawuza wrote:

I have similar problems with a HD 5770, even though I've found reports of this card working. See https://bbs.archlinux.org/viewtopic.php … 8#p1391318.  For what it's worth, I don't think it's a problem with the ATA controller, per se, since I get errors with other PCI devices as well. It seems to be pretty random, which one is affected.

You are right. The behavior is pretty random. Here's an incomplete list of problems that I remember occurring: ATA-errors on all connected devices, single port usb failures, full failure of all usb-devices, display port link failure (loss of picture), complete freeze of system, freeze + automatic reboot, and failure of SATA controller with IO_PAGE_FAULTS that resulted in corrupted root file system. Just about anything can happen.

To me this seems as if the GPU touches other devices' memory. My question about this is:

  • Is it possible for one PCIe device to directly access another one? If it's done through DMA, shouldn't the IOMMU prevent exactly this?

  • Is there any way to debug this?

Offline

#1402 2014-03-19 22:06:12

aw
Member
Registered: 2013-10-04
Posts: 921
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

rawuza wrote:
ahl wrote:
rawuza wrote:

I have similar problems with a HD 5770, even though I've found reports of this card working. See https://bbs.archlinux.org/viewtopic.php … 8#p1391318.  For what it's worth, I don't think it's a problem with the ATA controller, per se, since I get errors with other PCI devices as well. It seems to be pretty random, which one is affected.

You are right. The behavior is pretty random. Here's an incomplete list of problems that I remember occurring: ATA-errors on all connected devices, single port usb failures, full failure of all usb-devices, display port link failure (loss of picture), complete freeze of system, freeze + automatic reboot, and failure of SATA controller with IO_PAGE_FAULTS that resulted in corrupted root file system. Just about anything can happen.

To me this seems as if the GPU touches other devices' memory. My question about this is:

  • Is it possible for one PCIe device to directly access another one? If it's done through DMA, shouldn't the IOMMU prevent exactly this?

  • Is there any way to debug this?

The IOMMU can only provide isolation and translation for transactions that it sees.  If there's a multifunction device that allows peer-to-peer DMA within the component, then the IOMMU may never see the transaction.  This is the purpose of Access Control Services (ACS) to allow configuration and enforcement of whether all transactions are forwarded upstream.  This is why IOMMU groups put such devices together, so we can at least have a single entity manage them rather than allowing a guest to exploit the host.  Many folks here abuse the ACS override patch which was rejected upstream exactly because such peer-to-peer interactions can be unpredictable and difficult to debug.


http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Offline

#1403 2014-03-19 22:18:41

ahl
Member
Registered: 2014-03-16
Posts: 20

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

aw wrote:

The IOMMU can only provide isolation and translation for transactions that it sees.  If there's a multifunction device that allows peer-to-peer DMA within the component, then the IOMMU may never see the transaction.  This is the purpose of Access Control Services (ACS) to allow configuration and enforcement of whether all transactions are forwarded upstream.  This is why IOMMU groups put such devices together, so we can at least have a single entity manage them rather than allowing a guest to exploit the host.  Many folks here abuse the ACS override patch which was rejected upstream exactly because such peer-to-peer interactions can be unpredictable and difficult to debug.

But I have never even applied that patch. I'm using 3.14.0-rc7-00004-g9f8b483-dirty + fix_memleaks.patch.

Err, wait. I actually have it applied, but not activated. Surely it does not enable something automatically? I could try with clean source too.

Edit: I meant to test the patch, but I didn't find any documentation on how to evaluate when it might be needed. Like which devices should I try activating?

Last edited by ahl (2014-03-19 22:32:32)

Offline

#1404 2014-03-19 22:31:35

aw
Member
Registered: 2013-10-04
Posts: 921
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

ahl wrote:
aw wrote:

The IOMMU can only provide isolation and translation for transactions that it sees.  If there's a multifunction device that allows peer-to-peer DMA within the component, then the IOMMU may never see the transaction.  This is the purpose of Access Control Services (ACS) to allow configuration and enforcement of whether all transactions are forwarded upstream.  This is why IOMMU groups put such devices together, so we can at least have a single entity manage them rather than allowing a guest to exploit the host.  Many folks here abuse the ACS override patch which was rejected upstream exactly because such peer-to-peer interactions can be unpredictable and difficult to debug.

But I have never even applied that patch. I'm using 3.14.0-rc7-00004-g9f8b483-dirty + fix_memleaks.patch.

Err, wait. I actually have it applied, but not activated. Surely it does not enable something automatically? I could try with clean source too.

As I say, multifunction devices without ACS can do it too and there's not much we can do about it.  We're dealing with lots of consumer grade hardware in more of a precision operating environment.  It's really just pure speculation that this might be playing a factor.  The only way I can think to test would be to assign only one device from any IOMMU group, the other device can be disabled by writing zero to the PCI command register (setpci -s bus:dev.fn COMMAND=0).  That should prevent them from claiming any peer-to-peer traffic.  Test and incrementally add one device at a time.  Again, this is all just speculation, assuming peer-to-peer is occurring, how would you test for it.


http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Offline

#1405 2014-03-19 22:34:24

TripleSpeeder
Member
Registered: 2011-05-02
Posts: 47

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

noctavian wrote:
TripleSpeeder wrote:
Norcoen wrote:

Did anyone succeed using this with the Virtual Machine Managers (http://virt-manager.org/) XML-like configuration and could post it as an example?

I am trying to, but no success so far, see my post.
But it seems some people got it to work: Val532, kaeptnb

PS: This thread is at the same time a goldmine and a total mess :-D

This is my virt-manager config XML for a Windows 7 VM with AMD HD6770 GPU + HDMI, Audio controller and an usb controller passed to it. I made the vm using the virt-manager gui, I manually configured the number of sockets, cores and threads and after installing windows i used the gui to add the video card and various controllers to the vm. Also I have nested virtualization enabled if this makes a difference.

<!--
WARNING: THIS IS AN AUTO-GENERATED FILE. CHANGES TO IT ARE LIKELY TO BE
OVERWRITTEN AND LOST. Changes to this xml configuration should be made using:
  virsh edit Windows-7-Main
or other application using the libvirt API.
-->

<domain type='kvm'>
  <name>Windows-7-Main</name>
  <uuid>fbc03842-e6ea-d94d-eeba-e1dc4da0e1dd</uuid>
  <memory unit='KiB'>6291456</memory>
  <currentMemory unit='KiB'>6291456</currentMemory>
  <vcpu placement='static'>6</vcpu>
  <os>
    <type arch='x86_64' machine='pc-i440fx-1.7'>hvm</type>
    <boot dev='cdrom'/>
    <boot dev='hd'/>
  </os>
  <features>
    <acpi/>
    <apic/>
    <pae/>
  </features>
  <cpu mode='custom' match='exact'>
    <model fallback='allow'>SandyBridge</model>
    <vendor>Intel</vendor>
    <topology sockets='1' cores='6' threads='6'/>
    <feature policy='require' name='pbe'/>
    <feature policy='require' name='tm2'/>
    <feature policy='require' name='est'/>
    <feature policy='require' name='monitor'/>
    <feature policy='require' name='osxsave'/>
    <feature policy='require' name='smx'/>
    <feature policy='require' name='ss'/>
    <feature policy='require' name='vme'/>
    <feature policy='require' name='dtes64'/>
    <feature policy='require' name='ht'/>
    <feature policy='require' name='ds'/>
    <feature policy='require' name='pcid'/>
    <feature policy='require' name='tm'/>
    <feature policy='require' name='pdcm'/>
    <feature policy='require' name='vmx'/>
    <feature policy='require' name='ds_cpl'/>
    <feature policy='require' name='xtpr'/>
    <feature policy='require' name='acpi'/>
  </cpu>
  <clock offset='localtime'/>
  <on_poweroff>destroy</on_poweroff>
  <on_reboot>restart</on_reboot>
  <on_crash>restart</on_crash>
  <devices>
    <emulator>/usr/sbin/qemu-system-x86_64</emulator>
    <disk type='file' device='disk'>
      <driver name='qemu' type='qcow2'/>
      <source file='/var/lib/libvirt/images/windows_7_main.img'/>
      <target dev='vdb' bus='virtio'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x08' function='0x0'/>
    </disk>
    <disk type='file' device='cdrom'>
      <driver name='qemu' type='raw' cache='none'/>
      <target dev='hda' bus='ide'/>
      <readonly/>
      <address type='drive' controller='0' bus='0' target='0' unit='0'/>
    </disk>
    <disk type='file' device='cdrom'>
      <driver name='qemu' type='raw' cache='none'/>
      <target dev='hdc' bus='ide'/>
      <readonly/>
      <address type='drive' controller='0' bus='1' target='0' unit='0'/>
    </disk>
    <controller type='usb' index='0'>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x2'/>
    </controller>
    <controller type='pci' index='0' model='pci-root'/>
    <controller type='ide' index='0'>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x1'/>
    </controller>
    <controller type='sata' index='0'>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x05' function='0x0'/>
    </controller>
    <interface type='bridge'>
      <mac address='52:54:00:fe:3e:b0'/>
      <source bridge='br0'/>
      <model type='virtio'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x0'/>
    </interface>
    <serial type='pty'>
      <target port='0'/>
    </serial>
    <console type='pty'>
      <target type='serial' port='0'/>
    </console>
    <input type='tablet' bus='usb'/>
    <input type='mouse' bus='ps2'/>
    <graphics type='vnc' port='-1' autoport='yes'/>
    <sound model='ich6'>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x04' function='0x0'/>
    </sound>
    <video>
      <model type='vga' vram='9216' heads='1'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x0'/>
    </video>
    <hostdev mode='subsystem' type='pci' managed='yes'>
      <source>
        <address domain='0x0000' bus='0x00' slot='0x1b' function='0x0'/>
      </source>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x07' function='0x0'/>
    </hostdev>
    <hostdev mode='subsystem' type='pci' managed='yes'>
      <source>
        <address domain='0x0000' bus='0x00' slot='0x1a' function='0x0'/>
      </source>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x09' function='0x0'/>
    </hostdev>
    <hostdev mode='subsystem' type='pci' managed='yes'>
      <source>
        <address domain='0x0000' bus='0x01' slot='0x00' function='0x0'/>
      </source>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x0a' function='0x0'/>
    </hostdev>
    <hostdev mode='subsystem' type='pci' managed='yes'>
      <source>
        <address domain='0x0000' bus='0x01' slot='0x00' function='0x1'/>
      </source>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x0b' function='0x0'/>
    </hostdev>
    <memballoon model='virtio'>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x06' function='0x0'/>
    </memballoon>
  </devices>
</domain>

Thanks noctavian! However this is not exactly what I want to achieve. You are passing your GPU as secondary card using standard pci-assign method. But I'd like to use the GPU exclusively, so that SeaBios is already showing the boot screen on the real GPU (Using x-vga=on) and no emulated graphics adapter.

Offline

#1406 2014-03-19 23:00:42

rawuza
Member
Registered: 2013-12-22
Posts: 7

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

aw wrote:

As I say, multifunction devices without ACS can do it too and there's not much we can do about it.

We're dealing with lots of consumer grade hardware in more of a precision operating environment.  It's really just pure speculation that this might be playing a factor.  The only way I can think to test would be to assign only one device from any IOMMU group, the other device can be disabled by writing zero to the PCI command register (setpci -s bus:dev.fn COMMAND=0).  That should prevent them from claiming any peer-to-peer traffic.  Test and incrementally add one device at a time.  Again, this is all just speculation, assuming peer-to-peer is occurring, how would you test for it.

ACS is implemented in the bridge, right? So if my PCIe chipset support ACS (it's an AMD 990FX; how to check?) and I don't use the ACS override patch (which I don't), then this should not be possible, or am I missing something?

I don't understand how passing through the GPU can freeze the host, when the IOMMU should enforce isolation.

Offline

#1407 2014-03-20 00:12:48

Val532
Member
Registered: 2013-11-13
Posts: 35

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

TripleSpeeder wrote:
noctavian wrote:
TripleSpeeder wrote:

I am trying to, but no success so far, see my post.
But it seems some people got it to work: Val532, kaeptnb

PS: This thread is at the same time a goldmine and a total mess :-D

This is my virt-manager config XML for a Windows 7 VM with AMD HD6770 GPU + HDMI, Audio controller and an usb controller passed to it. I made the vm using the virt-manager gui, I manually configured the number of sockets, cores and threads and after installing windows i used the gui to add the video card and various controllers to the vm. Also I have nested virtualization enabled if this makes a difference.

<!--
WARNING: THIS IS AN AUTO-GENERATED FILE. CHANGES TO IT ARE LIKELY TO BE
OVERWRITTEN AND LOST. Changes to this xml configuration should be made using:
  virsh edit Windows-7-Main
or other application using the libvirt API.
-->

<domain type='kvm'>
  <name>Windows-7-Main</name>
  <uuid>fbc03842-e6ea-d94d-eeba-e1dc4da0e1dd</uuid>
  <memory unit='KiB'>6291456</memory>
  <currentMemory unit='KiB'>6291456</currentMemory>
  <vcpu placement='static'>6</vcpu>
  <os>
    <type arch='x86_64' machine='pc-i440fx-1.7'>hvm</type>
    <boot dev='cdrom'/>
    <boot dev='hd'/>
  </os>
  <features>
    <acpi/>
    <apic/>
    <pae/>
  </features>
  <cpu mode='custom' match='exact'>
    <model fallback='allow'>SandyBridge</model>
    <vendor>Intel</vendor>
    <topology sockets='1' cores='6' threads='6'/>
    <feature policy='require' name='pbe'/>
    <feature policy='require' name='tm2'/>
    <feature policy='require' name='est'/>
    <feature policy='require' name='monitor'/>
    <feature policy='require' name='osxsave'/>
    <feature policy='require' name='smx'/>
    <feature policy='require' name='ss'/>
    <feature policy='require' name='vme'/>
    <feature policy='require' name='dtes64'/>
    <feature policy='require' name='ht'/>
    <feature policy='require' name='ds'/>
    <feature policy='require' name='pcid'/>
    <feature policy='require' name='tm'/>
    <feature policy='require' name='pdcm'/>
    <feature policy='require' name='vmx'/>
    <feature policy='require' name='ds_cpl'/>
    <feature policy='require' name='xtpr'/>
    <feature policy='require' name='acpi'/>
  </cpu>
  <clock offset='localtime'/>
  <on_poweroff>destroy</on_poweroff>
  <on_reboot>restart</on_reboot>
  <on_crash>restart</on_crash>
  <devices>
    <emulator>/usr/sbin/qemu-system-x86_64</emulator>
    <disk type='file' device='disk'>
      <driver name='qemu' type='qcow2'/>
      <source file='/var/lib/libvirt/images/windows_7_main.img'/>
      <target dev='vdb' bus='virtio'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x08' function='0x0'/>
    </disk>
    <disk type='file' device='cdrom'>
      <driver name='qemu' type='raw' cache='none'/>
      <target dev='hda' bus='ide'/>
      <readonly/>
      <address type='drive' controller='0' bus='0' target='0' unit='0'/>
    </disk>
    <disk type='file' device='cdrom'>
      <driver name='qemu' type='raw' cache='none'/>
      <target dev='hdc' bus='ide'/>
      <readonly/>
      <address type='drive' controller='0' bus='1' target='0' unit='0'/>
    </disk>
    <controller type='usb' index='0'>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x2'/>
    </controller>
    <controller type='pci' index='0' model='pci-root'/>
    <controller type='ide' index='0'>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x1'/>
    </controller>
    <controller type='sata' index='0'>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x05' function='0x0'/>
    </controller>
    <interface type='bridge'>
      <mac address='52:54:00:fe:3e:b0'/>
      <source bridge='br0'/>
      <model type='virtio'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x0'/>
    </interface>
    <serial type='pty'>
      <target port='0'/>
    </serial>
    <console type='pty'>
      <target type='serial' port='0'/>
    </console>
    <input type='tablet' bus='usb'/>
    <input type='mouse' bus='ps2'/>
    <graphics type='vnc' port='-1' autoport='yes'/>
    <sound model='ich6'>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x04' function='0x0'/>
    </sound>
    <video>
      <model type='vga' vram='9216' heads='1'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x0'/>
    </video>
    <hostdev mode='subsystem' type='pci' managed='yes'>
      <source>
        <address domain='0x0000' bus='0x00' slot='0x1b' function='0x0'/>
      </source>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x07' function='0x0'/>
    </hostdev>
    <hostdev mode='subsystem' type='pci' managed='yes'>
      <source>
        <address domain='0x0000' bus='0x00' slot='0x1a' function='0x0'/>
      </source>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x09' function='0x0'/>
    </hostdev>
    <hostdev mode='subsystem' type='pci' managed='yes'>
      <source>
        <address domain='0x0000' bus='0x01' slot='0x00' function='0x0'/>
      </source>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x0a' function='0x0'/>
    </hostdev>
    <hostdev mode='subsystem' type='pci' managed='yes'>
      <source>
        <address domain='0x0000' bus='0x01' slot='0x00' function='0x1'/>
      </source>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x0b' function='0x0'/>
    </hostdev>
    <memballoon model='virtio'>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x06' function='0x0'/>
    </memballoon>
  </devices>
</domain>

Thanks noctavian! However this is not exactly what I want to achieve. You are passing your GPU as secondary card using standard pci-assign method. But I'd like to use the GPU exclusively, so that SeaBios is already showing the boot screen on the real GPU (Using x-vga=on) and no emulated graphics adapter.

Hi i see you whant to use Virt-Manager to stop and start your VM,
I think i can help you a bit.

This is an old working config xml for Virt-Manager, but i dont know if it work at this moment (i test a lot of different way to passthroug GPU to a VM)

<!--
WARNING: THIS IS AN AUTO-GENERATED FILE. CHANGES TO IT ARE LIKELY TO BE
OVERWRITTEN AND LOST. Changes to this xml configuration should be made using:
  virsh edit Windows-7
or other application using the libvirt API.
-->

<domain type='kvm' xmlns:qemu='http://libvirt.org/schemas/domain/qemu/1.0'>
  <name>Windows-7</name>
  <uuid>652e4382-4dc8-7349-cdbf-359d333aba08</uuid>
  <memory unit='KiB'>8388608</memory>
  <currentMemory unit='KiB'>8388608</currentMemory>
  <memoryBacking>
    <nosharepages/>
  </memoryBacking>
  <vcpu placement='static'>4</vcpu>
  <os>
    <type arch='x86_64' machine='pc-q35-1.7'>hvm</type>
    <boot dev='hd'/>
    <bootmenu enable='yes'/>
    <smbios mode='host'/>
  </os>
  <features>
    <acpi/>
    <apic/>
    <pae/>
  </features>
  <cpu mode='custom' match='exact'>
    <model fallback='allow'>Haswell</model>
    <vendor>Intel</vendor>
    <topology sockets='1' cores='4' threads='1'/>
    <feature policy='require' name='tm2'/>
    <feature policy='require' name='est'/>
    <feature policy='require' name='vmx'/>
    <feature policy='require' name='osxsave'/>
    <feature policy='require' name='smx'/>
    <feature policy='require' name='ss'/>
    <feature policy='require' name='ds'/>
    <feature policy='require' name='vme'/>
    <feature policy='require' name='dtes64'/>
    <feature policy='require' name='abm'/>
    <feature policy='require' name='ht'/>
    <feature policy='require' name='acpi'/>
    <feature policy='require' name='pbe'/>
    <feature policy='require' name='tm'/>
    <feature policy='require' name='pdcm'/>
    <feature policy='require' name='pdpe1gb'/>
    <feature policy='require' name='ds_cpl'/>
    <feature policy='require' name='rdrand'/>
    <feature policy='require' name='f16c'/>
    <feature policy='require' name='xtpr'/>
    <feature policy='require' name='monitor'/>
  </cpu>
  <clock offset='localtime'/>
  <on_poweroff>destroy</on_poweroff>
  <on_reboot>restart</on_reboot>
  <on_crash>restart</on_crash>
  <devices>
    <emulator>/usr/bin/qemu-system-x86_64</emulator>
    <disk type='file' device='disk'>
      <driver name='qemu' type='qcow2' cache='writethrough'/>
      <source file='/home/val/Image HDD/Windows-7-Test.img'/>
      <target dev='sda' bus='sata'/>
      <address type='drive' controller='0' bus='0' target='0' unit='0'/>
    </disk>
    <controller type='pci' index='0' model='pcie-root'/>
    <controller type='pci' index='1' model='dmi-to-pci-bridge'>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x0'/>
    </controller>
    <controller type='pci' index='2' model='pci-bridge'>
      <address type='pci' domain='0x0000' bus='0x01' slot='0x01' function='0x0'/>
    </controller>
    <controller type='usb' index='0' model='ich9-ehci1'>
      <address type='pci' domain='0x0000' bus='0x02' slot='0x01' function='0x7'/>
    </controller>
    <controller type='usb' index='0' model='ich9-uhci1'>
      <master startport='0'/>
      <address type='pci' domain='0x0000' bus='0x02' slot='0x05' function='0x0' multifunction='on'/>
    </controller>
    <controller type='usb' index='0' model='ich9-uhci2'>
      <master startport='2'/>
      <address type='pci' domain='0x0000' bus='0x02' slot='0x05' function='0x1'/>
    </controller>
    <controller type='usb' index='0' model='ich9-uhci3'>
      <master startport='4'/>
      <address type='pci' domain='0x0000' bus='0x02' slot='0x05' function='0x2'/>
    </controller>
    <controller type='sata' index='0'>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x1f' function='0x2'/>
    </controller>
    <interface type='bridge'>
      <mac address='52:54:00:f7:8e:2b'/>
      <source bridge='br0'/>
      <model type='e1000'/>
      <address type='pci' domain='0x0000' bus='0x02' slot='0x03' function='0x0'/>
    </interface>
    <hostdev mode='subsystem' type='usb' managed='yes'>
      <source>
        <vendor id='0x05e3'/>
        <product id='0x0607'/>
      </source>
    </hostdev>
    <hostdev mode='subsystem' type='usb' managed='yes'>
      <source>
        <vendor id='0x046d'/>
        <product id='0xc228'/>
      </source>
    </hostdev>
    <hostdev mode='subsystem' type='usb' managed='yes'>
      <source>
        <vendor id='0x046d'/>
        <product id='0xc229'/>
      </source>
    </hostdev>
    <hostdev mode='subsystem' type='usb' managed='yes'>
      <source>
        <vendor id='0x1532'/>
        <product id='0x002f'/>
      </source>
    </hostdev>
    <memballoon model='none'/>
  </devices>
  <qemu:commandline>
    <qemu:arg value='-device'/>
    <qemu:arg value='ioh3420,bus=pcie.0,addr=1c.0,multifunction=on,port=1,chassis=1,id=root.1'/>
    <qemu:arg value='-device'/>
    <qemu:arg value='vfio-pci,host=01:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on'/>
    <qemu:arg value='-device'/>
    <qemu:arg value='vfio-pci,host=01:00.1,bus=root.1,addr=00.1'/>
  </qemu:commandline>
</domain>

This work for passthrough your graphique card as a primary gpu and use vfio

And it's just an example !!!!

Offline

#1408 2014-03-20 01:55:26

Blind Tree Frog
Member
Registered: 2013-12-31
Posts: 27

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

OK, I'm having a problem and hopefully someone might be able to help....

As my earlier post said, I've got the 3 links from the first post built and installed (linux mainline, seabios, qemu..  done in that order with linux being installed again later to make sure I had pci-stub installed correctly... and I know I don't need seabios anymore, but I didn't know then).

I have my sole video card blacklisted.  On boot I see two or four lines of text and then no more updates.   I then SSH in, bind the video card to vfio and run qemu with this command:

qemu-system-x86_64 -nographic -enable-kvm -M q35 -m 1024 -cpu host \
  -cdrom cb.iso \
  -smp 6,sockets=1,cores=6,threads=1 \
  -bios /usr/share/qemu/bios.bin -vga none \
  -device ioh3420,bus=pcie.0,addr=1c.0,multifunction=on,port=1,chassis=1,id=root.1 \
  -device vfio-pci,host=03:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on \
  -device vfio-pci,host=03:00.1,bus=root.1,addr=00.1

where 'cb.iso' is a crunchbang live cd and '03:00.0' is my video card. I then hear the video card fan spin up and the screen flashes black and I get text dunmping to the screen from the VM.   Every single time it comes up with no bootable device however.   It should be loading the CDROM but it doesn't seem to be checking it.   I tried adding the '-boot order=d' option, but that didn't take and the menu option blows by before I get a chance to attempt to interact.

It will boot off an image for a floppy disk though.  So why can't I read a CDROM? (note, it also failed to read a fedora image and the physical bluray drive)

Offline

#1409 2014-03-20 02:30:18

alexis_evo
Member
Registered: 2013-07-04
Posts: 33

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Blind Tree Frog wrote:

It will boot off an image for a floppy disk though.  So why can't I read a CDROM? (note, it also failed to read a fedora image and the physical bluray drive)

Try the formatting listed in OP.

-drive file=/home/nbhs/windows.iso,id=isocd -device ide-cd,bus=ide.1,drive=isocd

Also try hitting the F12 key while starting to get into the boot menu. If the problem is the time it takes your monitor to activate/display the output (I had this same problem), that should open the boot menu and pause long enough for you to see it.

Edit: #! installer is a dual USB/CD ISO, that may be confusing kvm. Maybe you can try passing it as a raw disk image.

Last edited by alexis_evo (2014-03-20 02:37:48)

Offline

#1410 2014-03-20 02:34:38

alexis_evo
Member
Registered: 2013-07-04
Posts: 33

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Has anyone figured out the issue where the right half of the screen is shifted to the left half (or vice versa)? This has plagued every game I've tried to play in the VM. Doesn't happen as often as it originally did, but it still happens at least once every 4-5 hours of usage. It's very annoying when it happens in dota, as I'm playing with a gigantic handicap for the rest of the game sad.

Screenshot of it occurring in Mirror's Edge http://i.imgur.com/fuD5HkB.jpg

Offline

#1411 2014-03-20 03:09:01

Blind Tree Frog
Member
Registered: 2013-12-31
Posts: 27

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

alexis_evo wrote:
Blind Tree Frog wrote:

It will boot off an image for a floppy disk though.  So why can't I read a CDROM? (note, it also failed to read a fedora image and the physical bluray drive)

Try the formatting listed in OP.

-drive file=/home/nbhs/windows.iso,id=isocd -device ide-cd,bus=ide.1,drive=isocd

Also try hitting the F12 key while starting to get into the boot menu. If the problem is the time it takes your monitor to activate/display the output (I had this same problem), that should open the boot menu and pause long enough for you to see it.

Edit: #! installer is a dual USB/CD ISO, that may be confusing kvm. Maybe you can try passing it as a raw disk image.

The fedora live cd is a regular CD iso though.

The line from the OP works and I can boot in now.  I'm so used to using the '-cdrom' option that I forgot to check the first post for suggestions.  Thanks.


My next step is to figure out why I can't get the usb devices to pass through and this reset error I'm getting on my non-video card devices.  I'm going to post this here in case the answer jumps out to anybody and so I have a reference to check when I'm looking up answers away from my computer.

My qemu command:

qemu-system-x86_64 -nographic -enable-kvm -M q35 -m 1024 -cpu host \
  -drive file=/root/cb.iso,id=isocd -device ide-cd,bus=ide.1,drive=isocd \
  -smp 6,sockets=1,cores=6,threads=1 \
  -bios /usr/share/qemu/bios.bin -vga none \
  -device ioh3420,bus=pcie.0,addr=1c.0,multifunction=on,port=1,chassis=1,id=root.1 \
  -device vfio-pci,host=03:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on \
  -device vfio-pci,host=03:00.1,bus=root.1,addr=00.1 \
  -device vfio-pci,host=00:12.0,bus=pcie.0 \
  -device vfio-pci,host=00:12.2,bus=pcie.0 \
  -device vfio-pci,host=00:13.0,bus=pcie.0 \
  -device vfio-pci,host=00:13.2,bus=pcie.0 \
  -device vfio-pci,host=00:14.0,bus=pcie.0 \
  -device vfio-pci,host=00:14.2,bus=pcie.0 \
  -device vfio-pci,host=00:14.3,bus=pcie.0 \
  -device vfio-pci,host=00:14.4,bus=pcie.0 \
  -device vfio-pci,host=00:14.5,bus=pcie.0 \
  -device vfio-pci,host=00:16.0,bus=pcie.0 \
  -device vfio-pci,host=00:16.2,bus=pcie.0 \
  -device vfio-pci,host=07:00.0,bus=pcie.0 \

Output of it being run:

root@Pandora ~# sh qemu-test.sh
qemu-system-x86_64: vfio: Cannot reset device 0000:00:16.0, no available reset mechanism.
qemu-system-x86_64: vfio: Cannot reset device 0000:00:14.5, no available reset mechanism.
qemu-system-x86_64: vfio: Cannot reset device 0000:00:14.4, no available reset mechanism.
qemu-system-x86_64: vfio: Cannot reset device 0000:00:14.3, no available reset mechanism.
qemu-system-x86_64: vfio: Cannot reset device 0000:00:14.0, no available reset mechanism.
qemu-system-x86_64: vfio: Cannot reset device 0000:00:13.0, no available reset mechanism.
qemu-system-x86_64: vfio: Cannot reset device 0000:00:12.0, no available reset mechanism.
qemu-system-x86_64: vfio: Cannot reset device 0000:00:16.0, no available reset mechanism.
qemu-system-x86_64: vfio: Cannot reset device 0000:00:14.5, no available reset mechanism.
qemu-system-x86_64: vfio: Cannot reset device 0000:00:14.4, no available reset mechanism.
qemu-system-x86_64: vfio: Cannot reset device 0000:00:14.3, no available reset mechanism.
qemu-system-x86_64: vfio: Cannot reset device 0000:00:14.0, no available reset mechanism.
qemu-system-x86_64: vfio: Cannot reset device 0000:00:13.0, no available reset mechanism.
qemu-system-x86_64: vfio: Cannot reset device 0000:00:12.0, no available reset mechanism.

lspci output:

00:00.0 Host bridge: Advanced Micro Devices, Inc. [AMD/ATI] RD890 PCI to PCI bridge (external gfx0 port B) (rev 02)
00:00.2 IOMMU: Advanced Micro Devices, Inc. [AMD/ATI] RD990 I/O Memory Management Unit (IOMMU)
00:02.0 PCI bridge: Advanced Micro Devices, Inc. [AMD/ATI] RD890 PCI to PCI bridge (PCI express gpp port B)
00:04.0 PCI bridge: Advanced Micro Devices, Inc. [AMD/ATI] RD890 PCI to PCI bridge (PCI express gpp port D)
00:05.0 PCI bridge: Advanced Micro Devices, Inc. [AMD/ATI] RD890 PCI to PCI bridge (PCI express gpp port E)
00:09.0 PCI bridge: Advanced Micro Devices, Inc. [AMD/ATI] RD890 PCI to PCI bridge (PCI express gpp port H)
00:0a.0 PCI bridge: Advanced Micro Devices, Inc. [AMD/ATI] RD890 PCI to PCI bridge (external gfx1 port A)
00:0b.0 PCI bridge: Advanced Micro Devices, Inc. [AMD/ATI] RD890 PCI to PCI bridge (NB-SB link)
00:11.0 SATA controller: Advanced Micro Devices, Inc. [AMD/ATI] SB7x0/SB8x0/SB9x0 SATA Controller [AHCI mode] (rev 40)
00:12.0 USB controller: Advanced Micro Devices, Inc. [AMD/ATI] SB7x0/SB8x0/SB9x0 USB OHCI0 Controller
00:12.2 USB controller: Advanced Micro Devices, Inc. [AMD/ATI] SB7x0/SB8x0/SB9x0 USB EHCI Controller
00:13.0 USB controller: Advanced Micro Devices, Inc. [AMD/ATI] SB7x0/SB8x0/SB9x0 USB OHCI0 Controller
00:13.2 USB controller: Advanced Micro Devices, Inc. [AMD/ATI] SB7x0/SB8x0/SB9x0 USB EHCI Controller
00:14.0 SMBus: Advanced Micro Devices, Inc. [AMD/ATI] SBx00 SMBus Controller (rev 42)
00:14.2 Audio device: Advanced Micro Devices, Inc. [AMD/ATI] SBx00 Azalia (Intel HDA) (rev 40)
00:14.3 ISA bridge: Advanced Micro Devices, Inc. [AMD/ATI] SB7x0/SB8x0/SB9x0 LPC host controller (rev 40)
00:14.4 PCI bridge: Advanced Micro Devices, Inc. [AMD/ATI] SBx00 PCI to PCI Bridge (rev 40)
00:14.5 USB controller: Advanced Micro Devices, Inc. [AMD/ATI] SB7x0/SB8x0/SB9x0 USB OHCI2 Controller
00:15.0 PCI bridge: Advanced Micro Devices, Inc. [AMD/ATI] SB700/SB800/SB900 PCI to PCI bridge (PCIE port 0)
00:15.1 PCI bridge: Advanced Micro Devices, Inc. [AMD/ATI] SB700/SB800/SB900 PCI to PCI bridge (PCIE port 1)
00:15.2 PCI bridge: Advanced Micro Devices, Inc. [AMD/ATI] SB900 PCI to PCI bridge (PCIE port 2)
00:15.3 PCI bridge: Advanced Micro Devices, Inc. [AMD/ATI] SB900 PCI to PCI bridge (PCIE port 3)
00:16.0 USB controller: Advanced Micro Devices, Inc. [AMD/ATI] SB7x0/SB8x0/SB9x0 USB OHCI0 Controller
00:16.2 USB controller: Advanced Micro Devices, Inc. [AMD/ATI] SB7x0/SB8x0/SB9x0 USB EHCI Controller
00:18.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Family 10h Processor HyperTransport Configuration
00:18.1 Host bridge: Advanced Micro Devices, Inc. [AMD] Family 10h Processor Address Map
00:18.2 Host bridge: Advanced Micro Devices, Inc. [AMD] Family 10h Processor DRAM Controller
00:18.3 Host bridge: Advanced Micro Devices, Inc. [AMD] Family 10h Processor Miscellaneous Control
00:18.4 Host bridge: Advanced Micro Devices, Inc. [AMD] Family 10h Processor Link Control
01:00.0 PCI bridge: PLX Technology, Inc. Device 8747 (rev ba)
02:08.0 PCI bridge: PLX Technology, Inc. Device 8747 (rev ba)
02:10.0 PCI bridge: PLX Technology, Inc. Device 8747 (rev ba)
03:00.0 VGA compatible controller: Advanced Micro Devices, Inc. [AMD/ATI] Cape Verde XT [Radeon HD 7770 GHz Edition]
03:00.1 Audio device: Advanced Micro Devices, Inc. [AMD/ATI] Cape Verde/Pitcairn HDMI Audio [Radeon HD 7700/7800 Series]
05:00.0 SATA controller: ASMedia Technology Inc. ASM1062 Serial ATA Controller (rev 01)
06:00.0 SATA controller: ASMedia Technology Inc. ASM1062 Serial ATA Controller (rev 01)
07:00.0 USB controller: ASMedia Technology Inc. ASM1042 SuperSpeed USB Host Controller
0c:00.0 Ethernet controller: Realtek Semiconductor Co., Ltd. RTL8111/8168/8411 PCI Express Gigabit Ethernet Controller (rev 09)
0d:00.0 USB controller: ASMedia Technology Inc. ASM1042 SuperSpeed USB Host Controller
0e:00.0 USB controller: ASMedia Technology Inc. ASM1042 SuperSpeed USB Host Controller

vfio-pci.cfg

DEVICES="0000:00:12.0 0000:00:12.2 0000:00:13.0 0000:00:13.2  0000:00:14.0 0000:00:14.2 0000:00:14.3 0000:00:14.4 0000:00:14.5 0000:00:16.0 0000:00:16.2 0000:03:00.0 0000:03:00.1 0000:07:00.0"

/proc/cmdline

BOOT_IMAGE=/vmlinuz-linux root=UUID=6a991f03-8fd6-42f2-b7be-06eacf457e76 rw quiet iommu=on video=efifb:off pci-stub.ids=1002:4397,1002:4396,1002:4397,1002:4396,1002:4385,1002:4383,1002:4395,1002:4384,1002:4399,1002:4397,1002:4396,1002:683d,1002:aab0

interesting dmesg output that is probably a huge clue glaring at me:

]
[  189.799749] AMD-Vi: Event logged [IO_PAGE_FAULT device=00:12.2 domain=0x0024 address=0x0000000100000000 flags=0x0000]
[  189.800493] AMD-Vi: Event logged [IO_PAGE_FAULT device=00:12.2 domain=0x0024 address=0x00000000f000ff40 flags=0x0000]
[  189.801242] AMD-Vi: Event logged [IO_PAGE_FAULT device=00:12.2 domain=0x0024 address=0x00000000f000ff40 flags=0x0000]

I'm probably missing something obvious... now to find it.

Offline

#1412 2014-03-20 07:42:42

novist
Member
Registered: 2014-03-14
Posts: 47

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

i get reset error too. no idea why. however usb ports work just fine so i figured its not important.

Offline

#1413 2014-03-20 12:16:43

Blind Tree Frog
Member
Registered: 2013-12-31
Posts: 27

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

novist wrote:

i get reset error too. no idea why. however usb ports work just fine so i figured its not important.

I wasn't too worried about them if USB came up, but since my keyboard isn't working in the guest I figure I need to investigate them somewhat.

I'm more concerned about the dmesg error as I am seeing the same three strings repeated a lot.

edit:

i wonder if I'm setting the wrong bus in the qemu command

Last edited by Blind Tree Frog (2014-03-20 12:24:53)

Offline

#1414 2014-03-20 13:28:27

nbhs
Member
From: Montevideo, Uruguay
Registered: 2013-05-02
Posts: 402

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Blind Tree Frog wrote:
novist wrote:

i get reset error too. no idea why. however usb ports work just fine so i figured its not important.

I wasn't too worried about them if USB came up, but since my keyboard isn't working in the guest I figure I need to investigate them somewhat.

I'm more concerned about the dmesg error as I am seeing the same three strings repeated a lot.

edit:

i wonder if I'm setting the wrong bus in the qemu command

You need to passthrough both ehci and ohci controllers

lspci

...
00:16.0 USB controller: Advanced Micro Devices, Inc. [AMD/ATI] SB7x0/SB8x0/SB9x0 USB OHCI0 Controller
00:16.2 USB controller: Advanced Micro Devices, Inc. [AMD/ATI] SB7x0/SB8x0/SB9x0 USB EHCI Controller
...

Offline

#1415 2014-03-20 13:49:53

Blind Tree Frog
Member
Registered: 2013-12-31
Posts: 27

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

nbhs wrote:
Blind Tree Frog wrote:
novist wrote:

i get reset error too. no idea why. however usb ports work just fine so i figured its not important.

...
i wonder if I'm setting the wrong bus in the qemu command

You need to passthrough both ehci and ohci controllers

lspci

...
00:16.0 USB controller: Advanced Micro Devices, Inc. [AMD/ATI] SB7x0/SB8x0/SB9x0 USB OHCI0 Controller
00:16.2 USB controller: Advanced Micro Devices, Inc. [AMD/ATI] SB7x0/SB8x0/SB9x0 USB EHCI Controller
...

I thought I was:
(excerpt from above with newlines added to emphasis grouping)

 -device vfio-pci,host=00:12.0,bus=pcie.0 \
  -device vfio-pci,host=00:12.2,bus=pcie.0 \

  -device vfio-pci,host=00:13.0,bus=pcie.0 \
  -device vfio-pci,host=00:13.2,bus=pcie.0 \

  -device vfio-pci,host=00:14.0,bus=pcie.0 \
  -device vfio-pci,host=00:14.2,bus=pcie.0 \
  -device vfio-pci,host=00:14.3,bus=pcie.0 \
  -device vfio-pci,host=00:14.4,bus=pcie.0 \
  -device vfio-pci,host=00:14.5,bus=pcie.0 \

  -device vfio-pci,host=00:16.0,bus=pcie.0 \
  -device vfio-pci,host=00:16.2,bus=pcie.0 \

Except for device one device I stubbed everything out that was under the same root number and passed them all in to qemu at the same time.  The exception is 0000:07:00.0 which didn't have another.  I originally tried 0e:00.0 and 0d:00.0 as well but that caused all sorts of errors so I figured I'd come back to them.

Edit:

looking at my earlier post...

  -device vfio-pci,host=03:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on \
  -device vfio-pci,host=03:00.1,bus=root.1,addr=00.1

How do I determine what the bus needs to be set to and when do I set the addr field?  In this case I copied the pattern that the OP used, but I figured that there must be a deterministic way.

Off to the man pages I guess.

Last edited by Blind Tree Frog (2014-03-20 14:02:37)

Offline

#1416 2014-03-21 02:07:55

Jesse2004
Member
Registered: 2012-06-25
Posts: 9

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

GNA wrote:

/sys/kernel/iommu_groups is empty.
My system is supporting vt-d and it should be enabled. I dont get it.
Did i miss something obvious? Thanks for help!

I encountered this problem before, and adding intel_iommu=on to the kernel cmdline solved the problem.

Offline

#1417 2014-03-21 12:25:29

ahom
Member
Registered: 2014-03-21
Posts: 2

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

First off I would like to thank everybody contributing to this, it sure is exciting to see all the possibilities of these tools, I'm new to these forums but not new to archlinux and have been lurking around the vga passthrough for a longtime now.

I have a project, which is to have 2 windows VM (with each of them having one dedicated graphic card) running on the same archlinux host for my bf and I. From this thread I'm sure that it works with one, however, before buying all the stuff I need I would have liked some validation on some level that it could maybe work (if it definitely can't there's no point in spending money tongue). From the looks of it I would say it could work but I may be missing something as I never took a look at the actual code.

Here is a summary of the spec I want to buy and what I intend to do with it:

- CPU: AMD FX 8350 Black Edition
- RAM: 32 GB
- MB: I'm not sure yet, but one of
    - ASUS SABERTOOTH 990FX R2.0
    - MSI 990FXA-GD80
    - ASUS CROSSHAIR V FORMULA-Z
- GC: 2 AMD ones for the windows VMs (can't remember the model names), one more very small fanless one for the host
- Other parts are irrelevant to the issue I guess.

The setup of the VMs would be:

- Each Windows VM:
    - 3 cores
    - 12GB ram
    - One discrete GC
- Host:
    - 2 cores
    - 8 GB ram
    - One discrete GC

Here are my thoughts:

    - Will this certainly fail ? wrt passing 2 cards to 2 VM
    - Is there one of these motherboards that will not work properly? (BIOS wise or w/e ?)
    - Does having 16 lines PCIe x16 mandatory for performances on newest games/cards ? (I'm saying this because when you start adding up cards PCI lines satrts to be spread and there are not so much of them in the end for 3 cards, the second card usually gets 8x, it seems that the MSI one can work with 16x/16x/4x but I fear that the slots are not aligned in a way I can have a 2nd 2slot card)

Thanks!

Offline

#1418 2014-03-21 13:52:56

dwe11er
Member
Registered: 2014-03-18
Posts: 73

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

ahom wrote:

    - Does having 16 lines PCIe x16 mandatory for performances on newest games/cards ? (I'm saying this because when you start adding up cards PCI lines satrts to be spread and there are not so much of them in the end for 3 cards, the second card usually gets 8x, it seems that the MSI one can work with 16x/16x/4x but I fear that the slots are not aligned in a way I can have a 2nd 2slot card)
Thanks!

http://www.techpowerup.com/reviews/AMD/ … ng/25.html
http://www.techpowerup.com/reviews/Inte … ng/23.html
(seen few more in other places)

If you ask me, differences are negligible.

Offline

#1419 2014-03-21 14:02:01

Blind Tree Frog
Member
Registered: 2013-12-31
Posts: 27

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

ahom wrote:

Here is a summary of the spec I want to buy and what I intend to do with it:

- CPU: AMD FX 8350 Black Edition
- RAM: 32 GB
- MB: I'm not sure yet, but one of
    - ASUS SABERTOOTH 990FX R2.0
    - MSI 990FXA-GD80
    - ASUS CROSSHAIR V FORMULA-Z
- GC: 2 AMD ones for the windows VMs (can't remember the model names), one more very small fanless one for the host
- Other parts are irrelevant to the issue I guess.

I believe there are some preferences up thread to avoid ASUS mobo's and interest towards Gigabyte.  That said, that was also before ASUS's bios updates last year that seem to address the issues that people were concerned about.

Offline

#1420 2014-03-21 15:27:11

alexis_evo
Member
Registered: 2013-07-04
Posts: 33

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Blind Tree Frog wrote:
ahom wrote:

Here is a summary of the spec I want to buy and what I intend to do with it:

- CPU: AMD FX 8350 Black Edition
- RAM: 32 GB
- MB: I'm not sure yet, but one of
    - ASUS SABERTOOTH 990FX R2.0
    - MSI 990FXA-GD80
    - ASUS CROSSHAIR V FORMULA-Z
- GC: 2 AMD ones for the windows VMs (can't remember the model names), one more very small fanless one for the host
- Other parts are irrelevant to the issue I guess.

I believe there are some preferences up thread to avoid ASUS mobo's and interest towards Gigabyte.  That said, that was also before ASUS's bios updates last year that seem to address the issues that people were concerned about.

ASUS has been wonderful in terms of updating their BIOS to fix support. @GizmoChicken managed to get through to a tech that could get a fix working, gave us test BIOS with the fix, then eventually pushed out the update to everyone.

@ahom: I know the Sabertooth 990FX has a fixed BIOS, and I personally use an ASUS M5A99FX PRO R2.0 that works perfectly with the latest BIOS.

I'd suggest you be careful with how you limit 2 cores to a Windows VM with the FX-8350. Or avoid restricting it all together. The FX-8350 is actually four pairs of cores, with each pair sharing an FPU. In the worst case scenario, the 2 cores will perform as 1 core when the FPU bottlenecks the application in question (common for games). I find it's fine to overcommit CPUs to guests -- I run 6x cores on each guest for two Windows VMs, and use KVM to assign CPU priority to one of them. This gives the most out of the hardware to both VMs, as it isn't that often you'll 100% all four cores in both VMs.

Last edited by alexis_evo (2014-03-21 15:32:25)

Offline

#1421 2014-03-21 17:23:15

aw
Member
Registered: 2013-10-04
Posts: 921
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

novist wrote:

Im adding benchmark that does not make any sense. As you notice DirectCompute and especially DirectX 11 are more than one could wish for. However complex directx9 is right down the toilet. My brain cant figure out any scenario where this would make sense. It is sad because most of the games nowdays are still directx9 neutral
http://i61.tinypic.com/ictrgo.png

Further testing required, but an Nvidia Quadro K4000 (passed as a secondary GPU, no VGA) gives me a Passmark 3D Graphics mark of 2796.  The average for this card is 2797, so that's as good as we can hope for this card (this is without the debug register performance patches, so apparently not an issue on this test).  I'll update with results from Nvidia and Radeon in VGA mode as I'm able to run them.  Unfortunately I still saw the BSOD with passmark, but when it does it the installation gets broken and I'm unable to run it again even after uninstall-reinstall.  It seemed to run ok on a fresh win7x64 install with only the Nvidia driver installed.  No idea how to debug that.


http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Offline

#1422 2014-03-21 20:59:13

andy123
Member
Registered: 2011-11-04
Posts: 169
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

aw wrote:
novist wrote:

Im adding benchmark that does not make any sense. As you notice DirectCompute and especially DirectX 11 are more than one could wish for. However complex directx9 is right down the toilet. My brain cant figure out any scenario where this would make sense. It is sad because most of the games nowdays are still directx9 neutral
http://i61.tinypic.com/ictrgo.png

Further testing required, but an Nvidia Quadro K4000 (passed as a secondary GPU, no VGA) gives me a Passmark 3D Graphics mark of 2796.  The average for this card is 2797, so that's as good as we can hope for this card (this is without the debug register performance patches, so apparently not an issue on this test).  I'll update with results from Nvidia and Radeon in VGA mode as I'm able to run them.  Unfortunately I still saw the BSOD with passmark, but when it does it the installation gets broken and I'm unable to run it again even after uninstall-reinstall.  It seemed to run ok on a fresh win7x64 install with only the Nvidia driver installed.  No idea how to debug that.

The benchmark worked fine for me with my AMD Radeon HD5670 on Windows 8.1 (64). Since I have a strange/disproportional GPU+CPU config, I'm not sure if my results are comparable with what other people upload to the passmark service. In case you need more results, I can paste mine, but I'd like to ensure comparability.


i'm sorry for my poor english wirting skills…

Offline

#1423 2014-03-22 08:34:10

TripleSpeeder
Member
Registered: 2011-05-02
Posts: 47

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

ahom wrote:

First off I would like to thank everybody contributing to this, it sure is exciting to see all the possibilities of these tools, I'm new to these forums but not new to archlinux and have been lurking around the vga passthrough for a longtime now.

I have a project, which is to have 2 windows VM (with each of them having one dedicated graphic card) running on the same archlinux host for my bf and I. From this thread I'm sure that it works with one, however, before buying all the stuff I need I would have liked some validation on some level that it could maybe work (if it definitely can't there's no point in spending money tongue). From the looks of it I would say it could work but I may be missing something as I never took a look at the actual code.

Certainly an interesting project! How do you plan to assign peripherals like keyboard, mouse, etc. to the hosts? You might want to look for a motherboard that has at least 3 distinct USB controllers, so you can pass through one chip to each VM and have one left for the host.

Offline

#1424 2014-03-22 08:46:47

novist
Member
Registered: 2014-03-14
Posts: 47

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

andy123 wrote:
aw wrote:
novist wrote:

Im adding benchmark that does not make any sense. As you notice DirectCompute and especially DirectX 11 are more than one could wish for. However complex directx9 is right down the toilet. My brain cant figure out any scenario where this would make sense. It is sad because most of the games nowdays are still directx9 neutral
http://i61.tinypic.com/ictrgo.png

Further testing required, but an Nvidia Quadro K4000 (passed as a secondary GPU, no VGA) gives me a Passmark 3D Graphics mark of 2796.  The average for this card is 2797, so that's as good as we can hope for this card (this is without the debug register performance patches, so apparently not an issue on this test).  I'll update with results from Nvidia and Radeon in VGA mode as I'm able to run them.  Unfortunately I still saw the BSOD with passmark, but when it does it the installation gets broken and I'm unable to run it again even after uninstall-reinstall.  It seemed to run ok on a fresh win7x64 install with only the Nvidia driver installed.  No idea how to debug that.

The benchmark worked fine for me with my AMD Radeon HD5670 on Windows 8.1 (64). Since I have a strange/disproportional GPU+CPU config, I'm not sure if my results are comparable with what other people upload to the passmark service. In case you need more results, I can paste mine, but I'd like to ensure comparability.

in passmark options i picked other people with same gpu+cpu to compare with. i figured thats best to get a perspective of virtualized performance relative to bare metal. Would be really interesting to see your results.

Offline

#1425 2014-03-22 09:25:50

ahom
Member
Registered: 2014-03-21
Posts: 2

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

dwe11er wrote:

http://www.techpowerup.com/reviews/AMD/ … ng/25.html
http://www.techpowerup.com/reviews/Inte … ng/23.html
(seen few more in other places)

If you ask me, differences are negligible.

That's very good news! That was actually my main concern as both the VMs need to be performant.

alexis_evo wrote:

I'd suggest you be careful with how you limit 2 cores to a Windows VM with the FX-8350. Or avoid restricting it all together. The FX-8350 is actually four pairs of cores, with each pair sharing an FPU. In the worst case scenario, the 2 cores will perform as 1 core when the FPU bottlenecks the application in question (common for games). I find it's fine to overcommit CPUs to guests -- I run 6x cores on each guest for two Windows VMs, and use KVM to assign CPU priority to one of them. This gives the most out of the hardware to both VMs, as it isn't that often you'll 100% all four cores in both VMs.

I think I'll try first what you are suggesting about overcommiting CPUs, that seems like a right balance, if I perceive some issues I'll post here about my findings.

Offline

Board footer

Powered by FluxBB