Arch Linux

rabcor

Banned

Registered: 2013-02-09

Posts: 500

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Apart from that benchmark I linked to before, just for fun (I've always wanted to know the below) I added in an OpenGL windows vs OpenGL linux benchmark against DirectX11 on Unigine, I will blame Unigine for having poorly optimized OpenGL code though ad the max FPS difference on windows was roughly 20 FPS between DX11 and OpenGL. (Writing OpenGL is no joke, I'm not surprised) I wanted to test the mettle of Windows vs Linux nvidia proprietary drivers in OpenGL performance matching the settings as closely as I could, as it turns out the performance was significantly better on Linux than in Windows (unsurprising as OpenGL should be the main focus of the linux drivers whereas DX should be the main focus of the windows drivers) anyways, here we go (All running on the x16 bus)

Linux (3.14.5) driver version: 337.25
Windows (7 x64) driver version: 337.88
Unigine Heaven Settings: Extreme @1920x1080 Fullscreen

Note that while launching Unigine from a terminal in linux spat out for me the OpenGL version (3.2) I have no such output in windows, so I am only assuming windows uses 3.2 as well. I might be wrong however and it might be using 2.0 which would explain the (dramatic) performance differences between windows and linux when using OpenGL. Also note that they claim on the official website to be using 4.0

Nvidia GTX-670 Linux (OpenGL 3.2)

Min FPS:12.1
Max FPS:65.5
FPS:24.9
Score:627

Nvidia GTX-670 Windows (OpenGL 3.2)

Min FPS:6.2
Max FPS:51.6
FPS:20.6
Score:518

Nvidia GTX-670 Windows (DirectX11)

Min FPS:17.1
Max FPS:69.2
FPS:32.0
Score:807

It is worth noting that for the above scores on Linux I disabled flipping in the nvidia drivers. I ran it first with flipping enabled but this resulted in some ugly (what appeared to be) delayed frames which was literally driving me nuts, there was very minimal performance difference with flipping disabled/enabled however as the score and avg FPS were the exact same. (Min and Max FPS differed only slightly as max was 66.4 and min was 6.9 with flipping enabled) however take this as a tip from me to keep flipping disabled on Linux

It is also worth noting that the above benchmarks are probably not good DirectX vs OpenGL benchmarks since there is no telling which code was more optimized in Unigine Heaven (but by the looks of it DX11 was better optimized than OpenGL) Unigine also only seems to use OpenGL 3.2 rather than 4.4 or similar (I assume this was done so it could work on Mac as well but naturally the performance of GL will take a hit from this)

Hopefully I can later add to that the performance of the 670 as it's been passed through to windows rather than from a windows boot.

Update:
GTX 550-Ti results, these were done on the x4 bus but the performance difference between x4 and x16 on this card was so trivial it doesn't really matter (might be like 1FPS difference tops compared to x16)

Unigine Settings: Basic @1920x1080 Fullscreen

GTX 550-Ti Linux (OpenGL 3.2)

FPS:21.0
Score:530
Min FPS:8.7
Max FPS:34.6

GTX 550-Ti Windows (OpenGL 3.2)

FPS:13.9
Score:351
Min FPS:6.4
Max FPS:29.1

GTX 550-Ti Windows (DirectX 11)

FPS:21.2
Score:534
Min FPS:9.8
Max FPS:37.8

What shocked me with these results is that the Linux OpenGL benchmark was actually on par with the DX11 benchmark, I was doing these tests (a few comments below) because I had a suspicion everything wasn't all right in linux on the x4 bus (i.e. I was getting strangely mismatching results between x16 and x4 on there) but that seems to have been little more than a fluke because I was after all running this benchmark with settings way beyond the capabilities of this card (which is how I ended up trying settings the card could actually handle).

The fact that it matched up to DX11 was more than enough to convince me that everything is in fact performing outstandingly well in Linux and I was just worrying over nothing (like I said, a fluke).

It is also interesting to note that on Windows (in both benchmarks) the card would run at 72°C but on Linux the heat would never exceed 60°C. (Meaning it was performing on par with DX11 at lower temperatures)

There are only two reasons I can think of for why OpenGL performed on par with DX11 in this test and why it didn't in the 670 test.

1: The Fermi architecture works better for the Nvidia proprietary drivers on Linux (I find this highly unlikely though as when Nvidia started working at it's hardest ever on the linux drivers, the Kepler architecture which the 600 and 700 series use was already out)
2: OpenGL 3.2 is on par with DX11 in most areas excluding Tessellation which is disabled when the Basic preset is used (This is much likelier)

Last edited by rabcor (2014-06-11 14:46:27)

---

Making Logitech G510 Work on Linux!

aw

Member

Registered: 2013-10-04

Posts: 921

Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Follow up on that, I have not yet tried to pass through the primary card to a virtual machine (it wil be fun to see what happens) but I did decide that no one was giving me a proper answer for my case (outdated benchmarks and whatnot) so I simply tried running a benchmark myself. These are the results. I imagine that the less bandwidth your card is capable of using (for example the 670 has 192.2GB/s bandwidth) the less noticable the difference will be, where the difference starts to be noticable remains to be seen though.

I'm not sure I agree with your hypothesis that installing the card on a x4 electrical slot gives you more overclocking headroom.  I think it shows that that particular test was bus limited and the GPU was data starved and scaling back frequency.  Overclocking the GPU isn't likely to equalize the difference and a GPU-bound workload is still likely to generate the same heat load.  Also, a GTX670 does have a PCIe 3.0 interface (http://www.geforce.com/hardware/desktop … ifications), so if you're limited to PCIe 2.0, it's because of your motherboard/processor combination.

EDIT: ark confirms your i7-2600 only supports PCIe 2.0, I didn't see your motherboard specs in the post history to lookup those specs though

Last edited by aw (2014-06-11 02:15:15)

---

http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

rabcor

Banned

Registered: 2013-02-09

Posts: 500

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

I have an i7-2600 so I guess that's why. I slightly modified that answer to clarify that the overclocking thing was just me guessing. (Thanks for looking into this for me btw)

I have this board really though I just saw it said "PCI Express 3.0" on the board which was how I found out.

It's a shame that my CPU doesn't support pci-express 3.0 >_< god damn that pisses me off considering I would get a lot better performance out of that I think (which would further increase my need to use the primary slot)

As an update on my progress with using the primary card.

Running pci-stub on it works BUT after booting the console will use the stubbed card as it is in the primary slot and what GRUB used. To force X to ignore the card I had to add this to a file in xorg.conf.d/

Section "Device"
    Identifier     "Device0"
    Driver         "nvidia"
    VendorName     "NVIDIA Corporation"
    BoardName      "GeForce GTX 550 Ti"
    BusID          "PCI:2:0:0"
EndSection

I am pretty sure only the "Identifier" "Driver" and "BusID" values are important but I kept the others just for show. This will make the nvidia drivers in X think that the 550-Ti on bus 2:0:0 is my primary card, however when I try to start X with the 670 stubbed currently everything will freeze up. The things I'll try to do to fix this is to install that VGA_ARB patch for the nvidia drivers first, if that's a no go I will enable ACS Override and if that's a no go I'm running short on ideas.

Last edited by rabcor (2014-06-11 02:38:22)

---

Making Logitech G510 Work on Linux!

apporc

Member

Registered: 2014-06-09

Posts: 4

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Hi, siddharta.
To be clear, below is my information:
os: Linux Mint 17
Kernel: 3.15 (with no patches, no i915, no acs_override.) Link: http://kernel.ubuntu.com/~kernel-ppa/ma … c8-utopic/
Qemu: 2.0.0
libvirt: 1.2.2

CPU: Intel(R) Core(TM) i7-4770
GPU: radeon 7870
Motherboard: Asus Z87K

If your hardware is different from mine, it may be the reason why it don't work.
Use the software and hardware above, i almost did nothing, and it just worked.
I used to use pci_stub or radeon blacklist, vfio_bind and config .xml manually like this:

<domain type='kvm' xmlns:qemu='http://libvirt.org/schemas/domain/qemu/1.0'>
  <os>
    <type arch='x86_64' machine='pc-q35-2.0'>hvm</type>
    <boot dev='hd'/>
  </os>

 <qemu:commandline>
    <qemu:arg value='-vga'/>
    <qemu:arg value='none'/>
    <qemu:arg value='-device'/>
    <qemu:arg value='ioh3420,bus=pcie.0,addr=1c.0,multifunction=on,port=1,chassis=1,id=root.1'/>
    <qemu:arg value='-device'/>
    <qemu:arg value='vfio-pci,host=04:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on'/>
    <qemu:arg value='-device'/>
    <qemu:arg value='vfio-pci,host=04:00.1,bus=root.1,addr=00.1'/>
    <qemu:arg value='-bios'/>
    <qemu:arg value='/usr/share/qemu/bios.bin'/>
  </qemu:commandline>

But now all of above turned out to be not necessary.
The kernel/qemu i used now is all from linux mint' repository, with no patch.
I cleared the pci_stub in my kernel command line, just left "intel_iommu=on".
I unblacklisted radeon and fglrx in my /etc/modprobe.d/blacklist.conf.
And i didn't vfio_bind the radeon card on boot, i just removed /etc/init/vfio-bind.conf which i used at first:

# vfio-bind - Binds devices to vfio-pci
#

description "vfio binding devices"

start on runlevel [2345]
stop on runlevel [!2345]

exec /usr/local/bin/vfio-bind 0000:04:00.0 0000:04:00.1

The configuration i am still using is :

echo "options vfio_iommu_type1 allow_unsafe_interrupts=1" > /etc/modprobe.d/vfio_iommu_type1.conf
echo "options kvm ignore_msrs=1" >> /etc/modprobe.d/kvm.conf
#/etc/libvirt/qemu.conf, add:
user = "root"
group = "root"
clear_emulator_capabilities = 0

ls -1 /dev/vfio
#For ever number that appears, ensure its full path appears in the cgroup_device_acl configuration parameter. For example, mine looks like so:
cgroup_device_acl = [
    "/dev/null", "/dev/full", "/dev/zero",
    "/dev/random", "/dev/urandom",
    "/dev/ptmx", "/dev/kvm", "/dev/kqemu",
    "/dev/rtc","/dev/hpet", "/dev/vfio/vfio",
    "/dev/vfio/8"
]

After above, I just used virt-manager to create my win7 vm. I totally use virt-manager to add vga card and usb device to my vm.
Here is my xml:

<domain type='kvm' id='2'>
  <name>win7</name>
  <uuid>6cb6904a-ff53-6038-8718-c82a39eeab19</uuid>
  <memory unit='KiB'>4194304</memory>
  <currentMemory unit='KiB'>4194304</currentMemory>
  <vcpu placement='static'>4</vcpu>
  <resource>
    <partition>/machine</partition>
  </resource>
  <os>
    <type arch='x86_64' machine='pc-i440fx-2.0'>hvm</type>
    <boot dev='hd'/>
    <bootmenu enable='no'/>
  </os>
  <features>
    <acpi/>
    <apic/>
    <pae/>
  </features>
  <cpu mode='custom' match='exact'>
    <model fallback='allow'>Haswell</model>
    <vendor>Intel</vendor>
    <topology sockets='1' cores='2' threads='2'/>
    <feature policy='require' name='tm2'/>
    <feature policy='require' name='est'/>
    <feature policy='require' name='vmx'/>
    <feature policy='require' name='osxsave'/>
    <feature policy='require' name='smx'/>
    <feature policy='require' name='ss'/>
    <feature policy='require' name='ds'/>
    <feature policy='require' name='vme'/>
    <feature policy='require' name='dtes64'/>
    <feature policy='require' name='abm'/>
    <feature policy='require' name='ht'/>
    <feature policy='require' name='acpi'/>
    <feature policy='require' name='pbe'/>
    <feature policy='require' name='tm'/>
    <feature policy='require' name='pdcm'/>
    <feature policy='require' name='pdpe1gb'/>
    <feature policy='require' name='ds_cpl'/>
    <feature policy='require' name='rdrand'/>
    <feature policy='require' name='f16c'/>
    <feature policy='require' name='xtpr'/>
    <feature policy='require' name='monitor'/>
  </cpu>
  <clock offset='localtime'/>
  <on_poweroff>destroy</on_poweroff>
  <on_reboot>restart</on_reboot>
  <on_crash>restart</on_crash>
  <devices>
    <emulator>/usr/bin/kvm</emulator>
    <disk type='file' device='disk'>
      <driver name='qemu' type='qcow2'/>
      <source file='/var/lib/libvirt/images/win7.img'/>
      <target dev='vda' bus='virtio'/>
      <alias name='virtio-disk0'/>
      <address type='pci' domain='0x0000' bus='0x02' slot='0x06' function='0x0'/>
    </disk>
    <disk type='file' device='cdrom'>
      <driver name='qemu' type='raw'/>
      <source file='/home/apporc/Software/ISO/virtio-win-0.1-65.iso'/>
      <target dev='hdc' bus='ide'/>
      <readonly/>
      <alias name='ide0-1-0'/>
      <address type='drive' controller='0' bus='1' target='0' unit='0'/>
    </disk>
    <controller type='usb' index='0'>
      <alias name='usb0'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x2'/>
    </controller>
    <controller type='pci' index='0' model='pci-root'>
      <alias name='pci.0'/>
    </controller>
    <controller type='pci' index='1' model='pci-bridge'>
      <alias name='pci.1'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x0'/>
    </controller>
    <controller type='pci' index='2' model='pci-bridge'>
      <alias name='pci.2'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x0'/>
    </controller>
    <controller type='ide' index='0'>
      <alias name='ide0'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x1'/>
    </controller>
    <interface type='bridge'>
      <mac address='52:54:00:5a:a6:ef'/>
      <source bridge='br0'/>
      <target dev='vnet0'/>
      <model type='virtio'/>
      <alias name='net0'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x09' function='0x0'/>
    </interface>
    <input type='mouse' bus='ps2'/>
    <input type='keyboard' bus='ps2'/>
    <graphics type='spice' port='5900' autoport='yes' listen='127.0.0.1'>
      <listen type='address' address='127.0.0.1'/>
    </graphics>
    <sound model='ich6'>
      <alias name='sound0'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x05' function='0x0'/>
    </sound>
    <video>
      <model type='qxl' ram='65536' vram='65536' heads='1'/>
      <alias name='video0'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x06' function='0x0'/>
    </video>
    <hostdev mode='subsystem' type='pci' managed='yes'>
      <driver name='vfio'/>
      <source>
        <address domain='0x0000' bus='0x04' slot='0x00' function='0x0'/>
      </source>
      <alias name='hostdev0'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x07' function='0x0'/>
    </hostdev>
    <hostdev mode='subsystem' type='pci' managed='yes'>
      <driver name='vfio'/>
      <source>
        <address domain='0x0000' bus='0x04' slot='0x00' function='0x1'/>
      </source>
      <alias name='hostdev1'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x08' function='0x0'/>
    </hostdev>
    <hostdev mode='subsystem' type='usb' managed='yes'>
      <source>
        <vendor id='0x17ef'/>
        <product id='0x6047'/>
        <address bus='3' device='4'/>
      </source>
      <alias name='hostdev3'/>
    </hostdev>
    <memballoon model='virtio'>
      <alias name='balloon0'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x04' function='0x0'/>
    </memballoon>
  </devices>
  <seclabel type='none'/>
</domain>

The method virt-manager uses is now vfio_pci too, not pci_assign.

Here is another link i used to refer to: http://www.firewing1.com/howtos/fedora- … hrough-kvm.

Good luck!

aw

Member

Registered: 2013-10-04

Posts: 921

Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

I have an i7-2600 so I guess that's why. I slightly modified that answer to clarify that the overclocking thing was just me guessing. (Thanks for looking into this for me btw)

I have this board really though I just saw it said "PCI Express 3.0" on the board which was how I found out.

It's a shame that my CPU doesn't support pci-express 3.0 >_< god damn that pisses me off considering I would get a lot better performance out of that I think (which would further increase my need to use the primary slot)

Z77 only supports PCIe 2.0 as well - http://ark.intel.com/products/64024/Intel-BD82Z77-PCH

I think the PCIe 3.0/2.0 reference in the motherboard spec is referring the the PEG express slot, ie. the one hosted from the CPU.  So if you were to install a 3rd gen Core processor, you'd have PCIe 3.0 for that one slot.

As an update on my progress with using the primary card.

Running pci-stub on it works BUT after booting the console will use the stubbed card as it is in the primary slot and what GRUB used. To force X to ignore the card I had to add this to a file in xorg.conf.d/

Section "Device"
    Identifier     "Device0"
    Driver         "nvidia"
    VendorName     "NVIDIA Corporation"
    BoardName      "GeForce GTX 550 Ti"
    BusID          "PCI:2:0:0"
EndSection

I think requiring BusID is pretty standard for NVIDIA anytime you stray from a standard config.

I am pretty sure only the "Identifier" "Driver" and "BusID" values are important but I kept the others just for show. This will make the nvidia drivers in X think that the 550-Ti on bus 2:0:0 is my primary card, however when I try to start X with the 670 stubbed currently everything will freeze up. The things I'll try to do to fix this is to install that VGA_ARB patch for the nvidia drivers first, if that's a no go I will enable ACS Override and if that's a no go too I will start looking into third party solutions (like vgacon)

The host doesn't care about ACS stuff, so that's surely not going to do anything.

---

http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

apporc

Member

Registered: 2014-06-09

Posts: 4

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

About the noise of my Radeon 7870 card. It is really very loud.
Before i tried the guide of this thread, i used fglrx driver, and Radeon 7870 card didn't make noise.
I know if i used radeon driver, the noise will be there. Maybe just as they said, radeon driver doesn't handle the frequency well.

But after pci-stub, why Radeon 7870 is still noisy. I thought when Radeon is got stubed, nobody is using it, and nobody using it should lead to "no noise".
Surpringly it's not. No mater i blacklist radeon/fglrx module or pci-stub Radeon, the noise is all there. Until i boot my guest win7 vm, then win7's ATI driver make it quiet.
My question is, after host os(linux) booted, before guest win7 booted, Is there a way to make Radeon 7870 quiet?
I hope i don't need to keep my the guest win7 always booted, it's only one game machine.

Any suggestion is apreciated.
Thanks in advance.

Last edited by apporc (2014-06-11 03:08:32)

mv0lnicky

Member

Registered: 2014-03-05

Posts: 16

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Rabcor, is your "4x PCIE" electrical slot directly linked to the CPU or does it go through the the Z77 PCH? If it goes through the controller hub, it would explain why you're noticing a huge performance because you're passing through what used to be the south bridge. DMI 2.0 is limited to 20gbit/s. If it was directly linked to the CPU, I doubt you'd see a substantial difference in benchmarks. If your motherboard supports SLI or Crossfire, use the slots indicated for this by your manufacturer because they are the ones with direct access to the northbridge that is part of the CPU now.

I've seen the effect this has on graphic cards and it's quite noticeable with simple desktop compositing (Aero, Compiz, Mutter).

For reference, see : http://www.anandtech.com/show/5728/inte … nd-biostar

Last edited by mv0lnicky (2014-06-11 06:11:18)

rabcor

Banned

Registered: 2013-02-09

Posts: 500

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

VGA_ARB patch did not seem to help (I suspect it is needed either way) is it normal that xorg.log will say this when the VGA arb patch is enabled?

[    10.381] (WW) NVIDIA(0): [DRI2] Direct rendering is not supported when VGA arb is necessary for the device
[    10.381] (II) NVIDIA(0): The X server will not be able to send the VDPAU driver name to
[    10.381] (II) NVIDIA(0):     libvdpau.
[    10.381] (--) RandR disabled
[    10.381] (II) Found 2 VGA devices: arbiter wrapping enabled

I thought it disabled arbiter wrapping entirely.

What I think I need is a method for the kernel to load my secondary GPU as if it were my primary one. Think it's possible?

Edit: I found a way, my motherboard supports selecting a main display, it offers

iGPU
PCIe
PCI

Choosing PCI resulted in the motherboard reading my GTX 550 rather than my 670 as the primary card even if it is in the x4 bus. It would be neat to have an alternative way to do this in case someone has to deal with a motherboard that doesn't support doing this. (And I still need an answer to that VGA Arb question please)

Last edited by rabcor (2014-06-11 05:01:43)

---

Making Logitech G510 Work on Linux!

aw

Member

Registered: 2013-10-04

Posts: 921

Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

About the noise of my Radeon 7870 card. It is really very loud.
Before i tried the guide of this thread, i used fglrx driver, and Radeon 7870 card didn't make noise.
I know if i used radeon driver, the noise will be there. Maybe just as they said, radeon driver doesn't handle the frequency well.

But after pci-stub, why Radeon 7870 is still noisy. I thought when Radeon is got stubed, nobody is using it, and nobody using it should lead to "no noise".
Surpringly it's not. No mater i blacklist radeon/fglrx module or pci-stub Radeon, the noise is all there. Until i boot my guest win7 vm, then win7's ATI driver make it quiet.
My question is, after host os(linux) booted, before guest win7 booted, Is there a way to make Radeon 7870 quiet?
I hope i don't need to keep my the guest win7 always booted, it's only one game machine.

Sadly, that's probably the solution.  Without a driver that knows how to do power management, the cooling device on the card just runs at a default speed.  Ideally devices would turn the fan off (like AMD ZeroCore) or at least to a minimum level when put into the D3 power state, but they don't (I've tried).  It's probably still worthwhile to add some basic power management to vfio-pci to put devices in D3 while not in use, it saves 2-3 Watts on my system, but it's still quieter to run a guest with a driver that can silence the fan.

Last edited by aw (2014-06-11 05:09:23)

---

http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

mv0lnicky

Member

Registered: 2014-03-05

Posts: 16

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Edit: I found a way, my motherboard supports selecting a main display, it offers

iGPU
PCIe
PCI

Choosing PCI resulted in the motherboard reading my GTX 550 rather than my 670 as the primary card even if it is in the x4 bus. It would be neat to have an alternative way to do this in case someone has to deal with a motherboard that doesn't support doing this. (And I still need an answer to that VGA Arb question please)

PCI BIOS setting pretty much confirms what I said in my previous post. Your 4x slot is most likely routed through DMI 2.0. I don't think that the results of your research you linked in one of your previous post are accurate, seeing as you're hitting a major bottleneck in the chipset configuration. And if you had PCIE3.0 slots, or even 4.0 slots, you'd have the same problem.

How many physical 16x PCIE ports are present on your motherboard?

Last edited by mv0lnicky (2014-06-11 06:14:24)

ttouch

Member

From: /dev/null

Registered: 2012-05-27

Posts: 130

Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Umm, qemu adds an outer frame.
I change the resolution to 1920x1080, but the outer black frame remains. any advice?

---

GitHub | Git Server | Blog

mv0lnicky

Member

Registered: 2014-03-05

Posts: 16

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Umm, qemu adds an outer frame.
I change the resolution to 1920x1080, but the outer black frame remains. any advice?

You mean like underscan?

rabcor

Banned

Registered: 2013-02-09

Posts: 500

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Edit: I found a way, my motherboard supports selecting a main display, it offers

iGPU
PCIe
PCI

Choosing PCI resulted in the motherboard reading my GTX 550 rather than my 670 as the primary card even if it is in the x4 bus. It would be neat to have an alternative way to do this in case someone has to deal with a motherboard that doesn't support doing this. (And I still need an answer to that VGA Arb question please)

PCI BIOS setting pretty much confirms what I said in my previous post. Your 4x slot is most likely routed through DMI 2.0. I don't think that the results of your research you linked in one of your previous post are accurate, seeing as you're hitting a major bottleneck in the chipset configuration. And if you had PCIE3.0 slots, or even 4.0 slots, you'd have the same problem.

How many physical 16x PCIE ports are present on your motherboard?

These are my motherboard specs, manual is here too. I have very little idea what you're talking about.

But I did try running a benchmark on the 550-Ti too (I added this to superuser too) and in that case the performance difference was trivial (less than 1FPS) but my results seem like they are accurate from my perspective, matches what I expected before benchmarking (except that I was expecting the 670 to do significantly worse than it did on the x4 bus)

My understanding is that the x4 bus is connected to the CPU by first going through the x16 bus (meaning the x16 bus actually becomes an x12 bus in the process, I might want to add that to the superuser benchmark info as well, but I just don't know anything about this)

Edit: According to the manual the x16 bus card will remain at x16 mode even when a card is in use by the x4 bus. It does not say anything about DMI 2.0 or the PCI Express connecting to the CPU via it, but according to the reference you linked to earlier the pci express x16 is connected directly to the CPU and the pci express x4 is connected to a "pci express switch" which forwards it to the processor (not through the DMI)

I think you're confusing the pci express x16 operating at x4 bandiwdth with the actual pci express x4 bus (which is another thing entirely and much smaller, you wouldn't fit a GeForce card in there)

Edit2: I am however having a quite dubious performance difference for the GTX 550-Ti on linux's OpenGL between the x4 and x16, no really it is fucking amazingly bad.

GTX 550-Ti on x16 (Linux)

FPS:3.5
Score:89
Min FPS:2.6
Max FPS:6.6

GTX 550-Ti on x16(x4) (Linux)

FPS:1.3
Score:33
Min FPS:1.1
Max FPS:2.4

I'm having a hard time believing how huge that difference was (although when I think of it it really is only a 2.2 overall FPS difference, but this I find quite odd I need to investigate this, although it should be considered I am running it on settings the card clearly can't possibly handle so my first step should of course be to try running something it can handle and comparing between windows and linux)

Edit 3: The performance difference on the same settings between Windows and Linux on OpenGL 3.2 with the 550-Ti were so huge I am inclined to believe that that 89 score I got was some kind of fluke (maybe I forgot to enable the FXAA and force x16 anisotropic filtering, I'll look into this as well)

Either way here are my results.

GTX 550-Ti on x16 (Windows)

FPS:13.9
Score:351
Min FPS:6.4
Max FPS:29.1

GTX 550-Ti on x16(x4) (Linux)

FPS:21.0
Score:530
Min FPS:8.7
Max FPS:34.6

Settings were Unigines "basic" preset with 1920x1080 full screen resolution on OpenGL renderer. What I was doing here (as I am too lazy to put the 550-Ti back in the x16 slot it's just too much trouble to switch the cards up) was testing to confirm if the performance in Windows on OpenGL on this card wasn't still lower than on Linux. This benchmark was enough to convince me that the card is functioning at full efficiency (or like 99% since I think the x4 bus chips just a tiny little bit of performance away from this card, nothing as much as on the 670) on linux and I must have made a mistake in my settings somewhere to get so different results earlier. The performance difference was much bigger than between the 670 on linux and windows (meaning it was even higher than I expected and hoped it would be)

I am doing this because I know the card is working just as well on windows on this x4 bus as it was on the x16 bus (I triple checked with DX11 and it is indeed just getting roughly the same results as it did on the x16 pass)

One thing I did notice however is that 550-Ti on OpenGL on windows ran 10°C (72°C) hotter than it ever did on linux(60°C)  so it actually may be that the performance on linux is Slightly borked somehow (I mean this is the same behavior as I got when I switched 670 between x16 and x4, except that this isn't between buses but between operating systems, and possibly drivers, might be VGA arb related)

Edit 4: Just to check I tried installing the Nvidia drivers again (without the VGA arb patch) to see if they would perform any differently and well they did to a point. It seems like everything runs the same way as it did when I scored 89 earlier. except that after an X amount of time (which actually varies) it slows down and everything drops to around 1 FPS (temperature also refuses to exceed 60, perhaps that is the problem? perhaps nvidia's linux drivers don't want this card to exceed 60°C whereas the windows ones cap it at 70°C instead?)

Ah well I will give up on it, but I am pretty sure this is temperature and power management related, if not it must be a problem with the program's code somewhere (or drivers) at least on settings the card actually can handle it performs just as well as I would have expected it to.

Edit 5: Interesting thing to add is that the 550-Ti on OpenGL 3.2 in Unigine Heaven is actually on par with DX11

GTX 550-Ti on x16(x4) (Windows, DX11)

FPS:21.2
Score:534
Min FPS:9.8
Max FPS:37.8

Last edited by rabcor (2014-06-11 10:38:30)

---

Making Logitech G510 Work on Linux!

aw

Member

Registered: 2013-10-04

Posts: 921

Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

rabcor: refer back to your finding that DRI is disabled when multiple graphics cards are installed.  The difference you're seeing is a software limitation, not hardware.

mv0lnicky: DMI is more than sufficient to handle a x4 PCIe link.

---

http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

rabcor

Banned

Registered: 2013-02-09

Posts: 500

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

But I am technically only using one card. And DRI3 gave me no error (meaning DRI3 is working) but if you're referring to the benchmarks i was doing above that had me baffled, the one that scored twice as high as the other GL benchmarks on the same settings was taken with the same setup (with my 670 stubbed)

Last edited by rabcor (2014-06-11 14:51:47)

---

Making Logitech G510 Work on Linux!

mv0lnicky

Member

Registered: 2014-03-05

Posts: 16

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

P8Z77-V LX link -  I have very little idea what you're talking about.

My understanding is that the x4 bus is connected to the CPU by first going through the x16 bus (meaning the x16 bus actually becomes an x12 bus in the process, I might want to add that to the superuser benchmark info as well, but I just don't know anything about this)

Edit: According to the manual the x16 bus card will remain at x16 mode even when a card is in use by the x4 bus. It does not say anything about DMI 2.0 or the PCI Express connecting to the CPU via it, but according to the reference you linked to earlier the pci express x16 is connected directly to the CPU and the pci express x4 is connected to a "pci express switch" which forwards it to the processor (not through the DMI)

I think you're confusing the pci express x16 operating at x4 bandiwdth with the actual pci express x4 bus (which is another thing entirely and much smaller, you wouldn't fit a GeForce card in there)

Let me get this straight, when you're referring to 4x, you're talking about the black colored slot. No I am not talking about the little PCIe slots (they are actually 1x slots on your board).

Asus does not support multi-gpu (SLI/Xfire) on the LX motherboard as a result of product differentiation. From looking at the specs, this board seems artificially limited to running only one full 16x link. (directly attached to the CPU thus, no bottleneck on this one)

The confusion comes from the fact that in the ideal scenario, a board with two 16x PCIe slots - like the P8Z77-V PRO - will run on mode 1 (only one GPU) at full 16x, and on mode 2 (two GPU) at a reduced rate 8x + 8x. This limitation has been existing with Intel motherboards for a long time (think P35/X38 days) but it is not relevant in your case. If you took the same P8Z77-V PRO, stuck a GPU into slot 3, you'd have the same scenario as you currently have n your P8Z77-V LX.

I made a little picture since I guess it's hard to explain in words. http://oi59.tinypic.com/33xkndc.jpg EDIT: I made a bad circling on the the PRO board, the writing is on the PCI slot when it should be on the PCIE slot. Doesn't change anything anyway.

mv0lnicky: DMI is more than sufficient to handle a x4 PCIe link.

It is but DMI shares it's bandwidth with every other I/O (USB, SATA) and is max 20Gbit for everything with a 4x bus. I have not benchmarked it but going from two different PCIe slots, one on the PCH to one of the PCIe switch from the CPU is like night and day when using desktop compositing. Even switching workspace is noticably slower. Almost like a GPU switching power management modes even though they work at 100% clock rate. I suspect that access to system memory is significantly impacted by passing through DMI. For the gazillion SATA and USB ports, DMI is fine, but for serious 3D work, it's terrible.

A "normal" 4x PCIe slot should not be sharing it's entire bandwidth with everything under the sun.

Last edited by mv0lnicky (2014-06-11 15:41:09)

siddharta

Member

Registered: 2014-05-03

Posts: 30

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Hi, siddharta.
But now all of above turned out to be not necessary.
After above, I just used virt-manager to create my win7 vm. I totally use virt-manager to add vga card and usb device to my vm.

The method virt-manager uses is now vfio_pci too, not pci_assign.

Here is another link i used to refer to: http://www.firewing1.com/howtos/fedora- … hrough-kvm.

Good luck!

Thanks apporc, that's very interesting. I also like to use virsh and virt-manager to provision and manage guests. I now have VGA passthrough working (kernel 3.15 with i915 vga arbiter patch), can PCI assign those devices that can't be passed through with vfio (thanks to aw) and have USB device passthrough working again (I'm on OpenSUSE 13.1 and it seems qemu 2.0.0 in the virtualization repository is built with libusb disabled, so I had to rebuild that).

Now I'm struggling with converting everything to libvirt xml which doesn't go smootly... for example, if you provision a new guest through virt-manager, then change the chipset to q35 rather than i440 it complains that the USB controller needs a PCI bus. It's a good idea to try and stick with the virt-manager defined guest and work from there, I'll give that a go.

edit: I spoke in haste. While I don't get an error anymore when starting a guest that has a passthrough USB device in its definition, as was the case with the standard qemu 2.0.0 in OpenSUSE's virtualization repo, even a passthrough USB keyboard doesn't work in a guest. Not certain if the problem is with the updated libusb I had to install to build qemu with --enable-libusb or the qemu I built. This throws another stick in the gearbox (is that the expression?) so I might just see about building the whole thing on Arch as that seems the least painful-ish.

Last edited by siddharta (2014-06-11 17:33:17)

rabcor

Banned

Registered: 2013-02-09

Posts: 500

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Thanks mv0lnicky.

I have a problem, I'm setting up my system again (this time actually with arch as opposed to gentoo last time) I'm using the official arch kernel (but with ACS Override patch) here's what happens when I try to use vfio-bind (and I'm sure this is some really noobish mistake i'm making)

$ dmesg | grep vfio
[    2.613944] vfio-pci: probe of 0000:01:00.0 failed with error -22
[    2.616525] vfio-pci: probe of 0000:01:00.0 failed with error -22
[    2.616536] vfio-pci: probe of 0000:01:00.1 failed with error -22
[    2.619017] vfio-pci: probe of 0000:01:00.0 failed with error -22
[    2.619028] vfio-pci: probe of 0000:01:00.1 failed with error -22
[    2.619037] vfio-pci: probe of 0000:06:00.0 failed with error -22

Here is my cmdline

$ cat /proc/cmdline 
BOOT_IMAGE=/boot/vmlinuz-linux root=UUID=0773b55f-24c9-494a-a43b-db6b45a36a80 rw quiet pci-stub.ids=10de:1189,10de:0e0a pcie_acs_override=downstream

$ cat /etc/modprobe.d/kvm.conf 
options kvm ignore_msrs=1
options vfio_iommu_type1 allow_unsafe_interrupts=1
blacklist snd_ctxfi

(where ctxfi is the mysterious 06:00.0 soundblaster card) I know that pci-stub worked, I know that something fails between pci-stub and the vfio-bind. What am I doing wrong? "lspci -v" should show "kernel driver in use:vfio-pci" should it not? (instead I see no kernel driver line at all for the devices in question,  i get a ">?" sign of some sort instead)

/dev/vfio/vfio exists.

Edit: I think I know what the issue may be.

$ zgrep IOMMU_DEFAULT /proc/config.gz 
# CONFIG_INTEL_IOMMU_DEFAULT_ON is not set

How do I work aorund this? (Is there any other way than to recompile the kernel with the setting enabled?)

Last edited by rabcor (2014-06-11 17:43:19)

---

Making Logitech G510 Work on Linux!

aw

Member

Registered: 2013-10-04

Posts: 921

Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Edit: I think I know what the issue may be.

$ zgrep IOMMU_DEFAULT /proc/config.gz 
# CONFIG_INTEL_IOMMU_DEFAULT_ON is not set

How do I work aorund this? (Is there any other way than to recompile the kernel with the setting enabled?)

Add intel_iommu=on to your kernel boot options

---

http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

ttouch

Member

From: /dev/null

Registered: 2012-05-27

Posts: 130

Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Umm, qemu adds an outer frame.
I change the resolution to 1920x1080, but the outer black frame remains. any advice?

You mean like underscan?

Yes.

---

GitHub | Git Server | Blog

_pheinrich_

Member

Registered: 2014-05-26

Posts: 53

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Hi,

I have the same problem like rabcor.

vfio-bind results in

[root@ARCH iommu_groups]# dmesg | grep vfio
[    0.000000] Command line: BOOT_IMAGE=/boot/vmlinuz-linux root=UUID=bd845dd5-0a31-4cd0-ae57-875c43f15c30 rw intel_iommu=on vfio_iommu_type1.allow_unsafe_interrupts=1 pci-stub.ids=10de:0140 quiet
[    0.000000] Kernel command line: BOOT_IMAGE=/boot/vmlinuz-linux root=UUID=bd845dd5-0a31-4cd0-ae57-875c43f15c30 rw intel_iommu=on vfio_iommu_type1.allow_unsafe_interrupts=1 pci-stub.ids=10de:0140 quiet
[  209.145692] vfio-pci: probe of 0000:02:00.0 failed with error -22
[  473.422577] vfio-pci: probe of 0000:02:00.0 failed with error -22
[  832.559887] vfio-pci: probe of 0000:02:00.0 failed with error -22

I think that intel_iommu is enabled ... see kernel options above.

[root@ARCH tmp]# zgrep  IOMMU /proc/config.gz 
CONFIG_GART_IOMMU=y
CONFIG_CALGARY_IOMMU=y
CONFIG_CALGARY_IOMMU_ENABLED_BY_DEFAULT=y
CONFIG_IOMMU_HELPER=y
CONFIG_VFIO_IOMMU_TYPE1=m
CONFIG_IOMMU_API=y
CONFIG_IOMMU_SUPPORT=y
CONFIG_AMD_IOMMU=y
# CONFIG_AMD_IOMMU_STATS is not set
CONFIG_AMD_IOMMU_V2=m
CONFIG_INTEL_IOMMU=y
# CONFIG_INTEL_IOMMU_DEFAULT_ON is not set
CONFIG_INTEL_IOMMU_FLOPPY_WA=y
# CONFIG_IOMMU_DEBUG is not set
# CONFIG_IOMMU_STRESS is not set

CPU: i5 2500k
MB: Asus P8P67 EVO

aw

Member

Registered: 2013-10-04

Posts: 921

Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

CPU: i5 2500k
MB: Asus P8P67 EVO

Your CPU doesn't support VT-d - http://ark.intel.com/products/52210/Int … o-3_70-GHz

Thanks for playing

nbhs, maybe you could add a link to ark on the first post, we're getting this a lot.

---

http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

mv0lnicky

Member

Registered: 2014-03-05

Posts: 16

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

I'm surprised people can't verify for themselves since the first post mentions :

"NOTE: to get this working you'll need:

    AMD-VI/VT-D enabled and working."

Last edited by mv0lnicky (2014-06-11 21:17:49)

mv0lnicky

Member

Registered: 2014-03-05

Posts: 16

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Umm, qemu adds an outer frame.
I change the resolution to 1920x1080, but the outer black frame remains. any advice?

You mean like underscan?

Yes.

I doubt it's related to QEMU since it is just passing complete control of the hardware to the guest.

What operating system are you running as guest VM? Did you install the drivers for your graphic card?

aw

Member

Registered: 2013-10-04

Posts: 921

Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

I'm surprised people can't verify for themselves since the first post mentions :

"NOTE: to get this working you'll need:

    AMD-VI/VT-D enabled and working."

To be fair, the kernel does like to lie and say things like "Intel-IOMMU: enabled" when actually all it's done is register that you've passed intel_iommu=on.  The actual telling line that nobody seems to get right when trying to prove that it's enabled is "PCI-DMA: Intel(R) Virtualization Technology for Directed I/O".

---

http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users