Arch Linux

slis

Member

Registered: 2014-06-02

Posts: 127

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Hahahaha.

wulfspider

Member

From: Germany, Hesse, Darmstadt

Registered: 2015-05-05

Posts: 13

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Just for notice:

What's the difference between q35 and 440fx (I think this is the default one)?

aw

Member

Registered: 2013-10-04

Posts: 921

Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Just for notice:

What's the difference between q35 and 440fx (I think this is the default one)?

Q35 is a PCI-express chipset from around 2007, i440FX is a conventional PCI chipset from around 1996 used on Pentium Pro systems.  QEMU emulates both of these, 440FX better than Q35, IMO.  If you're running a Windows guest and can use OVMF (UEFI support for GPU and guest OS), then use the default 440FX, it will cause you far fewer problems and you can use libvirt without any hacks.

---

http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

wulfspider

Member

From: Germany, Hesse, Darmstadt

Registered: 2015-05-05

Posts: 13

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

I am trying to look for the EFI support of my graphics card (GTX 680) like this:

# cd /sys/bus/pci/devices/0000:01:00.0/
# echo 1 > rom
# cat rom > /tmp/image.rom
# echo 0 > rom

But I get an Input/Output error at the step with cat. I added the ids to the pci-stub boot parameter and added the nouveau to modprobe blacklist. Should I remove one of them?

aw

Member

Registered: 2013-10-04

Posts: 921

Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

I am trying to look for the EFI support of my graphics card (GTX 680) like this:

# cd /sys/bus/pci/devices/0000:01:00.0/
# echo 1 > rom
# cat rom > /tmp/image.rom
# echo 0 > rom

But I get an Input/Output error at the step with cat. I added the ids to the pci-stub boot parameter and added the nouveau to modprobe blacklist. Should I remove one of them?

Just try a try an OVMF VM with it, if you get the TianoCore splash screen it supports UEFI.

---

http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Duelist

Member

Registered: 2014-09-22

Posts: 358

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

But I get an Input/Output error at the step with cat.

Try using something like nvagetram or how was that software called.. There's a special piece of software to extract nvidia pre-POST ROMs from cards. Or you could get yours from baremetal GPU-z or techpowerup database. Or you can just follow aw's wise advise of just trying to kickstart OVMF without that unnecessary for GTX6XX cards check.

5000!  (sorry, had to do it)

Time to celebrate.

---

The forum rules prohibit requesting support for distributions other than arch.
I gave up. It was too late.
What I was trying to do.
The reference about VFIO and KVM VGA passthrough.

Fira

Member

Registered: 2011-09-24

Posts: 8

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Yo. I just upgraded to kernel 4.0.1, and applied the same old i915 patch....
But it doesn't seem to work anymore, even though the patch was successful.
I'm getting nothing anymore on my card output, and a bunch of "unclaimed register" errors in dmesg coming from i915.

Any tips ? Do anyone has a more recent version of the arbitration patch ?

Thanks

Edit: I've looked into OVMF, but sadly that means a whole system reinstall.
Actually been trying to reinstall on full EFI with OVMF, but I can't manage to boot any install ISO from OVMF.
VGA passthrough works perfect though.

Last edited by Fira (2015-05-05 22:46:04)

aw

Member

Registered: 2013-10-04

Posts: 921

Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Yo. I just upgraded to kernel 4.0.1, and applied the same old i915 patch....
But it doesn't seem to work anymore, even though the patch was successful.
I'm getting nothing anymore on my card output, and a bunch of "unclaimed register" errors in dmesg coming from i915.

Any tips ? Do anyone has a more recent version of the arbitration patch ?

Thanks

Edit: I've looked into OVMF, but sadly that means a whole system reinstall.
Actually been trying to reinstall on full EFI with OVMF, but I can't manage to boot any install ISO from OVMF.
VGA passthrough works perfect though.

Whole guest system re-install.  Well worth it to not be patching every new host kernel you get IMO.  I don't know where these ISO boot problems are coming from, but see the recent series on my blog, part 4 covers installing Windows 8.1 with OVMF using virt-manager.  If you're using Windows 7 with OVMF, I think some people have gotten that to work, but I haven't gotten that far in my howto series, maybe next week.

---

http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Fira

Member

Registered: 2011-09-24

Posts: 8

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

In fact I've done the exact same as you did in your blog, but on QEMU command line directly..
I've tried with Windows 8.1 and 7 images from MSDNAA as you did, and even an ArchLinux ISO.
Every time, OVMF fails to boot both with pure-EFI and CSM, then drops you to the shell indicating there is a valid GPT Label on the disks.
Any ideas ? I didn't have any trouble booting these under Seabios, and AFAIK they should all be EFI-aware.

Bronek

Member

From: London

Registered: 2014-02-14

Posts: 123

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

5000!  (sorry, had to do it)

You worked hard for the privilege :-P

Bronek

Member

From: London

Registered: 2014-02-14

Posts: 123

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

@aw I think my FirePro W7100 is either not meant to work with passthrough at all or is just faulty, since I have occasional artefacts on screen even before restarting the guest and when the card is secondary GPU in the system (i.e. should not be referred by host in any way). I think its not worth your time to try diagnosing my problems, I will switch to something else and put this card to other use.

toxster

Member

Registered: 2015-04-28

Posts: 5

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Hi again,

since my last post i have managed to go fully libvirt! no more vfio-bind scripts and long command lines...

For reference: the Nvidia Controlpanel defaults to vsync on (since i have a g-sync monitor) hence why i capped out at 60fps. I can get higher FPS in games now if i turn it off.

Here is my libvirt (since i was struggling to find a sample to base my own upon) i ended up creating a new machine in virt-manager and the virsh edit <name> to add the OVMF, kvm off, etc, required parts, left to do is trim out the unnecessary parts, serial port, ballooning etc.

<domain type='kvm' id='2'>
  <name>win8.1</name>
  <uuid>d9aaa8dd-6b43-421b-b165-706926e9181e</uuid>
  <memory unit='KiB'>16760832</memory>
  <currentMemory unit='KiB'>16760832</currentMemory>
  <vcpu placement='static' current='8'>32</vcpu>
  <resource>
    <partition>/machine</partition>
  </resource>
  <os>
    <type arch='x86_64' machine='pc-i440fx-2.3'>hvm</type>
    <loader readonly='yes' type='pflash'>/usr/share/ovmf/x64/ovmf_code_x64.bin</loader>
    <nvram template='/usr/share/ovmf/x64/ovmf_vars_x64.bin'>/var/lib/libvirt/qemu/nvram/win8.1_VARS.fd</nvram>
  </os>
  <features>
    <acpi/>
    <apic/>
    <pae/>
    <!-- if using NVIDIA, remove all elements inside of hyperv (Code 43 else) -->
    <hyperv>
    </hyperv>
    <!-- this kvm element was added to be able to use NVIDIA drivers. (Code 43 else) -->
    <kvm>
      <hidden state='on'/>
    </kvm>
  </features>
  <cpu mode='host-passthrough'>
    <topology sockets='1' cores='4' threads='8'/>
  </cpu>
  <clock offset='localtime'>
    <timer name='rtc' tickpolicy='catchup'/>
    <timer name='pit' tickpolicy='delay'/>
    <timer name='hpet' present='no'/>
    <timer name='hypervclock' present='no'/> <!-- NVIDIA: must be set to no -->
  </clock>
  <on_poweroff>destroy</on_poweroff>
  <on_reboot>restart</on_reboot>
  <on_crash>restart</on_crash>
  <pm>
    <suspend-to-mem enabled='no'/>
    <suspend-to-disk enabled='no'/>
  </pm>
  <devices>
    <emulator>/usr/sbin/qemu-system-x86_64</emulator>
    <disk type='block' device='disk'>
      <driver name='qemu' type='raw' cache='none' io='native'/>
      <source dev='/dev/sdb3'/>
      <backingStore/>
      <target dev='sda' bus='scsi'/>
      <alias name='scsi0-0-0-1'/>
      <address type='drive' controller='0' bus='0' target='0' unit='1'/>
    </disk>
    <disk type='block' device='disk'>
      <driver name='qemu' type='raw' cache='none' io='native'/>
      <source dev='/dev/sda'/>
      <backingStore/>
      <target dev='sdb' bus='scsi'/>
      <boot order='1'/>
      <alias name='scsi0-0-0-0'/>
      <address type='drive' controller='0' bus='0' target='0' unit='0'/>
    </disk>
    <disk type='file' device='cdrom'>
      <driver name='qemu' type='raw'/>
      <source file='/root/vm/virtio-win-0.1-100.iso'/>
      <backingStore/>
      <target dev='hda' bus='ide'/>
      <readonly/>
      <alias name='ide0-0-0'/>
      <address type='drive' controller='0' bus='0' target='0' unit='0'/>
    </disk>
    <controller type='usb' index='0' model='ich9-ehci1'>
      <alias name='usb0'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x06' function='0x7'/>
    </controller>
    <controller type='usb' index='0' model='ich9-uhci1'>
      <alias name='usb0'/>
      <master startport='0'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x06' function='0x0' multifunction='on'/>
    </controller>
    <controller type='usb' index='0' model='ich9-uhci2'>
      <alias name='usb0'/>
      <master startport='2'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x06' function='0x1'/>
    </controller>
    <controller type='usb' index='0' model='ich9-uhci3'>
      <alias name='usb0'/>
      <master startport='4'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x06' function='0x2'/>
    </controller>
    <controller type='scsi' index='0' model='virtio-scsi'>
      <alias name='scsi0'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x05' function='0x0'/>
    </controller>
    <controller type='pci' index='0' model='pci-root'>
      <alias name='pci.0'/>
    </controller>
    <controller type='ide' index='0'>
      <alias name='ide0'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x1'/>
    </controller>
    <controller type='virtio-serial' index='0'>
      <alias name='virtio-serial0'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x07' function='0x0'/>
    </controller>
    <interface type='network'>
      <mac address='52:54:00:f8:f3:dd'/>
      <source network='default' bridge='virbr0'/>
      <target dev='vnet0'/>
      <model type='virtio'/>
      <alias name='net0'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x0'/>
    </interface>
    <hostdev mode='subsystem' type='pci' managed='yes'>
      <driver name='vfio'/>
      <source>
        <address domain='0x0000' bus='0x03' slot='0x00' function='0x0'/>
      </source>
      <alias name='hostdev0'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x0'/>
    </hostdev>
    <hostdev mode='subsystem' type='usb' managed='yes'>
      <source>
        <vendor id='0x05ac'/>
        <product id='0x1006'/>
        <address bus='1' device='6'/>
      </source>
      <alias name='hostdev1'/>
    </hostdev>
    <hostdev mode='subsystem' type='usb' managed='yes'>
      <source>
        <vendor id='0x046d'/>
        <product id='0x0a01'/>
        <address bus='1' device='2'/>
      </source>
      <alias name='hostdev2'/>
    </hostdev>
    <hostdev mode='subsystem' type='usb' managed='yes'>
      <source>
        <vendor id='0x046d'/>
        <product id='0xc07d'/>
        <address bus='1' device='5'/>
      </source>
      <alias name='hostdev3'/>
    </hostdev>
    <hostdev mode='subsystem' type='usb' managed='yes'>
      <source>
        <vendor id='0x05ac'/>
        <product id='0x0250'/>
        <address bus='1' device='7'/>
      </source>
      <alias name='hostdev4'/>
    </hostdev>
    <memballoon model='virtio'>
      <alias name='balloon0'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x08' function='0x0'/>
    </memballoon>
  </devices>
</domain>

I hope this helps anyone..

Is there anything i can add to increase performance? Huge pages?

slis

Member

Registered: 2014-06-02

Posts: 127

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Yes hugepages,
and change <topology sockets='1' cores='4' threads='8'/> to <topology sockets='1' cores='8' threads='1'/> or something like that, using threads slows cpu down atleast for me...

and you should do static pinning on real cores no ht ones, and maybe emulatorpin, read more at

https://libvirt.org/formatdomain.html#elementsCPUTuning

Bronek

Member

From: London

Registered: 2014-02-14

Posts: 123

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Hi again,

since my last post i have managed to go fully libvirt! no more vfio-bind scripts and long command lines...

For reference: the Nvidia Controlpanel defaults to vsync on (since i have a g-sync monitor) hence why i capped out at 60fps. I can get higher FPS in games now if i turn it off.

Here is my libvirt (since i was struggling to find a sample to base my own upon) i ended up creating a new machine in virt-manager and the virsh edit <name> to add the OVMF, kvm off, etc, required parts, left to do is trim out the unnecessary parts, serial port, ballooning etc.

<domain type='kvm' id='2'>
  <name>win8.1</name>
  <uuid>d9aaa8dd-6b43-421b-b165-706926e9181e</uuid>
  <memory unit='KiB'>16760832</memory>
  <currentMemory unit='KiB'>16760832</currentMemory>
  <vcpu placement='static' current='8'>32</vcpu>
. . .

I hope this helps anyone..

Is there anything i can add to increase performance? Huge pages?

Yes go with hugepages. Also, giving your VM 32 vcpus will slow it down quite a bit, unless you actually have this many threads in host machine to give away!

nbhs

Member

From: Montevideo, Uruguay

Registered: 2013-05-02

Posts: 402

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

5000!  (sorry, had to do it)

And it's also exaclty 2 years old

Duelist

Member

Registered: 2014-09-22

Posts: 358

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

5000!  (sorry, had to do it)

And it's also exaclty 2 years old

That means a thousand posts per year. Not so much, if you think about it.

---

The forum rules prohibit requesting support for distributions other than arch.
I gave up. It was too late.
What I was trying to do.
The reference about VFIO and KVM VGA passthrough.

aw

Member

Registered: 2013-10-04

Posts: 921

Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

5000!  (sorry, had to do it)

And it's also exaclty 2 years old

That means a thousand posts per year. Not so much, if you think about it.

MATHS!

---

http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Duelist

Member

Registered: 2014-09-22

Posts: 358

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

And it's also exaclty 2 years old

That means a thousand posts per year. Not so much, if you think about it.

MATHS!

Crap. 2500. Still not so much.

Damn, that was awkward.

BTW, 772 of them are yours, 234 are mine.

Aaaalso, cruising through your blogposts.. libvirt doesn't support x-vga YET? Or there was a claim that it won't support it at all? That part 5 with selinux looks strange..

Last edited by Duelist (2015-05-06 19:04:45)

---

The forum rules prohibit requesting support for distributions other than arch.
I gave up. It was too late.
What I was trying to do.
The reference about VFIO and KVM VGA passthrough.

Fira

Member

Registered: 2011-09-24

Posts: 8

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

@aw: I looked further into your post and checked my system and... it appears that EDK2/OVMF doesn't detect the FAT filesystems on my ISOs. I'm only getting BLK mappings, no FS mappings, and therefore can't boot since it can't read EFI partitions..
Could you tell me which version you've been using ? I believe i'm on the latest from Jenkins. Stable releases don't appear to boot at all here.

Duelist

Member

Registered: 2014-09-22

Posts: 358

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Just found an article with benchmarks of Citrix XenApp+Grid K2 vGPU.

BEWARE, IT'S IN RUSSIAN.

Anyway, there's 1131/1183 pts in Unigine on GTX 650, and ~1400pts on K2 per instance.
So, like, get an expensive K2, and you can do like 4 VMs comparable in performance as... my setup on VFIO+2 HD7750. I get ~1200 when using crossfire. Duct-tape TightVNC server to it, provide tons of bandwidth, and voila.

---

The forum rules prohibit requesting support for distributions other than arch.
I gave up. It was too late.
What I was trying to do.
The reference about VFIO and KVM VGA passthrough.

deniv

Member

Registered: 2013-10-16

Posts: 27

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

I've seen some people here have GT610 with EFI support. Can anybody share the rom? My card comes only with VGA and I can't contact the vendor's support to get a newer rom from them.

aw

Member

Registered: 2013-10-04

Posts: 921

Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Aaaalso, cruising through your blogposts.. libvirt doesn't support x-vga YET? Or there was a claim that it won't support it at all? That part 5 with selinux looks strange..

The x- in front of the x-vga option is a big warning flag to libvirt to not support this feature because it's experimental.  It's experimental because VGA arbitration is so broken on Linux, mostly with the i915 driver.  I don't see how that's going to get fixed, so I don't have any immediate plans to make it non-experimental and I hope that with UEFI becoming more common, VGA may just go away.  Last I tried the selinux stuff worked, I didn't re-try it for that post though.

---

http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

aw

Member

Registered: 2013-10-04

Posts: 921

Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

@aw: I looked further into your post and checked my system and... it appears that EDK2/OVMF doesn't detect the FAT filesystems on my ISOs. I'm only getting BLK mappings, no FS mappings, and therefore can't boot since it can't read EFI partitions..
Could you tell me which version you've been using ? I believe i'm on the latest from Jenkins. Stable releases don't appear to boot at all here.

I'm on the latest from Gerd's repo too, I did a fresh update before doing that demo.  If you've got it setup like me, I'd suspect your media vs the guest.

---

http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

aw

Member

Registered: 2013-10-04

Posts: 921

Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Just found an article with benchmarks of Citrix XenApp+Grid K2 vGPU.

BEWARE, IT'S IN RUSSIAN.

Anyway, there's 1131/1183 pts in Unigine on GTX 650, and ~1400pts on K2 per instance.
So, like, get an expensive K2, and you can do like 4 VMs comparable in performance as... my setup on VFIO+2 HD7750. I get ~1200 when using crossfire. Duct-tape TightVNC server to it, provide tons of bandwidth, and voila.

A GTX 650 has 384 CUDA cores, a GRID K2 has two separate processor cores, each with 1536 CUDA cores.  Each core is really therefore more equivalent to a GTX 680.  According to the article (google translation), a K280Q configuration is effectively giving each VM a full GPU core, K260Q is half a core, K240Q is a quarter core.  So only in that last config is the core count equivalent to a GTX 650.  Also factor in that a GTX 650 is about $75 USD and a K2 is about $3000 USD and GeForce biting into those GRID margins, if we can run it in a VM, might make you suspect that Nvidia really is trying to prevent GeForce assignment.  But look at it from the other side that it would be completely ridiculous to put eight GTX 650s into a system while a GRID card is only a single double-slot card and you'd think that they'd embrace GeForce assignment as a gateway to a realistic density and far more flexible solution with GRID.

---

http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Duelist

Member

Registered: 2014-09-22

Posts: 358

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Just found an article with benchmarks of Citrix XenApp+Grid K2 vGPU.

BEWARE, IT'S IN RUSSIAN.

Anyway, there's 1131/1183 pts in Unigine on GTX 650, and ~1400pts on K2 per instance.
So, like, get an expensive K2, and you can do like 4 VMs comparable in performance as... my setup on VFIO+2 HD7750. I get ~1200 when using crossfire. Duct-tape TightVNC server to it, provide tons of bandwidth, and voila.

A GTX 650 has 384 CUDA cores, a GRID K2 has two separate processor cores, each with 1536 CUDA cores.  Each core is really therefore more equivalent to a GTX 680.  According to the article (google translation), a K280Q configuration is effectively giving each VM a full GPU core, K260Q is half a core, K240Q is a quarter core.  So only in that last config is the core count equivalent to a GTX 650.  Also factor in that a GTX 650 is about $75 USD and a K2 is about $3000 USD and GeForce biting into those GRID margins, if we can run it in a VM, might make you suspect that Nvidia really is trying to prevent GeForce assignment.  But look at it from the other side that it would be completely ridiculous to put eight GTX 650s into a system while a GRID card is only a single double-slot card and you'd think that they'd embrace GeForce assignment as a gateway to a realistic density and far more flexible solution with GRID.

I agree. I've tried to reply or add something, but failed multiple times.

I've seen some people here have GT610 with EFI support. Can anybody share the rom? My card comes only with VGA and I can't contact the vendor's support to get a newer rom from them.

Using envytools' nvagetbios -s prom, here's my GT610's VBIOS. It's a Gigabyte Gigabyte(sic!) GT610.
After 90 days the file will be automagically deleted, sorry about that.

---

The forum rules prohibit requesting support for distributions other than arch.
I gave up. It was too late.
What I was trying to do.
The reference about VFIO and KVM VGA passthrough.