Arch Linux

aw

Member

Registered: 2013-10-04

Posts: 921

Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

What would be a good choice for a motherboard at the moment, given that I still want to use it for a gaming VM and having at least two pcie x16 slots?

Are both slots for a single VM or split between host and guest?  Do they both need to be PCIe 3.0?  Integrated graphics or discrete for host?  Any other cards to install (for host or guest)?

---

http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

scoobydog

Member

Registered: 2014-04-14

Posts: 5

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Hi, I saw my system is in the Google Doc list, nice :-)

Here are the missing informations about my System:

Linux  3.16.0-4-amd64 #1 SMP Debian 3.16.7-ckt9-3~deb8u1 (2015-04-24) x86_64 GNU/Linux

QEMU emulator version 2.1.2 (Debian 1:2.1+dfsg-11), Copyright (c) 2003-2008 Fabrice Bellard

No special Paches, I'm using the kernel from Debian.

And it's also working with a Asus strix GTX970 OC Graphikcard, see a benchmark of it here https://www.youtube.com/watch?v=v1cFWrKtWis .

apex8

Member

Registered: 2014-03-29

Posts: 60

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

What would be a good choice for a motherboard at the moment, given that I still want to use it for a gaming VM and having at least two pcie x16 slots?

Are both slots for a single VM or split between host and guest?  Do they both need to be PCIe 3.0?  Integrated graphics or discrete for host?  Any other cards to install (for host or guest)?

I guess its not necessary, that both slots are for a single VM. Now I'm curious: does that make a difference? However I think it would be nice if they have PCIe 3.0.
Integrated Graphics is always a nice to have, I think. Two separete SATA controllers would be nice to have. Integrated or as extra card is not so important for me.
Also at the moment I use a dedicated sound card for the VM, but I think there isn't much benefit in this, is it? OK after all in my setup the speaker out goes directly into the line in of the integrated sound device used in the host

Last edited by apex8 (2015-05-16 10:13:41)

Duelist

Member

Registered: 2014-09-22

Posts: 358

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

...because 55w of electricity was equal to zero cents per month....

How do you figure that?  Free electric?  Surplus production?  55 watts * 24 hours = 1.32kW, which is 10 cents per day from my utility even at the lowest tier

Currency exchange rate allows me to say this. And I don't have the lowest prices, but they're quite low for my area. But that currency-exchange-magic works the other way, so i can't really sell my cards or buy a new one. Hence the obsession.

What would be a good choice for a motherboard at the moment, given that I still want to use it for a gaming VM and having at least two pcie x16 slots?

Let's start from the top:
What CPU do you want to use?
AMD? Use whatever with 990FX and tons of pci-e lanes. Intel? AW will give you more advice on this, especially about 9-series chipsets and ACS quirks.

And also... PCI-E 3.0 is needed when you don't have enough bandwidth in PCI-E 2.0. If your current cards run fine on pci-e 2.0 - don't bother. Maaaybe you might get some slight performance increase due to additional bandwidth. Especially you'll notice this if you're using some weird config(like me) or doing crossfire-over-pci-e aka XDMA CF(again, like me).
ATM most GPUs already support PCI-E 3.0, and there's rarely a motherboard-CPU combo without pci-e 3.0.(usually that's AMD)

Dedicated sound card for the VM... I admit, i've set that up too, but that's too much hassle(actually, now i continue using that card because of gameport and a joystick) in mixing host and guest sounds, so i'd suggest you digging to NetJack way of mixing outputs. Since we have localhost tun/tap network, we shouldn't stumble into bandwidth or lag issues, and something hints me that it should work better that embedded pulse audio or alsa stuff we have now. I'm not a linux sound specialist, so YMMV.

And why would you want to have a separate SATA controller? Why?
Virtio-pci gives you A LOT of speed and in some cases you can even benefit from host system caching. Oh noes, SMART and TRIM aren't working? Use virtio-scsi or whatever it was the controller variant.
Usually you'd want to have a separate SATA controller for the VM just to reduce the amount of guest-system-trickery with drivers(like windows having BSOD 7B). Otherwise it's not worth it, IMO.
And also, that controller, if it isn't a separate PCI card, may fall into a clustered IOMMU-group, requesting you to passthrough other stuff that you don't want to passthrough, like USB or ethernet controllers.

And also, i've exceeded my post quota on "And also" usage.

---

The forum rules prohibit requesting support for distributions other than arch.
I gave up. It was too late.
What I was trying to do.
The reference about VFIO and KVM VGA passthrough.

apex8

Member

Registered: 2014-03-29

Posts: 60

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

OK maybe I'm a bit too pessimistic on the disk performance, because currently I already use virt io with data-x-plane and the performance could be better. However the host runs on a SSD and this could be something I could do with the VM too.
At the sound configuration, you mentioned something like NetJack. Can you go into a bit more detail? Currently I'm using pulseaudio, but I suspect this thing of being responsible for trouble with google play movies and audio crackle. Why do I suspect pulseaudio for those issues? Both things worked in the past and the only major changes I did to the system was switching from KDE to LXDE and installing pulseaudio.

Last edited by apex8 (2015-05-16 11:35:42)

aw

Member

Registered: 2013-10-04

Posts: 921

Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

In the first post of my recent how-to series I talk about the hardware I use, not that I'd recommend it, but to talk about why it works for my config - http://vfio.blogspot.com/2015/05/vfio-g … dware.html

I won't be as generous as Duelist, skip AMD platforms.  They've admittedly given up on the single core performance race, you don't know what you're getting on an APU system for isolation, and their latest performance/workstation class chipset is quickly coming up on its four year anniversary since launch.

So that leaves Intel.  I assume that most people do not want to patch their kernel long term, so that means we want a config that avoids the i915 VGA arbiter issues and doesn't require the ACS override patch.  The VGA issue is easy, use a UEFI capable GPU and guest or use discrete graphics for the host.  The isolation problem is more complicated.  The reason I ask about PCIe 3.0 requirements is that PCIe 3.0 slots are pretty much always sourced from the processor root ports rather than the chipset root ports.  If you use a "client" processor, ie. Core i5/i7, or even a Xeon E3-1200 series, there's not going to be isolation between the processor root ports.  That means that whatever you plug into those root port slots either needs to be used exclusively by the host or exclusively by a single VM.  That leaves a lot of workable configs, but you must be conscious of the restrictions.  We have pretty good coverage for isolation of the PCH root ports, 9-series is coming in kernel 4.2, the reset should already be there.  This means that you get finer granularity of what you can assign for devices connected via the PCH root ports, which are generally the PCIe 2.0 slots as well as the integrated components.

If you don't want to sorry about this at all and want the most flexibility in your system config without patches, get a Xeon E5 or better with and X79 or X99 chipset.

If you really want to assign a disk controller to a VM, you're into risky territory, mostly because the cheap ones are all crap.  If you try to assign an integrated controller, the boot ROM for it may be integrated into the system BIOS, so you'll have no way to boot the VM unless you can extract it... and hope it supports UEFI if you're going the OVMF route with the VM.  You may have better luck with a discrete HBA, but you're still in the unknown/crap space.  Personally I find virtio-scsi w/ multi-queue and a fast backing setup to meet my needs.

If you want to assign a NIC, make it an Intel NIC, Realtek NICs are crap for assignment.  Broadcom can be ok, but it's a bit more of a gamble.  For anything else you might want to assign, try to avoid conventional PCI.  All conventional PCI devices are indistinguishable by the IOMMU, so there is no way to split them between host and guest and they often have poor support for interrupt masking.

---

http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Duelist

Member

Registered: 2014-09-22

Posts: 358

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

If you want to assign a NIC, make it an Intel NIC, Realtek NICs are crap for assignment.  Broadcom can be ok, but it's a bit more of a gamble.  For anything else you might want to assign, try to avoid conventional PCI.  All conventional PCI devices are indistinguishable by the IOMMU, so there is no way to split them between host and guest and they often have poor support for interrupt masking.

Oh god, I have Atheros flashbacks. Oh god. No, not the L1 again. Oh god... This card was incompatible with.. everything. Are they dead yet?

And why do you skip AMD platforms? Some people use debian, being the not-so-fresh-but-stable software, why not combine it with 4-years-old-but-already-patched-everywhere AMD chipset?

Some people prefer i5-2500k for it's insane overclocking abilities and cheapness, and it's 3-4 years old too. And what, it makes it a bad motherboard-cpu combo?

---

The forum rules prohibit requesting support for distributions other than arch.
I gave up. It was too late.
What I was trying to do.
The reference about VFIO and KVM VGA passthrough.

aw

Member

Registered: 2013-10-04

Posts: 921

Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

And why do you skip AMD platforms? Some people use debian, being the not-so-fresh-but-stable software, why not combine it with 4-years-old-but-already-patched-everywhere AMD chipset?

Some people prefer i5-2500k for it's insane overclocking abilities and cheapness, and it's 3-4 years old too. And what, it makes it a bad motherboard-cpu combo?

I'm not sure how 2500k is relevant here, it doesn't support VT-d, so for anything in this thread, yes it does make a bad combo.

AMD is just not interesting to me, so certainly factor in some personal bias.  990FX has reasonably good isolation, I have one myself.  Mine is also a dog compared to my Intel boxes.  If you head towards the newer APU-based AMD system, you're in a world of unknown.  We don't know what the isolation capabilities are of the chipset and I don't have a relationship with AMD to try to fix any gaps with quirks.  There's also no resource like ark to determine the various features of a given combo of AMD CPU and chipset.  We also have users reporting that they need to do strange things like disable hugepages on some AMD systems which implies there's some software or hardware problem out there, but we don't have enough data points, or interest, to figure it out.  So no, I can't in good faith recommend an AMD platform.

BTW, I forgot to mention that Gigabyte boards seem to have the best BIOS configuration for primary graphics, so if discrete host graphics was on my radar in choosing a motherboard, it would probably be a Gigabyte board.  I obviously can't guarantee anything there though.

---

http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

ceri

Member

Registered: 2013-10-12

Posts: 57

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

ISo that leaves Intel.  I assume that most people do not want to patch their kernel long term, so that means we want a config that avoids the i915 VGA arbiter issues and doesn't require the ACS override patch.  The VGA issue is easy, use a UEFI capable GPU and guest or use discrete graphics for the host.  The isolation problem is more complicated.  The reason I ask about PCIe 3.0 requirements is that PCIe 3.0 slots are pretty much always sourced from the processor root ports rather than the chipset root ports.  If you use a "client" processor, ie. Core i5/i7, or even a Xeon E3-1200 series, there's not going to be isolation between the processor root ports.  That means that whatever you plug into those root port slots either needs to be used exclusively by the host or exclusively by a single VM.  That leaves a lot of workable configs, but you must be conscious of the restrictions.  We have pretty good coverage for isolation of the PCH root ports, 9-series is coming in kernel 4.2, the reset should already be there.  This means that you get finer granularity of what you can assign for devices connected via the PCH root ports, which are generally the PCIe 2.0 slots as well as the integrated components.

I don't really know/understand the architecture of the interaction between PCI/e ports and the CPU & MB, and so I'm confused about the ACS isolation stuff, but from what you've written here and on your blog, would I be correct in assuming that:

a) if i wanted to use discrete graphics (w/ a UEFI bios) on a single guest, and intel integrated graphics on the host (with no other PCI/PCIe devices), I wouldn't have to worry about using any patches or isolation, nor the type of CPU/motherboard I get (except for CPU/GFX compatibility with the MB and VT-d, etc)?

b) in the case of (a) but also with a PCIe 1.0+ sound card for use on the host (e.g. put in a PCIe 2.0 slot), I wouldn't have to worry about patches or isolation with a pre-9 series chipset (or 9 series from linux 4.2+)?

c) in the case of (a), except that I would be using a second discrete GFX card for the host, I *would* have to worry about patches and isolation?

Last edited by ceri (2015-05-17 01:58:40)

marca311

Member

From: Canada

Registered: 2014-02-25

Posts: 5

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Hi all!

I have been following this thread for several weeks now and after a couple weeks of trying things on my computer, I have hit a brick wall.
Before we get any further, here are my specs:

Arch stock Kernel 4.0.2-1 or 3.19.3-1-ck+ACS override patch
Intel i7-5820k
MSI Gaming 7 X99S Motherboard
Intel X99 chipset
32GB RAM
EVGA 980 Superclocked
QEMU-git (pulled and compiled 16 May 2015)
OVMF-svn (pulled and compiled last week)

I have tried with both libvirt and straight up qemu.
Here is my QEMU file:

#!/bin/bash
vfio-bind 0000:03:00.0 0000:03:00.1
export QEMU_AUDIO_DAC_FIXED_SETTINGS=1
export QEMU_AUDIO_DAC_FIXED_FREQ=44100
export QEMU_AUDIO_DAC_FIXED_FMT=S16
export QEMU_SDL_SAMPLES=1024
qemu-system-x86_64 -enable-kvm -m 8192 -cpu host,kvm=off \
-soundhw hda \
-mem-path /dev/hugepages \
-drive if=pflash,format=raw,readonly,file=/usr/share/ovmf/x64/ovmf_code_x64.bin \
-drive if=pflash,format=raw,file=/usr/share/ovmf/x64/ovmf_vars_x64.bin \
-boot d /home/marcus/qemu/win8.1/c_drive.qcow2 \
-net nic \
-net user,smb=/run/Game-Drive \
-smp 4,sockets=1,cores=4,threads=1 \
-device vfio-pci,host=03:00.0,x-vga=on \
-device vfio-pci,host=03:00.1 \
-vga none \
-usb \
-device nec-usb-xhci,id=xhci \
-usbdevice host:046d:c52f \
-usbdevice host:045e:00b4

And here is my libvirt config:

<domain type='kvm'>
  <name>win8.1</name>
  <uuid>d9aaa8dd-6b43-421b-b165-706926e9181e</uuid>
  <memory unit='KiB'>8290304</memory>
  <currentMemory unit='KiB'>8290304</currentMemory>
  <vcpu placement='static' current='8'>32</vcpu>
  <resource>
    <partition>/machine</partition>
  </resource>
  <os>
    <type arch='x86_64' machine='pc-i440fx-2.3'>hvm</type>
    <loader readonly='yes' type='pflash'>/usr/share/ovmf/x64/ovmf_code_x64.bin</loader>
    <nvram template='/usr/share/ovmf/x64/ovmf_vars_x64.bin'>/var/lib/libvirt/qemu/nvram/win8.1_VARS.fd</nvram>
    <boot dev='hd'/>
  </os>
  <features>
    <acpi/>
    <apic/>
    <pae/>
    <hyperv>
    </hyperv>
    <kvm>
      <hidden state='on'/>
    </kvm>
  </features>
  <cpu mode='host-passthrough'>
    <topology sockets='1' cores='4' threads='8'/>
  </cpu>
  <clock offset='localtime'>
    <timer name='rtc' tickpolicy='catchup'/>
    <timer name='pit' tickpolicy='delay'/>
    <timer name='hpet' present='no'/>
    <timer name='hypervclock' present='no'/>
  </clock>
  <on_poweroff>destroy</on_poweroff>
  <on_reboot>restart</on_reboot>
  <on_crash>restart</on_crash>
  <pm>
    <suspend-to-mem enabled='no'/>
    <suspend-to-disk enabled='no'/>
  </pm>
  <devices>
    <emulator>/usr/sbin/qemu-system-x86_64</emulator>
    <disk type='block' device='cdrom'>
      <driver name='qemu' type='raw'/>
      <target dev='hda' bus='ide'/>
      <readonly/>
      <address type='drive' controller='0' bus='0' target='0' unit='0'/>
    </disk>
    <disk type='file' device='disk'>
      <driver name='qemu' type='qcow2'/>
      <source file='/home/marcus/qemu/win8.1/c_drive.qcow2'/>
      <target dev='hdb' bus='ide'/>
      <address type='drive' controller='0' bus='0' target='0' unit='1'/>
    </disk>
    <disk type='file' device='disk'>
      <driver name='qemu' type='qcow2'/>
      <source file='/home/marcus/qemu/win8.1/Witcher3.qcow2'/>
      <target dev='hdc' bus='ide'/>
      <address type='drive' controller='0' bus='1' target='0' unit='0'/>
    </disk>
    <disk type='file' device='disk'>
      <driver name='qemu' type='qcow2'/>
      <source file='/home/marcus/qemu/win8.1/GTAV.qcow2'/>
      <target dev='hdd' bus='ide'/>
      <address type='drive' controller='0' bus='1' target='0' unit='1'/>
    </disk>
    <controller type='usb' index='0' model='ich9-ehci1'>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x06' function='0x7'/>
    </controller>
    <controller type='usb' index='0' model='ich9-uhci1'>
      <master startport='0'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x06' function='0x0' multifunction='on'/>
    </controller>
    <controller type='usb' index='0' model='ich9-uhci2'>
      <master startport='2'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x06' function='0x1'/>
    </controller>
    <controller type='usb' index='0' model='ich9-uhci3'>
      <master startport='4'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x06' function='0x2'/>
    </controller>
    <controller type='scsi' index='0' model='virtio-scsi'>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x05' function='0x0'/>
    </controller>
    <controller type='pci' index='0' model='pci-root'/>
    <controller type='ide' index='0'>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x1'/>
    </controller>
    <controller type='virtio-serial' index='0'>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x07' function='0x0'/>
    </controller>
    <interface type='network'>
      <mac address='52:54:00:03:6e:f7'/>
      <source network='default'/>
      <model type='rtl8139'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x09' function='0x0'/>
    </interface>
    <sound model='ac97'>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x04' function='0x0'/>
    </sound>
    <sound model='es1370'>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x0a' function='0x0'/>
    </sound>
    <sound model='ich6'>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x0b' function='0x0'/>
    </sound>
    <sound model='ich9'>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x0c' function='0x0'/>
    </sound>
    <sound model='pcspk'/>
    <sound model='sb16'/>
    <hostdev mode='subsystem' type='pci' managed='yes'>
      <source>
        <address domain='0x0000' bus='0x03' slot='0x00' function='0x0'/>
      </source>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x0'/>
    </hostdev>
    <hostdev mode='subsystem' type='pci' managed='yes'>
      <source>
        <address domain='0x0000' bus='0x03' slot='0x00' function='0x1'/>
      </source>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x0'/>
    </hostdev>
    <hostdev mode='subsystem' type='usb' managed='yes'>
      <source>
        <vendor id='0x046d'/>
        <product id='0xc52f'/>
      </source>
    </hostdev>
    <hostdev mode='subsystem' type='usb' managed='yes'>
      <source>
        <vendor id='0x045e'/>
        <product id='0x00b4'/>
      </source>
    </hostdev>
    <memballoon model='virtio'>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x08' function='0x0'/>
    </memballoon>
  </devices>
</domain>

Here is what libvirt starts up

LC_ALL=C PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin QEMU_AUDIO_DRV=none /usr/sbin/qemu-system-x86_64 -name win8.1 -S -machine pc-i440fx-2.3,accel=kvm,usb=off -cpu host,kvm=off -drive file=/usr/share/ovmf/x64/ovmf_code_x64.bin,if=pflash,format=raw,unit=0,readonly=on -drive file=/var/lib/libvirt/qemu/nvram/win8.1_VARS.fd,if=pflash,format=raw,unit=1 -m 8096 -realtime mlock=off -smp 8,maxcpus=32,sockets=1,cores=4,threads=8 -uuid d9aaa8dd-6b43-421b-b165-706926e9181e -nographic -no-user-config -nodefaults -chardev socket,id=charmonitor,path=/var/lib/libvirt/qemu/win8.1.monitor,server,nowait -mon chardev=charmonitor,id=monitor,mode=control -rtc base=localtime,driftfix=slew -global kvm-pit.lost_tick_policy=discard -no-hpet -no-shutdown -global PIIX4_PM.disable_s3=1 -global PIIX4_PM.disable_s4=1 -boot strict=on -device ich9-usb-ehci1,id=usb,bus=pci.0,addr=0x6.0x7 -device ich9-usb-uhci1,masterbus=usb.0,firstport=0,bus=pci.0,multifunction=on,addr=0x6 -device ich9-usb-uhci2,masterbus=usb.0,firstport=2,bus=pci.0,addr=0x6.0x1 -device ich9-usb-uhci3,masterbus=usb.0,firstport=4,bus=pci.0,addr=0x6.0x2 -device virtio-scsi-pci,id=scsi0,bus=pci.0,addr=0x5 -device virtio-serial-pci,id=virtio-serial0,bus=pci.0,addr=0x7 -drive if=none,id=drive-ide0-0-0,readonly=on,format=raw -device ide-cd,bus=ide.0,unit=0,drive=drive-ide0-0-0,id=ide0-0-0 -drive file=/home/marcus/qemu/win8.1/c_drive.qcow2,if=none,id=drive-ide0-0-1,format=qcow2 -device ide-hd,bus=ide.0,unit=1,drive=drive-ide0-0-1,id=ide0-0-1,bootindex=1 -drive file=/home/marcus/qemu/win8.1/Witcher3.qcow2,if=none,id=drive-ide0-1-0,format=qcow2 -device ide-hd,bus=ide.1,unit=0,drive=drive-ide0-1-0,id=ide0-1-0 -drive file=/home/marcus/qemu/win8.1/GTAV.qcow2,if=none,id=drive-ide0-1-1,format=qcow2 -device ide-hd,bus=ide.1,unit=1,drive=drive-ide0-1-1,id=ide0-1-1 -netdev tap,fd=23,id=hostnet0 -device rtl8139,netdev=hostnet0,id=net0,mac=52:54:00:03:6e:f7,bus=pci.0,addr=0x9 -device AC97,id=sound0,bus=pci.0,addr=0x4 -device ES1370,id=sound1,bus=pci.0,addr=0xa -device intel-hda,id=sound2,bus=pci.0,addr=0xb -device hda-duplex,id=sound2-codec0,bus=sound2.0,cad=0 -device ich9-intel-hda,id=sound3,bus=pci.0,addr=0xc -device hda-duplex,id=sound3-codec0,bus=sound3.0,cad=0 -soundhw pcspk -device sb16,id=sound5 -device vfio-pci,host=03:00.0,id=hostdev0,bus=pci.0,addr=0x2 -device vfio-pci,host=03:00.1,id=hostdev1,bus=pci.0,addr=0x3 -device usb-host,hostbus=1,hostaddr=6,id=hostdev2 -device usb-host,hostbus=1,hostaddr=5,id=hostdev3 -device virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x8 -msg timestamp=on

Now then, that's out of the way.
My issue is in a few places.
The biggest one is that when I shut down the VM, the X server on the host freezes and if I try to shut it down, nothing happens.
If I try to shut the computer down, everything just freezes and I have to hard reboot. Finally, I have to shut off my power supply for approximately 15 mins before the motherboard even acknowledges me pressing the power button. I'm wondering if I'm damaging my hardware.

Error I get on shutdown right before the freeze (with qemu only, not libvirt):

qemu-system-x86_64: -device vfio-pci,host=03:00.0,x-vga=on: vfio: error, group 28 is not viable, please ensure all devices within the iommu_group are bound to their vfio bus driver.
qemu-system-x86_64: -device vfio-pci,host=03:00.0,x-vga=on: vfio: failed to get group 28
qemu-system-x86_64: -device vfio-pci,host=03:00.0,x-vga=on: Device initialization failed
qemu-system-x86_64: -device vfio-pci,host=03:00.0,x-vga=on: Device 'vfio-pci' could not be initialized

Yes, I do have the kernel arguments for PCI-Stub set up.

This issue happens with both qemu and libvirt, however qemu requires mouse something on the display to update first before it freezes, libvirt just freezes immediately.
I have been unable to find anything in journalctl or any relevant logs. Any ideas?

Last edited by marca311 (2015-05-17 03:51:00)

Duelist

Member

Registered: 2014-09-22

Posts: 358

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Hi all!

I have been following this thread for several weeks now and after a couple weeks of trying things on my computer, I have hit a brick wall.
Before we get any further, here are my specs:

Arch stock Kernel 4.0.2-1 or 3.19.3-1-ck+ACS override patch
Intel i7-5820k
MSI Gaming 7 X99S Motherboard
Intel X99 chipset
32GB RAM
EVGA 980 Superclocked
QEMU-git (pulled and compiled 16 May 2015)
OVMF-svn (pulled and compiled last week)

..SNIP...

Error I get on shutdown right before the freeze (with qemu only, not libvirt):

qemu-system-x86_64: -device vfio-pci,host=03:00.0,x-vga=on: vfio: error, group 28 is not viable, please ensure all devices within the iommu_group are bound to their vfio bus driver.
qemu-system-x86_64: -device vfio-pci,host=03:00.0,x-vga=on: vfio: failed to get group 28
qemu-system-x86_64: -device vfio-pci,host=03:00.0,x-vga=on: Device initialization failed
qemu-system-x86_64: -device vfio-pci,host=03:00.0,x-vga=on: Device 'vfio-pci' could not be initialized

Yes, I do have the kernel arguments for PCI-Stub set up.

This issue happens with both qemu and libvirt, however qemu requires mouse something on the display to update first before it freezes, libvirt just freezes immediately.
I have been unable to find anything in journalctl or any relevant logs. Any ideas?

List your iommu groups by doing

tree /sys/kernel/iommu_groups/

and check what's in group 28. Or just stuff it all here. Chances are that you have problems with device isolation, needing ACS patch and general 9-series intel chipset support that will be brought upstream in linux 4.2.

Last edited by Duelist (2015-05-17 11:57:11)

---

The forum rules prohibit requesting support for distributions other than arch.
I gave up. It was too late.
What I was trying to do.
The reference about VFIO and KVM VGA passthrough.

aw

Member

Registered: 2013-10-04

Posts: 921

Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

I don't really know/understand the architecture of the interaction between PCI/e ports and the CPU & MB, and so I'm confused about the ACS isolation stuff, but from what you've written here and on your blog, would I be correct in assuming that:

a) if i wanted to use discrete graphics (w/ a UEFI bios) on a single guest, and intel integrated graphics on the host (with no other PCI/PCIe devices), I wouldn't have to worry about using any patches or isolation, nor the type of CPU/motherboard I get (except for CPU/GFX compatibility with the MB and VT-d, etc)?

Yes, if the discrete graphics card for the guest is the only plugin card, then pretty much anything will work.  If your chipset is not quirked for PCH root port isolation, then you'd need to install it in a processor root port slot to keep it from getting grouped with onboard devices, if it is quirked, then any slot will work.

b) in the case of (a) but also with a PCIe 1.0+ sound card for use on the host (e.g. put in a PCIe 2.0 slot), I wouldn't have to worry about patches or isolation with a pre-9 series chipset (or 9 series from linux 4.2+)?

This is also an easy configuration, but you do somewhat need to worry about isolation and which card goes where.  If the guest discrete graphics is installed in a processor root port slot, then the host sound card would need to be installed in a PCH root port slot.  If you have a pre-9 series chipset, both could be installed in PCH root port slots.

c) in the case of (a), except that I would be using a second discrete GFX card for the host, I *would* have to worry about patches and isolation?

It's still achievable without patches, but you're potentially sacrificing performance.  The rule is that Intel Client processors (i5/i7) and Xeon E3-1200 series do not have isolation between the processor root ports.  Therefore anything plugged into those slots will be part of the same IOMMU group.  We cannot have split ownership of an IOMMU group between host and guest.  Most motherboard will only allow you to specify "PCIe" for graphics without slot granularity (Gigabyte seems to be the exception here), which means that you need to use the processor root port slots for your host graphics.  That means any other processor root port slots need to be used for host owned devices, for example you can put your x1 sound card in another processor slot.  The guest discrete graphics would need to use a PCH root port slot which is already quirked for isolation to avoid grouping with onboard devices (pre-9 series until kernel 4.2).  With BIOSes that allow slot selection of primary graphics, there's a bit more freedom that you can choose whether to put the host graphics in the processor or PCH root port, which can favor guest graphics performance.

---

http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

aw

Member

Registered: 2013-10-04

Posts: 921

Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

@Duelist, X99 is not a 9-series chipset, X99 quirks went in for 4.0

@marca311, your log doesn't make much sense, the group not viable error is a fatal qemu error, the VM shouldn't start but you're saying you get that error on shutdown.  I don't see how that's possible.  You say you have the kernel commandline enabled for pci-stub, but is it effective?  Does /proc/cmdline show the option?  Is pci-stub built statically into your kernel?  If pci-stub is a module then it becomes non-deterministic which pci-stub gets a shot at rebinding to the GPU before the native driver, it all depends on which module loaded first.  But event that's still confusing because your raw qemu script doesn't appear to be unbinding devices when finished.  I don't recommend vfio-bind anymore, and that itself should be causing you trouble if it's doing more than binding endpoint devices to vfio-pci.

---

http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

marca311

Member

From: Canada

Registered: 2014-02-25

Posts: 5

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

@marca311, your log doesn't make much sense, the group not viable error is a fatal qemu error, the VM shouldn't start but you're saying you get that error on shutdown.  I don't see how that's possible.  You say you have the kernel commandline enabled for pci-stub, but is it effective?  Does /proc/cmdline show the option?  Is pci-stub built statically into your kernel?  If pci-stub is a module then it becomes non-deterministic which pci-stub gets a shot at rebinding to the GPU before the native driver, it all depends on which module loaded first.  But event that's still confusing because your raw qemu script doesn't appear to be unbinding devices when finished.  I don't recommend vfio-bind anymore, and that itself should be causing you trouble if it's doing more than binding endpoint devices to vfio-pci.

So I double-checked my kernel config and it turns out that CONFIG_PCI_STUB was set to module, so I changed it and recompiled.
I haven't had a crash yet, but I haven't tested as much yet. Thanks so far though!

However this brings me to my next couple of issues: sound and CPU performance.
Running 3DMark gives me bad scores on my CPU (http://www.3dmark.com/3dm/7000311?) (See the physics score, the average for my processor is around 11940).
In addition, the GTA V benchmark gives me scores around 15-20fps on the GTX 980.
I have 4100 2048KB hugepages set up and QEMU is using nearly all of them.
Is this a chipset issue and I just have to be patient for kernel 4.2? Is there a patch other than the ACS one in the OP?

And finally on the sound front, I am getting heavily distorted sound from standalone QEMU setup, I'm using hda for the qemu soundhw option and the following flags:

export QEMU_ALSA_DAC_BUFFER_SIZE=512
export QEMU_ALSA_DAC_PERIOD_SIZE=170
export QEMU_AUDIO_DRV=alsa
export QEMU_AUDIO_DAC_FIXED_SETTINGS=1
export QEMU_AUDIO_DAC_FIXED_FREQ=48000
export QEMU_AUDIO_DAC_FIXED_FMT=S16
export QEMU_SDL_SAMPLES=1024

However on VM boot, I get the following errors:

alsa: Requested buffer size 512 was rejected, using 2048
alsa: Requested period size 170 was rejected, using 1024

Again, both of these are probably small things I'm missing, but I've been working on all these problems combined for a couple of weeks now, so I'm at my wits end.

Last edited by marca311 (2015-05-17 19:46:40)

aw

Member

Registered: 2013-10-04

Posts: 921

Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

@marca311, it looks like you've done no tuning aside from hugepages and host CPU type.  You're using emulated audio, default IDE disk, default rtl network, user mode networking, not attempting to pin vCPU threads...  you've got work to do before complaining about performance.  Nothing you're seeing has anything to do with chipset or the patch in 4.2 that's not even relevant to you.

---

http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Jimi

Member

From: Brooklyn, NY

Registered: 2009-09-25

Posts: 125

Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Does anyone have this working with an LGA 1150 CPU/mobo?

---

github

Duelist

Member

Registered: 2014-09-22

Posts: 358

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

patch in 4.2

If it's not X79 and X99, then what is 9-series chipsets?.. Those for LGA 1151(yeah, that one pin was very necessary) skylake(i7-5XXX) like H150 or what was it? Or what? Haven't seen that chipset belonging to people here...
Here: "On May 12, 2014, Intel announced the release of two 9-series chipsets, H97 and Z97."
Oh wait, then what is X79/X99?..

@marca311

You're using emulated audio, default IDE disk, default rtl network, user mode networking, not attempting to pin vCPU threads...

Emulated audio: i don't know what he meant...
Default IDE disk: Use virtio-blk-pci like -device virtio-blk-pci,drive=disk1 and try playing with x-data-plane and caching, maybe others will suggest you their best options combos.
Default rtl network and user mode networking: tun/tap networking is very fast and somewhat simple to manage: here, have a link to read, i'm not sure if it's not outdated, but you get the idea.
vCPU pinning: redhat's awesome guide, but it's for libvirt - for qemu command-line there was cpuset and even core NoHZ mode trickery with isolcpus, with which i'm not so familiar(since i personally don't have spare cores to disable from host usage). I think there was something about it in the op-post.

In general - try swooping through the redhat's guide, it's darn awesome for performance improvements.

Does anyone have this working with an LGA 1150 CPU/mobo?

Have you checked there?
I think filtering by the motherboard will help, though i can't help you much with that question.

Last edited by Duelist (2015-05-17 21:41:19)

---

The forum rules prohibit requesting support for distributions other than arch.
I gave up. It was too late.
What I was trying to do.
The reference about VFIO and KVM VGA passthrough.

aw

Member

Registered: 2013-10-04

Posts: 921

Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Does anyone have this working with an LGA 1150 CPU/mobo?

See part 1 of the guide in my sig

---

http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Duelist

Member

Registered: 2014-09-22

Posts: 358

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Does anyone have this working with an LGA 1150 CPU/mobo?

See part 1 of the guide in my sig

I've checked your blog a second after he posted this, and your xeon is 1155. Or am i misunderstanding something?

---

The forum rules prohibit requesting support for distributions other than arch.
I gave up. It was too late.
What I was trying to do.
The reference about VFIO and KVM VGA passthrough.

aw

Member

Registered: 2013-10-04

Posts: 921

Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Does anyone have this working with an LGA 1150 CPU/mobo?

See part 1 of the guide in my sig

I've checked your blog a second after he posted this, and your xeon is 1155. Or am i misunderstanding something?

Oops, you're right.  I do have another B85/Haswell setup, that works, but it has very limited I/O.

---

http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Jimi

Member

From: Brooklyn, NY

Registered: 2009-09-25

Posts: 125

Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

See part 1 of the guide in my sig

I've checked your blog a second after he posted this, and your xeon is 1155. Or am i misunderstanding something?

Oops, you're right.  I do have another B85/Haswell setup, that works, but it has very limited I/O.

I found a bunch of setups listed in https://docs.google.com/spreadsheet/ccc … _web#gid=2

Looks like my odds are good finally having Linux as the main OS on my home gaming PC.

---

github

mostlyharmless

Member

Registered: 2014-01-16

Posts: 72

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

So, managed to convert my existing qemu script Win 7 Bios setup to a OVMF/libvirt/Uefi setup without data loss or reinstallation.  However I note that Win XP Mode no longer works.  Anyone else trying to run Win XP mode under KVM? I tried hiding the VM with the <KVM> hidden=on without success.  Realize it's a little OT.

aw

Member

Registered: 2013-10-04

Posts: 921

Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

So, managed to convert my existing qemu script Win 7 Bios setup to a OVMF/libvirt/Uefi setup without data loss or reinstallation.  However I note that Win XP Mode no longer works.  Anyone else trying to run Win XP mode under KVM? I tried hiding the VM with the <KVM> hidden=on without success.  Realize it's a little OT.

Doesn't XP mode just require nested virt?  So you need to add options kvm_intel nested=1 (or kvm_amd) to a modprobe.d conf file, then expose the feature on the CPU using:

<cpu mode='host-passthrough'>
    <topology sockets='1' cores='4' threads='1'/>
    <feature policy='require' name='vmx'/>
  </cpu>

I haven't tried XP mode, but that makes cpu-z show the vt-x flag.

EDIT: Seems like just enabling the module option is enough, cpu-z shows vt-x regardless of the extra feature flag as long as nested is enabled in the kernel module.

Last edited by aw (2015-05-18 02:28:30)

---

http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Bronek

Member

From: London

Registered: 2014-02-14

Posts: 119

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

regarding motherboard selection, I have good experience with SuperMicro X9DA7 (which probably is not what you want because it's getting long in tooth and is 2 socket). From this experience I would guess that newer workstation class (single socket) X10 motherboards would be worth a try. Thinking about X10SRA in particular; its chipset is C612

Last edited by Bronek (2015-05-18 10:09:49)

mostlyharmless

Member

Registered: 2014-01-16

Posts: 72

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

I'll give the VMT option a try, but Windows isn't complaining about lack of virtualization ability.  It has a DCOM error in the Event viewer similar to http://superuser.com/questions/282994/w … ion-failed Bear in mind that the same installation worked under qemu script.