Arch Linux

thearcherblog

Member

Registered: 2014-10-30

Posts: 165

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Also, if you're sure that your host disks are okay, try preparing your windows to migrating on virtio-blk-pci instead of scsi-way. That involves creating a dummy-drive on virtio-blk-pci device and feeding windows with drivers from virtio.iso and then "reconnecting" the drive.

The disk seems good to me, which qemu options I need to use virtio-blk-pci?

First - you've got to create a -device entry with virtio-blk-pci and drive=null, where -drive=/dev/null,id=null,if=none,format=raw .
Second - you boot windows with virtio.iso plugged into it, load the drivers for that disk controller.
Third - you change the drive of given virtio-blk-pci device to your windows drive and it should boot. If it gets BSOD 7B during startup - drivers for disk controller aren't installed.

PS: what are the benefits of using virtio-blk instead of virtio-scsi?

With virtio-blk you can use x-data-plane=on, which increases IOPS in guest and lowers overhead in host. In windows guest I wasn't able to pass trim operations through virtio-blk as opposed to virtio-scsi. Later I found a workaround to this by using detect-zeroes=on with discard='unmap' and writing zeros to desired sectors (zero out unused disk space inside guest).

Can someone please share with me how I can enable virtio-blk-data-plane for the Libvirt Domain XML?

I couldn't transform these posts into my XML.

I also found some slides from redhat, but they don't work either. How does the relevant code look like in XML?

I am trying for some days now without luck.

thank you very much.

I didn't tried before but seems like google has the answer: http://blog.vmsplice.net/2013/03/new-in … irtio.html

Regards,

TheArcher

4kGamer

Member

Registered: 2014-10-29

Posts: 88

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

thank you TheArcher! Always there when needed

I found that page too. But on that page it doesn't mention anything about detect-zeroes=on and how do I "zero out unused disk space inside guest"?

4kGamer

Member

Registered: 2014-10-29

Posts: 88

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

and it says "Red Hat VirtIO SCSI Disk Device" in Device Manager in Windows. Is this correct for virtio-blk-data-plane?

and these are the relevant parts of my XML:

<domain type='kvm' xmlns:qemu='http://libvirt.org/schemas/domain/qemu/1.0'>

<disk type='block' device='disk'>
      <driver name='qemu' type='raw' cache='none' io='native' discard='unmap'/>
      <source dev='/dev/sda'/>
      <target dev='vda' bus='virtio'/>
      <boot order='1'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x0f' function='0x0'/>
    </disk>

<qemu:commandline>
    <qemu:arg value='-set'/>
    <qemu:arg value='device.virtio-disk0.scsi=off'/>
    <qemu:arg value='-set'/>
    <qemu:arg value='device.virtio-disk0.config-wce=off'/>
    <qemu:arg value='-set'/>
    <qemu:arg value='device.virtio-disk0.x-data-plane=on'/>
  </qemu:commandline>

Last edited by 4kGamer (2015-01-07 17:30:16)

Bronek

Member

From: London

Registered: 2014-02-14

Posts: 119

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Success story, it took me few days to get it right by seems pretty stable. The biggest obstacle was ZFS on Linux as root partition (still causing hiccups - obviously not related to topic at all) and few silly mistakes in network setup etc. Other issue was selection of hardware passed-thru eventually I settled for either of my two GPUs (AMD Turks ie. FirePro V4900 and Hawaii ie. 290X - both work nicely), all USB controllers (but one) including PCIe cards and keyboard (attached to only USB controller remaining to host). From the start the intent was to not to leave any graphic capability to host, and both GPU cards are meant for virtual machines. For this I simply blacklisted radeon and configured pci-stub to take the ownership of both cards.

Following Alex's advice I set to use FX440 rather than Q35, also right from the start I installed Windows on Virtio thus being spared abysmal perfomance of emulated disk. Also I do not use HDMI audio output and do not plan on using it, my audio is from DAC attached to one of passed-thru USB controllers. I had BSODs or other problems if I explicitly declared CPU type to something different than qemu64 (or host), tried SandyBridge and Nehalem. I also run almost exactly same command to start Linux guest (mint 17.1) which also runs nicely - obvious differences being -rtc base=utc, different -drive for disk backing store and virtio network card rather than e1000 (I might try virtio on Windows again actually, there may be better versions avialable).

My goal is to split my machine into 3 machines - one "visible" Windows that you will see after starting the computer, "invisible" host which provides zfs for user files shared over virtual network and another "invisible" linux machine for general hacking and experimentation. Both linux machines to be accessed through shh "as if they were over network". I might eventually split the machine to run concurrently two "visible" Windows since it has two GPUs , enough CPU power and enough RAM for that. Only a question of finding space on desk for another monitor and keyboard.

Here is my qemu command:

#!/bin/bash
qemu-system-x86_64 -enable-kvm \
-M pc -m 16G -mem-path /dev/hugepages \
-smp cpus=8,cores=4,threads=2,sockets=2 \
-cpu host,hv-time,hv_relaxed,hv_vapic,hv_spinlocks=0x1000 \
-bios /usr/share/qemu/bios.bin \
-rtc base=localtime,clock=host  \
-nodefaults \
-nographic -vga none \
-monitor stdio \
-boot order=c \
-device vfio-pci,host=02:00.0,addr=08.0,romfile=ROM/Hawaii-1.rom,x-vga=on \
-device vfio-pci,host=00:1b.0,addr=09.0 \
-device vfio-pci,host=00:1d.0,addr=0a.0 \
-device vfio-pci,host=00:1a.0,addr=0b.0 \
-device vfio-pci,host=81:00.0,addr=0e.0 \
-device vfio-pci,host=83:00.0,addr=0f.0 \
-usb -usbdevice host:04D9:2011 \
-drive file=/dev/zvol/zdata/image1,id=disk0,format=raw,if=virtio \
-drive file=/home/bronek/windows7.iso,id=isocd0,if=none -device ide-cd,drive=isocd0 \
-drive file=/home/bronek/virtio-win-0.1-94.iso,id=isocd1,if=none -device ide-cd,drive=isocd1\
-netdev bridge,br=br0,id=net0 -net nic,netdev=net0,macaddr=52:54:01:23:45:f5,model=e1000

My software versions are qemu-2.2.0 (from testing), linux-3.17.7 (modified linux 3.17.6 package with newer version and own config) and zfs 0.6.3-r1.2 + few patches from zfsonlinux (modified archzfs package).

I also switched from systemd-networkd to netctl because I liked its configuration better and enabled sysctl for definition of iptables overrides for bridge and for initialisation of huge pages:

root@gdansk ~ # cat /etc/sysctl.d/80-hugepages.conf
# Reserve 40GB in 2MB pages for virtual machines
vm.nr_hugepages = 20480;

root@gdansk ~ # cat /etc/sysctl.d/81-bridge.conf   
# Disable ip tables on bridge frames
net.bridge.bridge-nf-call-ip6tables = 0
net.bridge.bridge-nf-call-iptables = 0
net.bridge.bridge-nf-call-arptables = 0

Didnt run any games yet - this machine is actually not "for gaming" although I will certainly allow my children to play  games.

Next step is to replace qemu command with libvirt (for CPU pinning, automatic start and managment) but without GUI managment part since the host is headless and I only access it through SSH (and I like command line in Linux better than GUI tools). I also plan to setup smb sharing of zfs setup on the host, with the guest. Tempted to make it Kerberos with help of samba4 running on host but not sure, I might not have time for that.

Last part I was struggling to find qemu guest agent for Windows binaries (for VSS support) prepared by someone but even without it I can take backups of my guest, the only difference being I need to do it offline. Building qemu guest agent myself, from qemu sources proved to be bit more difficult than I expected it to be. Although I might try it again later.

Last edited by Bronek (2015-01-07 19:38:34)

aw

Member

Registered: 2013-10-04

Posts: 921

Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Success story, it took me few days to get it right by seems pretty stable.

Thanks for the report and congrats!  Tiny tip, add -serial none -parallel none to your QEMU commandline to avoid the terminal needing a reset after guest shutdown.  I think some recent upstream changes preventing multiple things being muxed to stdio will require this for the next release anyway.

---

http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Bronek

Member

From: London

Registered: 2014-02-14

Posts: 119

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Regarding virtio Ethernet adapter for Windows, version virtio-win-0.1-94 didn't work for me, but version virtio-win-0.1-81 did, so I'm no longer using e1000. Virtio SCSI driver worked from both versions. I'm running Windows 7 Ultimate.

Now to that libvirt setup .... hope it wont be that hard.

Bronek

Member

From: London

Registered: 2014-02-14

Posts: 119

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Thanks for the report and congrats!  Tiny tip, add -serial none -parallel none to your QEMU commandline to avoid the terminal needing a reset after guest shutdown.  I think some recent upstream changes preventing multiple things being muxed to stdio will require this for the next release anyway.

Thanks for the tip, I'm using -nodefaults since that seemed the only way to get rid of bogus floppy drive and also to get -monitor stdio working (wont need it after migration to libvirt, but now its very useful) but I will add the above nevertheless

Any other tips will be hugely appreciated

thearcherblog

Member

Registered: 2014-10-30

Posts: 165

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

thank you TheArcher! Always there when needed

I found that page too. But on that page it doesn't mention anything about detect-zeroes=on and how do I "zero out unused disk space inside guest"?

Hello sir,

According to http://qemu.11.n7.nabble.com/is-x-data- … ml#a293049, dataplane is deprecated and will be removed on QEMU 2.2.

I do not know exactly the problem you have with the performance but, in my case, with normal image (raw), my windows 8.1 boots in 2 seconds on a SSD non dedicated one. I think is enough performance for me

Best regards,

TheArcher

4kGamer

Member

Registered: 2014-10-29

Posts: 88

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

hello! I didn't know that. Thank you very much. This is valuable information for me. Some of us here are obviously confused with what's best as a storage device.

To tell you the truth I don't even know how to rate my performance. I guess all the Windows benchmark tools for assessing hardware cannot be regarded as reliable.

What benchmark tools do you use or can you recommend? Especially for storage (IOPS and 4k random read/write in particular)

Last edited by 4kGamer (2015-01-08 07:14:51)

thearcherblog

Member

Registered: 2014-10-30

Posts: 165

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

I never use benchmark utilities because normally are synthetic ones and are just numbers that can be really different from the real performance I'm getting. Well, to be honest I run them on the beginning... you know, the normal ones, some for graphic card and some for general... but I have been too many years working with computers to know that best benchmark utilities and results does not mean that is better performance... talking about phones... look antutu results for different phones and read the feelings... No one is agree!

For me is just a feeling, I mean, if I can run my games and boots ok, can copy files in a reasonable time, etc... is ok. Sometimes I notice that is slower or not giving me the right performance and then I look for some information about but, talking about the QEMU/KVM... I'm so happy about how it works. Performance is really good even when I'm not checking it with benchmarks.

Cheers,

TheArcher

4kGamer

Member

Registered: 2014-10-29

Posts: 88

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

yeah you're definitely right! I completely agree.

I am not after certain numbers. It's only that sometimes I experience issues in Games and I wonder if it's related to poor disk performance. I therefore need to make sure that I have the right tools to test NOW so I don't have to worry LATER.

So it's only to get the configuration right. And I am happy you made me aware that I was trying a depriaciated technique. Before I also tried SATA passthrough which worked fine for most parts, but then I had issues in Games and I still don't know why ...

some tools would definitely help for now

thearcherblog

Member

Registered: 2014-10-30

Posts: 165

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

But... what kind of issues do you have?

As I'm not expert, maybe if you put your hardware configuration, config files and so on, and describe as detailed as possible the problems... maybe a guru from here can help you

Cheers

4kGamer

Member

Registered: 2014-10-29

Posts: 88

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

ok, I will post my complete config in a few hours. Here is my issue:

what I experienced a few times already: all of a sudden there is "a glitch" in the Picture, just for a second. And right at that instant CPU usage jumps to 100 % and then it goes down again. Maybe it's related to disk performance, maybe it's just the CPU or maybe it's just a mistake of the Game. I experienced this when I play offline Fifa 15 with friends. (Normally my CPU usage is about 20-30 % when I play Fifa 15 at 4k resolution)
I also tried to replicate the issue when I use bare metal Windows, but it didn't occur in it so far.

Last edited by 4kGamer (2015-01-08 09:03:03)

thearcherblog

Member

Registered: 2014-10-30

Posts: 165

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

ok, I will post my complete config in a few hours. Here is my issue:

what I experienced a few times already: all of a sudden there is "a glitch" in the Picture, just for a second. And right at that instant CPU usage jumps to 100 % and then it goes down again. Maybe it's related to disk performance, maybe it's just the CPU or maybe it's just a mistake of the Game. I experienced this when I play offline Fifa 15 with friends. I also tried to replicate the issue when I use bare metal Windows, but it didn't occur in it so far.

Don't know for sure but can be several things:

- As is on playing internet... can be the ethernet? Are you passing it through or emulating?
- Can be NVIDIA drivers? (you know, loading an effect or similar)
- If it's not happening with any other game... can be the game?

Hope we can solve it together

Cheers

4kGamer

Member

Registered: 2014-10-29

Posts: 88

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

I am passing through my Ethernet. And it happened offline when I played with friends. I couldn't test thoroughly with other games. I barely played on it so far. A few days back we had a Soccer session here and it happened a few times. Never before and I am familiar with the game.

So my guess: It could be Nvidia Drivers or the CPU or the disk. But maybe it's something completely different

I will post my complete settings in a few hours. I am very grateful for your help!

thearcherblog

Member

Registered: 2014-10-30

Posts: 165

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

I am passing through my Ethernet. And it happened offline when I played with friends. I couldn't test thoroughly with other games. I barely played on it so far. A few days back we had a Soccer session here and it happened a few times. Never before and I am familiar with the game.

So my guess: It could be Nvidia Drivers or the CPU or the disk. But maybe it's something completely different

I will post my complete settings in a few hours. I am very grateful for your help!

Ok

As you have quite similar hadrware as me... Are you passing through the intel LAN isn't it? I know that the realtek ones give me problems

Anyway... yes, can be but... my suggestion to troubleshoot this would be:

Try different things (different stadiums, weather conditions,... to try to find if there is any pattern)
Different NVIDIA drivers... sometimes the last version is not the best, talking by experience
I do not know how many vCPU you have assigned or if you have different setup. In my case I assign 4 because if I assign 8 the heat increase really high and I didn't notice any performance improvement.

Anyway, I will wait for your config files

Cheers.

4kGamer

Member

Registered: 2014-10-29

Posts: 88

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

So here is my setup and config files:

Motherboard:ASROCK Z97 Extreme6
CPU: i7-4790k
GPU: MSI GTX 980 Gaming 4G
RAM: 12 GB RAM
Storage for Windows 8.1: Samsung 840 EVO Basic 1TB (/dev/sda)
Linux Kernel: linux-3.17.6-1 (untouched)

Packages installed:
qemu 2.1
libvirt
virt-manager

cat /proc/cmdline
"rw root=/dev/mapper/VG-root rdblacklist=nouveau nouveau.modeset=0 elevator=noop vfio_iommu_type1 intel_iommu=on,igfx_off panic=10 pci-stub.ids=10de:13c0,10de:0fbb,8086:15a1,8086:8cb1 intel_pstate=disable pcie_acs_override=downstream vfio_iommu_type1.allow_unsafe_interrupts=1 quiet"

/etc/modprobe.d/modprobe.conf:

options kvm ignore_msrs=1
options intel_kvm emulate_invalid_guest_state=0
options intel_kvm nested=1
options intel_kvm enable_shadow_vmcs=1
options intel_kvm enable_apicv=1
options intel_kvm ept=1
#options vfio_pci nointxmask=1
options vfio_iommu_type1 disable_hugepages=1
options vfio_iommu_type1 allow_unsafe_interrupts=1

/etc/libvirt/qemu/vm1.xml:

<!--
WARNING: THIS IS AN AUTO-GENERATED FILE. CHANGES TO IT ARE LIKELY TO BE
OVERWRITTEN AND LOST. Changes to this xml configuration should be made using:
  virsh edit vm1
or other application using the libvirt API.
-->

<domain type='kvm'>
  <name>vm1</name>
  <uuid>2459c3f0-4269-4823-55d4-43528ff0d991</uuid>
  <memory unit='KiB'>10485760</memory>
  <currentMemory unit='KiB'>10485760</currentMemory>
  <vcpu placement='static' cpuset='0-7'>8</vcpu>
  <cputune>
    <vcpupin vcpu='0' cpuset='0,4'/>
    <vcpupin vcpu='1' cpuset='1,5'/>
    <vcpupin vcpu='2' cpuset='2,6'/>
    <vcpupin vcpu='3' cpuset='3,7'/>
    <vcpupin vcpu='4' cpuset='0,4'/>
    <vcpupin vcpu='5' cpuset='1,5'/>
    <vcpupin vcpu='6' cpuset='2,6'/>
    <vcpupin vcpu='7' cpuset='3,7'/>
  </cputune>
  <os>
    <type arch='x86_64' machine='pc-i440fx-2.1'>hvm</type>
    <loader type='pflash'>/var/lib/libvirt/images/VM1-OVMF.fd</loader>
    <bootmenu enable='no'/>
  </os>
  <features>
    <acpi/>
    <apic/>
    <pae/>
    <kvm>
      <hidden state='on'/>
    </kvm>
  </features>
  <cpu mode='host-passthrough'>
    <topology sockets='1' cores='4' threads='2'/>
  </cpu>
  <clock offset='localtime'/>
  <on_poweroff>destroy</on_poweroff>
  <on_reboot>restart</on_reboot>
  <on_crash>restart</on_crash>
  <pm>
    <suspend-to-mem enabled='no'/>
    <suspend-to-disk enabled='no'/>
  </pm>
  <devices>
    <emulator>/usr/sbin/qemu-system-x86_64</emulator>
    <disk type='block' device='disk'>
      <driver name='qemu' type='raw' cache='none' io='native'/>
      <source dev='/dev/sda'/>
      <target dev='vda' bus='virtio'/>
      <boot order='1'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x0f' function='0x0'/>
    </disk>
    <disk type='file' device='cdrom'>
      <driver name='qemu' type='raw' cache='none'/>
      <source file='/root/virtio-win-0.1-94.iso'/>
      <target dev='hda' bus='ide'/>
      <readonly/>
      <boot order='2'/>
      <address type='drive' controller='0' bus='0' target='0' unit='0'/>
    </disk>
    <disk type='file' device='cdrom'>
      <driver name='qemu' type='raw' cache='none'/>
      <source file='/root/windows.iso'/>
      <target dev='hdb' bus='ide'/>
      <readonly/>
      <boot order='3'/>
      <address type='drive' controller='0' bus='0' target='0' unit='1'/>
    </disk>
    <controller type='usb' index='0' model='ich9-ehci1'>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x0e' function='0x7'/>
    </controller>
    <controller type='usb' index='0' model='ich9-uhci1'>
      <master startport='0'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x05' function='0x0' multifunction='on'/>
    </controller>
    <controller type='usb' index='0' model='ich9-uhci2'>
      <master startport='2'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x05' function='0x1'/>
    </controller>
    <controller type='usb' index='0' model='ich9-uhci3'>
      <master startport='4'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x05' function='0x2'/>
    </controller>
    <controller type='pci' index='0' model='pci-root'/>
    <controller type='scsi' index='1' model='virtio-scsi'>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x09' function='0x0'/>
    </controller>
    <controller type='ide' index='0'>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x1'/>
    </controller>
    <hostdev mode='subsystem' type='pci' managed='yes'>
      <driver name='vfio'/>
      <source>
        <address domain='0x0000' bus='0x01' slot='0x00' function='0x0'/>
      </source>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x0'/>
    </hostdev>
    <hostdev mode='subsystem' type='pci' managed='yes'>
      <driver name='vfio'/>
      <source>
        <address domain='0x0000' bus='0x01' slot='0x00' function='0x1'/>
      </source>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x0'/>
    </hostdev>
    <hostdev mode='subsystem' type='pci' managed='yes'>
      <driver name='vfio'/>
      <source>
        <address domain='0x0000' bus='0x00' slot='0x19' function='0x0'/>
      </source>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x04' function='0x0'/>
    </hostdev>
    <hostdev mode='subsystem' type='pci' managed='yes'>
      <driver name='vfio'/>
      <source>
        <address domain='0x0000' bus='0x00' slot='0x14' function='0x0'/>
      </source>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x0b' function='0x0'/>
    </hostdev>
    <hostdev mode='subsystem' type='pci' managed='yes'>
      <driver name='vfio'/>
      <source>
        <address domain='0x0000' bus='0x00' slot='0x1a' function='0x0'/>
      </source>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x0c' function='0x0'/>
    </hostdev>
    <hostdev mode='subsystem' type='pci' managed='yes'>
      <driver name='vfio'/>
      <source>
        <address domain='0x0000' bus='0x00' slot='0x1d' function='0x0'/>
      </source>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x0d' function='0x0'/>
    </hostdev>
    <memballoon model='virtio'>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x07' function='0x0'/>
    </memballoon>
  </devices>
</domain>

Last edited by 4kGamer (2015-01-08 14:00:15)

4kGamer

Member

Registered: 2014-10-29

Posts: 88

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

If I forgot something just tell me and I'll provide it

oh and yes I pass through the Intel LAN

and I forgot to post this:

lsblk

00:00.0 Host bridge: Intel Corporation 4th Gen Core Processor DRAM Controller (rev 06)
00:01.0 PCI bridge: Intel Corporation Xeon E3-1200 v3/4th Gen Core Processor PCI Express x16 Controller (rev 06)
00:02.0 VGA compatible controller: Intel Corporation Xeon E3-1200 v3/4th Gen Core Processor Integrated Graphics Controller (rev 06)
00:03.0 Audio device: Intel Corporation Xeon E3-1200 v3/4th Gen Core Processor HD Audio Controller (rev 06)
00:14.0 USB controller: Intel Corporation 9 Series Chipset Family USB xHCI Controller
00:16.0 Communication controller: Intel Corporation 9 Series Chipset Family ME Interface #1
00:19.0 Ethernet controller: Intel Corporation Ethernet Connection (2) I218-V
00:1a.0 USB controller: Intel Corporation 9 Series Chipset Family USB EHCI Controller #2
00:1c.0 PCI bridge: Intel Corporation 9 Series Chipset Family PCI Express Root Port 1 (rev d0)
00:1c.2 PCI bridge: Intel Corporation 9 Series Chipset Family PCI Express Root Port 3 (rev d0)
00:1c.3 PCI bridge: Intel Corporation 9 Series Chipset Family PCI Express Root Port 4 (rev d0)
00:1c.4 PCI bridge: Intel Corporation 9 Series Chipset Family PCI Express Root Port 5 (rev d0)
00:1c.6 PCI bridge: Intel Corporation 9 Series Chipset Family PCI Express Root Port 7 (rev d0)
00:1d.0 USB controller: Intel Corporation 9 Series Chipset Family USB EHCI Controller #1
00:1f.0 ISA bridge: Intel Corporation 9 Series Chipset Family Z97 LPC Controller
00:1f.2 SATA controller: Intel Corporation 9 Series Chipset Family SATA Controller [AHCI Mode]
00:1f.3 SMBus: Intel Corporation 9 Series Chipset Family SMBus Controller
01:00.0 VGA compatible controller: NVIDIA Corporation GM204 [GeForce GTX 980] (rev a1)
01:00.1 Audio device: NVIDIA Corporation GM204 High Definition Audio Controller (rev a1)
03:00.0 Ethernet controller: Realtek Semiconductor Co., Ltd. RTL8111/8168/8411 PCI Express Gigabit Ethernet Controller (rev 11)
04:00.0 PCI bridge: ASMedia Technology Inc. Device 1184
05:01.0 PCI bridge: ASMedia Technology Inc. Device 1184
05:03.0 PCI bridge: ASMedia Technology Inc. Device 1184
05:05.0 PCI bridge: ASMedia Technology Inc. Device 1184
05:07.0 PCI bridge: ASMedia Technology Inc. Device 1184
07:00.0 SATA controller: ASMedia Technology Inc. ASM1062 Serial ATA Controller (rev 02)
09:00.0 SATA controller: ASMedia Technology Inc. ASM1062 Serial ATA Controller (rev 02)
0a:00.0 Multimedia controller: Digital Devices GmbH Octopus DVB Adapter
0b:00.0 USB controller: ASMedia Technology Inc. ASM1042A USB 3.0 Host Controller

the bold ones are the pci devices I passed through (successfully)

Last edited by 4kGamer (2015-01-08 13:58:10)

thearcherblog

Member

Registered: 2014-10-30

Posts: 165

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

As you make CPU pin with the libvirt.... is happening the same using the command line? and not using them? I don't use any CPU pin

4kGamer

Member

Registered: 2014-10-29

Posts: 88

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

I haven't tried without CPU Pin, but that is something I am not really sure if I did it correctly. Also hugepages, which are disabled in my setup. I think I have to review both. And like you know I am really uncertain of the storage device.

aw

Member

Registered: 2013-10-04

Posts: 921

Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

So here is my setup and config files:

Motherboard:ASROCK Z97 Extreme6
CPU: i7-4790k
GPU: MSI GTX 980 Gaming 4G
RAM: 12 GB RAM
Storage for Windows 8.1: Samsung 840 EVO Basic 1TB (/dev/sda)
Linux Kernel: linux-3.17.6-1 (untouched)

Packages installed:
qemu 2.1
libvirt
virt-manager

cat /proc/cmdline
"rw root=/dev/mapper/VG-root rdblacklist=nouveau nouveau.modeset=0 elevator=noop vfio_iommu_type1 intel_iommu=on,igfx_off panic=10 pci-stub.ids=10de:13c0,10de:0fbb,8086:15a1,8086:8cb1 intel_pstate=disable pcie_acs_override=downstream vfio_iommu_type1.allow_unsafe_interrupts=1 quiet"

If you're using an "untouched" kernel, the pcie_acs_override option isn't doing anything.  You also shouldn't need the vfio option here.

/etc/modprobe.d/modprobe.conf:

options kvm ignore_msrs=1
options intel_kvm emulate_invalid_guest_state=0
options intel_kvm nested=1
options intel_kvm enable_shadow_vmcs=1
options intel_kvm enable_apicv=1
options intel_kvm ept=1
#options vfio_pci nointxmask=1
options vfio_iommu_type1 disable_hugepages=1
options vfio_iommu_type1 allow_unsafe_interrupts=1

Could you not find any more random option to add?  You should not need any of the vfio ones, and by disabling huge pages, you're making the IOMMU work harder.  BTW, the Intel kvm module is called kvm_intel, not intel_kvm, so all of those options are not doing anything.  I don't think you want to be touching any of those option anyway.

/etc/libvirt/qemu/vm1.xml:

<!--
WARNING: THIS IS AN AUTO-GENERATED FILE. CHANGES TO IT ARE LIKELY TO BE
OVERWRITTEN AND LOST. Changes to this xml configuration should be made using:
  virsh edit vm1
or other application using the libvirt API.
-->

<domain type='kvm'>
  <name>vm1</name>
  <uuid>2459c3f0-4269-4823-55d4-43528ff0d991</uuid>
  <memory unit='KiB'>10485760</memory>
  <currentMemory unit='KiB'>10485760</currentMemory>
  <vcpu placement='static' cpuset='0-7'>8</vcpu>
  <cputune>
    <vcpupin vcpu='0' cpuset='0,4'/>
    <vcpupin vcpu='1' cpuset='1,5'/>
    <vcpupin vcpu='2' cpuset='2,6'/>
    <vcpupin vcpu='3' cpuset='3,7'/>
    <vcpupin vcpu='4' cpuset='0,4'/>
    <vcpupin vcpu='5' cpuset='1,5'/>
    <vcpupin vcpu='6' cpuset='2,6'/>
    <vcpupin vcpu='7' cpuset='3,7'/>

You really want to pin to a single CPU, so I'd eliminate the a,b options, pick one.  I recognize that you're trying to pin to a core, which should share cache, but I don't think this is helping you.  Also, you're assigning 100% of your CPU power to the guest.  Are you running a very minimal host?  How is the host supposed to do anything without stealing CPU from the guest?  In my setup I assign the cores to the guest and leave the threads for the host.  Huge pages are another tuning option here.

---

http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

kaeptnb

Member

Registered: 2013-09-04

Posts: 30

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

ok, I will post my complete config in a few hours. Here is my issue:

what I experienced a few times already: all of a sudden there is "a glitch" in the Picture, just for a second. And right at that instant CPU usage jumps to 100 % and then it goes down again. Maybe it's related to disk performance, maybe it's just the CPU or maybe it's just a mistake of the Game. I experienced this when I play offline Fifa 15 with friends. (Normally my CPU usage is about 20-30 % when I play Fifa 15 at 4k resolution)
I also tried to replicate the issue when I use bare metal Windows, but it didn't occur in it so far.

You really should try to pin your vcpus 1:1 to real cpus; and keep hyper-threading in mind to prevent cache-trashing (which very well could be the cause for your micro-stutters);

With a i7 920 Host CPU (8 virtual cpus; hyperthreading) i use this:

  <vcpu placement='static'>4</vcpu>
  <cputune>
    <vcpupin vcpu='0' cpuset='2'/>
    <vcpupin vcpu='1' cpuset='6'/>
    <vcpupin vcpu='2' cpuset='3'/>
    <vcpupin vcpu='3' cpuset='7'/>
    <emulatorpin cpuset='0-1,4-5'/>
  </cputune>
  <os>
    <type arch='x86_64' machine='pc-i440fx-2.1'>hvm</type>
    <loader readonly='yes' type='pflash'>/etc/libvirt/efi/edk2.git/ovmf-x64/OVMF_CODE-pure-efi.fd</loader>
    <nvram template='/etc/libvirt/efi/edk2.git/ovmf-x64/OVMF_VARS-pure-efi.fd'>/var/lib/libvirt/qemu/nvram/win_VARS.fd</nvram>
    <bootmenu enable='yes'/>
  </os>
  <features>
    <acpi/>
    <apic/>
    <pae/>
    <hyperv>
      <relaxed state='on'/>
      <vapic state='on'/>
      <spinlocks state='on' retries='8191'/>
    </hyperv>
  </features>
  <cpu mode='host-passthrough'>
    <topology sockets='1' cores='2' threads='2'/>
  </cpu>

Looking forward to hear if that helps!
Greetings

Last edited by kaeptnb (2015-01-08 15:21:24)

4kGamer

Member

Registered: 2014-10-29

Posts: 88

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Alex, thank you very much for having a closer look. I will adjust accordingly. I didn't know many of the things you wrote. I have assigned all CPUs to guest to make sure I have enough power. It's just for now when I am still trying to find out what I really need. I will leave 25 - 50 % to the host as I am not doing anything other than using the console at the moment - but that might change in time

Alex, is this okay?
       

<vcpu placement='static' cpuset='0-5'>8</vcpu>
    <vcpupin vcpu='0' cpuset='0'/>
    <vcpupin vcpu='1' cpuset='1'/>
    <vcpupin vcpu='2' cpuset='2'/>
    <vcpupin vcpu='3' cpuset='3'/>
    <vcpupin vcpu='4' cpuset='4'/>
    <vcpupin vcpu='5' cpuset='5'/>

You said you only assign cores to the guest. How is this done within the XML?

4kGamer

Member

Registered: 2014-10-29

Posts: 88

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

ok, I will post my complete config in a few hours. Here is my issue:

what I experienced a few times already: all of a sudden there is "a glitch" in the Picture, just for a second. And right at that instant CPU usage jumps to 100 % and then it goes down again. Maybe it's related to disk performance, maybe it's just the CPU or maybe it's just a mistake of the Game. I experienced this when I play offline Fifa 15 with friends. (Normally my CPU usage is about 20-30 % when I play Fifa 15 at 4k resolution)
I also tried to replicate the issue when I use bare metal Windows, but it didn't occur in it so far.

You really should try to pin your vcpus 1:1 to real cpus; and keep hyper-threading in mind to prevent cache-trashing (which very well could be the cause for your micro-stutters);

With a i7 920 Host CPU (8 virtual cpus; hyperthreading) i use this:

  <vcpu placement='static'>4</vcpu>
  <cputune>
    <vcpupin vcpu='0' cpuset='2'/>
    <vcpupin vcpu='1' cpuset='6'/>
    <vcpupin vcpu='2' cpuset='3'/>
    <vcpupin vcpu='3' cpuset='7'/>
    <emulatorpin cpuset='0-1,4-5'/>
  </cputune>
  <os>
    <type arch='x86_64' machine='pc-i440fx-2.1'>hvm</type>
    <loader readonly='yes' type='pflash'>/etc/libvirt/efi/edk2.git/ovmf-x64/OVMF_CODE-pure-efi.fd</loader>
    <nvram template='/etc/libvirt/efi/edk2.git/ovmf-x64/OVMF_VARS-pure-efi.fd'>/var/lib/libvirt/qemu/nvram/win_VARS.fd</nvram>
    <bootmenu enable='yes'/>
  </os>
  <features>
    <acpi/>
    <apic/>
    <pae/>
    <hyperv>
      <relaxed state='on'/>
      <vapic state='on'/>
      <spinlocks state='on' retries='8191'/>
    </hyperv>
  </features>
  <cpu mode='host-passthrough'>
    <topology sockets='1' cores='2' threads='2'/>
  </cpu>

Looking forward to hear if that helps!
Greetings

Thank you for sharing your configs! I will definitely pin 1:1 from now on.

About hyperv: If I understand correctly I can't enable hyper-threading because I use an Nvidia GPU. It would give me Code 43 on my GTX 980.

Last edited by 4kGamer (2015-01-08 15:39:12)

aw

Member

Registered: 2013-10-04

Posts: 921

Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Alex, thank you very much for having a closer look. I will adjust accordingly. I didn't know many of the things you wrote. I have assigned all CPUs to guest to make sure I have enough power. It's just for now when I am still trying to find out what I really need. I will leave 25 - 50 % to the host as I am not doing anything other than using the console at the moment - but that might change in time

Alex, is this okay?
       

<vcpu placement='static' cpuset='0-5'>8</vcpu>
    <vcpupin vcpu='0' cpuset='0'/>
    <vcpupin vcpu='1' cpuset='1'/>
    <vcpupin vcpu='2' cpuset='2'/>
    <vcpupin vcpu='3' cpuset='3'/>
    <vcpupin vcpu='4' cpuset='4'/>
    <vcpupin vcpu='5' cpuset='5'/>

You said you only assign cores to the guest. How is this done within the XML?

I don't think you have the right idea about what a thread is.  A thread is not a full execution unit, so your example here has vcpus 4 & 5 partially sharing vcpus 0 & 1.  I don't know how you'd represent that to the guest.  You also need to realize that a thread only adds maybe 10-20% performance, so if you assign 4 cores (without threads), you're already providing the VM with perhaps 75%+ of the processing power of the socket.  I do something more like kaeptnb, although I don't know how they came up with those cpuset values.  AFAIK, cpusets are the same as processor numbers, where the cores are generally enumerated before the threads.  Thus on a 4-core w/ threads processor, 0-3 are cores and 4-7 are threads, where 4 is the thread sibling of 0, 5 of 1, etc.

---

http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users