[SOLVED] KVM/QEMU GPU Passthrough Performance Issues

Woodiii · 2022-10-21 14:19:53

Hello Community!

I bought myself a new cpu (Ryzen 7 5700G) with an internal graphic chip, so I can get rid of Windows bare metal and move to gaming in a vm.
So I loaded the vfio-pci kernel driver onto my Nvidia GTX 1070 as you can see here:

10:00.0 VGA compatible controller: NVIDIA Corporation GP104 [GeForce GTX 1070] (rev a1)
Subsystem: Micro-Star International Co., Ltd. [MSI] Device 3301
Kernel driver in use: vfio-pci
Kernel modules: nouveau, nvidia_drm, nvidia
10:00.1 Audio device: NVIDIA Corporation GP104 High Definition Audio Controller (rev a1)
Subsystem: Micro-Star International Co., Ltd. [MSI] Device 3301
Kernel driver in use: vfio-pci
Kernel modules: snd_hda_intel

I also checked the iommu groups, which you can see here:

IOMMU group 9
10:00.0 VGA compatible controller [0300]: NVIDIA Corporation GP104 [GeForce GTX 1070] [10de:1b81] (rev a1)
10:00.1 Audio device [0403]: NVIDIA Corporation GP104 High Definition Audio Controller [10de:10f0] (rev a1)

I then passed through the graphics card in kvm, loaded a custom rom to get rid of error 43 and installed virt-io-guest tools.

I did also some CPU pinning. I also isolated my cpus from Linux with this script:

systemctl set-property --runtime -- user.slice AllowedCPUs=0,1,8,9
systemctl set-property --runtime -- system.slice AllowedCPUs=0,1,8,9
systemctl set-property --runtime -- init.scope AllowedCPUs=0,1,8,9

And after the vm is shut down i allow Linux to use all cores again.

So am I missing something here, because my performance is not good. Windows is lagging completely and I have 2 fps in PUBG.

I have created two storage pools in KVM, one on my SSD where the Windows OS gets installed and one on my HDD for the games and stuff like that.

I know it would be better to pass through an entire SSD, but with my current setup this won't work, because then I'd have to install Linux on the HDD, which I definetely won't do!

Or is there an other way to do it, and which hardware should I pass through to the vm to increase the performance?

Hopefully someone can help

Regards,
Woodiii

EDIT: I've tried some things...

I added more Hyper-V options.
I changed clock settings.
I changed disks and Video to use virtio.
I used Windows 11 instead of Windows 10
It all does not help. I don't know what else to do...

EDIT 2: Issue has been solved, just read my last reply.
here you can see my working xml configuration of the vm:

<domain type="kvm">
<name>win10</name>
<uuid>0d7aa8f3-e906-446c-9897-19e7b650d0a8</uuid>
<metadata>
<libosinfo:libosinfo xmlns:libosinfo="http://libosinfo.org/xmlns/libvirt/domain/1.0">
<libosinfo:os id="http://microsoft.com/win/10"/>
</libosinfo:libosinfo>
</metadata>
<memory unit="KiB">20971520</memory>
<currentMemory unit="KiB">20971520</currentMemory>
<vcpu placement="static">12</vcpu>
<iothreads>2</iothreads>
<cputune>
<vcpupin vcpu="0" cpuset="2"/>
<vcpupin vcpu="1" cpuset="3"/>
<vcpupin vcpu="2" cpuset="4"/>
<vcpupin vcpu="3" cpuset="5"/>
<vcpupin vcpu="4" cpuset="6"/>
<vcpupin vcpu="5" cpuset="7"/>
<vcpupin vcpu="6" cpuset="10"/>
<vcpupin vcpu="7" cpuset="11"/>
<vcpupin vcpu="8" cpuset="12"/>
<vcpupin vcpu="9" cpuset="13"/>
<vcpupin vcpu="10" cpuset="14"/>
<vcpupin vcpu="11" cpuset="15"/>
<emulatorpin cpuset="0-1"/>
<iothreadpin iothread="1" cpuset="0-1"/>
<iothreadpin iothread="2" cpuset="2-3"/>
</cputune>
<os>
<type arch="x86_64" machine="pc-q35-7.1">hvm</type>
<loader readonly="yes" type="pflash">/usr/share/edk2-ovmf/x64/OVMF_CODE.fd</loader>
<nvram>/var/lib/libvirt/qemu/nvram/win10_VARS.fd</nvram>
<boot dev="hd"/>
</os>
<features>
<acpi/>
<apic/>
<hyperv mode="custom">
<relaxed state="on"/>
<vapic state="on"/>
<spinlocks state="on" retries="8191"/>
<vpindex state="on"/>
<synic state="on"/>
<stimer state="on">
<direct state="on"/>
</stimer>
<reset state="on"/>
<frequencies state="on"/>
<reenlightenment state="on"/>
<tlbflush state="on"/>
<ipi state="on"/>
</hyperv>
<kvm>
<hidden state="on"/>
</kvm>
<vmport state="off"/>
</features>
<cpu mode="host-passthrough" check="none" migratable="on">
<topology sockets="1" dies="1" cores="6" threads="2"/>
</cpu>
<clock offset="localtime">
<timer name="rtc" present="no" tickpolicy="catchup"/>
<timer name="pit" present="no" tickpolicy="delay"/>
<timer name="hpet" present="no"/>
<timer name="kvmclock" present="no"/>
<timer name="hypervclock" present="yes"/>
</clock>
<on_poweroff>destroy</on_poweroff>
<on_reboot>restart</on_reboot>
<on_crash>destroy</on_crash>
<pm>
<suspend-to-mem enabled="no"/>
<suspend-to-disk enabled="no"/>
</pm>
<devices>
<emulator>/usr/bin/qemu-system-x86_64</emulator>
<disk type="file" device="disk">
<driver name="qemu" type="raw"/>
<source file="/var/lib/libvirt/images/win10-os.img"/>
<target dev="sda" bus="sata"/>
<address type="drive" controller="0" bus="0" target="0" unit="0"/>
</disk>
<disk type="file" device="cdrom">
<driver name="qemu" type="raw"/>
<source file="/home/jakob/isoFiles/Win10_22H2_EnglishInternational_x64.iso"/>
<target dev="sdb" bus="sata"/>
<readonly/>
<address type="drive" controller="0" bus="0" target="0" unit="1"/>
</disk>
<controller type="usb" index="0" model="qemu-xhci" ports="15">
<address type="pci" domain="0x0000" bus="0x02" slot="0x00" function="0x0"/>
</controller>
<controller type="pci" index="0" model="pcie-root"/>
<controller type="pci" index="1" model="pcie-root-port">
<model name="pcie-root-port"/>
<target chassis="1" port="0x10"/>
<address type="pci" domain="0x0000" bus="0x00" slot="0x02" function="0x0" multifunction="on"/>
</controller>
<controller type="pci" index="2" model="pcie-root-port">
<model name="pcie-root-port"/>
<target chassis="2" port="0x11"/>
<address type="pci" domain="0x0000" bus="0x00" slot="0x02" function="0x1"/>
</controller>
<controller type="pci" index="3" model="pcie-root-port">
<model name="pcie-root-port"/>
<target chassis="3" port="0x12"/>
<address type="pci" domain="0x0000" bus="0x00" slot="0x02" function="0x2"/>
</controller>
<controller type="pci" index="4" model="pcie-root-port">
<model name="pcie-root-port"/>
<target chassis="4" port="0x13"/>
<address type="pci" domain="0x0000" bus="0x00" slot="0x02" function="0x3"/>
</controller>
<controller type="pci" index="5" model="pcie-root-port">
<model name="pcie-root-port"/>
<target chassis="5" port="0x14"/>
<address type="pci" domain="0x0000" bus="0x00" slot="0x02" function="0x4"/>
</controller>
<controller type="pci" index="6" model="pcie-root-port">
<model name="pcie-root-port"/>
<target chassis="6" port="0x15"/>
<address type="pci" domain="0x0000" bus="0x00" slot="0x02" function="0x5"/>
</controller>
<controller type="pci" index="7" model="pcie-root-port">
<model name="pcie-root-port"/>
<target chassis="7" port="0x16"/>
<address type="pci" domain="0x0000" bus="0x00" slot="0x02" function="0x6"/>
</controller>
<controller type="pci" index="8" model="pcie-root-port">
<model name="pcie-root-port"/>
<target chassis="8" port="0x17"/>
<address type="pci" domain="0x0000" bus="0x00" slot="0x02" function="0x7"/>
</controller>
<controller type="pci" index="9" model="pcie-root-port">
<model name="pcie-root-port"/>
<target chassis="9" port="0x18"/>
<address type="pci" domain="0x0000" bus="0x00" slot="0x03" function="0x0" multifunction="on"/>
</controller>
<controller type="pci" index="10" model="pcie-root-port">
<model name="pcie-root-port"/>
<target chassis="10" port="0x19"/>
<address type="pci" domain="0x0000" bus="0x00" slot="0x03" function="0x1"/>
</controller>
<controller type="pci" index="11" model="pcie-root-port">
<model name="pcie-root-port"/>
<target chassis="11" port="0x1a"/>
<address type="pci" domain="0x0000" bus="0x00" slot="0x03" function="0x2"/>
</controller>
<controller type="pci" index="12" model="pcie-root-port">
<model name="pcie-root-port"/>
<target chassis="12" port="0x1b"/>
<address type="pci" domain="0x0000" bus="0x00" slot="0x03" function="0x3"/>
</controller>
<controller type="pci" index="13" model="pcie-root-port">
<model name="pcie-root-port"/>
<target chassis="13" port="0x1c"/>
<address type="pci" domain="0x0000" bus="0x00" slot="0x03" function="0x4"/>
</controller>
<controller type="pci" index="14" model="pcie-root-port">
<model name="pcie-root-port"/>
<target chassis="14" port="0x1d"/>
<address type="pci" domain="0x0000" bus="0x00" slot="0x03" function="0x5"/>
</controller>
<controller type="sata" index="0">
<address type="pci" domain="0x0000" bus="0x00" slot="0x1f" function="0x2"/>
</controller>
<controller type="virtio-serial" index="0">
<address type="pci" domain="0x0000" bus="0x03" slot="0x00" function="0x0"/>
</controller>
<interface type="network">
<mac address="52:54:00:a0:28:f5"/>
<source network="default"/>
<model type="e1000e"/>
<address type="pci" domain="0x0000" bus="0x01" slot="0x00" function="0x0"/>
</interface>
<serial type="pty">
<target type="isa-serial" port="0">
<model name="isa-serial"/>
</target>
</serial>
<console type="pty">
<target type="serial" port="0"/>
</console>
<input type="mouse" bus="ps2"/>
<input type="keyboard" bus="ps2"/>
<sound model="ich9">
<address type="pci" domain="0x0000" bus="0x00" slot="0x1b" function="0x0"/>
</sound>
<audio id="1" type="none"/>
<video>
<model type="none"/>
</video>
<hostdev mode="subsystem" type="pci" managed="yes">
<source>
<address domain="0x0000" bus="0x10" slot="0x00" function="0x0"/>
</source>
<address type="pci" domain="0x0000" bus="0x04" slot="0x00" function="0x0"/>
</hostdev>
<hostdev mode="subsystem" type="pci" managed="yes">
<source>
<address domain="0x0000" bus="0x10" slot="0x00" function="0x1"/>
</source>
<address type="pci" domain="0x0000" bus="0x05" slot="0x00" function="0x0"/>
</hostdev>
<hostdev mode="subsystem" type="usb" managed="yes">
<source>
<vendor id="0x1532"/>
<product id="0x021e"/>
</source>
<address type="usb" bus="0" port="1"/>
</hostdev>
<hostdev mode="subsystem" type="usb" managed="yes">
<source>
<vendor id="0x1532"/>
<product id="0x005c"/>
</source>
<address type="usb" bus="0" port="2"/>
</hostdev>
<memballoon model="virtio">
<address type="pci" domain="0x0000" bus="0x06" slot="0x00" function="0x0"/>
</memballoon>
</devices>
</domain>

Last edited by Woodiii (2022-10-26 15:01:22)

manifault · 2022-10-23 23:56:41

I'm by no means a KVM expert, though I've worked quite a bit on other hypervisors. I wouldn't expect this to matter because the VM shouldn't be doing an excessive amount of I/O with all the RAM you're giving it, but just out of curiosity, does it help if you pin the iothreads to unused cores? Right now iothread 1 could be contending with cores on your host OS, and iothread 2 could be contending with vcpu0 and vcpu1.

Also, and probably more importantly, I assume you've enabled AMD-V in your BIOS?

Woodiii · 2022-10-25 13:15:57

Hello,

Yes I've enabled AMD-V in my BIOS.
I've tried to adjust the cpu settings in kvm, but it didn't lead to a satisfying result...

I did some more research and did a few benchmarks. I have 120FPS in the benchmarks, but I have 5 FPS in the display. That's why I did some research on the Display spice theme. I've found that looking glass seems to work better than display spice. Therefore I have 2 questions:

1. In your experience, does looking-glass bring a significant performance boost compared to display spice?
2. Is it possible that I just plug my HDMI cable into the graphics card and when I start the VM the VM output is just displayed on the monitor? So just skip display spice/looking glass, so to speak.

Regards,
Woodiii

Last edited by Woodiii (2022-10-25 13:17:44)

unnilquadium · 2022-10-26 12:25:14

Woodiii wrote:

2. Is it possible that I just plug my HDMI cable into the graphics card and when I start the VM the VM output is just displayed on the monitor? So just skip display spice/looking glass, so to speak.

That's exactly what you are supposed to do. You must use your graphics card output for graphics.
Remember that you are passing through a piece of hardware, so you must use it as if it were connected to a real computer. Check if your monitor has a second input and plug your VM's GPU to it, then just change inputs when you start your VM.

As a suggestion, ditch all spice devices from your VM including spice graphics, spice redirectors, etc...
If you want to use a usb device on your VM (for example keyboard, mouse) just pass those through to the VM (you will not be able to use those on your host while the VM is running, but you will not have to deal with spice and its wonkiness.
Another option is to get a pcie usb controller card (you can easily find these for around €15 ~ €20) and pass that to the VM as well. Then just plug the usb stuff that you want to use with your VM into that controller card.

Woodiii · 2022-10-26 13:05:20

unnilquadium wrote:

That's exactly what you are supposed to do. You must use your graphics card output for graphics.
Remember that you are passing through a piece of hardware, so you must use it as if it were connected to a real computer. Check if your monitor has a second input and plug your VM's GPU to it, then just change inputs when you start your VM.
As a suggestion, ditch all spice devices from your VM including spice graphics, spice redirectors, etc...
If you want to use a usb device on your VM (for example keyboard, mouse) just pass those through to the VM (you will not be able to use those on your host while the VM is running, but you will not have to deal with spice and its wonkiness.

So,
I have to install Windows 10, install the NVIDIA drivers and the virtio guest tools. After I've verified that the graphics card is working properly I shut down the VM.
Then I remove all the display spice stuff and if I spin up the VM, it automatically connects to my monitor, which is plugged in my NVIDIA GPU. (My second monitor is plugged in my internal GPU of my CPU)

Or do I have to configure something in kvm, so that the vm uses the graphic card as display output?

Regards,
Woodiii

Woodiii · 2022-10-26 14:59:28

Wuhuu!

I've finally got it running. I will update my VM config.xml in the leading post.
I installed Windows 10, the NVIDIA Display Drivers, made all the Windows updates.
After this I removed everything related to spice and set Video to none.
I also passed through my keyboard and mouse, which works just fine.
Now when I boot the VM I get even better performance than on my native install
I hope Anti-Cheat won't fuck up the expierience, but we'll see...

Thank you for your help!

Regards,
Woodiii

unnilquadium · 2022-10-26 17:52:08

Congrtats. Hope you enjoy it.
Anti-cheat shouldn't be any problem since for all that matters it is running under a Windows OS.
Regards.

Arch Linux

#1 2022-10-21 14:19:53

[SOLVED] KVM/QEMU GPU Passthrough Performance Issues

#2 2022-10-23 23:56:41

Re: [SOLVED] KVM/QEMU GPU Passthrough Performance Issues

#3 2022-10-25 13:15:57

Re: [SOLVED] KVM/QEMU GPU Passthrough Performance Issues

#4 2022-10-26 12:25:14

Re: [SOLVED] KVM/QEMU GPU Passthrough Performance Issues

#5 2022-10-26 13:05:20

Re: [SOLVED] KVM/QEMU GPU Passthrough Performance Issues

#6 2022-10-26 14:59:28

Re: [SOLVED] KVM/QEMU GPU Passthrough Performance Issues

#7 2022-10-26 17:52:08

Re: [SOLVED] KVM/QEMU GPU Passthrough Performance Issues

Board footer