Arch Linux

Ansa89

Member

Registered: 2014-08-30

Posts: 20

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

I'm using a dedicated physical disk through lvm (I created a virtual disk using the whole physical disk and passed it to qemu).

Maybe it's a collision of the host and guest LVM since you're using a whole disk and it will be visible to both.  Is the guest volume group active in the host?

1) LVM is used only on the host to create a virtual disk for the guest (while the guest doesn't use LVM and sees the virtual disk as a normal one).
2) The physical disk is visible only on the host; the virtual disk is visible to both host and guest (again: the guest sees it as a normal disk).
3) The LVM virtual disk is used only by the guest (and by qemu on the host).

PS: what are the benefits of using virtio-blk instead of virtio-scsi?

JohnyPea

Member

Registered: 2014-07-17

Posts: 17

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

2. I can't figure out how to pipe only the sound through spice. Work-around: use Xen for video card type.

It works when you have the VM client open and connected (virt-manager, spicy). This requires virtual graphics, which can be disabled in quest, but I think a better solution is to use pulseaudio server directly. I have found information on this somewhere earlier in this thread:

/etc/pulse/default.pa:
..
load-module module-native-protocol-unix auth-anonymous=1
..

and with added configuration in VM xml:

  <qemu:commandline>
    <qemu:env name='QEMU_AUDIO_DAC_FIXED_FREQ' value='48000'/>
    <qemu:env name='QEMU_AUDIO_ADC_FIXED_FREQ' value='48000'/>
    <qemu:env name='QEMU_PA_SAMPLES' value='1024'/>
    <qemu:env name='QEMU_AUDIO_DRV' value='pa'/>
    <qemu:env name='QEMU_PA_SERVER' value='/run/user/1000/pulse/native'/>
  </qemu:commandline>

In guest, you are then using virtual sound card configured in xml the standard way.
It doesnt work if you are running qemu as root (or not dropping capabilities through libvirt..?), but thanks to aws qemu wrapper script it works for me perfectly (permissions problems without it). You have to have logged in session on host, too.
I didn't dive into security of pulseaudio server after this, there could be some problems.

JohnyPea

Member

Registered: 2014-07-17

Posts: 17

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

PS: what are the benefits of using virtio-blk instead of virtio-scsi?

With virtio-blk you can use x-data-plane=on, which increases IOPS in guest and lowers overhead in host. In windows guest I wasn't able to pass trim operations through virtio-blk as opposed to virtio-scsi. Later I found a workaround to this by using detect-zeroes=on with discard='unmap' and writing zeros to desired sectors (zero out unused disk space inside guest).

4kGamer

Member

Registered: 2014-10-29

Posts: 88

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Hi, I have a weird issue. I haven't been able to solve this. When I start Windows 8.1 a message pops up in Arch. It says:

vcpu 2 ignored rdmsr kvm
vcpu 4 ignored rdmsr kvm
vcpu 6 ignored rdmsr kvm

This repeats itself for as long as I am running the VM.

I have a i7-4790k and here is some more info about it:

lscpu:

Architecture:          x86_64
CPU op-mode(s):             32-bit, 64-bit
Byte Order:                       Little Endian
CPU(s):                            8
On-line CPU(s) list:   0-7
Thread(s) per core:    2
Core(s) per socket:    4
Socket(s):                  1
NUMA node(s):          1
Vendor ID:                 GenuineIntel
CPU family:               6
Model:                       60
Model name:            Intel(R) Core(TM) i7-4790K CPU @ 4.00GHz
Stepping:                  3
CPU MHz:                4392.031
CPU max MHz:        4400.0000
CPU min MHz:         800.0000
BogoMIPS:               8000.34
Virtualization:            VT-x
L1d cache:                32K
L1i cache:                 32K
L2 cache:                  256K
L3 cache:                  8192K
NUMA node0 CPU(s):     0-7

virsh capabilities shows this

<topology>
      <cells num='1'>
        <cell id='0'>
          <memory unit='KiB'>12061780</memory>
          <pages unit='KiB' size='4'>3015445</pages>
          <pages unit='KiB' size='2048'>0</pages>
          <distances>
            <sibling id='0' value='10'/>
          </distances>
          <cpus num='8'>
            <cpu id='0' socket_id='0' core_id='0' siblings='0,4'/>
            <cpu id='1' socket_id='0' core_id='1' siblings='1,5'/>
            <cpu id='2' socket_id='0' core_id='2' siblings='2,6'/>
            <cpu id='3' socket_id='0' core_id='3' siblings='3,7'/>
            <cpu id='4' socket_id='0' core_id='0' siblings='0,4'/>
            <cpu id='5' socket_id='0' core_id='1' siblings='1,5'/>
            <cpu id='6' socket_id='0' core_id='2' siblings='2,6'/>
            <cpu id='7' socket_id='0' core_id='3' siblings='3,7'/>
          </cpus>
        </cell>
      </cells>
    </topology>

this is what my XML file looks like. 

<vcpu current="7" placement="static">8</vcpu>
-<cputune>
<vcpupin cpuset="0" vcpu="0"/>
<vcpupin cpuset="1" vcpu="1"/>
<vcpupin cpuset="2" vcpu="2"/>
<vcpupin cpuset="3" vcpu="3"/>
<vcpupin cpuset="4" vcpu="4"/>
<vcpupin cpuset="5" vcpu="5"/>
<vcpupin cpuset="6" vcpu="6"/>
</cputune>

I also tried different settings like only this:

<vcpu placement='static'>7</vcpu>

or like this:

<vcpu placement='static' current='4'>8</vcpu>
  <cputune>
    <vcpupin vcpu='0' cpuset='0,4'/>
    <vcpupin vcpu='1' cpuset='1,5'/>
    <vcpupin vcpu='2' cpuset='2,6'/>
    <vcpupin vcpu='3' cpuset='3,7'/>
  </cputune>

I have to say - and maybe you guys can even tell from my post, I am not even sure what's relevant here and what not. So if you need any more info just tell me what you need.
I am really grateful for your help.

Last edited by 4kGamer (2014-12-29 20:51:49)

Duelist

Member

Registered: 2014-09-22

Posts: 358

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Hi, I have a weird issue. I haven't been able to solve this. When I start Windows 8.1 a message pops up in Arch. It says:

vcpu 2 ignored rdmsr kvm
vcpu 4 ignored rdmsr kvm
vcpu 6 ignored rdmsr kvm

It says that virtual CPU number X didn't reply to command "read MSR(model specific register) KVM". Do you passthrough an nvidia card? If so, i suppose, you have kvm=off somewhere, hiding KVM? Something hints me that it is related to your errors. BTW, that seems like just a warning, so you can safely ignore it.

And a thing to test:

Some applications like Passmark Performance Test and SiSoftware Sandra crash the VM without this:

echo "options kvm ignore_msrs=1" >> /etc/modprobe.d/kvm.conf

That may help too.

When i've launched some software like CPU-Z, i'd get some of these messages, but generally they're harmless.

---

The forum rules prohibit requesting support for distributions other than arch.
I gave up. It was too late.
What I was trying to do.
The reference about VFIO and KVM VGA passthrough.

4kGamer

Member

Registered: 2014-10-29

Posts: 88

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Yes, I do have an Nvidia Card. The MSI 980 GTX.

I'm relieved, if that error message is harmless.

I do have the option kvm ignore_msrs=1. I also have other options (like vfio_iommu_type1 disable_hugepages=1), but they are all under one file: /etc/modprobe.d/modprobe.conf

Does it matter what name the .conf file in /etc/modprobe.d/ has?

adejour

Member

Registered: 2014-04-29

Posts: 6

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Whaaaa !
Incredible..

I'm stuck since 1 year on this no-reboot issue, and with this driver, it works (I'm not passing audio for the moment) !!

I was able to install the drivers but upon reboot, got stuck in a boot loop.

Figured it could be driver related so I tried the latest catalyst beta drivers (14.11.2) and it worked! Audio and video are passing through with no problems.

The above also fixed an issue I had with video card freezing (stuck image) on reboot/shutdown so no need for no-reboot.

The only problem I have now is that I can't see the bios(seabios) screen but that's minor given it still boots into whatever drive is first. Strangely enough, OVMF(UEFI) works fine. Maybe something with the card.

In any case, glad to get it working after a week!

Thanks AW and others for the help.

To those running an r9 290x...use the 14.11.2 drivers!

Just a little resumé of my configuration :
MB : ASUS H87M-PLUS
GPU : Radeon R7 260X
RAM : 8Gb (4Gb passed to guest)
CPU : i5 4570
Kernel : 3.15.1 compiled locally for pci passthrough
Guest : Windows 7 Home Prem. x64
CCC : 14.11.2 beta drivers  (17/11/2014)

And I have to notice that the last 14.12 drivers (they say "14.501.1003" Display Driver ver. - 9/12/2014) don't work (still the no-reboot problems)!!!!

So, I would advice to everyone having (reboot) issue to passthrough R7 & R9 to try these drivers.

Furthermore, the Windows Benchmark gives some good results I guess :
CPU : 5.9
Memory : 7.3
Graphic : 7.7
Gaming graphic : 7.7
HD : 7.9

Remaing issue for the moment :

- sometimes (can't reproduce), I have a blue screen "MEMORY MANAGEMENT"
- tried to launch DOTA 2, but CPU get to 100% and the screen of the game stay black (I can come back to the desktop & kill the dota 2 process)

Here is my config, I saw it is quite different than kidchunks one (https://bbs.archlinux.org/viewtopic.php … 5#p1481075)  :
If you have any advice :

/usr/bin/vfio-bind 0000:01:00.0 0000:01:00.1 # Radeon R7
/usr/bin/vfio-bind 0000:00:1d.0 # USB on back side
qemu-system-x86_64 \
-M q35 -enable-kvm -balloon none -rtc clock=host -cpu host \
-m 4G -smp 1,sockets=1,cores=1,threads=1 \
-bios /usr/share/qemu/bios.bin \
-acpitable file=/sys/firmware/acpi/tables/SLIC \
-serial null -parallel null \
-net nic,model=virtio,macaddr=xx:xx:xx:xx:xx:xx \
-net nic -net tap,ifname=tap2,script=no,downscript=no \
-monitor none -display none \
-vga none \
-device ioh3420,bus=pcie.0,addr=1c.0,multifunction=on,port=1,chassis=1,id=root.1 \
-device virtio-scsi-pci,id=scsi \
-drive file=VM/win7.img,id=disk0,format=raw -device scsi-hd,drive=disk0 \
-device vfio-pci,host=01:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on \
-device vfio-pci,host=00:1d.0,bus=pcie.0 \
-drive file=iso/Win7HomePrem64.X15-65739.iso,id=isocd -device scsi-cd,drive=isocd \
-drive file=iso/virtio-win-0.1-94.iso,id=virtiocd -device ide-cd,bus=ide.1,drive=virtiocd

Silar

Member

Registered: 2014-12-24

Posts: 2

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Dump the ROM or get it from techpowerup database and use the romfile= option to provide the ROM to the guest.

I tried both, dumping die ROM and get it from techpowerup. Now i get no more error messages and qemu starts but i get no video signal from the guests graphic card. In addition the host freeze after a few seconds. And i don't know what happens and how to fix this problem.

nbhs

Member

From: Montevideo, Uruguay

Registered: 2013-05-02

Posts: 402

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Dump the ROM or get it from techpowerup database and use the romfile= option to provide the ROM to the guest.

I tried both, dumping die ROM and get it from techpowerup. Now i get no more error messages and qemu starts but i get no video signal from the guests graphic card. In addition the host freeze after a few seconds. And i don't know what happens and how to fix this problem.

I had a similar problem with my GTX470, the first run worked great, but after a guest reboot i had the same problem as you do.

What i did was, reboot the host, start the guest up with the GTX470 as primary, dump the rom with gpu-z, copy it back to your host, shut down the guest, then load the gpu rom it like this:

-device vfio-pci,host=07:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on,romfile=/path/to/your/gpu/bios.bin

I found out that in my case, if i ran the guest for a second time WITHOUT the romfile, i wouldnt be able to start it again even if i actually loaded the romfile.

Last edited by nbhs (2014-12-30 11:32:17)

Bronek

Member

From: London

Registered: 2014-02-14

Posts: 123

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

I need a bit of help here. My system is Supermicro X9DA7 with two Xeon E5-2630v2 . Kernel version 3.18.1 mainline with no patches, qemu version 2.2.0 (with no patches either), seabios 1.7.5-2

Relevant dmesg excerpt:

[bronek@gdansk ~]$ dmesg | grep -iE "iommu|stub"                                                                              
[    0.000000] Command line: BOOT_IMAGE=../vmlinuz-linux zfs=zroot rw quiet edac_core.edac_mc_panic_on_ue=1 elevator=noop pci-stub.ids=1002:6749,1002:aa90,1002:67b0,1002:aac8 intel_iommu=on spl.spl_hostid=0xbb51d4d8 initrd=../initramfs-linux.img
[    0.000000] Kernel command line: BOOT_IMAGE=../vmlinuz-linux zfs=zroot rw quiet edac_core.edac_mc_panic_on_ue=1 elevator=noop pci-stub.ids=1002:6749,1002:aa90,1002:67b0,1002:aac8 intel_iommu=on spl.spl_hostid=0xbb51d4d8 initrd=../initramfs-linux.img
[    0.000000] Intel-IOMMU: enabled
[    0.086314] dmar: IOMMU 0: reg_base_addr fbffe000 ver 1:0 cap d2078c106f0466 ecap f020de
[    0.086320] dmar: IOMMU 1: reg_base_addr bfffc000 ver 1:0 cap d2078c106f0466 ecap f020de
[    0.086451] IOAPIC id 3 under DRHD base  0xfbffe000 IOMMU 0
[    0.086453] IOAPIC id 0 under DRHD base  0xbfffc000 IOMMU 1
[    0.086454] IOAPIC id 2 under DRHD base  0xbfffc000 IOMMU 1
[    1.135393] IOMMU 0 0xfbffe000: using Queued invalidation
[    1.135394] IOMMU 1 0xbfffc000: using Queued invalidation
[    1.135400] IOMMU: Setting RMRR:
[    1.135423] IOMMU: Setting identity map for device 0000:00:1a.0 [0x7f23d000 - 0x7f24bfff]
[    1.135470] IOMMU: Setting identity map for device 0000:00:1d.0 [0x7f23d000 - 0x7f24bfff]
[    1.135507] IOMMU: Setting identity map for device 0000:07:00.0 [0x7f23d000 - 0x7f24bfff]
[    1.135555] IOMMU: Setting identity map for device 0000:08:00.0 [0x7f23d000 - 0x7f24bfff]
[    1.135608] IOMMU: Setting identity map for device 0000:83:00.0 [0x7f23d000 - 0x7f24bfff]
[    1.135686] IOMMU: Setting identity map for device 0000:84:00.0 [0x7f23d000 - 0x7f24bfff]
[    1.135742] IOMMU: Prepare 0-16MiB unity mapping for LPC
[    1.135760] IOMMU: Setting identity map for device 0000:00:1f.0 [0x0 - 0xffffff]
[    1.151027] pci-stub: add 1002:6749 sub=FFFFFFFF:FFFFFFFF cls=00000000/00000000
[    1.151034] pci-stub 0000:03:00.0: claimed by stub
[    1.151042] pci-stub: add 1002:AA90 sub=FFFFFFFF:FFFFFFFF cls=00000000/00000000
[    1.151059] pci-stub 0000:03:00.1: claimed by stub
[    1.151067] pci-stub: add 1002:67B0 sub=FFFFFFFF:FFFFFFFF cls=00000000/00000000
[    1.151090] pci-stub 0000:82:00.0: claimed by stub
[    1.151107] pci-stub: add 1002:AAC8 sub=FFFFFFFF:FFFFFFFF cls=00000000/00000000
[    1.151137] pci-stub 0000:82:00.1: claimed by stub

As you can see, I have two GPU cards from AMD, both claimed by pci-stub

[bronek@gdansk ~]$ lspci -nn | grep 1002
03:00.0 VGA compatible controller [0300]: Advanced Micro Devices, Inc. [AMD/ATI] Turks GL [FirePro V4900] [1002:6749]
03:00.1 Audio device [0403]: Advanced Micro Devices, Inc. [AMD/ATI] Turks/Whistler HDMI Audio [Radeon HD 6000 Series] [1002:aa90]
82:00.0 VGA compatible controller [0300]: Advanced Micro Devices, Inc. [AMD/ATI] Hawaii XT [Radeon R9 290X] [1002:67b0]
82:00.1 Audio device [0403]: Advanced Micro Devices, Inc. [AMD/ATI] Device [1002:aac8]

Also both cards are passed successfully to vfio-pci by script presented in the first post on this thread:

[bronek@gdansk ~]$ ls -al /sys/bus/pci/devices/0000:03:00.1/driver
lrwxrwxrwx 1 root root 0 Dec 30 12:59 /sys/bus/pci/devices/0000:03:00.1/driver -> ../../../../bus/pci/drivers/vfio-pci
[bronek@gdansk ~]$ ls -al /sys/bus/pci/devices/0000:03:00.0/driver
lrwxrwxrwx 1 root root 0 Dec 30 12:59 /sys/bus/pci/devices/0000:03:00.0/driver -> ../../../../bus/pci/drivers/vfio-pci
[bronek@gdansk ~]$ ls -al /sys/bus/pci/devices/0000:82:00.1/driver
lrwxrwxrwx 1 root root 0 Dec 30 12:59 /sys/bus/pci/devices/0000:82:00.1/driver -> ../../../../bus/pci/drivers/vfio-pci
[bronek@gdansk ~]$ ls -al /sys/bus/pci/devices/0000:82:00.0/driver
lrwxrwxrwx 1 root root 0 Dec 30 12:59 /sys/bus/pci/devices/0000:82:00.0/driver -> ../../../../bus/pci/drivers/vfio-pci

When trying VGA passthrough I get the following:

[root@gdansk bronek]# cat trypc.sh 
#!/bin/bash
qemu-system-x86_64 -enable-kvm -M pc -m 1024 -cpu host -smp 6 \
-bios /usr/share/qemu/bios.bin \
-nographic -vga none \
-device vfio-pci,host=03:00.0,addr=08.0,multifunction=on,x-vga=on \
-device vfio-pci,host=03:00.1,addr=08.1
[root@gdansk bronek]# ./trypc.sh 
qemu-system-x86_64: -device vfio-pci,host=03:00.0,addr=08.0,multifunction=on,x-vga=on: vfio: No available IOMMU models
qemu-system-x86_64: -device vfio-pci,host=03:00.0,addr=08.0,multifunction=on,x-vga=on: vfio: failed to setup container for group 18
qemu-system-x86_64: -device vfio-pci,host=03:00.0,addr=08.0,multifunction=on,x-vga=on: vfio: failed to get group 18
qemu-system-x86_64: -device vfio-pci,host=03:00.0,addr=08.0,multifunction=on,x-vga=on: Device initialization failed.
qemu-system-x86_64: -device vfio-pci,host=03:00.0,addr=08.0,multifunction=on,x-vga=on: Device 'vfio-pci' could not be initialized

I receive the same error for both cards and also when using -M q35 (with ioh3420). In case that matters, this machine is intended as headless VDI server and has no X windows installed. Radeon driver is blacklisted. I do not have ACS patch as it does not seem necessary in this case:

[bronek@gdansk ~]$ ls /sys/bus/pci/devices/0000:03:00.0/iommu_group/devices  
0000:03:00.0  0000:03:00.1

Also, I am not using IGP thus no arb patch and also no option allow_unsafe_interrupts=1, as it seems not necessary for my motherboard. I did try it though, with no effect.

I was following this thread from start (yes, started last year!) and I did try this feature previously using older kernel and older qemu version, with little more success (was able to install Linux guest, but not Windows). This time I'm stuck at the very beginning. Perhaps if I downgraded qemu or kernel it might help a little ?

Last edited by Bronek (2014-12-30 14:14:54)

aw

Member

Registered: 2013-10-04

Posts: 921

Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

[root@gdansk bronek]# cat trypc.sh 
#!/bin/bash
qemu-system-x86_64 -enable-kvm -M pc -m 1024 -cpu host -smp 6 \
-bios /usr/share/qemu/bios.bin \
-nographic -vga none \
-device vfio-pci,host=03:00.0,addr=08.0,multifunction=on,x-vga=on \
-device vfio-pci,host=03:00.1,addr=08.1
[root@gdansk bronek]# ./trypc.sh 
qemu-system-x86_64: -device vfio-pci,host=03:00.0,addr=08.0,multifunction=on,x-vga=on: vfio: No available IOMMU models
qemu-system-x86_64: -device vfio-pci,host=03:00.0,addr=08.0,multifunction=on,x-vga=on: vfio: failed to setup container for group 18
qemu-system-x86_64: -device vfio-pci,host=03:00.0,addr=08.0,multifunction=on,x-vga=on: vfio: failed to get group 18
qemu-system-x86_64: -device vfio-pci,host=03:00.0,addr=08.0,multifunction=on,x-vga=on: Device initialization failed.
qemu-system-x86_64: -device vfio-pci,host=03:00.0,addr=08.0,multifunction=on,x-vga=on: Device 'vfio-pci' could not be initialized

What does 'grep VFIO /boot/config-`uname -r`' show?  lsmod | grep vfio?

---

http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Bronek

Member

From: London

Registered: 2014-02-14

Posts: 123

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Hi Alex thanks for prompt reply. I think ArchLinux does not save a config copy to /boot , but I have this:

[bronek@gdansk ~]$ zcat /proc/config.gz | grep VFIO
CONFIG_VFIO_IOMMU_TYPE1=m
CONFIG_VFIO=m
CONFIG_VFIO_PCI=m
CONFIG_VFIO_PCI_VGA=y
CONFIG_KVM_VFIO=y

[bronek@gdansk ~]$  lsmod | grep vfio
vfio_pci               31566  0 
vfio                   18519  1 vfio_pci

aw

Member

Registered: 2013-10-04

Posts: 921

Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Hi Alex thanks for prompt reply. I think ArchLinux does not save a config copy to /boot , but I have this:

[bronek@gdansk ~]$ zcat /proc/config.gz | grep VFIO
CONFIG_VFIO_IOMMU_TYPE1=m
CONFIG_VFIO=m
CONFIG_VFIO_PCI=m
CONFIG_VFIO_PCI_VGA=y
CONFIG_KVM_VFIO=y

[bronek@gdansk ~]$  lsmod | grep vfio
vfio_pci               31566  0 
vfio                   18519  1 vfio_pci

So somehow vfio_iommu_type1 is not getting auto-loaded.  The way it's supposed to work is that vfio-pci depends on vfio, so loading vfio-pci always brings in vfio.  vfio then does a module request for any available iommu drivers to load those.  This is where vfio_iommu_type1 is supposed to get loaded.  Maybe something has broken in your userspace that the module request is not being serviced?  Anyway, you should be able to manually `modprobe vfio_iommu_type1`

---

http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Bronek

Member

From: London

Registered: 2014-02-14

Posts: 123

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

This is where vfio_iommu_type1 is supposed to get loaded.  Maybe something has broken in your userspace that the module request is not being serviced?  Anyway, you should be able to manually `modprobe vfio_iommu_type1`

Thank you, this has helped!

[root@gdansk bronek]# modprobe vfio_iommu_type1
[root@gdansk bronek]# lsmod | grep vfio 
vfio_iommu_type1       17163  0 
vfio_pci               31566  0 
vfio                   18519  2 vfio_iommu_type1,vfio_pci

I will add this module to /etc/mkinitcpio.conf to ensure it is always automatically loaded.

Also,  I suppose I need your other arb patch to which you refer in your  blog http://vfio.blogspot.co.uk/2014/08/vfiovga-faq.html question 4  - since after closing qemu (SIGTERM from other session) I got a bit of mess on my VGA screen.

EDIT Never mind, I see your patch is already applied in my kernel version (3.18.1) ! I guess the mess I've got is simply result of qemu being cautions on handling SIGTERM

Last edited by Bronek (2014-12-30 15:32:29)

adejour

Member

Registered: 2014-04-29

Posts: 6

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Hello,

About my last post (vga passthrough working with beta ATI drivers 14.11.2) https://bbs.archlinux.org/viewtopic.php … 2#p1488792

I succeed to make audio working adding  -device vfio-pci,host=01:00.1,bus=pcie.0 to the command line.
Now, all games are working well (tested dota 2 and bioshock infinite).

thank you all !

pkim

Member

Registered: 2014-12-27

Posts: 9

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

2. I can't figure out how to pipe only the sound through spice. Work-around: use Xen for video card type.

It works when you have the VM client open and connected (virt-manager, spicy). This requires virtual graphics, which can be disabled in quest, but I think a better solution is to use pulseaudio server directly. I have found information on this somewhere earlier in this thread:

/etc/pulse/default.pa:
..
load-module module-native-protocol-unix auth-anonymous=1
..

and with added configuration in VM xml:

  <qemu:commandline>
    <qemu:env name='QEMU_AUDIO_DAC_FIXED_FREQ' value='48000'/>
    <qemu:env name='QEMU_AUDIO_ADC_FIXED_FREQ' value='48000'/>
    <qemu:env name='QEMU_PA_SAMPLES' value='1024'/>
    <qemu:env name='QEMU_AUDIO_DRV' value='pa'/>
    <qemu:env name='QEMU_PA_SERVER' value='/run/user/1000/pulse/native'/>
  </qemu:commandline>

In guest, you are then using virtual sound card configured in xml the standard way.
It doesnt work if you are running qemu as root (or not dropping capabilities through libvirt..?), but thanks to aws qemu wrapper script it works for me perfectly (permissions problems without it). You have to have logged in session on host, too.
I didn't dive into security of pulseaudio server after this, there could be some problems.

OK, I tried doing what you suggested, but I couldn't get any sound once I deleted the graphics and video section in the xml file.

On the other hand, I did figure out the glitch/stutter/click issues on the sound. It turns out others are having the same issue with ICH6 and ICH9 - here.

I set up the audio for AC'97 in virt-manager and installed the AC'97 driver in Windows 7 - worked without any glitch! :-)

The only "real" issue now is the PCIe reset - can't reboot without suspending/resuming the host.

Duelist

Member

Registered: 2014-09-22

Posts: 358

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

I'm feeling like my GPU's vendor dumped me.
Some months ago i've tried to update my card's VBIOS on UEFI, it downloaded the image but failed to apply it due to lack of drivers or some other stuff - it wasn't my pc, so i didn't want to mess with owner's windows system like installing drivers.

But i got the ROM. For one card only, but that's fine.
Tried passing it wth romfile and OVMF just hanged, aw suggested to test it on real hardware to see if it's just poorly written and broken. So i gave up for some time.

Now, i've installed windows system on a bare metal system using UEFI+GPT and even secureboot was working. But ASUS UEFI Update tools now says that... my GPU doesn't support UEFI. Considering the fact i have the actual ROM... It seems strange.

So, it's time to begin my journey on writing messages to their support team.

I've got some news i've decided to share with you, folks.

I wrote to the support team, but they're being all "jelly-brained" and stuff. I think that is related to holidays.
Anyway, they've requested a screenshot of main GPU-Z window and provided a VBIOS image that was lacking serial number information but was the same as original. It hints that this information is left for decoration purposes only(or maybe some other non-essential stuff), the main meat-and-bones of ROM will work without it.

That yet remains open my two questions - why the hell their tool won't work anymore, and is it right to flash the bios or it's considered buggy. The journey shall continue.

But i definitely can tell one important thing:
If any of you are trying to use OVMF with a card that "may be updated to support UEFI" - think twice before doing that, updating it may not be so easy.

Last edited by Duelist (2014-12-31 01:06:03)

---

The forum rules prohibit requesting support for distributions other than arch.
I gave up. It was too late.
What I was trying to do.
The reference about VFIO and KVM VGA passthrough.

Hoeze

Member

Registered: 2014-12-30

Posts: 2

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Hi,
how can I disable/re-enable my GPU after stopping a guest?
I'd like to disable my GPU until it's needed to save power...

Last edited by Hoeze (2015-01-02 13:45:26)

rawuza

Member

Registered: 2013-12-22

Posts: 7

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Hi,

I've got a couple questions regarding performance optimizations:

1) Neste Page Tables: I found suggestions to disable NPT (also in this thread). Now why would I want to do this? NPT should speed up memory access. Or is there some interaction between NPT and PCI passthrough that makes it less performant?
2) VCPU pinning: Again, I found suggestions to pin each QEMU thread to a single CPU core. I guess it might in fact help to pin each VCPU thread to a core. But QEMU uses more threads than VCPUS (I guess for IO and additional worker threads). So a naive approach might pin multiple VCPU threads to a single core. Is this suggestion then plane wrong, or am I missing something here? Can I determine which threads are VCPU threads? (I'm talking SMP here, not NUMA, so the effect might be negligible anyway).

aw

Member

Registered: 2013-10-04

Posts: 921

Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Hi,

I've got a couple questions regarding performance optimizations:

1) Neste Page Tables: I found suggestions to disable NPT (also in this thread). Now why would I want to do this? NPT should speed up memory access. Or is there some interaction between NPT and PCI passthrough that makes it less performant?

The suggestions has been do disable nesting support, ie. the ability to run a hypervisor within the guest, not to disable three dimensional paging.  I haven't seen actual benchmarks from doing this, so I'm not sure how worthwhile it is.

2) VCPU pinning: Again, I found suggestions to pin each QEMU thread to a single CPU core. I guess it might in fact help to pin each VCPU thread to a core. But QEMU uses more threads than VCPUS (I guess for IO and additional worker threads). So a naive approach might pin multiple VCPU threads to a single core. Is this suggestion then plane wrong, or am I missing something here? Can I determine which threads are VCPU threads? (I'm talking SMP here, not NUMA, so the effect might be negligible anyway).

Pinning all vCPUs to the same single core would be the worst optimization ever.  The suggestion is to pin each vCPU to a separate pCPU, which gives you cache locality and makes the host scheduler's job easier.  info cpus lists the vCPU threads.  libvirt also has the ability to do vCPU pinning.

---

http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

aw

Member

Registered: 2013-10-04

Posts: 921

Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Hi,
how can I disable/re-enable my GPU after stopping a guest?
I'd like to disable my GPU until it's needed to save power...

Unless you have power controllable slots, the best suggestion might be to leave the guest running.  The guest driver can probably put the card in a fairly low power state when it's idle.  I don't expect to see a lot of power savings from D3 in the host, which is about the best we can offer without device specific drivers.

---

http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

rawuza

Member

Registered: 2013-12-22

Posts: 7

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

The suggestions has been do disable nesting support, ie. the ability to run a hypervisor within the guest, not to disable three dimensional paging.

So that would be the "nested" parameter of kvm_amd. That makes more sense. The comments I found suggested to set "npt" to 0.

Pinning all vCPUs to the same single core would be the worst optimization ever.

Well, yes, I meant pinning each vCPU to a separate core.

info cpus lists the vCPU threads.

That's exactly what I need. Thanks a lot.

PS: I previously had trouble with my old GPU. I've got a new one and it finally works. Quite impressive, I must say.

4kGamer

Member

Registered: 2014-10-29

Posts: 88

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Hi, I'd like to ask Alex and others here about something I discovered. I ran Open Hardware Monitor and under section "Clocks" I get no value for CPU Core #1 till #4.

It's not only for benchmarking, but my case has a screen where I can follow stuff like that constantly. And it's annoying that it says 4x0 Mhz.

Is this expected behaviour or is something wrong?

aw

Member

Registered: 2013-10-04

Posts: 921

Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Hi, I'd like to ask Alex and others here about something I discovered. I ran Open Hardware Monitor and under section "Clocks" I get no value for CPU Core #1 till #4.

It's not only for benchmarking, but my case has a screen where I can follow stuff like that constantly. And it's annoying that it says 4x0 Mhz.

Is this expected behaviour or is something wrong?

Personally I wouldn't necessarily expect a hardware monitor to work in a virtual machine.

---

http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

4kGamer

Member

Registered: 2014-10-29

Posts: 88

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

it works pretty well. It's just the CPU Clock. So something is preventing it from showing the right value.