Arch Linux

kainet

Member

Registered: 2014-12-22

Posts: 9

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

You need to use OVMF, or patch kernel.

Thanks! What patches i should install ?

slis

Member

Registered: 2014-06-02

Posts: 127

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

https://lkml.org/lkml/2014/5/9/517

This one.

Are you having latest qemu? whezzy's is too old i guess.

JohnyPea

Member

Registered: 2014-07-17

Posts: 17

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Hello everybody,

I was watching this thread from its first post, bud had very little success passing through my cards in the beginning. Also didn't beg for help because I'm doing this on ubuntu (please, don't laugh... too much) and tried to figure things out on my own. Thanks to so many useful advices in this thread I have learnt a lot, and also tried a few successful configurations. So thanks all for the contributions.
My actual working configuration:

Host:
- FX-8350, Sabertooth 990FX, 2xNV660Ti 3GB, 32GB RAM
- SSD Samsung 840Pro, Adaptec 2405 RAID0, And a few various big drives
- Ubuntu 15.04
- Self compiled kernel (based on arch config) 3.18.1, 1000HZ, voluntary preemption, builtin pci-back, and other tweaks; works also with stock ubuntu kernel if pci-back is in initramfs, but everything is slower

BOOT_IMAGE=/boot/vmlinuz-3.18.1-custom root=UUID=7ca2c85e-f29e-404d-91c7-92ad1e087465 ro splash ivrs_ioapic[9]=00:14.0 ivrs_ioapic[10]=00:00.1 iommu=pt iommu=1 xen-pciback.hide=(06:00.0)(06:00.1) default_hugepagesz=1G hugepagesz=1G panic=10 hugepages=6 isolcpus=2-7 vt.handoff=7

- qemu 2.2.50 with DEVID patch
- propietery nvidia drivers for primary card on host

Guest(s):
Almost completely through libvirt (thanks to aws wrapper script its pretty neat), I have just the pci-back > vfio-pci in init script (probably can be done in libvirt pre-start script)
Im also using taskset in libvirt event script for compacting all running processes to cores 0-1 and releasing them to all cores after guest shutdown
aws qemu wrapper script was the last thing to tidy up libvirt config and solve the privilege mess, genial:

dzon@xenhost:~$ cat /usr/bin/qemu-vfio
#!/bin/sh
exec qemu-system-x86_64 `echo "\$@" | sed 's|06:00.0|06:00.0,x-vga=on,x-vid=0x10DE,x-did=0x11FA,x-ss-vid=0x10DE,x-ss-did=0x097C,romfile=/usr/share/qemu/NVIDIA.QuadroK4000.3072.120813.rom|g'`

Win8.1, libvirt, xml:

<domain type='kvm' xmlns:qemu='http://libvirt.org/schemas/domain/qemu/1.0'>
  <name>Win81</name>
  <uuid>b61f94a6-08de-455b-9073-95c0c3316501</uuid>
  <memory unit='KiB'>6291456</memory>
  <currentMemory unit='KiB'>6291456</currentMemory>
  <memoryBacking>
    <hugepages/>
  </memoryBacking>
  <vcpu placement='static' cpuset='0-7'>6</vcpu>
  <iothreads>2</iothreads>
  <cputune>
    <vcpupin vcpu='0' cpuset='2'/>
    <vcpupin vcpu='1' cpuset='3'/>
    <vcpupin vcpu='2' cpuset='4'/>
    <vcpupin vcpu='3' cpuset='5'/>
    <vcpupin vcpu='4' cpuset='6'/>
    <vcpupin vcpu='5' cpuset='7'/>
    <emulatorpin cpuset='0-1'/>
    <iothreadpin iothread='1' cpuset='0-1'/>
  </cputune>
  <os>
    <type arch='x86_64' machine='pc-i440fx-2.2'>hvm</type>
    <loader readonly='yes' type='pflash'>/mnt/home/dzon/App/virt/ovmf-x64/OVMF_CODE-pure-efi.fd</loader>
    <nvram>/var/lib/libvirt/qemu/nvram/Win81_VARS.fd</nvram>
    <bios useserial='yes' rebootTimeout='0'/>
  </os>
  <features>
    <acpi/>
    <apic eoi='on'/>
    <pae/>
    <hyperv>
      <relaxed state='on'/>
      <vapic state='on'/>
      <spinlocks state='on' retries='8191'/>
    </hyperv>
    <kvm>
      <hidden state='on'/>
    </kvm>
    <pvspinlock state='on'/>
  </features>
  <cpu mode='host-passthrough'>
    <topology sockets='1' cores='6' threads='1'/>
  </cpu>
  <clock offset='localtime'>
    <timer name='rtc' tickpolicy='catchup'/>
    <timer name='pit' tickpolicy='delay'/>
    <timer name='hpet' present='no'/>
    <timer name='hypervclock' present='yes'/>
  </clock>
  <on_poweroff>destroy</on_poweroff>
  <on_reboot>destroy</on_reboot>
  <on_crash>destroy</on_crash>
  <pm>
    <suspend-to-mem enabled='no'/>
    <suspend-to-disk enabled='no'/>
  </pm>
  <devices>
    <emulator>/usr/bin/qemu-vfio</emulator>
    <disk type='file' device='disk'>
      <driver name='qemu' type='qcow2' cache='none' io='native' discard='unmap'/>
      <source file='/home/VMSSD/Win81.qcow2'/>
      <target dev='sda' bus='scsi'/>
      <boot order='1'/>
      <address type='drive' controller='0' bus='0' target='0' unit='0'/>
    </disk>
    <disk type='file' device='disk'>
      <driver name='qemu' type='raw' cache='none' io='native'/>
      <source file='/dev/sda2'/>
      <target dev='sdc' bus='scsi'/>
      <address type='drive' controller='0' bus='0' target='0' unit='2'/>
    </disk>
    <disk type='file' device='disk'>
      <driver name='qemu' type='qcow2' cache='none' io='native'/>
      <source file='/home/VMSSD/Win81Data.qcow2'/>
      <target dev='sdd' bus='scsi'/>
      <address type='drive' controller='0' bus='0' target='0' unit='3'/>
    </disk>
    <disk type='file' device='disk'>
      <driver name='qemu' type='raw'/>
      <source file='/home/dzon/VM/Win81q.raw'/>
      <target dev='sde' bus='scsi'/>
      <address type='drive' controller='0' bus='0' target='0' unit='4'/>
    </disk>
    <controller type='usb' index='0' model='nec-xhci'>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x05' function='0x0'/>
    </controller>
    <controller type='pci' index='0' model='pci-root'/>
    <controller type='virtio-serial' index='0'>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x06' function='0x0'/>
    </controller>
    <controller type='scsi' index='0' model='virtio-scsi'>
      <driver queues='6'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x07' function='0x0'/>
    </controller>
    <interface type='network'>
      <mac address='52:54:00:a5:e1:5a'/>
      <source network='default'/>
      <model type='virtio'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x0'/>
    </interface>
    <serial type='pty'>
      <target port='0'/>
    </serial>
    <console type='pty'>
      <target type='serial' port='0'/>
    </console>
    <channel type='spicevmc'>
      <target type='virtio' name='com.redhat.spice.0'/>
      <address type='virtio-serial' controller='0' bus='0' port='1'/>
    </channel>
    <input type='mouse' bus='ps2'/>
    <input type='keyboard' bus='ps2'/>
    <graphics type='spice' autoport='yes'/>
    <sound model='ich6'>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x04' function='0x0'/>
    </sound>
    <video>
      <model type='qxl' ram='65536' vram='65536' vgamem='8192' heads='1'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x0'/>
    </video>
    <hostdev mode='subsystem' type='usb' managed='yes'>
      <source>
        <vendor id='0x046d'/>
        <product id='0xc52b'/>
      </source>
    </hostdev>
    <hostdev mode='subsystem' type='pci' managed='yes'>
      <source>
        <address domain='0x0000' bus='0x06' slot='0x00' function='0x0'/>
      </source>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x09' function='0x0'/>
    </hostdev>
    <hostdev mode='subsystem' type='pci' managed='yes'>
      <source>
        <address domain='0x0000' bus='0x06' slot='0x00' function='0x1'/>
      </source>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x0a' function='0x0'/>
    </hostdev>
    <redirdev bus='usb' type='spicevmc'>
    </redirdev>
    <redirdev bus='usb' type='spicevmc'>
    </redirdev>
    <redirdev bus='usb' type='spicevmc'>
    </redirdev>
    <redirdev bus='usb' type='spicevmc'>
    </redirdev>
    <memballoon model='virtio'>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x08' function='0x0'/>
    </memballoon>
  </devices>
  <qemu:commandline>
    <qemu:env name='QEMU_AUDIO_DAC_FIXED_FREQ' value='48000'/>
    <qemu:env name='QEMU_AUDIO_ADC_FIXED_FREQ' value='48000'/>
    <qemu:env name='QEMU_PA_SAMPLES' value='1024'/>
    <qemu:env name='QEMU_AUDIO_DRV' value='pa'/>
    <qemu:env name='QEMU_PA_SERVER' value='/run/user/1000/pulse/native'/>
  </qemu:commandline>
</domain>

dzon@xenhost:~$ cat /etc/libvirt/hooks/qemu
#!/bin/bash
logPath=/home/dzon/libvirt_start
echo "skript_start" $1 $2 >> $logPath
if [ $2 == "start" ] ; then
 echo "start" >> $logPath
 for i in `pgrep -v rcu`; do
  taskset -pc 0-1 $i >> $logPath;
 done
fi
if [ $2 == "release" ] ; then
 echo "shutdown" >> $logPath
 for i in `pgrep -v rcu`; do
  taskset -pc 0-7 $i >> $logPath;
 done
fi
exit 0

What works:
- primary spice graphics with keyboard/mouse grab and usb redirection
- usb redirection (I'm using a sript forr ssh-ing into host through putty and disconnect my redirected keyb/mouse from guset with doubleclick, the same in host for redirection back in live guest)
- secondary passed through NV660Ti quadrified to K4000 with latest drivers in guest
- GPU performance 100% (weirdly sometimes above this as compared to native win install, probably some timing issues)
- CPU performance allmost 100% (6 cores), depends on applications
- HDD performace is great, also using partition passthrough, but not as boot drive - there are references in this thread on how to achieve that
- audio through pulseaudio daemon without a need to have connected client, also the HD audio on card is working fine
- no BSODs or lockups at all, very stable

What doesnt:
- in latest OMVF config the machine isn’t able to reboot, instead a new boot it just shuts down (correctly though)
- one anomaly is that without quadryfying my card the guest thinks that it is in pcieX1 mode, although on host I see it in full x16, after changing IDs and ROM for guest everything is ok
- there is a lot of host kernel time in some games during loading (not iowait) which causes temporary lagging, it seems according to perf that it is caused by _raw_spin_lock which could hint on multiple guest threads racing for timer gets - here I'm stuck but its not a game breaker. According to some of the posts here this could be AMD specific. Still looking for solution, if its possible.

Also tried and worked combinations:
linux guests, Win7, i440fx, q35, qemu cmd line, few low-end AMD and Nvidia cards

Some of the config parts maybe leftovers from previous tests, not everything is necessary for working configuration
Hope some of this info is usefull. Thanks all.

edit:
corrected module name to pci-back - thats what I'm using because of the two identical cards, added (dirty) hook script

Last edited by JohnyPea (2014-12-23 12:36:36)

Duelist

Member

Registered: 2014-09-22

Posts: 358

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

  <on_reboot>destroy</on_reboot>
- in latest OMVF config the machine isn’t able to reboot, instead a new boot it just shuts down (correctly though)

Errr, isn't that's the way it's supposed to work? Also, great setup, great info. Thanks. I think i'm going to adapt some of your settings for my system.

By the way, what benchmarks did you use to check CPU and GPU performance?

---

The forum rules prohibit requesting support for distributions other than arch.
I gave up. It was too late.
What I was trying to do.
The reference about VFIO and KVM VGA passthrough.

JohnyPea

Member

Registered: 2014-07-17

Posts: 17

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

  <on_reboot>destroy</on_reboot>
- in latest OMVF config the machine isn’t able to reboot, instead a new boot it just shuts down (correctly though)

Errr, isn't that's the way it's supposed to work? Also, great setup, great info. Thanks. I think i'm going to adapt some of your settings for my system.

By the way, what benchmarks did you use to check CPU and GPU performance?

OMFG, i'm an idiot. Time to learn how to read again. Thank you very much.
I tried a lot of benchmarks, but basic synthetic ones were SuperPI, Furmark, Fluidmark. For drive performance ATTO disk benchmak. In Win7, I was comparing also builtin performance evaluation. (its possible in Win8, but its too much hassle for me). Also for guite a while I was using benchmark with Metro: Last Light - when the physics engine was acting weird, it is heavy on resources and most errors in machine config manifested even when synthetic benchmarks were running fine.
Noticed just now, that I left a Virtio SCSI disk types in the mentioned guest config. This way the iothreads config part isn't working (only with virtio). AFAIK the SCSI is goot only for trim operations in win guest and single virtual controller for multiple drives for now. Not performance gains. Virtio type is probably a better choice.
If you find anything usefull, I'm glad I could help.
edit: added ATTO benchmark

Last edited by JohnyPea (2014-12-23 18:36:24)

kainet

Member

Registered: 2014-12-22

Posts: 9

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

I managed to install latest nvidia drivers (344.75) on Windows 7 (32 bit) and everything looks ok. Still trying to install driver on Windows 7 64 bit. Still using unpatched kernel and qemu (2.1) from debian wheezy-backports repository.

P.S. Simple tutorial how to compile kernel with optimisations for vm in debian  http://edencomputing.com/index.php/2014 … ebian-way/

Last edited by kainet (2014-12-24 07:06:03)

ailcp

Member

Registered: 2014-12-20

Posts: 4

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

750Ti is better than HD7750 or rare R7 250E. It eats less power, works faster, heats less. It should work. HD7750 isn't compatible with UEFI out-of-the-box, while 750Ti is.(aw has 750 too) Is there any related messages related to VFIO or your GPU in dmesg? Or, maybe, windows doesn't enable MSI? Can you show us "sudo lspci -v" when your VM is running?

I got the "AMD-Vi: Event logged [IO_PAGE_FAULT device=01:00.0" error when the VM is running.  Besides, I borrowed a i5-4460 and H97 motherboard for testing and turned out the same 750Ti worked smoothly on that machine with the same XML.  I guess the code 43 error really lies with FM2/FM2+  CPU + Motherboard + Nvidia GPU combo.  Looking at the google doc again, there is a user "Renfast" who tested two FM2+ motherboard.  He/She was successful with the HD7970 but not the GTX 680.

qquark

Member

Registered: 2014-12-24

Posts: 2

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Hi everyone,

I'm thinking of doing a similar setup, but before starting messing with my system, I wanted to make sure that I'm not wasting my time; I have a few questions that someone here might be able to answer, as I wasn't able to google the answers unfortunately:

I currently have a P6T motherboard with shitty DMAR tables, changing it is not an option, and I sent a support request to Asus, but I don't have any satisfying answer yet (and probably won't get one). There are rumors of a "beta" bios working, but trying to find any info about it seems like a dead end from what I saw for now.

I noticed that apparently one of the RMRR entries is correct, and the other one is not.
Since this is very easy to do, I enabled intel_iommu=on to see what happens, and Linux tells me that while setting up identity mappings it detected that my bios is broken (true, true ...); *however*, after a lengthy log full of call stacks about the devices it tried to map at the second RMRR entry, I get a log showing success (or at least no error) for those same devices at the first RMRR entry.

My first question from this result is whether I have a chance of getting VGA passthrough working in those conditions (?).
Can Linux manage to work with only one entry by ignoring the error in the other ? (assuming the first one is correct AND working) ? And Is there a way to override this entry (I do not know what the RMRR actually represents, but I suspect the answer would be negative).
Also, in my dmesg log, I only find this identity mapping happening for the USB controller devices (or subdevices ?), should I also expect to see my current graphics card there ?

Second topic: I see that most people are generally using setups with {intel,ATI}+{nvidia,ATI}, but I would like to know if there is any problem with an nvidia+nvidia setup (?), as I only have a spare 9800GT I can try in addition to my GTX660, and do not have any recent extra ATI card, nor an integrated intel GP (i7 920). From what I thought I understood, just being able to unbind/rebind the appropriate card should be enough to remove any conflict between host and guest, why are people also blacklisting drivers ? Do they interfere even though they're not bound to the card ? Or is it just an easier way to do boot-time non-binding ?

Thanks for any help in advance !

slis

Member

Registered: 2014-06-02

Posts: 127

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

I can only help you with second topic.
You can use any combination, but you need to patch nvidia host driver for nvidia+nvidia etc... (same/similar goes for radeon).
People blacklist driver instead of using pci-stub I guess to achieve same effect easier (works if u don't have both cards that use same driver).

qquark

Member

Registered: 2014-12-24

Posts: 2

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

I see, thanks !

That's a step in the right direction

Jimaklas

Member

Registered: 2014-12-24

Posts: 2

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Hello there, congratulations on the excellent advice and information found in this thread, it is by far the most helpful corner of the net as far as VGA Passthrough with KVM is concerned! Following the guide in the OP and solving most of my problems by searching the thread i managed to successfully pass my PCIe VGA adapter to a VM. By "successfully" i mean that there are no error messages issued during VM script execution. My problem is that i get no VGA output on the passthrough-ed card. Let me be more specific:

Motherboard: Intel DQ45CB (supports Intel's virtualization extensions + VT-d)
CPU: Intel Core2 Quad Q9450 @ 2.66GHz (supports Intel's virtualization extensions + VT-d)
GPU1: on chip Intel GMA 4500 (set as primary in BIOS)
GPU2: Nvidia GTX 550 Ti (passed-through in the VM)
RAM: 8GB Dual Channel DDR2
OS: openSuSE 13.2 x86_64

qemu: 2.1.0
seabios: 1.7.5 (also tried the git version)
kernel: 3.18.1 with acs override + i915 vga arbiter patches applied

I have patched the kernel with the patches provided in the OP (as found in linux-mainline-3.18.0.tar.gz), patch process went fine although i use kernel 3.18.1 instead of 3.18.0.
Some relevant kernel configuration parameters:

dimitris@openSuSE:~> cat /proc/config.gz | gzip -d | grep -e CONFIG_VIRTUALIZATION= -e CONFIG_KVM= -e CONFIG_KVM_INTEL= -e CONFIG_PCI_STUB= -e CONFIG_VFIO= -e CONFIG_VFIO_PCI= -e CONFIG_VFIO_PCI_VGA= -e CONFIG_VHOST_NET= -e CONFIG_VIRTIO_PCI= -e CONFIG_VIRTIO_BALLOON= -e CONFIG_VGA_ARB=

CONFIG_PCI_STUB=y
CONFIG_VHOST_NET=m
CONFIG_VGA_ARB=y
CONFIG_VFIO=y
CONFIG_VFIO_PCI=y
CONFIG_VFIO_PCI_VGA=y
CONFIG_VIRTIO_PCI=m
CONFIG_VIRTIO_BALLOON=m
CONFIG_VIRTUALIZATION=y
CONFIG_KVM=y
CONFIG_KVM_INTEL=y

I am booting the kernel with:

dimitris@openSuSE:~> cat /proc/cmdline

BOOT_IMAGE=/boot/vmlinuz-3.18.1-2.kvmpcipass-desktop root=UUID=4b1fbf1a-5770-4424-8609-a2b384ce6c41 resume=/dev/disk/by-uuid/2b281131-8310-48d9-94a0-9f81531469f3 splash=silent quiet showopts drm_kms_helper.edid_firmware=VGA-1:edid/lg-L1982U.bin pci-stub.ids=10de:1244,10de:0bee intel_iommu=on vfio_iommu_type1.allow_unsafe_interrupts=1 i915.enable_hd_vgaarb=1

I am using a VGA bios from techpowerup site which compares equal to the VGA bios that i extracted from the card itself.

I am using the following script to boot the VM:

#!/bin/bash

/usr/bin/qemu-system-x86_64 \
-enable-kvm -M q35 -m 4096 -cpu host \
-smp 2,sockets=1,cores=2,threads=1 \
-bios /home/dimitris/Documents/build/seabios/out/bios.bin \
-vga none \
-device ioh3420,bus=pcie.0,addr=1c.0,multifunction=on,port=1,chassis=1,id=root.1 \
-device vfio-pci,host=01:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on,romfile=/home/dimitris/Gigabyte.GTX550Ti.1024.111121.rom \
-device vfio-pci,host=01:00.1,bus=root.1,addr=00.1

and i am getting no output on the monitor attached to the Nvidia card (tried VGA, DVI and HDMI connections).

All searches lead me to the answer: "patch your kernel with i915 vga arbiter patch and enable it in the kernel command line of grub".
As far as i can understand, since the patch is already applied (CONFIG_VGA_ARB=y) and the grub command line includes i915.enable_hd_vgaarb=1, i should already have VGA output on my passed-through card.

I also tried to pass the card as secondary adapter:

#!/bin/bash

/usr/bin/qemu-system-x86_64 \
-enable-kvm -M q35 -m 4096 -cpu host \
-smp 2,sockets=1,cores=2,threads=1 \
-bios /home/dimitris/Documents/build/seabios/out/bios.bin \
-device ioh3420,bus=pcie.0,addr=1c.0,multifunction=on,port=1,chassis=1,id=root.1 \
-device vfio-pci,host=01:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on,romfile=/home/dimitris/Gigabyte.GTX550Ti.1024.111121.rom \
-device vfio-pci,host=01:00.1,bus=root.1,addr=00.1 \
-device virtio-scsi-pci,id=scsi \
-drive file=/dev/mapper/os-win7,id=disk,format=raw -device scsi-hd,drive=disk \
-drive file=/home/dimitris/Downloads/GRMCPRXFREO_EL_DVD.iso,id=isocd -device scsi-cd,drive=isocd \
-drive file=/home/dimitris/Downloads/virtio-win-0.1-94.iso,id=virtiocd -device ide-cd,bus=ide.1,drive=virtiocd

in which case the VM booted normally, Windows installation completed and the card could be seen in device manager. The Nvidia driver installation finished successfully (tried both 331.82 and 344.75) but the card was giving Error Code 12 mentioning that there are not enough resources to allocate to it. Then i removed used the "-vga none" parameter and rebooted with no output from the card. This problem (Error Code 12 in windows device manager) was caused to another user of this thread because he hadn't applied the i915 vga arbiter patch to his kernel and was solved after the patch was applied. It seems the patch doesn't get activated in my situation and i am clueless as to why this could be happening. Any ideas?

Last edited by Jimaklas (2014-12-24 14:01:50)

Duelist

Member

Registered: 2014-09-22

Posts: 358

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

750Ti is better than HD7750 or rare R7 250E. It eats less power, works faster, heats less. It should work. HD7750 isn't compatible with UEFI out-of-the-box, while 750Ti is.(aw has 750 too) Is there any related messages related to VFIO or your GPU in dmesg? Or, maybe, windows doesn't enable MSI? Can you show us "sudo lspci -v" when your VM is running?

I got the "AMD-Vi: Event logged [IO_PAGE_FAULT device=01:00.0" error when the VM is running.  Besides, I borrowed a i5-4460 and H97 motherboard for testing and turned out the same 750Ti worked smoothly on that machine with the same XML.  I guess the code 43 error really lies with FM2/FM2+  CPU + Motherboard + Nvidia GPU combo.  Looking at the google doc again, there is a user "Renfast" who tested two FM2+ motherboard.  He/She was successful with the HD7970 but not the GTX 680.

Try -realtime mlock=on and having firefox(or chrome or whatever) eating enough memory, just about 1.5G.

---

The forum rules prohibit requesting support for distributions other than arch.
I gave up. It was too late.
What I was trying to do.
The reference about VFIO and KVM VGA passthrough.

Duelist

Member

Registered: 2014-09-22

Posts: 358

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

All searches lead me to the answer: "patch your kernel with i915 vga arbiter patch and enable it in the kernel command line of grub".
As far as i can understand, since the patch is already applied (CONFIG_VGA_ARB=y) and the grub command line includes i915.enable_hd_vgaarb=1, i should already have VGA output on my passed-through card.

As far as i know, i915 vgaarb patch isn't upstream yet, so you'll need to MANUALLY patch the kernel with, you know, patch command.
Config option isn't much related to it, it just enables any vga arbitration, and you need fixes from the patch to get i915's vgaarb logic working.

BTW, nice to see that this tech works on older, socket775 platforms too.

---

The forum rules prohibit requesting support for distributions other than arch.
I gave up. It was too late.
What I was trying to do.
The reference about VFIO and KVM VGA passthrough.

Jimaklas

Member

Registered: 2014-12-24

Posts: 2

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

All searches lead me to the answer: "patch your kernel with i915 vga arbiter patch and enable it in the kernel command line of grub".
As far as i can understand, since the patch is already applied (CONFIG_VGA_ARB=y) and the grub command line includes i915.enable_hd_vgaarb=1, i should already have VGA output on my passed-through card.

As far as i know, i915 vgaarb patch isn't upstream yet, so you'll need to MANUALLY patch the kernel with, you know, patch command.
Config option isn't much related to it, it just enables any vga arbitration, and you need fixes from the patch to get i915's vgaarb logic working.

BTW, nice to see that this tech works on older, socket775 platforms too.

I am sorry i wasn't clear enough in the 1st post. Indeed, the i915 vgaarb patch isn't upstream yet, so i have actually patched the kernel myself. The patch process went like a breeze and i have also checked that every relevant file is patched according to the diffs (i also diffed the patched files against the originals and got the diffs i applied with the above patch). Is there a way to tell if the patch indeed does it's job? Here is dmesg output in case it helps:

dimitris@openSuSE:~> dmesg | grep -i vgaarb
[    0.000000] Command line: BOOT_IMAGE=/boot/vmlinuz-3.18.1-2.kvmpcipass-desktop root=UUID=4b1fbf1a-5770-4424-8609-a2b384ce6c41 resume=/dev/disk/by-uuid/2b281131-8310-48d9-94a0-9f81531469f3 splash=silent quiet showopts drm_kms_helper.edid_firmware=VGA-1:edid/lg-L1982U.bin pci-stub.ids=10de:1244,10de:0bee intel_iommu=on vfio_iommu_type1.allow_unsafe_interrupts=1 i915.enable_hd_vgaarb=1
[    0.000000] Kernel command line: BOOT_IMAGE=/boot/vmlinuz-3.18.1-2.kvmpcipass-desktop root=UUID=4b1fbf1a-5770-4424-8609-a2b384ce6c41 resume=/dev/disk/by-uuid/2b281131-8310-48d9-94a0-9f81531469f3 splash=silent quiet showopts drm_kms_helper.edid_firmware=VGA-1:edid/lg-L1982U.bin pci-stub.ids=10de:1244,10de:0bee intel_iommu=on vfio_iommu_type1.allow_unsafe_interrupts=1 i915.enable_hd_vgaarb=1
[    0.115067] vgaarb: setting as boot device: PCI:0000:00:02.0
[    0.115067] vgaarb: device added: PCI:0000:00:02.0,decodes=io+mem,owns=io+mem,locks=none
[    0.115067] vgaarb: device added: PCI:0000:01:00.0,decodes=io+mem,owns=none,locks=none
[    0.115067] vgaarb: loaded
[    0.115067] vgaarb: bridge control possible 0000:01:00.0
[    0.115067] vgaarb: no bridge control possible 0000:00:02.0
[    4.249183] vgaarb: device changed decodes: PCI:0000:00:02.0,olddecodes=io+mem,decodes=none:owns=io+mem

Yes KVM passthrough works on socket 775 platforms and is effectively a quite cheap solution with very decent performance

Silar

Member

Registered: 2014-12-24

Posts: 2

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Hi guys! I just start playing around with VM and KVM  VGA-Passthrough
My Hardware
OS: Latest Arch Linux with patched kernel from the first post (linux-mainline-3.18.0)
MB: Asrock Z68 Gen3 Extreme 3
CPU: i7 3770
Host GPU: Intel HD
Guest: GPU: nVidia GTX560Ti

I start qemu with the following parameters:

qemu-system-x86_64 -enable-kvm -M q35 -m 8192 -cpu host \
-smp 6,sockets=1,cores=6,threads=1 \
-vga none \
-device ioh3420,bus=pcie.0,addr=1c.0,multifunction=on,port=1,chassis=1,id=root.
-device vfio-pci,host=01:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on \
-device vfio-pci,host=01:00.1,bus=root.1,addr=00.1 \
-device vfio-pci,host=00:1a.0,bus=pcie.0 \
-device virtio-scsi-pci,id=scsi \
-drive file=/mnt/windows.img,id=windisk,format=raw,if=virtio

The first time i start the VM everything is ok. But after the guest was shutdown and start again I get this error:

qemu-system-x86_64: vfio-pci: Cannot read device rom at 0000:01:00.0
Device option ROM contents are probably invalid (check dmesg).
Skip option ROM probe with rombar=0, or load from file with romfile=

dmesg also says:

[ 3496.604688] vfio-pci 0000:00:1a.0: enabling device (0400 -> 0402)
[ 3496.707482] vfio_cap_init: 0000:00:1a.0 hiding cap 0xa
[ 3500.034452] vfio-pci 0000:01:00.0: Invalid ROM contents
[ 3500.883294] kvm: zapping shadow pages for mmio generation wraparound

Does anybody know this error and can help me to fix it?

A second question:

I use qemu-git from AUR. How can i use -cpu kvm=off to hide the hypervisor to install latest nvidia driver?

aw

Member

Registered: 2013-10-04

Posts: 921

Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

I noticed that apparently one of the RMRR entries is correct, and the other one is not.
Since this is very easy to do, I enabled intel_iommu=on to see what happens, and Linux tells me that while setting up identity mappings it detected that my bios is broken (true, true ...); *however*, after a lengthy log full of call stacks about the devices it tried to map at the second RMRR entry, I get a log showing success (or at least no error) for those same devices at the first RMRR entry.

My first question from this result is whether I have a chance of getting VGA passthrough working in those conditions (?).
Can Linux manage to work with only one entry by ignoring the error in the other ? (assuming the first one is correct AND working) ? And Is there a way to override this entry (I do not know what the RMRR actually represents, but I suspect the answer would be negative).
Also, in my dmesg log, I only find this identity mapping happening for the USB controller devices (or subdevices ?), should I also expect to see my current graphics card there ?

RMRRs (Reserved Memory Region Reporting) entries in the DMAR table are things that should see very little use, typically only USB and integrated graphics.  They tell the IOMMU driver to configure a persistent identity map for the specified memory range and device(s).  Google'ing for the problem on this system seems to indicate that the RMRR points to a memory region that isn't listed as reserved by the e820 table.  If that's the only problem, you could fix it using this following boot option:

        memmap=nn[KMG]$ss[KMG]
                        [KNL,ACPI] Mark specific memory as reserved.
                        Region of memory to be reserved is from ss to ss+nn.
                        Example: Exclude memory from 0x18690000-0x1869ffff
                                 memmap=64K$0x18690000
                                 or
                                 memmap=0x10000$0x18690000

Second topic: I see that most people are generally using setups with {intel,ATI}+{nvidia,ATI}, but I would like to know if there is any problem with an nvidia+nvidia setup (?), as I only have a spare 9800GT I can try in addition to my GTX660, and do not have any recent extra ATI card, nor an integrated intel GP (i7 920). From what I thought I understood, just being able to unbind/rebind the appropriate card should be enough to remove any conflict between host and guest, why are people also blacklisting drivers ? Do they interfere even though they're not bound to the card ? Or is it just an easier way to do boot-time non-binding ?

Don't expect that you can bind/unbind graphics cards from their driver as easily as you can a NIC.  That's why people generally use pci-stub.ids= to make the stub driver claim devices rather than the host graphics driver.  As long as the two cards have different device IDs, which they obviously will in this case, there's no problem using a host and assigned GPU by the same vendor.  Note that there have been an maybe still are issues with using nvidia in the host regardless of the second card due to VGA arbiter locking.  nouveau shouldn't be an issue.  Ideally with a GTX660 you can get a UEFI ROM for it and use OVMF.

---

http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

aw

Member

Registered: 2013-10-04

Posts: 921

Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Hello there, congratulations on the excellent advice and information found in this thread, it is by far the most helpful corner of the net as far as VGA Passthrough with KVM is concerned! Following the guide in the OP and solving most of my problems by searching the thread i managed to successfully pass my PCIe VGA adapter to a VM. By "successfully" i mean that there are no error messages issued during VM script execution. My problem is that i get no VGA output on the passthrough-ed card. Let me be more specific:

Motherboard: Intel DQ45CB (supports Intel's virtualization extensions + VT-d)
CPU: Intel Core2 Quad Q9450 @ 2.66GHz (supports Intel's virtualization extensions + VT-d)
GPU1: on chip Intel GMA 4500 (set as primary in BIOS)
GPU2: Nvidia GTX 550 Ti (passed-through in the VM)
RAM: 8GB Dual Channel DDR2
OS: openSuSE 13.2 x86_64

qemu: 2.1.0
seabios: 1.7.5 (also tried the git version)
kernel: 3.18.1 with acs override + i915 vga arbiter patches applied

You may be in a unique situation with this dinosaur.  The i915 patch fixes VGA arbitration for CPU integrated Intel graphics.  Your graphics is integrated into the chipset, not the CPU, so I don't expect the i915 or kernel boot option is going to help you.  If you're not getting VGA output, then it's possible that Intel broke VGA routing even prior to the switch from PCH to CPU based IGD.  IOW, you're probably out of luck unless you're able to do some debugging on your own.  Your options with this system would be to disable integrated graphics and use a discrete card for the host or use an assigned GPU and guest capable of going the UEFI/OVMF route.

---

http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

aw

Member

Registered: 2013-10-04

Posts: 921

Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

750Ti is better than HD7750 or rare R7 250E. It eats less power, works faster, heats less. It should work. HD7750 isn't compatible with UEFI out-of-the-box, while 750Ti is.(aw has 750 too) Is there any related messages related to VFIO or your GPU in dmesg? Or, maybe, windows doesn't enable MSI? Can you show us "sudo lspci -v" when your VM is running?

I got the "AMD-Vi: Event logged [IO_PAGE_FAULT device=01:00.0" error when the VM is running.  Besides, I borrowed a i5-4460 and H97 motherboard for testing and turned out the same 750Ti worked smoothly on that machine with the same XML.  I guess the code 43 error really lies with FM2/FM2+  CPU + Motherboard + Nvidia GPU combo.  Looking at the google doc again, there is a user "Renfast" who tested two FM2+ motherboard.  He/She was successful with the HD7970 but not the GTX 680.

Try -realtime mlock=on and having firefox(or chrome or whatever) eating enough memory, just about 1.5G.

This sounds an awful lot like voodoo

---

http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

aw

Member

Registered: 2013-10-04

Posts: 921

Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Hi guys! I just start playing around with VM and KVM  VGA-Passthrough
My Hardware
OS: Latest Arch Linux with patched kernel from the first post (linux-mainline-3.18.0)
MB: Asrock Z68 Gen3 Extreme 3
CPU: i7 3770
Host GPU: Intel HD
Guest: GPU: nVidia GTX560Ti

I start qemu with the following parameters:

qemu-system-x86_64 -enable-kvm -M q35 -m 8192 -cpu host \
-smp 6,sockets=1,cores=6,threads=1 \
-vga none \
-device ioh3420,bus=pcie.0,addr=1c.0,multifunction=on,port=1,chassis=1,id=root.
-device vfio-pci,host=01:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on \
-device vfio-pci,host=01:00.1,bus=root.1,addr=00.1 \
-device vfio-pci,host=00:1a.0,bus=pcie.0 \
-device virtio-scsi-pci,id=scsi \
-drive file=/mnt/windows.img,id=windisk,format=raw,if=virtio

The first time i start the VM everything is ok. But after the guest was shutdown and start again I get this error:

qemu-system-x86_64: vfio-pci: Cannot read device rom at 0000:01:00.0
Device option ROM contents are probably invalid (check dmesg).
Skip option ROM probe with rombar=0, or load from file with romfile=

dmesg also says:

[ 3496.604688] vfio-pci 0000:00:1a.0: enabling device (0400 -> 0402)
[ 3496.707482] vfio_cap_init: 0000:00:1a.0 hiding cap 0xa
[ 3500.034452] vfio-pci 0000:01:00.0: Invalid ROM contents
[ 3500.883294] kvm: zapping shadow pages for mmio generation wraparound

Does anybody know this error and can help me to fix it?

Dump the ROM or get it from techpowerup database and use the romfile= option to provide the ROM to the guest.

---

http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Duelist

Member

Registered: 2014-09-22

Posts: 358

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

I got the "AMD-Vi: Event logged [IO_PAGE_FAULT device=01:00.0" error when the VM is running.  Besides, I borrowed a i5-4460 and H97 motherboard for testing and turned out the same 750Ti worked smoothly on that machine with the same XML.  I guess the code 43 error really lies with FM2/FM2+  CPU + Motherboard + Nvidia GPU combo.  Looking at the google doc again, there is a user "Renfast" who tested two FM2+ motherboard.  He/She was successful with the HD7970 but not the GTX 680.

Try -realtime mlock=on and having firefox(or chrome or whatever) eating enough memory, just about 1.5G.

This sounds an awful lot like voodoo

More like cargo-cult.

mlock is default:on, btw. But it works.
I still have no idea what's broken. Any ideas where do i dig to investigate the page faults? I'm not so familiar with deep hardware fiddling yet.

Also, there are performance drawbacks on some applications.

My system needs a full overhaul.

---

The forum rules prohibit requesting support for distributions other than arch.
I gave up. It was too late.
What I was trying to do.
The reference about VFIO and KVM VGA passthrough.

Ansa89

Member

Registered: 2014-08-30

Posts: 20

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Any hint about this?

Duelist

Member

Registered: 2014-09-22

Posts: 358

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Any hint about this?

As a "better than nothing" hint, i'll tell you: check whatever you use for your vm's disk. Be it filesystem's(you use file as a storage) health or actual hardware disk.
It seems to me like a kernel's problem, not a VM one.
Found this bugreport, seems somewhat related:
https://bugzilla.redhat.com/show_bug.cgi?id=1167001

Also, if you're sure that your host disks are okay, try preparing your windows to migrating on virtio-blk-pci instead of scsi-way. That involves creating a dummy-drive on virtio-blk-pci device and feeding windows with drivers from virtio.iso and then "reconnecting" the drive.

---

The forum rules prohibit requesting support for distributions other than arch.
I gave up. It was too late.
What I was trying to do.
The reference about VFIO and KVM VGA passthrough.

Myranti

Member

Registered: 2010-03-04

Posts: 20

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Has anyone run into an issue with black lines flickering on the host system when the secondary GPU is being passed through? (Edit below) It disappears when I shut the guest down but comes back when the guest starts up. I otherwise have things working, though haven't tested any games yet.

I'm using two Radeon cards, R5 230 for host and R9 270X for guest with the open source drivers but have the second card bound to pci-stub. My google-fu hasn't found anything relevant yet, but if it's an issue that's already been posted that someone remembers I can work through earlier posts, perhaps with some targeted direction.

If people feel I need to provide more information to try determine what might be the cause, I'm happy to do so.

Edit: Okay, I've tested a GTX460 (nouveau) and an R7 250 (radeon) and don't get flickering on either of them, so it looks like it's related to the R5 230 I'm using. Will try to replace this with another card in the upcoming week.

Last edited by Myranti (2014-12-28 09:12:40)

Duelist

Member

Registered: 2014-09-22

Posts: 358

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Has anyone run into an issue with black lines flickering on the host system when the secondary GPU is being passed through? It disappears when I shut the guest down but comes back when the guest starts up. I otherwise have things working, though haven't tested any games yet.

I'm using two Radeon cards, R5 230 for host and R9 270X for guest with the open source drivers but have the second card bound to pci-stub. My google-fu hasn't found anything relevant yet, but if it's an issue that's already been posted that someone remembers I can work through earlier posts, perhaps with some targeted direction.

If people feel I need to provide more information to try determine what might be the cause, I'm happy to do so.

I don't know the solution, but i think motherboard and cpu information would be handy.

---

The forum rules prohibit requesting support for distributions other than arch.
I gave up. It was too late.
What I was trying to do.
The reference about VFIO and KVM VGA passthrough.

Myranti

Member

Registered: 2010-03-04

Posts: 20

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

I don't know the solution, but i think motherboard and cpu information would be handy.

The motherboard i'm using is an Asus P9D WS (VT-d and virtualization options enabled in BIOS of course) and the CPU is a Xeon E3-1241v3.

Last edited by Myranti (2014-12-25 16:27:54)