Arch Linux

jonascj

Member

Registered: 2014-02-28

Posts: 20

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

As much as I appreciate the original post for sharing information about this great setup, it is, to say the least, very confusing.

So I've decided to do only what made sense and what could be confirmed elsewhere. See the below short story for a quick overview of my setup and my problem. See the long story for more in depth description:

#### Short story
I am trying to make this work with a minimum of configuration and alterations to different things. This is my setup:

Kernel: Stock 3.13.6 from Arch base install
QEMU: 1.7.0 installed via 'pacman -S qemu'
Seabios:  1.7.3.1-2 (already installed, maybe a dependency of qemu)

PCI-stub or blacklisting: Neither (according to #kvm@irc.freenode.net it is not necessary, unbind is sufficient)

BOOT options: I have only added "intel_iommu=on" to my boot line in grub.cfg

I can sucessfully unbind the Radeon card and attach it to vfio-pci:

[jonas@jonascj-pc ~]$ ls -la /sys/bus/pci/devices/0000\:01\:00.0/driver
lrwxrwxrwx 1 root root 0 Apr  3 03:52 /sys/bus/pci/devices/0000:01:00.0/driver -> ../../../../bus/pci/drivers/radeon
[jonas@jonascj-pc ~]$ ls -la /sys/bus/pci/devices/0000\:01\:00.1/driver
lrwxrwxrwx 1 root root 0 Apr  3 03:51 /sys/bus/pci/devices/0000:01:00.1/driver -> ../../../../bus/pci/drivers/snd_hda_intel
[jonas@jonascj-pc ~]$ sudo vfio-bind 0000:01:00.1 0000:01:00.0
[sudo] password for jonas: 
[jonas@jonascj-pc ~]$ ls -la /sys/bus/pci/devices/0000\:01\:00.0/driver
lrwxrwxrwx 1 root root 0 Apr  3 03:53 /sys/bus/pci/devices/0000:01:00.0/driver -> ../../../../bus/pci/drivers/vfio-pci
[jonas@jonascj-pc ~]$ ls -la /sys/bus/pci/devices/0000\:01\:00.1/driver
lrwxrwxrwx 1 root root 0 Apr  3 03:53 /sys/bus/pci/devices/0000:01:00.1/driver -> ../../../../bus/pci/drivers/vfio-pci
[jonas@jonascj-pc ~]$

I am testing with this script:

#!/bin/bash
qemu-system-x86_64 -enable-kvm -M q35 -m 1024 -cpu host \
-smp 6,sockets=1,cores=6,threads=1 \
-bios /usr/share/qemu/bios.bin -vga none \
-device ioh3420,bus=pcie.0,addr=1c.0,multifunction=on,port=1,chassis=1,id=root.1 \
-device vfio-pci,host=01:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on \
-device vfio-pci,host=01:00.1,bus=root.1,addr=00.1

I get a blank window on the host and the monitor connected to my radeon card goes from its standby-mode (blinking led and pitch black screen / no backlight) to its there-is-signal-mode (constant led and lighter black / backlight on). The image on my host monitor (displaying my terminal, blank qemu window etc.) gets miscolored. If I change to tty2 and back to tty1 (ctrl+alt+f2 -> ctrl+alt+f1) the miscoloration disappears but the Radeon screen stays blank and so does the qemu window on the host.

If I use the kernel from the original post I get output on the Radeon monitor. So the difference seems to be solely with the kernel. Which of those provided patches might be the needed one and where can I find more information about it? I do not like to blindly accept this preconfigured kernel with patches from who knows where. I think I've already shown (see long story for details) that quite a few things listd in the original post is not necessary and so have other people posting replies. Maybe only a single patch to the kernel is necessary?

#### Long story
## Software requirements

Ingredients

• qemu-git.tar.gz

• seabios-git.tar.gz

• linux-mainline.tar.gz (3.13.6 includes acs override patch, i935 vga arbiter fixes, memleak fix by xani, debug registers patch)

You can also use these packages from AUR but you might run into problems:

• qemu-git

• linux-mainline

1. A kernel >= 3.9 (consider using kernel 3.12.0 or later)  is required  or if you're building it yourself it must be configured with the following:

   • CONFIG_VFIO_IOMMU_TYPE1=y

   • CONFIG_VFIO=y

   • CONFIG_VFIO_PCI=y

   • CONFIG_VFIO_PCI_VGA=y

2. You'll need qemu >= 1.5.0 (consider using qemu 1.7 or later)  or qemu-git

3. And seabios

Why is there two or three lists here without any clear separation?

I have chosen to use the stock kernel 3.13.6 which came with my base installation of Arch.

This kernel is / was configured as (according to 'zless /proc/config.gz')

CONFIG_VFIO_IOMMU_TYPE1=m
CONFIG_VFIO=m
CONFIG_VFIO_PCI=m
CONFIG_VFIO_PCI_VGA=y

(Note the three m's, not y's)

For qemu I've just installed the one from the official repo ('pacman -S qemu'). This is qemu 1.7.0 according to 'qemu-system-x86_64 -version'.

For seabios I've just used the version which was already installed (maybe it was installed as a requirement of qemu): seabios-1.7.3.1-2.

## Preparing the GPU for vfio binding

Preparing the GPU so we can bind it to vfio

For the next step we need to:

1. Blacklist radeon or nouveau or nvidia or fglrx on /etc/modprobe.d/blacklist.conf

   Example, blacklisting the opensource radeon module:

   echo "blacklist radeon" >> /etc/modprobe.d/blacklist.conf 

2. Use pci-stub

   In my case since i have 2 radeon cards blacklisting the radeon module is not an option, so i use pci-stub.

   NOTE: If pci-stub was built as a module, you'll need to modify /etc/mkinitcpio.conf and add pci-stub in the MODULES section, after that you need to update your initramfs like this.

   mkinitcpio -p linux-mainline

   lspci

   07:00.0 VGA compatible controller: Advanced Micro Devices, Inc. [AMD/ATI] Cayman PRO [Radeon HD 6950] <-- radeon 6950
   07:00.1 Audio device: Advanced Micro Devices, Inc. [AMD/ATI] Cayman/Antilles HDMI Audio [Radeon HD 6900 Series] <-- radeon 6950 audio

   lspci -n

   07:00.0 0300: 1002:6719 <-- radeon 6950
   07:00.1 0403: 1002:aa80 <-- radeon 6950 audio

   Now add this to your grub cfg file:

   pci-stub.ids=1002:6719,1002:aa80

   dmesg | grep pci-stub

   [    2.096151] pci-stub: add 1002:6719 sub=FFFFFFFF:FFFFFFFF cls=00000000/00000000
   [    2.096160] pci-stub 0000:07:00.0: claimed by stub
   [    2.096165] pci-stub: add 1002:AA80 sub=FFFFFFFF:FFFFFFFF cls=00000000/00000000
   [    2.096174] pci-stub 0000:07:00.1: claimed by stub
   [    2.096178] pci-stub: add 1B21:1042 sub=FFFFFFFF:FFFFFFFF cls=00000000/00000000

According to to people in #kvm@irc.freenode.net it is not necessary to blacklist modules or attach the device (gfx card) to pci-stub before attaching it to vfio-pci. It should be sufficient to unbind it which is already done in/by the vfio-bind script (see next section).

Do you guys still think this is necessary? I haven't been able to confirm this anywhere else.

## Setting up vfio and kvm modules

Setting up vfio and kvm modules

If your board doesn't enable interrupt remapping, you need to add this to your grub cfg:

vfio_iommu_type1.allow_unsafe_interrupts=1

Or if vfio-pci was built as a module ( default on arch )

echo "options vfio_iommu_type1 allow_unsafe_interrupts=1" > /etc/modprobe.d/vfio_iommu_type1.conf

Some applications like Passmark Performance Test and SiSoftware Sandra crash the VM without this:

echo "options kvm ignore_msrs=1" >> /etc/modprobe.d/kvm.conf

I don't know about this. I have no idea how to verify if my board enables interrupt remapping or not. According to http://www.linux-kvm.org/page/How_to_as … T-d_in_KVM ('dmesg | grep -e DMAR -e IOMMU') IOMMU is enabled.

Anyone who knows how to find out if these options are needed (i.e. how to verify if interrupt remapping is enabled)?

## Binding a device to vfio-pci

Binding a device to vfio-pci

Assuming the kernel, qemu and seabios are built and working, lets bind some devices.
You can use this script to make life easier:

#!/bin/bash

modprobe vfio-pci

for dev in "$@"; do
        vendor=$(cat /sys/bus/pci/devices/$dev/vendor)
        device=$(cat /sys/bus/pci/devices/$dev/device)
        if [ -e /sys/bus/pci/devices/$dev/driver ]; then
                echo $dev > /sys/bus/pci/devices/$dev/driver/unbind
        fi
        echo $vendor $device > /sys/bus/pci/drivers/vfio-pci/new_id
done

Save it as /usr/bin/vfio-bind

chmod 755 /usr/bin/vfio-bind

Bind the GPU:

vfio-bind 0000:07:00.0 0000:07:00.1

This here makes really good sense. The script unbinds the device (gfx card) if it is already attached to a driver and afterwards it attaches it to vfio-pci.

I would say this works perfectly for me. This is my lspci output:

[jonas@jonascj-pc ~]$ lspci | grep -i radeon
01:00.0 VGA compatible controller: Advanced Micro Devices, Inc. [AMD/ATI] Tahiti XT [Radeon HD 7970]
01:00.1 Audio device: Advanced Micro Devices, Inc. [AMD/ATI] Tahiti XT HDMI Audio [Radeon HD 7970 Series]

After booting my stock kernel without any blacklisting or pci-stub'ing I can see the Radeon devices attached to one driver and after doing vfio-bind they are attached to vfio-pci:

[jonas@jonascj-pc ~]$ ls -la /sys/bus/pci/devices/0000\:01\:00.0/driver
lrwxrwxrwx 1 root root 0 Apr  3 03:52 /sys/bus/pci/devices/0000:01:00.0/driver -> ../../../../bus/pci/drivers/radeon
[jonas@jonascj-pc ~]$ ls -la /sys/bus/pci/devices/0000\:01\:00.1/driver
lrwxrwxrwx 1 root root 0 Apr  3 03:51 /sys/bus/pci/devices/0000:01:00.1/driver -> ../../../../bus/pci/drivers/snd_hda_intel
[jonas@jonascj-pc ~]$ sudo vfio-bind 0000:01:00.1 0000:01:00.0
[sudo] password for jonas: 
[jonas@jonascj-pc ~]$ ls -la /sys/bus/pci/devices/0000\:01\:00.0/driver
lrwxrwxrwx 1 root root 0 Apr  3 03:53 /sys/bus/pci/devices/0000:01:00.0/driver -> ../../../../bus/pci/drivers/vfio-pci
[jonas@jonascj-pc ~]$ ls -la /sys/bus/pci/devices/0000\:01\:00.1/driver
lrwxrwxrwx 1 root root 0 Apr  3 03:53 /sys/bus/pci/devices/0000:01:00.1/driver -> ../../../../bus/pci/drivers/vfio-pci
[jonas@jonascj-pc ~]$

## Testing if its working out

Testing if its working out

Lets test if its working, as root:

qemu-system-x86_64 -enable-kvm -M q35 -m 1024 -cpu host \
-smp 6,sockets=1,cores=6,threads=1 \
-bios /usr/share/qemu/bios.bin -vga none \
-device ioh3420,bus=pcie.0,addr=1c.0,multifunction=on,port=1,chassis=1,id=root.1 \
-device vfio-pci,host=07:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on \
-device vfio-pci,host=07:00.1,bus=root.1,addr=00.1

NOTE: -m | VM memory
NOTE: -cpu host | Expose host cpu features.
NOTE: -smp 6,sockets=1,cores=6,threads=1 | Expose 1 cpu with 6 cores and 1 thread each.
NOTE: -bios | Bios image to load.
NOTE: -vga none | Disable cirrus emulated vga.
NOTE: -m q35 | Enable the new q35 emulated chipset, qemu defaults to the old i440fx chipset.

You should see a black qemu window on your main display, and seabios ouput on your monitor from your passthru'd card saying it cant find anything to boot.
Its important to use -M q35 which is the new qemu emulated intel chipset with pcie support,
"-device ioh3420,bus=pcie.0,addr=1c.0,multifunction=on,port=1,chassis=1,id=root.1" creates a pcie root port and we attach our gpu to it.
We must also use x-vga=on option on our gpu (and -vga none) otherwise it wont work

If you're using an intel cpu and nothing happens try this:

modprobe -r kvm_intel
modprobe kvm_intel emulate_invalid_guest_state=0

NOTE: There might be some problems using nvidia/fglrx/nouveau drivers on the host gpu, see the ISSUES section below on how to solve this.

I am testing with this script:

#!/bin/bash
qemu-system-x86_64 -enable-kvm -M q35 -m 1024 -cpu host \
-smp 6,sockets=1,cores=6,threads=1 \
-bios /usr/share/qemu/bios.bin -vga none \
-device ioh3420,bus=pcie.0,addr=1c.0,multifunction=on,port=1,chassis=1,id=root.1 \
-device vfio-pci,host=01:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on \
-device vfio-pci,host=01:00.1,bus=root.1,addr=00.1

I get a blank window on the host and the monitor connected to my radeon card goes from its standby-mode (blinking led and pitch black screen / no backlight) to its there-is-signal-mode (constant led and lighter black / backlight on). The image on my host monitor (displaying my terminal, blank qemu window etc.) gets miscolored. If I change to tty2 and back to tty1 (ctrl+alt+f2 -> ctrl+alt+f1) the miscoloration disappears but the Radeon screen stays blank and so does the qemu window on the host.

Last edited by jonascj (2014-04-03 01:10:42)

jonascj

Member

Registered: 2014-02-28

Posts: 20

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Thanks to nbhs and everyone else involved in this thread, I've got VGA passthrough working using this guide with the following setup:

Intel 4770
Asus Z87-Plus
Gigabyte "Windforce" GTX 770 GV-N770OC-2GD

I'm starting X with the onboard Intel GPU and getting near native Windows performance on a passthroughed 770.

For others and my future self, here are the steps involved. I think this is everything:

Install qemu-git
Install seabios-git

The i915 kernel patch is required. I couldn't get the provided mainline kernel to work. I built the latest kernel and added the i915 patch with the following:

pacman -S abs base-devel
ABSROOT=. abs core/linux
cd core/linux/
wget https://gist.githubusercontent.com/anonymous/732acf9551136a78e91b/raw/5b97d50e7fb7faced258ad8948b7a82127d25f31/i915_313rc4.patch
wget https://gist.githubusercontent.com/etcet/9864922/raw/e49d204ddaf2d6130c91d94d0ce6a9123c6197cb/i915-pkgbuild.patch
patch -p3 < i915-pkgbuild.patch
makepkg
pacman -U linux-i915-headers-3.13.7-1-x86_64.pkg.tar.xz
pacman -U linux-i915-3.13.7-1-x86_64.pkg.tar.xz

More: https://wiki.archlinux.org/index.php/Ke … ild_System

Hi etcet

I am trying to figure out what the minimum requirements in terms of patches and configuration is (for my setup, I cannot test other setups).

My setup is very similar to yours:
Z87 chipset (Asrock Z87M Extreme 4)
Intel Core i7 4770,
Intel integrated gpu for the host
A single dedicated card (Radeon R9 280X / HD7970)

So far I've gotten output on my monitor connected to the dedicated card with these steps:

1. Use main line kernel provided by OP

2. Use qemu and seabios from offical repo ('pacman -S qemu seabios) which results in qemu 1.7.0 and seabios 1.7.3.1-2

3. vfio-bind 0000:01:00.0 0000:01:00.1    (no pci-stub or blacklisting)

4. Test with this script and I get seabios output on the monitor connected to the dedicated card:

   #!/bin/bash
   qemu-system-x86_64 -enable-kvm -M q35 -m 1024 -cpu host \
   -smp 6,sockets=1,cores=6,threads=1 \
   -bios /usr/share/qemu/bios.bin -vga none \
   -device ioh3420,bus=pcie.0,addr=1c.0,multifunction=on,port=1,chassis=1,id=root.1 \
   -device vfio-pci,host=01:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on \
   -device vfio-pci,host=01:00.1,bus=root.1,addr=00.1

Now I just wonder what the minimum requirements for the kernel is. The stock 3.13.6 kernel (from an Arch base-install) does not work. It gives me blank screen on the monitor connected to the dedicated card and miscolors my host display (the Intel integrated gpu). Then  saw your post about "the i915 patch is needed" and thought maybe that is my minimum requirement.

But how did you know what the "i915 kernel patch" is and where to obtain it? You use these patches (one for the kernel code itself and one for the pkgbuild file):
https://gist.githubusercontent.com/anon … 3rc4.patch
https://gist.githubusercontent.com/etce … uild.patch

How did you find those patches, who made them, where do you plan on finding updates to those if you suddenly decide to upgrade your kernel in a few months?

Edit: I have patched the 3.13.8 core kernel with the two patches you linked in your post, build it, booted it and it works (i.e. gives me seabios output on the monitor connected to the dedicated Radeon card). So this seems to be the only patching to the kernel I need (I of course haven't gone any further in the process - installing windows, drivers, played games etc. - maybe I need more patches later on due to performance etc.). The question remains the same though - where do these patches come from etc.?

Best regards
Jonas

Last edited by jonascj (2014-04-03 12:05:06)

novist

Member

Registered: 2014-03-14

Posts: 47

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

headsup - some kvm improvements at http://lkml.iu.edu/hypermail/linux/kern … 03070.html

For x86 there are optimizations for debug registers, which trigger on
some Windows games, and other important fixes for Windows guests. We now
expose to the guest Broadwell instruction set extensions and also Intel
MPX. There's also a fix/workaround for OS X guests, nested virtualization
features (preemption timer), and a couple kvmclock refinements.

Cant test myself just yet

aw

Member

Registered: 2013-10-04

Posts: 921

Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Edit: I have patched the 3.13.8 core kernel with the two patches you linked in your post, build it, booted it and it works (i.e. gives me seabios output on the monitor connected to the dedicated Radeon card). So this seems to be the only patching to the kernel I need (I of course haven't gone any further in the process - installing windows, drivers, played games etc. - maybe I need more patches later on due to performance etc.). The question remains the same though - where do these patches come from etc.?

They come from here:

http://git.kernel.org/cgit/linux/kernel … f81dea8a7d
http://git.kernel.org/cgit/linux/kernel … ba24389bee

And I agree, they should really be the only patches required and hopefully none are required if you don't have Intel graphics in your system.  Some people are experiencing problems with memory being freed and apply another patch, I suspect that has something to do with the toolchain they're using to build.  The other popular patch is the ACS override, which allows devices to be separated out of IOMMU groups.  There's code going into 3.15 to help many Intel chipsets with this problem and I just posted a patch this week that adds IDs for X79 that will hopefully also make 3.15.

Why are the "i915 patches" needed and why aren't they upstream?  VGA is a very old standard and was an evolution of things like EGA and CGA.  It was developed back when you were lucky to have one graphics card and ROMs were a new thing.  Like many legacy things, it stuck because it was widely supported and there was nothing better.  One of the problems with VGA is that it lives at a fixed address and only one device can occupy that address space at a time.  PCI attempts to solve this problem by having a VGA enable bit in bridges allowing the chipset to select which single device receives VGA accesses.  There are still complications when two VGA devices live on the same bus, but I'll skip as it's not important here.  To manage where VGA is routed we have something in the kernel called the VGA arbiter.  The arbiter requires that any access to VGA space be wrapped in locking that allows the arbiter to switch chipset routing.  When you specify x-vga=on for a device then the vfio-pci driver will claim accesses to the VGA space and forward them to the vfio-pci kernel module, which does this locking.  If everyone plays nice with the VGA arbiter, things work.

The trouble is that not everyone place nice, some drivers don't play at all.  The vgacon driver in the kernel doesn't participate in VGA arbitration.  The i915 driver lies and claims that it doesn't claim VGA accesses, effectively an opt-out of arbitration, while actually it does claim VGA accesses.  This means that our VM comes along and does a VGA access, it gets forwarded to the kernel and vfio takes the VGA arbiter lock, the arbiter sees that i915 doesn't claim VGA and the VGA enable bit is set on the bridge for our device and proceeds to give us the lock.  Our access then goes straight to i915.  This is the reason you see both the corruption on the host display and lack of anything happening on the guest display, the accesses are going to the wrong card.

The first patch above fixes this, making i915 play correctly with VGA arbitration and the second patch cleans up some ordering to avoid a big white box drawn on the screen.  It works and makes everyone happy... except Xorg on the host.  The DRI code in Xorg, so I'm told, wants to mmap(2) VGA MMIO space.  Doing so is incompatible with the VGA arbiter because Xorg would need to hold the VGA arbitration lock for the life of that mmap, effectively a permanent lock.  At that point VGA arbitration is deadlocked.  Rather than do that, Xorg detects that there are multiple VGA devices and disables DRI.  The result is that making i915 not lie about its VGA usage kills desktop performance for anyone with multiple graphics cards.  This is the reason the two patches were reverted.

Solutions to this are:
a) don't use VGA space in the guest (DanaGoyette attempted to do this using UEFI/OVMF in place of Seabios, also accomplished on some cards using assigned graphics as a secondary device)
b) create a VGA arbiter interface that would allow mmaps (I have some prototype code to do this, but I don't have time to figure out how to make Xorg use it)
c) figure out why DRI wants to mmap VGA space and avoid it
d) your idea here

On the subject of whether you need to blacklist or otherwise avoid allowing the graphics device to attach to host drivers before you unbind it for the guest, it's a matter of personal preference and host driver capabilities.  In the non-graphics case of PCI device assignment, it's not needed because Linux has pretty good support for hotplug and thus the bugs have been worked out for detaching and re-attaching devices from drivers.  The hotplug story for graphics cards on Linux has not been as good, so those drivers tend to lag in reliably and cleanly releasing the device.  The nvidia proprietary driver for instance will allow you to unbind the driver with X up and running on the device and X continues to run after unbind.  I filed a bug with Nvidia about this, but they already closed it with no code change.  If you have no intention of using the device in the host, it's often the safest and easiest path just to blacklist the driver or force it to bind to pci-stub.  If unbinding works for you, do it.

---

http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

nbhs

Member

From: Montevideo, Uruguay

Registered: 2013-05-02

Posts: 402

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Kernel 3.14 is up on the front page

gneville

Member

Registered: 2014-04-02

Posts: 27

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Just tried the linux-mainline 3.14 Kernel on the OP as posted by nbhs, still hasn't resolved my issue however I now see some interesting messages in dmesg. I still get video output and can install windows etc however drivers won't install and even while booting windows I get these messages. Tested with multiple qemu versions as well:

VM starts fine, video displays, I see these messages as always:

[  351.920675] vfio_ecap_init: 0000:01:00.0 hiding ecap 0x19@0x270
[  351.920684] vfio_ecap_init: 0000:01:00.0 hiding ecap 0x1b@0x2d0

Then all of a sudden dmesg gets flooded with this:

[  377.186286] ------------[ cut here ]------------
[  377.186288] WARNING: CPU: 3 PID: 618 at arch/x86/kvm/x86.c:1114 kvm_get_msr_common+0x793/0x960 [kvm]()
[  377.186288] Modules linked in: tun vfio_pci vfio_iommu_type1 vfio bridge stp llc snd_hda_codec_realtek snd_hda_codec_generic dm_mod coretemp x86_pkg_temp_thermal intel_powerclamp kvm_intel kvm crct10dif_pclmul crc32_pclmul crc32c_intel joydev ghash_clmulni_intel mousedev iTCO_wdt aesni_intel btusb iTCO_vendor_support bluetooth aes_x86_64 lrw gf128mul glue_helper ablk_helper cryptd 6lowpan_iphc xpad ff_memless rfkill microcode snd_hda_intel led_class hid_generic snd_hda_codec igb snd_hwdep e1000e snd_pcm hwmon psmouse dca serio_raw snd_timer mei_me pcspkr lpc_ich i2c_i801 ptp snd pps_core mei soundcore fan thermal battery evdev shpchp mac_hid processor nfs lockd sunrpc fscache usbhid hid sd_mod crc_t10dif crct10dif_common ata_generic pata_acpi atkbd libps2 ata_piix libata ehci_pci xhci_hcd ehci_hcd
[  377.186305]  scsi_mod usbcore usb_common i8042 serio pci_stub vfat fat ext4 crc16 mbcache jbd2 i915 video button intel_gtt i2c_algo_bit drm_kms_helper drm i2c_core
[  377.186309] CPU: 3 PID: 618 Comm: qemu-system-x86 Tainted: G        W    3.14.0-2-mainline #1
[  377.186310] Hardware name:    / , BIOS 4.6.5 07/26/2013
[  377.186310]  0000000000000009 ffff880415a81c10 ffffffff814d563c 0000000000000000
[  377.186312]  ffff880415a81c48 ffffffff81064f1d 0000000040000020 ffff880415a81cd8
[  377.186313]  ffff880413bd3ec0 ffff8803378a4000 ffff8803378a4040 ffff880415a81c58
[  377.186314] Call Trace:
[  377.186315]  [<ffffffff814d563c>] dump_stack+0x4d/0x6f
[  377.186316]  [<ffffffff81064f1d>] warn_slowpath_common+0x7d/0xa0
[  377.186317]  [<ffffffff81064ffa>] warn_slowpath_null+0x1a/0x20
[  377.186320]  [<ffffffffa0efa453>] kvm_get_msr_common+0x793/0x960 [kvm]
[  377.186322]  [<ffffffffa13170c2>] vmx_get_msr+0x102/0x370 [kvm_intel]
[  377.186323]  [<ffffffffa131735a>] handle_rdmsr+0x2a/0x140 [kvm_intel]
[  377.186325]  [<ffffffffa131c9a5>] vmx_handle_exit+0xb5/0xa20 [kvm_intel]
[  377.186326]  [<ffffffff81074c02>] ? __set_task_blocked+0x32/0x70
[  377.186328]  [<ffffffffa1314360>] ? vmx_invpcid_supported+0x20/0x20 [kvm_intel]
[  377.186330]  [<ffffffffa0effb60>] kvm_arch_vcpu_ioctl_run+0xc70/0x11c0 [kvm]
[  377.186333]  [<ffffffffa0efb85e>] ? kvm_arch_vcpu_load+0x4e/0x1e0 [kvm]
[  377.186336]  [<ffffffffa0ee85c2>] kvm_vcpu_ioctl+0x2b2/0x5b0 [kvm]
[  377.186336] systemd-journald[168]: /dev/kmsg buffer overrun, some messages lost.
[  377.186338]  [<ffffffff811e4ab8>] ? fsnotify+0x228/0x2f0
[  377.186340]  [<ffffffff810a9bb8>] ? __wake_up_locked_key+0x18/0x20
[  377.186341]  [<ffffffff811b6180>] do_vfs_ioctl+0x2e0/0x4c0
[  377.186342]  [<ffffffff811bff9e>] ? __fget+0x6e/0xb0
[  377.186344]  [<ffffffff811b63e1>] SyS_ioctl+0x81/0xa0
[  377.186345]  [<ffffffff814e3069>] system_call_fastpath+0x16/0x1b
[  377.186346] ---[ end trace 73327e2f3359a7b9 ]---
[  377.186413] systemd-journald[168]: /dev/kmsg buffer overrun, some messages lost.

On top of this I have just installed the OP linux-mainline (3.13.6-1-mainline),seabios (1.7.4.r1788.g8abc6a8-1) & qemu (2.0.r31714.g03d5142-1) versions (Just added reset patch and manually compiled (https://lists.gnu.org/archive/html/qemu … 0767.html))

and still no change

Hi All,

I'm hoping someone will be able to help me out please. I've been trying to get vga passthrough working with qemu and vfio for a few months on and off but having no luck. I'm getting similar problems to users redger & DanaGoyette.

My hardware: CPU: i7 4770s, MB: Jetway NF9J-Q87 GFX: AMD R9 290

I used to use Xen 4.3 and had vga passthrough working just passing through the intel hd4600 controller to hosts, but since I got the AMD R9 290 card Xen doesn't seem to passthrough the card whatsoever, so thought I would try qemu and vfio.

I've tried using different linux versions and qemu versions but not having much luck. I am able to successfully vfio-bind my gfx card and I can start QEMU and get video via the GFX Card HDMI cable, I'm also able to successfully install Windows 8.1. However after installation I'm getting problems with either windows installing the amd drivers or using the catalyst driver installer, the host just reboots.

My latest setup:

linux 3.13.7-1 + i915 patch
Seabios git 1.7.4-66 (31Mar14)
Qemu 2.0.0 RC0 + patched pcibus_reset & assertion (https://lists.nongnu.org/archive/html/q … 05901.html & https://lists.gnu.org/archive/html/qemu … 00767.html)

Before the pcibus_reset patch I kept getting the following messages when the host rebooted while trying to install the drivers:

"qemu-system-x86_64 shows: hw/pci/pci.c:250: pcibus_reset: Assertion `bus->irq_count[\i] == 0' failed."

dmesg shows:  2805.839608 vfio-pci 0000:01:00.0: irq 46 for MSI/MSI-X

After applying the pcibust_reset patch the host no longer reboots after installing the drivers however the screen just goes blank so I have to manually kill the host. I do see the following messages:

qemu-system-x86_64 shows: ehci warning: guest updated active QH

dmesg shows:  4681.189513 vfio-pci 0000:01:00.0: irq 46 for MSI/MSI-X

This is my QEMU config:

/usr/local/qemu-2-0-0-rc0-patched-pci/bin/qemu-system-x86_64 -name win8 -enable-kvm -M q35 -m 6144  \
-cpu Haswell,hv-time,hv_relaxed,hv_vapic,hv_spinlocks=0x1000 \
-boot order=dc \
-smp 2,sockets=1,cores=2,threads=1 \
-bios /root/vfio/bios.bin -vga none \
-usb -usbdevice tablet \
-spice port=5902,disable-ticketing \
-vnc 0.0.0.0:0 \
-device ioh3420,bus=pcie.0,addr=1c.0,multifunction=on,port=1,chassis=1,id=root.1 \
-device ahci,bus=pcie.0,id=ahci \
-net nic \
-net tap,ifname=tap0,script=no,downscript=no \
-device vfio-pci,host=01:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on \
-device vfio-pci,host=01:00.1,bus=root.1,addr=00.1 \
-device vfio-pci,host=00:14.0,bus=pcie.0 \
-drive file=/dev/mapper/vg1-winv1,media=disk,index=1,id=disk,cache=none,if=virtio \
-drive file=/dev/mapper/vg1-winv1data,media=disk,index=2,id=diskb,cache=none,if=virtio \
-drive file=/home/xen/iso/windows/en-gb_windows_8_1_enterprise_x64_dvd_2971910.iso,boot=on,media=cdrom,index=3,id=isocd  \
-device ide-cd,bus=ahci.1,drive=isocd \
-drive file=/root/virtio-win-0.1-74.iso,media=cdrom,index=4,id=virtio  \
-device ide-cd,bus=ahci.2,drive=virtio

If I enable "-vga cirrus" and VNC to the host I can install the AMD drivers successfully and after a reboot it shows in device manager the AMD card but with a yellow exclamation and code 43. If I then go back to "-vga none" windows no longer wants to boot and I need up in a boot loop (seabios>windows booting>seabios>windows booting).

Any ideas please?

jonascj

Member

Registered: 2014-02-28

Posts: 20

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Thank you, aw, for that very insightful reply.

I have a couple of follow-up questions and/or need a few clarifications.

The first patch above fixes this, making i915 play correctly with VGA arbitration and the second patch cleans up some ordering to avoid a big white box drawn on the screen.  It works and makes everyone happy... except Xorg on the host.  The DRI code in Xorg, so I'm told, wants to mmap(2) VGA MMIO space.  Doing so is incompatible with the VGA arbiter because Xorg would need to hold the VGA arbitration lock for the life of that mmap, effectively a permanent lock.  At that point VGA arbitration is deadlocked.  Rather than do that, Xorg detects that there are multiple VGA devices and disables DRI. The result is that making i915 not lie about its VGA usage kills desktop performance for anyone with multiple graphics cards.  This is the reason the two patches were reverted.

Solutions to this are:
a) don't use VGA space in the guest (DanaGoyette attempted to do this using UEFI/OVMF in place of Seabios, also accomplished on some cards using assigned graphics as a secondary device)
b) create a VGA arbiter interface that would allow mmaps (I have some prototype code to do this, but I don't have time to figure out how to make Xorg use it)
c) figure out why DRI wants to mmap VGA space and avoid it
d) your idea here

Regarding what is marked in red:

So applying the the first patch (which makes i915 work correctly with VGA arbitration) kills overall performance (because it makes Xorg  disable DRI which forces everyone to make indirect rendering) and because of that that patch was removed / reverted upstream?

Does this mean that kernels with this patch reapplied (OP's kernel for example, or etcet's core repo kernel with i915 patch) suffer from this problem (i.e. bad overall performance)?

Regarding the text marked in blue:
Solutions to which problem? Solutions to the i915 arbiter problem - i.e. an alternative to the current patch?

aw

Member

Registered: 2013-10-04

Posts: 921

Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Thank you, aw, for that very insightful reply.

I have a couple of follow-up questions and/or need a few clarifications.

The first patch above fixes this, making i915 play correctly with VGA arbitration and the second patch cleans up some ordering to avoid a big white box drawn on the screen.  It works and makes everyone happy... except Xorg on the host.  The DRI code in Xorg, so I'm told, wants to mmap(2) VGA MMIO space.  Doing so is incompatible with the VGA arbiter because Xorg would need to hold the VGA arbitration lock for the life of that mmap, effectively a permanent lock.  At that point VGA arbitration is deadlocked.  Rather than do that, Xorg detects that there are multiple VGA devices and disables DRI. The result is that making i915 not lie about its VGA usage kills desktop performance for anyone with multiple graphics cards.  This is the reason the two patches were reverted.

Solutions to this are:
a) don't use VGA space in the guest (DanaGoyette attempted to do this using UEFI/OVMF in place of Seabios, also accomplished on some cards using assigned graphics as a secondary device)
b) create a VGA arbiter interface that would allow mmaps (I have some prototype code to do this, but I don't have time to figure out how to make Xorg use it)
c) figure out why DRI wants to mmap VGA space and avoid it
d) your idea here

Regarding what is marked in red:

So applying the the first patch (which makes i915 work correctly with VGA arbitration) kills overall performance (because it makes Xorg  disable DRI which forces everyone to make indirect rendering) and because of that that patch was removed / reverted upstream?

Does this mean that kernels with this patch reapplied (OP's kernel for example, or etcet's core repo kernel with i915 patch) suffer from this problem (i.e. bad overall performance)?

In both cases yes, when multiple VGA devices are present (and possibly only on the i915 devices - radeon and nouveau appear to participate in VGA arbitration just fine).

Regarding the text marked in blue:
Solutions to which problem? Solutions to the i915 arbiter problem - i.e. an alternative to the current patch?

Solutions to either fix or avoid the problem that we can't have multiple VGA devices participate in arbitration without losing host DRI, which means fixing the VGA arbiter and/or fixing Xorg's use of the VGA arbiter, or avoiding VGA space in the guest.

---

http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

jonascj

Member

Registered: 2014-02-28

Posts: 20

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Does this mean that kernels with this patch reapplied (OP's kernel for example, or etcet's core repo kernel with i915 patch) suffer from this problem (i.e. bad overall performance)?

In both cases yes, when multiple VGA devices are present (and possibly only on the i915 devices - radeon and nouveau appear to participate in VGA arbitration just fine).

Sorry for being so thick, but this means that everyone using the i915 module/driver suffer from disabled DRI while they use kernels patched with this patch? I.e. everyone using Intel IGPU for their host while having a dedicated gfx passed through to a KVM guest have DRI disabled on their host.

aw

Member

Registered: 2013-10-04

Posts: 921

Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Does this mean that kernels with this patch reapplied (OP's kernel for example, or etcet's core repo kernel with i915 patch) suffer from this problem (i.e. bad overall performance)?

In both cases yes, when multiple VGA devices are present (and possibly only on the i915 devices - radeon and nouveau appear to participate in VGA arbitration just fine).

Sorry for being so thick, but this means that everyone using the i915 module/driver suffer from disabled DRI while they use kernels patched with this patch? I.e. everyone using Intel IGPU for their host while having a dedicated gfx passed through to a KVM guest have DRI disabled on their host.

Yes

---

http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

winie

Member

Registered: 2014-03-01

Posts: 17

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

The trouble is that not everyone place nice, some drivers don't play at all.  The vgacon driver in the kernel doesn't participate in VGA arbitration.  The i915 driver lies and claims that it doesn't claim VGA accesses, effectively an opt-out of arbitration, while actually it does claim VGA accesses.  This means that our VM comes along and does a VGA access, it gets forwarded to the
Solutions to this are:
a) don't use VGA space in the guest (DanaGoyette attempted to do this using UEFI/OVMF in place of Seabios, also accomplished on some cards using assigned graphics as a secondary device)
b) create a VGA arbiter interface that would allow mmaps (I have some prototype code to do this, but I don't have time to figure out how to make Xorg use it)
c) figure out why DRI wants to mmap VGA space and avoid it
d) your idea here.

Does wayland or mir solve this problem?

MattSpe

Member

Registered: 2014-04-04

Posts: 2

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Hi Everyone
I've spent the last few days fighting with my VGA passthrough.
No matter what I try, I always get the same result : passthrough is successful only on the first launch after host reset. Any following guest launch fail (I see the screen flicking but it stays black).
My last attempt was with content from first post (qemu-git seabios-git linux-mainline 3.14) on Arch. Same result.

My config is : X5680 + Sapphire X58 Pure Black + GT610 + GTX560Ti.
I tried both combinations 610 primary/560 secondary and 560 primary/610 secondary, no difference.
If I try to pass the secondary, only first attempt is successful. If I try to pass the primary not even the first attempt is working (stays black after flickering).
I tried to pass a romfile (I get an error msg on second launch if I don't pass, no need on the first launch), no difference.

Could someone point me in the right direction ?
Do I miss some patch/config ? Is it a hardware issue ?

Thanks !
Matthieu

anickname

Member

Registered: 2014-01-07

Posts: 23

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

I installed 3.14 from the first page and it didn't work well. The log is full of :

------------[ cut here ]------------
WARNING: CPU: 0 PID: 1367 at arch/x86/kvm/x86.c:1114 kvm_get_msr_common+0x793/0x960 [kvm]()
Modules linked in: nls_utf8 cifs dns_resolver fscache ebt_ip ebtable_filter bridge stp llc ebtables x_tables nct6775 hwmon_vid coretemp iTCO_wdt iTCO_vendor_support mxm_wmi intel_rapl
 raid1 md_mod sd_mod crc_t10dif atkbd libps2 crct10dif_pclmul crct10dif_common crc32_pclmul crc32c_intel ghash_clmulni_intel aesni_intel aes_x86_64 lrw gf128mul glue_helper ablk_helpe
 ehci_pci
 ehci_hcd r8169 mii usbcore usb_common i8042 serio pci_stub
CPU: 0 PID: 1367 Comm: windows_81 Tainted: G        W    3.14.0-2-mainline #1
Hardware name: To Be Filled By O.E.M. To Be Filled By O.E.M./Z77 Professional, BIOS P1.70B 11/12/2013
 0000000000000009 ffff8802571afc10 ffffffff814d563c 0000000000000000
 ffff8802571afc48 ffffffff81064f1d 0000000040000020 ffff8802571afcd8
 ffff88025726bec0 ffff88025717c000 ffff88025717c040 ffff8802571afc58
Call Trace:
 [<ffffffff814d563c>] dump_stack+0x4d/0x6f
 [<ffffffff81064f1d>] warn_slowpath_common+0x7d/0xa0
 [<ffffffff81064ffa>] warn_slowpath_null+0x1a/0x20
 [<ffffffffa03da453>] kvm_get_msr_common+0x793/0x960 [kvm]
 [<ffffffffa00740c2>] vmx_get_msr+0x102/0x370 [kvm_intel]
 [<ffffffffa007435a>] handle_rdmsr+0x2a/0x140 [kvm_intel]
 [<ffffffffa00799a5>] vmx_handle_exit+0xb5/0xa20 [kvm_intel]
 [<ffffffffa0071360>] ? vmx_invpcid_supported+0x20/0x20 [kvm_intel]
 [<ffffffffa03dfb60>] kvm_arch_vcpu_ioctl_run+0xc70/0x11c0 [kvm]
 [<ffffffff8101e9d6>] ? ___preempt_schedule+0x56/0xb0
 [<ffffffffa03c85c2>] kvm_vcpu_ioctl+0x2b2/0x5b0 [kvm]
 [<ffffffff814d7909>] ? __schedule+0x3c9/0x900
 [<ffffffff811b6180>] do_vfs_ioctl+0x2e0/0x4c0
 [<ffffffff811bff9e>] ? __fget+0x6e/0xb0
 [<ffffffff811b63e1>] SyS_ioctl+0x81/0xa0
 [<ffffffff814e3069>] system_call_fastpath+0x16/0x1b
---[ end trace 1bf431db809123d0 ]---

Before, I had 3.13.6 also from the first page and worked without any issues.

My config is :

qemu-system-x86_64 \
    -daemonize \
    -pidfile "/run/qemu-$VM.pid" \
    -nodefconfig -nodefaults \
    -name "$VM",process="$VM" \
    -enable-kvm \
    -boot menu=on \
    -rtc base=localtime \
    -machine q35 \
    -balloon none \
    -m 8192 -mem-path /dev/hugepages -mem-prealloc \
    -cpu Haswell,hv-time \
    -smp sockets=1,cores=4,threads=1 \
    -bios bios-256k.bin \
    -chardev file,path="$VM-seabios.log",id=seabios -device isa-debugcon,iobase=0x402,chardev=seabios \
    -vga none \
    -nographic \
    -monitor unix:/run/qemu-$VM.monitor,server,nowait \
    -serial none \
    -parallel none \
    -device ioh3420,bus=pcie.0,addr=1c.0,multifunction=on,port=1,chassis=1,id=root.1 \
    -device vfio-pci,host=01:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on,romfile=$KVM_ROOT/roms/Sapphire_RadeonHD_7790.rom \
    -device vfio-pci,host=00:1b.0,bus=pcie.0 \
    -device vfio-pci,host=06:00.0,bus=pcie.0 \
    -device vfio-pci,host=09:00.0,bus=pcie.0 \
    -device vfio-pci,host=0b:00.0,bus=pcie.0

dwe11er

Member

Registered: 2014-03-18

Posts: 73

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

I installed 3.14 from the first page and it didn't work well. The log is full of :

------------[ cut here ]------------
...
---[ end trace 1bf431db809123d0 ]---

Before, I had 3.13.6 also from the first page and worked without any issues.

My config is :

qemu-system-x86_64 \
    -daemonize \
...
    -device vfio-pci,host=0b:00.0,bus=pcie.0

I think hv-time might be doing bad things. It's working great without tho.

Last edited by dwe11er (2014-04-05 14:51:04)

anickname

Member

Registered: 2014-01-07

Posts: 23

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

I will try, thanks.
I upgraded to 3.14 hoping for some speed improvement from hv-time.

My processor is Sandy Bridge and in the config I have Haswell. Does this creates any problems ?
I have Haswell because with Sandy Bridge it wasn't stable; crashes from time to time.

EDIT: Yes, without hv-time is working with 3.14. The problems is that now, the cpu load on the host when
running Windows 8.1 idle is with ~20-25% higher than when running 3.13.6 with hv-time. With 3.13.6 was
~45-50%.
I don't know if it is because of hv-time or because of the new kernel.

Last edited by anickname (2014-04-05 18:50:10)

apex8

Member

Registered: 2014-03-29

Posts: 60

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Using the provided packages with Kernel 3.14 I got rid of the host crashes during VM startup. Graphics pass through is working, as far as I can tell (Windows performance indicator).
Now I`m trying to pass through my creative recon3d sound card. Originally I tried XEN where this was working without problems, but now I`m getting the following error during startup:

vfio: error, group 6 is not viable

The corresponding part of my script is:

-device vfio-pci,host=03:00.0 \

lspci -k | grep 03
03:00.0 Audio device: Creative Labs Device 0012 (rev 01)

lspci -nnn | grep 03:00.0
03:00.0 Audio device [0403]: Creative Labs Device [1102:0012] (rev 01)

cat /proc/cmdline
... pci-stub.ids=10de:1187,10de:0e0a,1102:0012,8086:1c2d

Does someone have an idea, whats wrong here?

BR

dwe11er

Member

Registered: 2014-03-18

Posts: 73

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

http://pastebin.com/KWD4DVcS

So, if anyone interested, I did my own launching script with ability to unbind devices and rebind back to their previous driver (useful if passing hd audio) after vm is down.

eliwallace

Member

Registered: 2014-04-05

Posts: 2

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Hi, I've been trying to get this working with a GTX 770 on the host passing a GTX 660 through to the guest, but I ran into

qemu-system-x86_64: -device vfio-pci,host=08:00.0,bus=root.1,addr=00.0,multifunction=on,rombar=0,romfile=/usr/local/share/qemu/vgabios-660.rom,x-vga=on: vfio: Device does not support requested feature x-vga
qemu-system-x86_64: -device vfio-pci,host=08:00.0,bus=root.1,addr=00.0,multifunction=on,rombar=0,romfile=/usr/local/share/qemu/vgabios-660.rom,x-vga=on: vfio: failed to get device 0000:08:00.0
qemu-system-x86_64: -device vfio-pci,host=08:00.0,bus=root.1,addr=00.0,multifunction=on,rombar=0,romfile=/usr/local/share/qemu/vgabios-660.rom,x-vga=on: Device initialization failed.
qemu-system-x86_64: -device vfio-pci,host=08:00.0,bus=root.1,addr=00.0,multifunction=on,rombar=0,romfile=/usr/local/share/qemu/vgabios-660.rom,x-vga=on: Device 'vfio-pci' could not be initialized

Using kernel 3.13, qemu 1.7, FX-8350 cpu, Crosshair V F-Z mobo, proprietary Nvidia drivers on host

launching qemu with

qemu-system-x86_64 -enable-kvm -M q35 -m 1024 -cpu host -smp 6,sockets=1,cores=6,threads=1 -bios /usr/share/qemu/bios.bin -vga none -device ioh3420,bus=pcie.0,addr=1c.0,multifunction=on,port=1,chassis=1,id=root.1 -device vfio-pci,host=08:00.0,bus=root.1,addr=00.0,multifunction=on,rombar=0,romfile=/usr/local/share/qemu/vgabios-660.rom,x-vga=on -device vfio-pci,host=08:00.1,bus=root.1,addr=00.1

lspci -vvvv:

08:00.0 VGA compatible controller: NVIDIA Corporation GK106 [GeForce GTX 660] (rev a1) (prog-if 00 [VGA controller])
	Subsystem: eVga.com. Corp. Device 2662
	Control: I/O+ Mem+ BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
	Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
	Interrupt: pin A routed to IRQ 32
	Region 0: Memory at f4000000 (32-bit, non-prefetchable) [size=16M]
	Region 1: Memory at c0000000 (64-bit, prefetchable) [size=128M]
	Region 3: Memory at c8000000 (64-bit, prefetchable) [size=32M]
	Region 5: I/O ports at a000 [size=128]
	Expansion ROM at f5000000 [disabled] [size=512K]
	Capabilities: [60] Power Management version 3
		Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0-,D1-,D2-,D3hot-,D3cold-)
		Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=0 PME-
	Capabilities: [68] MSI: Enable- Count=1/1 Maskable- 64bit+
		Address: 0000000000000000  Data: 0000
	Capabilities: [78] Express (v2) Endpoint, MSI 00
		DevCap:	MaxPayload 256 bytes, PhantFunc 0, Latency L0s unlimited, L1 <64us
			ExtTag+ AttnBtn- AttnInd- PwrInd- RBE+ FLReset-
		DevCtl:	Report errors: Correctable- Non-Fatal- Fatal- Unsupported-
			RlxdOrd- ExtTag- PhantFunc- AuxPwr- NoSnoop+
			MaxPayload 128 bytes, MaxReadReq 512 bytes
		DevSta:	CorrErr- UncorrErr- FatalErr- UnsuppReq- AuxPwr- TransPend-
		LnkCap:	Port #0, Speed 8GT/s, Width x16, ASPM L0s L1, Exit Latency L0s <512ns, L1 <4us
			ClockPM+ Surprise- LLActRep- BwNot-
		LnkCtl:	ASPM Disabled; RCB 64 bytes Disabled- CommClk+
			ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
		LnkSta:	Speed 2.5GT/s, Width x16, TrErr- Train- SlotClk+ DLActive- BWMgmt- ABWMgmt-
		DevCap2: Completion Timeout: Range AB, TimeoutDis+, LTR-, OBFF Not Supported
		DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis-, LTR-, OBFF Disabled
		LnkCtl2: Target Link Speed: 8GT/s, EnterCompliance- SpeedDis-
			 Transmit Margin: Normal Operating Range, EnterModifiedCompliance- ComplianceSOS-
			 Compliance De-emphasis: -6dB
		LnkSta2: Current De-emphasis Level: -3.5dB, EqualizationComplete-, EqualizationPhase1-
			 EqualizationPhase2-, EqualizationPhase3-, LinkEqualizationRequest-
	Capabilities: [b4] Vendor Specific Information: Len=14 <?>
	Capabilities: [100 v1] Virtual Channel
		Caps:	LPEVC=0 RefClk=100ns PATEntryBits=1
		Arb:	Fixed- WRR32- WRR64- WRR128-
		Ctrl:	ArbSelect=Fixed
		Status:	InProgress-
		VC0:	Caps:	PATOffset=00 MaxTimeSlots=1 RejSnoopTrans-
			Arb:	Fixed- WRR32- WRR64- WRR128- TWRR128- WRR256-
			Ctrl:	Enable+ ID=0 ArbSelect=Fixed TC/VC=ff
			Status:	NegoPending- InProgress-
	Capabilities: [128 v1] Power Budgeting <?>
	Capabilities: [600 v1] Vendor Specific Information: ID=0001 Rev=1 Len=024 <?>
	Capabilities: [900 v1] #19
	Kernel driver in use: vfio-pci

08:00.1 Audio device: NVIDIA Corporation GK106 HDMI Audio Controller (rev a1)
	Subsystem: eVga.com. Corp. Device 2662
	Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
	Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
	Latency: 0, Cache Line Size: 64 bytes
	Interrupt: pin B routed to IRQ 33
	Region 0: Memory at f5080000 (32-bit, non-prefetchable) [size=16K]
	Capabilities: [60] Power Management version 3
		Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0-,D1-,D2-,D3hot-,D3cold-)
		Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=0 PME-
	Capabilities: [68] MSI: Enable- Count=1/1 Maskable- 64bit+
		Address: 0000000000000000  Data: 0000
	Capabilities: [78] Express (v2) Endpoint, MSI 00
		DevCap:	MaxPayload 256 bytes, PhantFunc 0, Latency L0s unlimited, L1 <64us
			ExtTag+ AttnBtn- AttnInd- PwrInd- RBE+ FLReset-
		DevCtl:	Report errors: Correctable- Non-Fatal- Fatal- Unsupported-
			RlxdOrd- ExtTag- PhantFunc- AuxPwr- NoSnoop+
			MaxPayload 128 bytes, MaxReadReq 512 bytes
		DevSta:	CorrErr- UncorrErr- FatalErr- UnsuppReq- AuxPwr- TransPend-
		LnkCap:	Port #0, Speed 8GT/s, Width x16, ASPM L0s L1, Exit Latency L0s <512ns, L1 <4us
			ClockPM+ Surprise- LLActRep- BwNot-
		LnkCtl:	ASPM Disabled; RCB 64 bytes Disabled- CommClk+
			ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
		LnkSta:	Speed 2.5GT/s, Width x16, TrErr- Train- SlotClk+ DLActive- BWMgmt- ABWMgmt-
		DevCap2: Completion Timeout: Range AB, TimeoutDis+, LTR-, OBFF Not Supported
		DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis-, LTR-, OBFF Disabled
		LnkSta2: Current De-emphasis Level: -3.5dB, EqualizationComplete-, EqualizationPhase1-
			 EqualizationPhase2-, EqualizationPhase3-, LinkEqualizationRequest-
	Kernel driver in use: vfio-pci

dmesg | egrep -i "amd-vi|pci-stub|vfio|iommu"

[    0.000000] Command line: BOOT_IMAGE=/boot/vmlinuz-3.13.7 root=UUID=5e9c875a-9b8a-418a-b461-b1e08d8a1791 ro quiet iommu=pt iommu=1 amd_iommu=on
[    0.000000] Kernel command line: BOOT_IMAGE=/boot/vmlinuz-3.13.7 root=UUID=5e9c875a-9b8a-418a-b461-b1e08d8a1791 ro quiet iommu=pt iommu=1 amd_iommu=on
[    0.273684] [Firmware Bug]: AMD-Vi: IOAPIC[9] not in IVRS table
[    0.273757] [Firmware Bug]: AMD-Vi: IOAPIC[10] not in IVRS table
[    0.273806] [Firmware Bug]: AMD-Vi: No southbridge IOAPIC found
[    0.273853] AMD-Vi: Disabling interrupt remapping
[    1.220732] AMD-Vi: Found IOMMU at 0000:00:00.2 cap 0x40
[    1.220863] AMD-Vi: Initialized for Passthrough Mode
[    1.306447] AMD IOMMUv2 driver by Joerg Roedel <joerg.roedel@amd.com>
[    1.306448] AMD IOMMUv2 functionality not available on this system
[    3.703453] pci-stub: add 10DE:11C0 sub=FFFFFFFF:FFFFFFFF cls=00000000/00000000
[    3.703463] pci-stub 0000:08:00.0: claimed by stub
[    3.703468] pci-stub: add 10DE:0E0B sub=FFFFFFFF:FFFFFFFF cls=00000000/00000000
[    3.703474] pci-stub 0000:08:00.1: claimed by stub
[    9.215347] pci-stub 0000:08:00.0: enabling device (0000 -> 0003)
[   38.196490] vboxpci: IOMMU found
[  344.289349] VFIO - User Level meta-driver version: 0.3
[  464.944491] vfio_ecap_init: 0000:08:00.0 hiding ecap 0x19@0x900
[  664.101872] vfio_ecap_init: 0000:08:00.0 hiding ecap 0x19@0x900
[  743.049314] vfio_ecap_init: 0000:08:00.0 hiding ecap 0x19@0x900
[ 5881.675217] vfio_ecap_init: 0000:08:00.0 hiding ecap 0x19@0x900
[ 5906.191754] vfio_ecap_init: 0000:08:00.0 hiding ecap 0x19@0x900
[ 5945.493057] vfio_ecap_init: 0000:08:00.0 hiding ecap 0x19@0x900
[ 8566.737047] vfio_ecap_init: 0000:08:00.0 hiding ecap 0x19@0x900
[ 9020.289413] vfio_ecap_init: 0000:08:00.0 hiding ecap 0x19@0x900

cat /boot/config-`uname -r` | grep -i "vfio"

CONFIG_VFIO_IOMMU_TYPE1=m
CONFIG_VFIO=m
CONFIG_VFIO_PCI=m
CONFIG_VFIO_PCI_VGA=y
CONFIG_KVM_VFIO=y

EDIT: Solved by bios update

Last edited by eliwallace (2014-04-06 18:23:45)

SXX

Member

Registered: 2014-03-31

Posts: 15

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Hey want to say thanks because I successfully set VM on my hardware:
i7 4771, ASRock Z87 Extreme6, HD 6950.
Though I use Kubuntu 14.04 with:
Vanilla seabios and QEMU (version 1.7.91 (Debian 2.0.0~rc1+dfsg-0ubuntu1)) and seabios (1.7.4-4).
And kernel 3.14 with OP patches. I not yet sure what options I actually need and which are useless.

Just wonder about that problem with i915. Is this why I only have LLVMPipe on host when boot with patched kernel?

Last edited by SXX (2014-04-05 23:44:16)

Nex7

Member

Registered: 2014-04-06

Posts: 5

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

So I've pored through this thread and Google, and can't find anyone even claiming to be seeing the symptoms I'm seeing. Maybe I've missed something, or someone has an idea?

Motherboard: SuperMicro X10SAE-O
CPU: E3-1275v3
1st VGA: i915
2nd VGA (in top PCI-e x18 slot): GeForce GTX 770 "Windforce"

After much trial and error, using the latest (3.14) kernel with CONFIG_VFIO stuff compiled in & qemu-git from AUR (OP's fail to compile), plus OP's vfio-bind stuff and pci-stub set in kernel cmdline for the GTX770, I have almost full success. The Windows guest boots fine, reboots fine, can even be shut down and restarted without rebooting host.

However, I can't install the damn nVidia drivers in the Windows 7 x64 guest. They install fine, but upon reboot, Windows boots past the initial logo and then into a blank screen, I see my USB devices flicker a few times during a rather long pause, and then the VM reboots and sticks you at the 'Windows failed to boot' menu asking what to do.

I've tried a rather large number of things to resolve the issue based on every Google search I could craft, too many to list here, and I'm stumped. Anybody seen this/have an idea?

kernel cmdline:

BOOT_IMAGE=/vmlinuz-linux-mainline root=UUID=f4e70249-85fe-430b-9039-fcd9d7f89327 rw intel_iommu=on vfio_iommu_type1.allow_unsafe_interrupts=1 pci-stub.ids=10de:1184,10de:0e0a quiet

run line:

qemu-system-x86_64 -enable-kvm -M q35 -m 8192 -cpu qemu64 -smp 4,sockets=1,cores=4,threads=1 -bios /usr/share/qemu/bios.bin \
-rtc base=localtime -nodefaults \
-vga none -nographic -device ioh3420,bus=pcie.0,addr=1c.0,multifunction=on,port=1,chassis=1,id=root.1 \
-device vfio-pci,host=01:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on \
-device vfio-pci,host=01:00.1,bus=root.1,addr=00.1 \
-net nic,model=virtio,macaddr=52:54:01:01:01:01 -net bridge,br=br0 \
-device virtio-scsi-pci,id=scsi \
-usb -usbdevice host:1532:0016 -usbdevice host:24f0:0137 \
-usbdevice host:1b1c:0a0a -usbdevice host:04a9:1909 \
-drive file=/mnt/homesan/vm-win7-game/os_disk.raw,id=disk,format=raw,cache=none -device scsi-hd,drive=disk \
-drive file=/mnt/homesan/vm-win7-game/steam_disk.raw,id=game-disk,format=raw,cache=none -device scsi-hd,drive=game-disk \
-drive file=/usr/share/qemu/win7.iso,id=installcd -device ide-cd,bus=ide.1,drive=installcd \
-drive file=/usr/share/qemu/virtio-win-0.1-74.iso,id=virtiocd -device ide-cd,bus=ide.2,drive=virtiocd

There's no output from qemu when I run it, and nothing in dmesg beyond some crap about USB that seems unimportant (as it's always there, nvidia driver or not), I'm at the point of buying a Radeon, but I see people posting just days ago that they had a working GTX 770, so I have slight hope left.  Help?

Last edited by Nex7 (2014-04-06 03:31:39)

Cubex

Member

Registered: 2014-04-06

Posts: 24

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Hello Nex7, I am issuing the same problems, everything works until I install the nvidia driver on Win7, I posted the problem in awilliam's github but I guess that he didn't see or something, anyways this is the link https://github.com/awilliam/qemu-vfio/issues/2

My 770 is a Gigabyte 770 WindForce (4GB) so we are in similar situation here...
CPU: i7 4770 (without any letter)
MB: Asrock Extreme 6 Z87
GPU for host: Nvidia 430 at PCIE-3 x16 slot 1
GPU for qemu: Nvidia 770 at PCIE-3 x8 slot 2

I want to add something extra, the 430 succefully works (when I swich their roles and PCIE slots) and nvidia driver installs and works correctly (nvidia panel recognizes the card and the attached monitor) so it's something within Gigabyte 770 VBIOS maybe? maybe because card has UEFI VBIOS? I'm looking for help too...

Last edited by Cubex (2014-04-06 16:04:35)

Blind Tree Frog

Member

Registered: 2013-12-31

Posts: 27

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

dmesg | egrep -i "amd-vi|pci-stub|vfio|iommu"

[    0.273684] [Firmware Bug]: AMD-Vi: IOAPIC[9] not in IVRS table
[    0.273757] [Firmware Bug]: AMD-Vi: IOAPIC[10] not in IVRS table
[    0.273806] [Firmware Bug]: AMD-Vi: No southbridge IOAPIC found

Updating your BIOS should fix this issue I expect.  May not help with anything else, but FYI.

gneville

Member

Registered: 2014-04-02

Posts: 27

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

I've got the same problem but with my AMD R9 290 card, everythings fine until the drivers install. So I don't think this is exclusive to Nvidia, could be because of UEFI VBIOS......

Hello Nex7, I am issuing the same problems, everything works until I install the nvidia driver on Win7, I posted the problem in awilliam's github but I guess that he didn't see or something, anyways this is the link https://github.com/awilliam/qemu-vfio/issues/2

My 770 is a Gigabyte 770 WindForce (4GB) so we are in similar situation here...
CPU: i7 4770 (without any letter)
MB: Asrock Extreme 6 Z87
GPU for host: Nvidia 430 at PCIE-3 x16 slot 1
GPU for qemu: Nvidia 770 at PCIE-3 x8 slot 2

I want to add something extra, the 430 succefully works (when I swich their roles and PCIE slots) and nvidia driver installs and works correctly (nvidia panel recognizes the card and the attached monitor) so it's something within Gigabyte 770 VBIOS maybe? maybe because card has UEFI VBIOS? I'm looking for help too...

eliwallace

Member

Registered: 2014-04-05

Posts: 2

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

dmesg | egrep -i "amd-vi|pci-stub|vfio|iommu"

[    0.273684] [Firmware Bug]: AMD-Vi: IOAPIC[9] not in IVRS table
[    0.273757] [Firmware Bug]: AMD-Vi: IOAPIC[10] not in IVRS table
[    0.273806] [Firmware Bug]: AMD-Vi: No southbridge IOAPIC found

Updating your BIOS should fix this issue I expect.  May not help with anything else, but FYI.

Thanks, that fixed all the immediate problems

Nex7

Member

Registered: 2014-04-06

Posts: 5

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

So if I go into Safe Mode after one of these failed boots, the popup says:

Problem signature:
  Problem Event Name:    BlueScreen
  OS Version:    6.1.7601.2.1.0.256.4
  Locale ID:    1033

Additional information about the problem:
  BCCode:    116
  BCP1:    FFFFFA8009E404E0
  BCP2:    FFFFF8800F18EE20
  BCP3:    0000000000000000
  BCP4:    0000000000000002
  OS Version:    6_1_7601
  Service Pack:    1_0
  Product:    256_1

Google for this is not helpful - lots of 'update your drivers' and 'the card might be broken'. I did find one thing of some interest, this quote:

"It's not a true crash, in the sense that the bluescreen was initiated only because the combination of video driver and video hardware was being unresponsive, and not because of any synchronous processing exception".

Since Vista, the "Timeout Detection and Recovery" (TDR) components of the OS video subsystem have been capable of doing some truly impressive things to try to recover from issues which would have caused earlier OSs like XP to crash.

As a last resort, the TDR subsystem sends the video driver a "please restart yourself now!" command and waits a few seconds.

If there's no response, the OS concludes that the video driver/hardware combo has truly collapsed in a heap, and it fires off that stop 0x116 BSOD.

If playing with video driver versions hasn't helped, make sure the box is not overheating.

Try removing a side panel and aiming a big mains fan straight at the motherboard and GPU.

Obviously I do not believe my card is overheating, but perhaps that explanation of how you get to a BCC code 116 might spark something in one of the local expert's minds about what's going on?

I'm going to try playing with BIOS settings some more..