Arch Linux

Cubex

Member

Registered: 2014-04-06

Posts: 24

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Weee got Crysis 3 working ^^, now im trying to hunt the audio issue, nbhs, there is any equivalent of disabling nested page table thing under Intel?

PD: also found this, those who want to optimize might found interesting http://pic.dhe.ibm.com/infocenter/lnxin … ng_pdf.pdf

tswindell

Member

Registered: 2014-04-10

Posts: 3

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Managed to get this working out-of-the-box with Ubuntu 14.04 RC

Steps:
  - Added intel_iommu=on to /etc/default/grub
  - Added i915, nouveau, nvidia to /etc/modprobe.d/blacklist.conf
  - sudo update-initramfs -u
  - sudo update-grub
  - rebooted and installed qemu-kvm
  - Used vfio-bind script from OP
  - Executed qemu as OP

Hardware:
  - Intel i7 4771
  - ASUS Z87 Plus
  - 16GB DDR3 RAM
  - 2x nVidia GeForce 680 GTX

I have only one issue now, and that is that both nVidia cards are in the same iommu group, I'm guessing this is an issue with my motherboard, and it means I can't run two guests and dedicate a card to each, which is my end goal.

I guess this is what the ACS override patch is for, so I'm going to apply V2 of that and see if that works.

Last edited by tswindell (2014-04-10 22:38:16)

lildigiman

Member

Registered: 2014-04-02

Posts: 3

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Managed to get this working out-of-the-box with Ubuntu 14.04 RC

Steps:
  - Added intel_iommu=on to /etc/default/grub
  - Added i915, nouveau, nvidia to /etc/modprobe.d/blacklist.conf
  - sudo update-initramfs -u
  - sudo update-grub
  - rebooted and installed qemu-kvm
  - Used vfio-bind script from OP
  - Executed qemu as OP

Hardware:
  - Intel i7 4771
  - ASUS Z87 Plus
  - 16GB DDR3 RAM
  - 2x nVidia GeForce 680 GTX

I have only one issue now, and that is that both nVidia cards are in the same iommu group, I'm guessing this is an issue with my motherboard, and it means I can't run two guests and dedicate a card to each, which is my end goal.

I guess this is what the ACS override patch is for, so I'm going to apply V2 of that and see if that works.

Are you using the Intel GPU for the host?

beckend

Member

Registered: 2014-03-16

Posts: 5

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Eb, I have no idea what your missing, however, here is my PKGBUILD and acsv2 patch in pastebin.
PKGBUILD: http://pastebin.com/G7huLq6p
acs.patch: http://pastebin.com/gpTZiuXv

tswindell

Member

Registered: 2014-04-10

Posts: 3

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Are you using the Intel GPU for the host?

Yes, indeed I am, though blacklisting i915 is the only way I can get the VM to boot, presumably because of the VGA arbitration issues. Not a problem though as I'm running the host practically headless. Only issue for me like I said is separating the two 680 GTX cards from the iommu group they're in to be able to use each in a separate VM container.

Cubex

Member

Registered: 2014-04-06

Posts: 24

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Sadly the V2 patch doesn't work for me, it groups the first and second PCI-E slots on same group >.<

/sys/bus/pci/devices/0000:01:00.0/iommu_group/devices:
0000:00:01.0@  0000:00:01.1@  0000:01:00.0@  0000:01:00.1@  0000:02:00.0@  0000:02:00.1@

/sys/bus/pci/devices/0000:01:00.1/iommu_group/devices:
0000:00:01.0@  0000:00:01.1@  0000:01:00.0@  0000:01:00.1@  0000:02:00.0@  0000:02:00.1@

/sys/bus/pci/devices/0000:02:00.0/iommu_group/devices:
0000:00:01.0@  0000:00:01.1@  0000:01:00.0@  0000:01:00.1@  0000:02:00.0@  0000:02:00.1@

/sys/bus/pci/devices/0000:02:00.1/iommu_group/devices:
0000:00:01.0@  0000:00:01.1@  0000:01:00.0@  0000:01:00.1@  0000:02:00.0@  0000:02:00.1@

I will unapply the v2 patch and put again the old acs override, that in my case works great...

EDIT:

I also post the dmesg showing that the patch was active...

dmesg | grep ACS                             
[    0.181833] pci 0000:00:1c.0: Intel PCH root port ACS workaround enabled
[    0.181949] pci 0000:00:1c.1: Intel PCH root port ACS workaround enabled
[    0.182066] pci 0000:00:1c.3: Intel PCH root port ACS workaround enabled

Last edited by Cubex (2014-04-11 14:49:39)

aw

Member

Registered: 2013-10-04

Posts: 921

Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Sadly the V2 patch doesn't work for me, it groups the first and second PCI-E slots on same group >.<

/sys/bus/pci/devices/0000:01:00.0/iommu_group/devices:
0000:00:01.0@  0000:00:01.1@  0000:01:00.0@  0000:01:00.1@  0000:02:00.0@  0000:02:00.1@

/sys/bus/pci/devices/0000:01:00.1/iommu_group/devices:
0000:00:01.0@  0000:00:01.1@  0000:01:00.0@  0000:01:00.1@  0000:02:00.0@  0000:02:00.1@

/sys/bus/pci/devices/0000:02:00.0/iommu_group/devices:
0000:00:01.0@  0000:00:01.1@  0000:01:00.0@  0000:01:00.1@  0000:02:00.0@  0000:02:00.1@

/sys/bus/pci/devices/0000:02:00.1/iommu_group/devices:
0000:00:01.0@  0000:00:01.1@  0000:01:00.0@  0000:01:00.1@  0000:02:00.0@  0000:02:00.1@

I will unapply the v2 patch and put again the old acs override, that in my case works great...

EDIT:

I also post the dmesg showing that the patch was active...

dmesg | grep ACS                             
[    0.181833] pci 0000:00:1c.0: Intel PCH root port ACS workaround enabled
[    0.181949] pci 0000:00:1c.1: Intel PCH root port ACS workaround enabled
[    0.182066] pci 0000:00:1c.3: Intel PCH root port ACS workaround enabled

The upstream patch is for PCH root ports only, 00:1c.*.  Complain to Intel about processor based root ports.

---

http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Eb

Member

Registered: 2014-04-08

Posts: 11

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Eb, I have no idea what your missing, however, here is my PKGBUILD and acsv2 patch in pastebin.
PKGBUILD: http://pastebin.com/G7huLq6p
acs.patch: http://pastebin.com/gpTZiuXv

Thanks, rebuilt with what you provided with no issues.

Unfortunately, with v2 of the patch I'm getting the same problem where I have an overly large IOMMU group containing all my video devices and then some (See: https://bbs.archlinux.org/viewtopic.php … #p1402357).

I have the downstream option enabled for pcie_acs_override in my boot options, as well as intel_iommu.

I had this working before, but that was before installing another video card to try to fix some terrible graphical artifacts on the host when starting the KVM. I believe I was using linux-mainline 3.13 at that point as well.

Is there anything else I can try?

novist

Member

Registered: 2014-03-14

Posts: 47

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Just wanted to share my solution to those horrible directx9 performance issues i had. I think it was just me overlooking some things as first post did not put them entirely in detail. Basically what i did was:
1. Added kvm-amd.npt=0 to grub command line. i had config in /etc/modprobe.d/.. but it did not work because kvm stuff is compiled in kernel and not as module that.
2. Did not use any patches i most likely dont need (like intel fixes since i have amd cpu/chipset):

patch -p1 -i "${srcdir}/0001-Bluetooth-allocate-static-minor-for-vhci.patch"
patch -p1 -i "${srcdir}/0002-module-allow-multiple-calls-to-MODULE_DEVICE_TABLE-p.patch"
patch -p1 -i "${srcdir}/0003-module-remove-MODULE_GENERIC_TABLE.patch"
patch -p1 -i "${srcdir}/radeon_load_vbios_from_file.patch"
patch -p1 -i "${srcdir}/override_for_missing_acs_capabilities.patch"

3. Copied timer subsystem configs from config suplied in first post:

CONFIG_TICK_ONESHOT=y
CONFIG_NO_HZ_COMMON=y
# CONFIG_HZ_PERIODIC is not set
CONFIG_NO_HZ_IDLE=y
# CONFIG_NO_HZ_FULL is not set
# CONFIG_NO_HZ is not set
CONFIG_HIGH_RES_TIMERS=y

after recompiling kernel performance is great. passmark still scores about 90% of bare metal speed on directx9 complex benchmark but hell i am ready to sacrifice that much since gpu is good enough to leave some room for penalties. directx10/11 benchmarks score in ~98% of bare metal performance though.

I suppose i missed all this stuff because of being rather new to linux and working on linux mint instead of arch. So newbies - make sure you put configs in right spot (modprobe.d or grub) and be aware of additional kernel build options!

@nbhs: it would be great if you could update first post with details of any other important kernel build options because not everyone might wish to use those configs supplied. Also all this information you shared made me and i bet lots of other people very happy pandas. Thanks bro.

Eb

Member

Registered: 2014-04-08

Posts: 11

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Eb, I have no idea what your missing, however, here is my PKGBUILD and acsv2 patch in pastebin.
PKGBUILD: http://pastebin.com/G7huLq6p
acs.patch: http://pastebin.com/gpTZiuXv

Thanks, rebuilt with what you provided with no issues.

Unfortunately, with v2 of the patch I'm getting the same problem where I have an overly large IOMMU group containing all my video devices and then some (See: https://bbs.archlinux.org/viewtopic.php … #p1402357).

I have the downstream option enabled for pcie_acs_override in my boot options, as well as intel_iommu.

I had this working before, but that was before installing another video card to try to fix some terrible graphical artifacts on the host when starting the KVM. I believe I was using linux-mainline 3.13 at that point as well.

Is there anything else I can try?

I've removed my secondary video card, and got this:

ls -l /sys/kernel/iommu_groups/1/devices
total 0
lrwxrwxrwx 1 root root 0 Apr  9 11:33 0000:00:01.0 -> ../../../../devices/pci0000:00/0000:00:01.0
lrwxrwxrwx 1 root root 0 Apr  9 11:33 0000:01:00.0 -> ../../../../devices/pci0000:00/0000:00:01.0/0000:01:00.0
lrwxrwxrwx 1 root root 0 Apr  9 11:33 0000:02:08.0 -> ../../../../devices/pci0000:00/0000:00:01.0/0000:01:00.0/0000:02:08.0
lrwxrwxrwx 1 root root 0 Apr  9 11:33 0000:02:10.0 -> ../../../../devices/pci0000:00/0000:00:01.0/0000:01:00.0/0000:02:10.0
lrwxrwxrwx 1 root root 0 Apr  9 11:33 0000:02:11.0 -> ../../../../devices/pci0000:00/0000:00:01.0/0000:01:00.0/0000:02:11.0
lrwxrwxrwx 1 root root 0 Apr  9 11:33 0000:03:00.0 -> ../../../../devices/pci0000:00/0000:00:01.0/0000:01:00.0/0000:02:08.0/0000:03:00.0
lrwxrwxrwx 1 root root 0 Apr  9 11:33 0000:03:00.1 -> ../../../../devices/pci0000:00/0000:00:01.0/0000:01:00.0/0000:02:08.0/0000:03:00.1

The IOMMU group is still quite large, but when I try to run the KVM, it runs.

After putting my secondary video card back in, I'm running into the same old errors. I've tried messing with my boot line a bit, running pcie_acs_override=downstream seems to not make any changes. Perhaps I patched it in wrong? Maybe I'm not installing the new kernel correctly? I'm patching (via PKGBUILD), then building via makepkg, then using pacman -U <local_package_name> to install it. Am I missing something here? I'm using the PKGBUILD and ACS patch as provided in the quote at the top of this post.

When I check dmesg for anything mentioning acs, I only get the following:

dmesg | grep acs
[    0.000000] Command line: BOOT_IMAGE=../vmlinuz-linux root=UUID=30b74524-9b44-46bc-bcca-6f7ebafdbf0a rw intel_iommu=on pci-stub.ids=1002:6810,1002:aab0 pcie_acs_override=downstream initrd=../initramfs-linux.img
[    0.000000] Kernel command line: BOOT_IMAGE=../vmlinuz-linux root=UUID=30b74524-9b44-46bc-bcca-6f7ebafdbf0a rw intel_iommu=on pci-stub.ids=1002:6810,1002:aab0 pcie_acs_override=downstream initrd=../initramfs-linux.img

Checking for ACS instead of acs gives:

dmesg | grep ACS
[    0.000000] ACPI: FACS 000000007c172080 000040

Looking for mentions of IOMMU gives:

dmesg | grep IOMMU
[    0.000000] Intel-IOMMU: enabled
[    0.015176] dmar: IOMMU 0: reg_base_addr fed90000 ver 1:0 cap c0000020660462 ecap f0101a
[    0.015180] dmar: IOMMU 1: reg_base_addr fed91000 ver 1:0 cap d2008020660462 ecap f010da
[    0.015245] IOAPIC id 2 under DRHD base  0xfed91000 IOMMU 1
[    0.498206] IOMMU 0 0xfed90000: using Queued invalidation
[    0.498207] IOMMU 1 0xfed91000: using Queued invalidation
[    0.498208] IOMMU: Setting RMRR:
[    0.498217] IOMMU: Setting identity map for device 0000:00:02.0 [0x7f800000 - 0x8f9fffff]
[    0.499450] IOMMU: Setting identity map for device 0000:00:1d.0 [0x7c00d000 - 0x7c01afff]
[    0.499470] IOMMU: Setting identity map for device 0000:00:1a.0 [0x7c00d000 - 0x7c01afff]
[    0.499488] IOMMU: Setting identity map for device 0000:00:14.0 [0x7c00d000 - 0x7c01afff]
[    0.499500] IOMMU: Prepare 0-16MiB unity mapping for LPC
[    0.499505] IOMMU: Setting identity map for device 0000:00:1f.0 [0x0 - 0xffffff]
[    1.889788] AMD IOMMUv2 driver by Joerg Roedel <joerg.roedel@amd.com>
[    1.889790] AMD IOMMUv2 functionality not available on this system

I'm running an Intel core, not AMD, so I don't think that the AMD IOMMUv2 messages at the end matter.

Edit

D'oh, was never actually booting into the new kernel. However, still getting the same issue of IOMMU group containing the video cards being too large. The patch seems to be working however, as I have many, many more groups.

Last edited by Eb (2014-04-11 20:04:23)

gneville

Member

Registered: 2014-04-02

Posts: 27

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

I've just found something interesting with my setup.

I can get around the BSOD on Windows boot after installing the drivers, providing I do a fresh reload of the Host. I also had to remove the HDMI Audio from the ioh3420 device.

Edit: Confirmed this is the same for Win7 & Win8 as a Guest

But provided I reboot the Host everytime I boot the Windows Guest with the below qemu commands it will start up and use the AMD drivers correctly, if I then reboot the Guest or Shutdown and restart the Qemu again from Command line then Windows will BSOD on boot with the "PAGE_FAULT_IN_NONPAGED_AREA" error.

Any ideas what can be causing this please?

/usr/local/qemu-xen-4-4-git-patched/bin/qemu-system-x86_64 -name win8 -enable-kvm -M q35 -m 6144  \
-cpu Haswell,hv-time,hv_relaxed,hv_vapic,hv_spinlocks=0x1000 \
-boot order=dc \
-smp 2,sockets=1,cores=2,threads=1 \
-bios /home/xen/vfio/bios.bin -vga none \
-usb -usbdevice tablet \
-spice port=5902,disable-ticketing \
-vnc 0.0.0.0:0 \
-net nic \
-net tap,ifname=tap0,script=no,downscript=no \
-device ioh3420,bus=pcie.0,addr=1c.0,multifunction=on,port=1,chassis=1,id=root.1 \
-device vfio-pci,host=01:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on \
-device vfio-pci,host=01:00.1,bus=pcie.0 \
-device vfio-pci,host=00:14.0,bus=pcie.0 \
-device ahci,bus=pcie.0,id=ahci \
-drive file=/dev/mapper/vg1-winv1,media=disk,index=1,id=disk,cache=none,if=virtio \
-drive file=/dev/mapper/vg1-winv1data,media=disk,index=2,id=diskb,cache=none,if=virtio \
-drive file=/home/xen/iso/windows/en-gb_windows_8_1_enterprise_x64_dvd_2971910.iso,boot=on,media=cdrom,index=3,id=isocd  \
-device ide-cd,bus=ahci.1,drive=isocd \
-drive file=/root/virtio-win-0.1-74.iso,media=cdrom,index=4,id=virtio  \
-device ide-cd,bus=ahci.2,drive=virtio

Still not having much luck with vfio vga passthrough on my AMD R9 290

I've tried pretty much all the suggestions I've seen on here but no matter what I try the guest always crashes after installing video drivers.

I did swap from Win8 to Win7 and noticed that after installing the video drivers and rebooting when windows boots up it crashes with a BSOD saying

"PAGE_FAULT_IN_NONPAGED_AREA"

I've tried the following:
multiple linux kernels all with the patches rolled in.
multiple Qemu versions (1.7.0, 1.7.1, 2.0.0RC0,1 &2) and have applied the vga reset patches
Tried both machine types of q35 and i440fx, of which both successfully passthrough the vga card but crash after installing the drivers and rebooting windows
Tried just passing through the Video card leaving out the HDMI Audio.
Tried installing drivers without installing the Catalyst Control Centre as suggested by some other forums.
Tried both AMD drivers and Windows Provided drivers via Windows Updates.
Tried using RomFiles

I'm guessing the problem lies within Qemu somewhere.

On top of this I have just installed the OP linux-mainline (3.13.6-1-mainline),seabios (1.7.4.r1788.g8abc6a8-1) & qemu (2.0.r31714.g03d5142-1) versions (Just added reset patch and manually compiled (https://lists.gnu.org/archive/html/qemu … 0767.html))

and still no change

Hi All,

I'm hoping someone will be able to help me out please. I've been trying to get vga passthrough working with qemu and vfio for a few months on and off but having no luck. I'm getting similar problems to users redger & DanaGoyette.

My hardware: CPU: i7 4770s, MB: Jetway NF9J-Q87 GFX: AMD R9 290

I used to use Xen 4.3 and had vga passthrough working just passing through the intel hd4600 controller to hosts, but since I got the AMD R9 290 card Xen doesn't seem to passthrough the card whatsoever, so thought I would try qemu and vfio.

I've tried using different linux versions and qemu versions but not having much luck. I am able to successfully vfio-bind my gfx card and I can start QEMU and get video via the GFX Card HDMI cable, I'm also able to successfully install Windows 8.1. However after installation I'm getting problems with either windows installing the amd drivers or using the catalyst driver installer, the host just reboots.

My latest setup:

linux 3.13.7-1 + i915 patch
Seabios git 1.7.4-66 (31Mar14)
Qemu 2.0.0 RC0 + patched pcibus_reset & assertion (https://lists.nongnu.org/archive/html/q … 05901.html & https://lists.gnu.org/archive/html/qemu … 00767.html)

Before the pcibus_reset patch I kept getting the following messages when the host rebooted while trying to install the drivers:

After applying the pcibust_reset patch the host no longer reboots after installing the drivers however the screen just goes blank so I have to manually kill the host. I do see the following messages:

This is my QEMU config:

If I enable "-vga cirrus" and VNC to the host I can install the AMD drivers successfully and after a reboot it shows in device manager the AMD card but with a yellow exclamation and code 43. If I then go back to "-vga none" windows no longer wants to boot and I need up in a boot loop (seabios>windows booting>seabios>windows booting).

Any ideas please?

Last edited by gneville (2014-04-11 19:55:10)

Nex7

Member

Registered: 2014-04-06

Posts: 5

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

So I've discovered what I believe to be my problem, but I'm confused why nobody else seems to be hitting it.

https://lists.gnu.org/archive/html/qemu … 01858.html

That's what's hitting me - I'm getting c0000221 stop's on boot on my VM, at this point, and seemingly this is due to virtio-scsi. I wonder why nobody else is hitting this?

EDIT: Of course, starting over with a fresh Win7 x64 install using ide instead of virtio, it works fine up until I try to install Windows updates, even only a few of them, at which point after reboot it dies with a different BSOD, code c000021a. -sigh-

Last edited by Nex7 (2014-04-12 02:55:53)

novist

Member

Registered: 2014-03-14

Posts: 47

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

So I've discovered what I believe to be my problem, but I'm confused why nobody else seems to be hitting it.

https://lists.gnu.org/archive/html/qemu … 01858.html

That's what's hitting me - I'm getting c0000221 stop's on boot on my VM, at this point, and seemingly this is due to virtio-scsi. I wonder why nobody else is hitting this?

EDIT: Of course, starting over with a fresh Win7 x64 install using ide instead of virtio, it works fine up until I try to install Windows updates, even only a few of them, at which point after reboot it dies with a different BSOD, code c000021a. -sigh-

you are not alone here. i made attempt to use virtio-scsi too. however i had OS installed on IDE drive and later on switched to virtio-scsi. what i experienced was various files being seemingly corrupted (software throwing errors due to crc32 missmatch, applications failing to read files etc). switching back to virtio made it all work fine again. seems like virtio-scsi does cause some reading problems and what not. so just stick to virtio i guess. its really fast anyway.

Last edited by novist (2014-04-12 03:30:48)

Eb

Member

Registered: 2014-04-08

Posts: 11

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Success! ..kind of. I've compiled with the old patch and everything seems to be working proper. After installing Windows 8.1 I decided to install the drivers for the video card. Windows seems to be running smoothly, but now I have no video output. Looking into exactly what may be happening here now..

Edit:

After trying both beta and non beta drivers, I'm consistantly getting a black screen as soon as I finish installing the drivers and the VM switches from basic output to using the card. Wiping the VM fixed this until installing the drivers once more.  Any ideas how I can get video back?

Last edited by Eb (2014-04-12 07:16:00)

Krobar

Member

Registered: 2014-04-12

Posts: 17

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Adding -curses to the qemu-system options fixed this for me.

Try:

-cpu qemu64

and remove the -smp line

If that worked you might have built qemu without kvm support

#qemu-system-x86_64 -enable-kvm -M q35 -cpu qemu64 \ -bios /usr/share/qemu/bios.bin -vga none \ -device ioh3420,bus=pcie.0,addr=1c.0,multifunction=on,port=1,chassis=1,id=root.1 \ -device vfio-pci,host=01:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on \ -device vfio-pci,host=01:00.1,bus=root.1,addr=00.1

qemu-system-x86_64: -cpu qemu64: could not open disk image  -bios: No such file or directory

I'm looking into reinstalling qemu but like i said, you removed the link from your original post and i don't have any different source to build from than my current one except the git one, which is v 1.6 and has seabios intergrated, and i don't know how to patch the intergrated seabios, i couldn't find out how (but otherwise that is what i would do).

i tried removing the backslashes and the VM will start if i run it from X.

qemu-system-x86_64 -enable-kvm -M q35 -cpu host -smp 6,sockets=1,cores=6,threads=1 -bios /usr/share/qemu/bios.bin -vga none -device ioh3420,bus=pcie.0,addr=1c.0,multifunction=on,port=1,chassis=1,id=root.1 -device vfio-pci,host=01:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on -device vfio-pci,host=01:00.1,bus=root.1,addr=00.1

qemu-system-x86_64: -device vfio-pci,host=01:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on: Warning, device 0000:01:00.0 does not support reset
qemu-system-x86_64: -device vfio-pci,host=01:00.1,bus=root.1,addr=00.1: Warning, device 0000:01:00.1 does not support reset
qemu-system-x86_64: Attempt to reset PCI bus for VGA support failed (Inappropriate ioctl for device).  VGA may not work.

if not running from x (terminal) then i get

Could not initialize SDL(No available video device) - exiting

instead of

qemu-system-x86_64: Attempt to reset PCI bus for VGA support failed (Inappropriate ioctl for device).  VGA may not work.

Nex7

Member

Registered: 2014-04-06

Posts: 5

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

you are not alone here. i made attempt to use virtio-scsi too. however i had OS installed on IDE drive and later on switched to virtio-scsi. what i experienced was various files being seemingly corrupted (software throwing errors due to crc32 missmatch, applications failing to read files etc). switching back to virtio made it all work fine again. seems like virtio-scsi does cause some reading problems and what not. so just stick to virtio i guess. its really fast anyway.

Maybe I'm confused - can you show me your commandline, that is using 'virtio' but not 'virtio-scsi'?

In other news, I did get this working on IDE, but have not yet retried virtio-scsi. The symptom was every time I installed the Windows 7 64-bit OS it was fine UNTIL I did ANY Windows Updates. I tried a few times, with different packages, always the same. This got me thinking it isn't the updates themselves, but something they have in common - going down that list, I hit upon 'Restore Points', as those happen any time you do any Windows Updates. So I turned off Restore Points on the OS disk entirely, and after that, did Windows Update, and it came back up after reboot w/o BSOD. It is now completely stable. I have no idea why Restore Points screw the pooch, but they seem to, at least on IDE (haven't retried with virtio).

Eb

Member

Registered: 2014-04-08

Posts: 11

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Success! ..kind of. I've compiled with the old patch and everything seems to be working proper. After installing Windows 8.1 I decided to install the drivers for the video card. Windows seems to be running smoothly, but now I have no video output. Looking into exactly what may be happening here now..

Edit:

After trying both beta and non beta drivers, I'm consistantly getting a black screen as soon as I finish installing the drivers and the VM switches from basic output to using the card. Wiping the VM fixed this until installing the drivers once more.  Any ideas how I can get video back?

I've booted directly into the Windows system (8.1) and the card works perfectly.  Back to KVM and it doesn't work. I've even tried to install manufacturer drivers, no dice there either. I'm not sure completely if this is an issue with drivers or my hardware, but since it's only having trouble with QEMU I'm hoping that there's some options I can change when booting the KVM. Are there any QEMU boot options I can experiment with? Should just return the card and try my luck with a different model?

Eagerly waiting for a reply, I don't have much of a window left to return the card.

nbhs

Member

From: Montevideo, Uruguay

Registered: 2013-05-02

Posts: 402

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Success! ..kind of. I've compiled with the old patch and everything seems to be working proper. After installing Windows 8.1 I decided to install the drivers for the video card. Windows seems to be running smoothly, but now I have no video output. Looking into exactly what may be happening here now..

Edit:

After trying both beta and non beta drivers, I'm consistantly getting a black screen as soon as I finish installing the drivers and the VM switches from basic output to using the card. Wiping the VM fixed this until installing the drivers once more.  Any ideas how I can get video back?

I've booted directly into the Windows system (8.1) and the card works perfectly.  Back to KVM and it doesn't work. I've even tried to install manufacturer drivers, no dice there either. I'm not sure completely if this is an issue with drivers or my hardware, but since it's only having trouble with QEMU I'm hoping that there's some options I can change when booting the KVM. Are there any QEMU boot options I can experiment with? Should just return the card and try my luck with a different model?

Eagerly waiting for a reply, I don't have much of a window left to return the card.

Have you tried swapping card to check if you get the same result? Have you tried to passthrough only the gpu without the audio device?

apex8

Member

Registered: 2014-03-29

Posts: 60

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Hi,
I have a question regarding disk performance. Originally I was using a file on a XFS partition as hard disk image:

-drive if=none,file=/home/vms/drives/winimg.img,id=vdisk1,cache=none,format=raw,aio=native \
-device virtio-blk,scsi=off,config-wce=off,drive=vdisk1,id=disk1 \

The physical hard disk is not an SSD

With this I can't use the virtio option x-data-plane=on.
However in the Windows Performance Indicator the Harddisk had a score of 6.5

Then I switched to using an physical hard disk partition with the following configuration:

-drive if=none,file=/dev/sdb7,id=vdisk1,cache=none,format=raw,aio=native \
-device virtio-blk,scsi=off,config-wce=off,drive=vdisk1,id=disk1,x-data-plane=on \

I thought performance would increase a bit, but the performance indicator dropped to 5.9.

I there some configuarion on the host, I missed?

BR

spam

Member

Registered: 2013-06-15

Posts: 3

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Hello,

I have been trying vga-passthroug but I have managed to get it properly yet.

My hardware config is:
ASRock Z87 Extreme 4
i7 4771
GTX 660 (ASUS DC2O)

I have used the OP packages and with and without this patch  http://lists.nongnu.org/archive/html/qe … 00597.html 

I have not read all thread yet and it is a bit confusing with patches here and there. I can only get a code 43 in windows. I believe I have to enable acs  but atm do I need a patch or only a kernel parameter for linux-mainline ?

I boot in my MB in UEFI mode (at least I believe so), does it matter ?

./lsgroup added
### Group 0 ###
    00:00.0 Host bridge: Intel Corporation 4th Gen Core Processor DRAM Controller (rev 06)
### Group 1 ###
    00:01.0 PCI bridge: Intel Corporation Xeon E3-1200 v3/4th Gen Core Processor PCI Express x16 Controller (rev 06)
    01:00.0 VGA compatible controller: NVIDIA Corporation GK106 [GeForce GTX 660] (rev a1)
    01:00.1 Audio device: NVIDIA Corporation GK106 HDMI Audio Controller (rev a1)
### Group 2 ###
    00:02.0 VGA compatible controller: Intel Corporation Xeon E3-1200 v3/4th Gen Core Processor Integrated Graphics Controller (rev 06)
### Group 3 ###
    00:03.0 Audio device: Intel Corporation Xeon E3-1200 v3/4th Gen Core Processor HD Audio Controller (rev 06)
### Group 4 ###
    00:14.0 USB controller: Intel Corporation 8 Series/C220 Series Chipset Family USB xHCI (rev 05)
### Group 5 ###
    00:16.0 Communication controller: Intel Corporation 8 Series/C220 Series Chipset Family MEI Controller #1 (rev 04)
### Group 6 ###
    00:19.0 Ethernet controller: Intel Corporation Ethernet Connection I217-V (rev 05)
### Group 7 ###
    00:1a.0 USB controller: Intel Corporation 8 Series/C220 Series Chipset Family USB EHCI #2 (rev 05)
### Group 8 ###
    00:1b.0 Audio device: Intel Corporation 8 Series/C220 Series Chipset High Definition Audio Controller (rev 05)
### Group 9 ###
    00:1c.0 PCI bridge: Intel Corporation 8 Series/C220 Series Chipset Family PCI Express Root Port #1 (rev d5)
    00:1c.2 PCI bridge: Intel Corporation 8 Series/C220 Series Chipset Family PCI Express Root Port #3 (rev d5)
    00:1c.5 PCI bridge: Intel Corporation 8 Series/C220 Series Chipset Family PCI Express Root Port #6 (rev d5)
    03:00.0 SATA controller: ASMedia Technology Inc. ASM1062 Serial ATA Controller (rev 01)
    04:00.0 Network controller: Qualcomm Atheros AR9287 Wireless Network Adapter (PCI-Express) (rev 01)
### Group 10 ###
    00:1d.0 USB controller: Intel Corporation 8 Series/C220 Series Chipset Family USB EHCI #1 (rev 05)
### Group 11 ###
    00:1f.0 ISA bridge: Intel Corporation Z87 Express LPC Controller (rev 05)
    00:1f.2 SATA controller: Intel Corporation 8 Series/C220 Series Chipset Family 6-port SATA Controller 1 [AHCI mode] (rev 05)
    00:1f.3 SMBus: Intel Corporation 8 Series/C220 Series Chipset Family SMBus Controller (rev 05)

I add a bit of information on my software configuration:

$ cat /proc/cmdline 
BOOT_IMAGE=/vmlinuz-linux-mainline root=UUID=bc42943b-cfbd-47df-931f-40a88e0a4ef9 rw quiet intel_iommu=on pci-stub.ids=10de:11c0,10de:0e0b

$ cat /etc/modprobe.d/*
blacklist nouveau
blacklist nvidia
options kvm ignore_msrs=1
options pci-stub ids=10de:11c0,10de:0e0b
options vfio_iommu_type1 allow_unsafe_interrupts=1

$ cat /etc/modules-load.d/*
pci-stub

And to finish the qemu command I have been trying:

qemu-system-x86_64 -enable-kvm -M q35 -m 4096 -cpu host -smp 4,sockets=1,cores=4,threads=1 -bios /usr/share/qemu/bios.bin -vga none \
-device ioh3420,bus=pcie.0,addr=1c.0,multifunction=on,port=1,chassis=1,id=root.1 \
-device vfio-pci,host=01:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on \
-device vfio-pci,host=01:00.1,bus=root.1,addr=00.1 \
-drive file=/home/spam/windows.img,id=disk,format=raw \
-device ide-hd,bus=ide.0,drive=disk \
-drive file=/home/spam/windows.iso,id=isocd \
-device ide-cd,bus=ide.1,drive=isocd \
-usb -usbdevice host:003.006

And to finish, the last dmesg

[ 5663.995723] VFIO - User Level meta-driver version: 0.3
[ 6337.160714] vfio-pci 0000:01:00.0: enabling device (0000 -> 0003)
[ 6337.160846] vfio_ecap_init: 0000:01:00.0 hiding ecap 0x19@0x900
[ 6376.195200] kvm: zapping shadow pages for mmio generation wraparound

Last edited by spam (2014-04-13 19:51:17)

Torello

Member

Registered: 2014-04-01

Posts: 6

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Hello,
Finally, vga-passthrough work. ( Using kernel with debug registers patch, qemu and seabios from first post ).
But I have a strange issue, which makes keyboard and mouse in guest os - unusable.

My specs:
Core i7 4771, GA-Z87X-OC , Radeon 6950@6970 for guest and integrated intel video for host.
Keyboard Microsoft Natural 4000 ( usb ), mouse Logitech Mx510 ( usb ).

My startup script is:
qemu-system-x86_64 -enable-kvm -M q35 -m 16384 -cpu host \
-smp 4,sockets=1,cores=4,threads=1 \
-vga none \
-device ioh3420,bus=pcie.0,addr=1c.0,multifunction=on,port=1,chassis=1,id=root.1 \
-device vfio-pci,host=01:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on \
-device vfio-pci,host=01:00.1,bus=root.1,addr=00.1 \
-drive file=/home/torello/virt_images/windows7.img,id=disk,format=raw -device ide-hd,bus=ide.0,drive=disk \
-drive file=/home/torello/distr/win7_msdn.iso,id=isocd -device ide-cd,bus=ide.1,drive=isocd \
-boot menu=on

If I boot vm without "-vga none".
I can switch to qemu window, and it grabs my keyboard and mouse input.

If I boot with "-vga none", I can't use keyboard in guest at all, and with my mouse I can do some operations like scrolling the window using the thumb, or change window place ( using mouse ), but If I want to click a button I must click 20-30 times, trying different corners, before I was able to press any onscreen button.

Do you have any suggestions, how I can debug/solve that issue?

Last edited by Torello (2014-04-14 05:23:42)

novist

Member

Registered: 2014-03-14

Posts: 47

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

@Torello: pass-through usb ports too.

FruitieX

Member

Registered: 2008-07-03

Posts: 18

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Hi,

After updating to 3.14 kernel with all patches as posted in the OP, I'm getting a *massive* amount of spam in dmesg while my VM is running. (so much that the systemd journal can't keep up and uses 100% CPU)
As a result I get terrible performance in the VM, whereas with the 3.13 kernel I get very good performance and can play Windows games at near native speeds.
Here's the dmesg output that gets repeated:

[   99.118043] WARNING: CPU: 2 PID: 1519 at arch/x86/kvm/x86.c:1114 kvm_get_msr_common+0x793/0x960 [kvm]()
[   99.118047] Modules linked in: vfio_pci vfio_iommu_type1 vfio ipt_MASQUERADE iptable_nat nf_nat_ipv4 nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_conntrack nf_conntrack ipt_REJECT xt_
CHECKSUM iptable_mangle xt_tcpudp bridge stp llc ip6table_filter ip6_tables iptable_filter ip_tables x_tables nls_iso8859_1 nls_cp437 vfat fat ext4 mbcache jbd2 snd_hda_codec_hdmi iTCO
_wdt iTCO_vendor_support ppdev hid_logitech_dj coretemp hwmon x86_pkg_temp_thermal intel_powerclamp kvm_intel kvm crct10dif_pclmul crc32_pclmul ghash_clmulni_intel aesni_intel aes_x86_
64 lrw gf128mul glue_helper ablk_helper cryptd btusb microcode bluetooth pcspkr 6lowpan_iphc psmouse serio_raw i2c_i801 e1000e ptp pps_core lpc_ich rfkill crc16 mei_me mei snd_usb_audi
o parport_pc tpm_tis snd_usbmidi_lib mousedev evdev joydev snd_rawmidi snd_hda_codec_realtek
[   99.118064]  snd_seq_device tpm mac_hid snd_hda_codec_generic i915 nuvoton_cir parport battery rc_core snd_hda_intel snd_hda_codec drm_kms_helper snd_hwdep video drm snd_pcm button
intel_gtt i2c_algo_bit snd_timer i2c_core snd soundcore shpchp processor btrfs xor raid6_pq hid_generic usbhid hid bcache sd_mod crc_t10dif crct10dif_common atkbd libps2 crc32c_intel a
hci libahci libata scsi_mod ehci_pci xhci_hcd ehci_hcd usbcore usb_common i8042 serio pci_stub
[   99.118078] CPU: 2 PID: 1519 Comm: qemu-system-x86 Tainted: G        W    3.14.0-ARCH #1
[   99.118079] Hardware name: To Be Filled By O.E.M. To Be Filled By O.E.M./Z87M Pro4, BIOS P2.00 11/07/2013
[   99.118080]  0000000000000009 ffff8800b9505c08 ffffffff814cea8a 0000000000000000
[   99.118081]  ffff8800b9505c40 ffffffff81064b1d 0000000040000020 ffff8800b9505cd0
[   99.118083]  ffff8804132e3ec0 ffff8803fbaf0000 ffff8803fbaf0040 ffff8800b9505c50
[   99.118084] Call Trace:
[   99.118085]  [<ffffffff814cea8a>] dump_stack+0x4d/0x6f
[   99.118086]  [<ffffffff81064b1d>] warn_slowpath_common+0x7d/0xa0
[   99.118087]  [<ffffffff81064bfa>] warn_slowpath_null+0x1a/0x20
[   99.118091]  [<ffffffffa0952233>] kvm_get_msr_common+0x793/0x960 [kvm]
[   99.118093]  [<ffffffffa0ad1310>] ? vmx_get_msr+0x370/0x370 [kvm_intel]
[   99.118095]  [<ffffffffa0ad10a2>] vmx_get_msr+0x102/0x370 [kvm_intel]
[   99.118097]  [<ffffffffa0ad133a>] handle_rdmsr+0x2a/0x140 [kvm_intel]
[   99.118099]  [<ffffffffa0ad1b8f>] ? handle_exception+0x16f/0x3e0 [kvm_intel]
[   99.118101]  [<ffffffffa0ad6975>] vmx_handle_exit+0xb5/0xa20 [kvm_intel]
[   99.118103]  [<ffffffffa0ace360>] ? vmx_invpcid_supported+0x20/0x20 [kvm_intel]
[   99.118107]  [<ffffffffa09577a0>] kvm_arch_vcpu_ioctl_run+0xc70/0x11c0 [kvm]
[   99.118110]  [<ffffffffa095363e>] ? kvm_arch_vcpu_load+0x4e/0x1e0 [kvm]
[   99.118113]  [<ffffffffa09405c2>] kvm_vcpu_ioctl+0x2b2/0x5b0 [kvm]
[   99.118115]  [<ffffffff814d11fd>] ? __schedule+0x86d/0x900
[   99.118116]  [<ffffffff8101e966>] ? ___preempt_schedule+0x56/0xb0
[   99.118118]  [<ffffffff811b3de0>] do_vfs_ioctl+0x2e0/0x4c0
[   99.118119]  [<ffffffff811bdace>] ? __fget+0x6e/0xb0
[   99.118121]  [<ffffffff811b4041>] SyS_ioctl+0x81/0xa0
[   99.118122]  [<ffffffff814dc469>] system_call_fastpath+0x16/0x1b
[   99.118126] ---[ end trace 8d24f63038d6e9d0 ]---

I'm noticing that if I remove the hv-time flag from my qemu launch options I don't get this dmesg spam anymore, but VM performance is still really bad up to a point where even moving the mouse cursor is laggy.
Is anyone else experiencing this issue, and any ideas how to fix this? Again, the 3.13 kernel was fine.

System specs:
CPU: Intel i5-4670
GPU: AMD Radeon 280X
Mobo: ASRock Z87M Pro4

FruitieX

Last edited by FruitieX (2014-04-14 10:55:31)

anickname

Member

Registered: 2014-01-07

Posts: 23

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Hi,

After updating to 3.14 kernel with all patches as posted in the OP, I'm getting a *massive* amount of spam in dmesg while my VM is running. (so much that the systemd journal can't keep up and uses 100% CPU)
As a result I get terrible performance in the VM, whereas with the 3.13 kernel I get very good performance and can play Windows games at near native speeds.
Here's the dmesg output that gets repeated:

I'm noticing that if I remove the hv-time flag from my qemu launch options I don't get this dmesg spam anymore, but VM performance is still really bad up to a point where even moving the mouse cursor is laggy.
Is anyone else experiencing this issue, and any ideas how to fix this? Again, the 3.13 kernel was fine.

System specs:
CPU: Intel i5-4670
GPU: AMD Radeon 280X
Mobo: ASRock Z87M Pro4

FruitieX

https://bugzilla.kernel.org/show_bug.cgi?id=73721
Still waiting for an answer.

Val532

Member

Registered: 2013-11-13

Posts: 35

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Hi,

After updating to 3.14 kernel with all patches as posted in the OP, I'm getting a *massive* amount of spam in dmesg while my VM is running. (so much that the systemd journal can't keep up and uses 100% CPU)
As a result I get terrible performance in the VM, whereas with the 3.13 kernel I get very good performance and can play Windows games at near native speeds.
Here's the dmesg output that gets repeated:

I'm noticing that if I remove the hv-time flag from my qemu launch options I don't get this dmesg spam anymore, but VM performance is still really bad up to a point where even moving the mouse cursor is laggy.
Is anyone else experiencing this issue, and any ideas how to fix this? Again, the 3.13 kernel was fine.

System specs:
CPU: Intel i5-4670
GPU: AMD Radeon 280X
Mobo: ASRock Z87M Pro4

FruitieX

https://bugzilla.kernel.org/show_bug.cgi?id=73721
Still waiting for an answer.

Hi,

I do not understand your problem because i do not have it !!

i use

qemu-system-x86_64 \
-M q35 -m 8G -enable-kvm -cpu host,hv-time,hv_relaxed,hv_vapic,hv_spinlocks=0x1000 \
-no-user-config -nodefaults -vga none -boot d -nographic \
-smp 4,sockets=1,cores=4,threads=1 \
...

And i have not any problem.

Qemu 2.0.0-RC2 + original kernel 3.14 from kernel.org + i915, acs (and other litle patch on KVM side but without it, it works the same).

On a Intel Core i5-4670S + Asrock Z87 Extrem 6 and two HD 6870.