Arch Linux

rb3

Member

Registered: 2012-09-17

Posts: 9

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Hey guys, I'm trying to set up the vfio passthrough on my mid-2014 rMBP, as I have documented here. I'm getting stuck at the "Testing if its working out" section of the original post on this thread. When I execute the indicated command (modified to provide my ROM file, which I'm not sure supports EFI), I get a QEMU window with a prompt, and no output in the connected monitor. Can anyone steer me in the right direction to get this to work? Thanks!

ghormoon

Member

Registered: 2015-05-20

Posts: 17

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

can you SSH to the guest to check if the device is listed there? if yes, get lspci -v to compare with host, if it looks ok or not.
EDIT: if you mean qemu with prompt as login prompt, you don't even need ssh.
but that brings me to another question - I assume the graphics are switchable,  not dedicated one for display, one for external one. you'd need to switch to the passed in graphics to see anything.

Last edited by ghormoon (2015-06-24 04:48:35)

CharlieBra7o

Member

Registered: 2014-03-01

Posts: 20

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

@CharlieBra7o

Maybe the card doesn't really work with MSI either, interrupts are surprisingly difficult for hardware to get right.  I'd pursue the broken intx masking path if MSI doesn't immediately work.

Quick update on the X-Fi issue:

While trying the nointmask=1 option, I found out, that for some reason my modprobe.d conf file doesnt even get loaded on boot... Nevertheless, with nointmask=1 enabled, the VM can't bind the PCI-E device, so I discarded that option and tried the MSI option again..:

I found out, that once this "IRQ n: nobody cared [...] Disabling IRQ #n" error occurs (ie. when I install some sort of official X-Fi drivers, there is no way (reboot/reinstall) of getting sound again, until the host gets rebooted. Then, after a host reboot and a Windows reinstall, the X-Fi works again without MSI. Without installing any official X-Fi drivers and using Windows' built-in "HDA" drivers, which give me sound, and with MSISupported=0 I can then reboot the VM and no IRQ issues come up. Trying MSISupported=1 successfully enables MSI and the sound still works after rebooting the VM.

The problem with the X-Fi is, that it has this so-called "FlexiJack" input, which defaults to "Line-In" and can only be set to "Microphone" with the official X-Fi drivers (at least as far as my research goes). So without the official drivers, there is no way of using a Microphone except connecting a front panel via the "HD audio" pin connectors on the sound card with front panel headphones/microphone connectors - which can be used right away, without prior configuration. My front panel, unfortunally, introduces a lot of noise into the signal, which makes me crazy... Nevertheless, I'm now gonna stick to this somewhat 'dirty' workaround...

The_Moves

Member

Registered: 2015-01-06

Posts: 59

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

I've not had these issues with my Sound Blaster Z. Windows won't install any default drivers. But once I install the Creative Drivers it works. That was me using Windows 7.

mostlyharmless

Member

Registered: 2014-01-16

Posts: 72

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Just a quick update in case anyone has the same problem:  I had a working Seabios qemu command line q35 Windows 7 which eventually I converted over to OVMF/libvirt/440fx, no reinstall, though I had to "reauthenticate" with Microsoft a couple of times.  Everything worked fine except for XP mode which quit with a cryptic "Unable to start server message" (see 5122-5125)

Windows Virtual PC

Cannot start Windows Virtual PC Host Process. Check the System event log for more details

Server execution failed
[OK]
The Event log had:

The server {567DUXXX1-4678-A214-61DXXXXXXXXX} did not register with DCOM within the required timeout

I noticed that since the change to OVMF etc, Windows now thinks I have a Xeon E312x instead of a i7-4930k (which is the actual hardware). I had tried changing to "copy host configuration" and incorporated the other changes suggested by aw, but had no luck with XP mode and when I tried changing to cpu host got a bluescreen crash.  I then noticed that the default configuration in libvirt was 6 sockets, 1 core and 1 thread, and changed it manually to 1 socket, 6 cores and 1 thread.  Figured a i7 system should only have one socket    Voila, no bluescreen with cpu host: Windows still thinks I have a Xeon, Device Manager is unchanged with 6 cpus, but now XP mode works again. Performance seems the same regardless.

Last edited by mostlyharmless (2015-06-25 15:26:43)

Denso

Member

Registered: 2014-08-30

Posts: 179

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

XP mode

You need to enable "Nested Virtualization" :

$ nano /etc/modprobe.d/nested.conf

options kvm_intel nested=1
options kvm_intel enable_shadow_vmcs=1

reboot and test XP Mode

Edit : Oops , didn't realize you got it working already !

Last edited by Denso (2015-06-25 16:26:52)

mostlyharmless

Member

Registered: 2014-01-16

Posts: 72

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Thanks, yeah, I already had those things enabled, furthermore, XP mode had worked before on the old setup, so it couldn't have been a host configuration problem, it had to be a VM configuration problem, logically speaking....

Duelist

Member

Registered: 2014-09-22

Posts: 358

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

I noticed that since the change to OVMF etc, Windows now thinks I have a Xeon E312x instead of a i7-4930k (which is the actual hardware).

  <cpu mode='host-passthrough'>
    <topology sockets='SOCKETS' cores='CORES PER SOCKET' threads='THREADS PER CORE'/>
  </cpu>

So it'd be 1-4-2 for a full, real 4930k since you have hyper-threading.
Or 1-3-2 for your 6 vCPU config.

But the main thing is - host-passthrough cpu mode, which fixed my CPUID to the real one.

---

The forum rules prohibit requesting support for distributions other than arch.
I gave up. It was too late.
What I was trying to do.
The reference about VFIO and KVM VGA passthrough.

Peetipablo

Member

Registered: 2015-06-26

Posts: 16

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

So I've been banging my head against a wall trying to get this to work for a few days with no real success.  I'm able to get a VM up and running, but when I try to turn vga off and pass my graphics card to the VM I don't get any signal at all.  If anyone could take a quick glance at my setup and point me in the right direction, I'd be grateful.

CPU: Intel i5 4690k
GPU: AMD r9 290
Mobo: Asrock Z97 Extreme 4
Memory: 16GB 1600Mhz
Monitor1: Asus 144hz, Displayport ------> IGPU, DVI-D ----------> Discrete GPU
Monitor 2:  Random 60hz, HDMI -----> IGPU

Trying to use my IGP as a host display adapter, and pass through my r9 290 to windows 8.1 VM.

Vfio-bind script:

#!/bin/bash

#!/bin/sh

DEVS="0000:01:00.0 0000:01:00.1"

for DEV in $DEVS; do
    echo "vfio-pci" > /sys/bus/pci/devices/$DEV/driver_override
done

modprobe -i vfio-pci

Vfio-bind script is working correctly (I believe):

[peetipablo@donbot ~]$ tree /sys/bus/pci/devices/0000\:01\:00.0/driver
/sys/bus/pci/devices/0000:01:00.0/driver
├── 0000:01:00.0 -> ../../../../devices/pci0000:00/0000:00:01.0/0000:01:00.0
├── 0000:01:00.1 -> ../../../../devices/pci0000:00/0000:00:01.0/0000:01:00.1
├── bind
├── module -> ../../../../module/vfio_pci
├── new_id
├── remove_id
├── uevent
└── unbind

pci-stub enabled in /etc/mkinitcpio.conf, stub.ids set in /etc/default/grub and bootloader/kernel updated, will attach output if requested.

Starting up in either virt-manager, or from console it seems that the VM is booted and running fine, just zero video output.

/var/log/libvirt/qemu/win8.1.log:

2015-06-26 08:03:16.529+0000: starting up libvirt version: 1.2.16, qemu version: 2.3.0
LC_ALL=C PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin QEMU_AUDIO_DRV=none /usr/sbin/qemu-system-x86_64 -name win8.1 -S -mac$
qemu: terminating on signal 15 from pid 2107
2015-06-26 08:08:44.561+0000: shutting down

Relavent journalctl:

journalctl -b:
Jun 26 04:26:23 donbot libvirtd[2107]: failed to create directory '/var/lib/libvirt/images': File exists
Jun 26 04:26:25 donbot NetworkManager[608]: <info>  (vnet0): carrier is OFF
Jun 26 04:26:25 donbot NetworkManager[608]: <info>  (vnet0): new Tun device (driver: 'unknown' ifindex: 6)
Jun 26 04:26:25 donbot NetworkManager[608]: <info>  (vnet0): exported as /org/freedesktop/NetworkManager/Devices/5
Jun 26 04:26:25 donbot NetworkManager[608]: <info>  (virbr0): bridge port vnet0 was attached
Jun 26 04:26:25 donbot NetworkManager[608]: <info>  (vnet0): enslaved to virbr0
Jun 26 04:26:25 donbot NetworkManager[608]: <error> [1435307185.502845] [devices/nm-device.c:2086] nm_device_generate_connection(): (v
Jun 26 04:26:25 donbot kernel: device vnet0 entered promiscuous mode
Jun 26 04:26:25 donbot NetworkManager[608]: <info>  (vnet0): link connected
Jun 26 04:26:25 donbot kernel: virbr0: port 2(vnet0) entered listening state
Jun 26 04:26:25 donbot kernel: virbr0: port 2(vnet0) entered listening state
Jun 26 04:26:25 donbot dbus[612]: [system] Activating via systemd: service name='org.freedesktop.machine1' unit='dbus-org.freedesktop.
Jun 26 04:26:25 donbot systemd[1]: Starting Virtual Machine and Container Registration Service...
Jun 26 04:26:25 donbot dbus[612]: [system] Successfully activated service 'org.freedesktop.machine1'
Jun 26 04:26:25 donbot systemd[1]: Started Virtual Machine and Container Registration Service.
Jun 26 04:26:25 donbot systemd-machined[2906]: New machine qemu-win8.1.
Jun 26 04:26:25 donbot systemd[1]: Started Virtual Machine qemu-win8.1.
Jun 26 04:26:25 donbot systemd[1]: Starting Virtual Machine qemu-win8.1.
Jun 26 04:26:25 donbot kernel: vfio_ecap_init: 0000:01:00.0 hiding ecap 0x19@0x270
Jun 26 04:26:25 donbot kernel: vfio_ecap_init: 0000:01:00.0 hiding ecap 0x1b@0x2d0
Jun 26 04:26:27 donbot ntpd[1053]: bind(23) AF_INET6 fe80::fc54:ff:feab:ec15%6#123 flags 0x11 failed: Cannot assign requested address
Jun 26 04:26:27 donbot ntpd[1053]: unable to create socket on vnet0 (10) for fe80::fc54:ff:feab:ec15%6#123
Jun 26 04:26:27 donbot ntpd[1053]: failed to init interface for address fe80::fc54:ff:feab:ec15%6
Jun 26 04:26:27 donbot avahi-daemon[2477]: Registering new address record for fe80::fc54:ff:feab:ec15 on vnet0.*.
Jun 26 04:26:27 donbot kernel: virbr0: port 2(vnet0) entered learning state
Jun 26 04:26:28 donbot sudo[2877]: pam_unix(sudo:session): session closed for user root
Jun 26 04:26:29 donbot ntpd[1053]: Listen normally on 11 vnet0 [fe80::fc54:ff:feab:ec15%6]:123
Jun 26 04:26:29 donbot ntpd[1053]: new interface(s) found: waking up resolver
Jun 26 04:26:29 donbot NetworkManager[608]: <info>  (virbr0): link connected
Jun 26 04:26:29 donbot kernel: virbr0: topology change detected, propagating
Jun 26 04:26:29 donbot kernel: virbr0: port 2(vnet0) entered forwarding state

Duelist

Member

Registered: 2014-09-22

Posts: 358

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

So I've been banging my head against a wall trying to get this to work for a few days with no real success.  I'm able to get a VM up and running, but when I try to turn vga off and pass my graphics card to the VM I don't get any signal at all.  If anyone could take a quick glance at my setup and point me in the right direction, I'd be grateful.

CPU: Intel i5 4690k
GPU: AMD r9 290
Mobo: Asrock Z97 Extreme 4
Memory: 16GB 1600Mhz
Monitor1: Asus 144hz, Displayport ------> IGPU, DVI-D ----------> Discrete GPU
Monitor 2:  Random 60hz, HDMI -----> IGPU

Trying to use my IGP as a host display adapter, and pass through my r9 290 to windows 8.1 VM.

Well.. That vfio-bind script is not needed when libvirt is used.

Here lies a fresh and working guide on how to set it up. The information on that blog is 100% accurate, as it's written by the author of vfio module.
There may be some details with i915(for IGPU) driver and 9-series chipsets, but i'm not familiar with them. It'd be nice if you provide your distro and kernel versions.

Last edited by Duelist (2015-06-26 10:20:55)

---

The forum rules prohibit requesting support for distributions other than arch.
I gave up. It was too late.
What I was trying to do.
The reference about VFIO and KVM VGA passthrough.

Peetipablo

Member

Registered: 2015-06-26

Posts: 16

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

I'm running latest updated  arch, and have been trying both the linked linux-vfio kernel on page 1 of this thread and linux-mainline.  I actually was referencing that blog last night. I was under the impression that the i915 fix wasn't necessary if booting with UEFI.

aw

Member

Registered: 2013-10-04

Posts: 921

Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

It's a little bit unclear to me what you're doing with pci-stub versus that vfio bind script.  As I document in my howto series, there are ways to use scripts like that in your initramfs to avoid pci-stub, but not generally in combination with pci-stub.  You are apparently getting the device bound to vfio-pci in the end though.  Your paste of the libvirt log is truncating the qemu commandline, so there's really nothing to see there.

The i915 patch is only required for VGA-mode, it should not be necessary for OVMF.  Did you check whether the ROM on your assigned GPU supports UEFI?

Since the GPU is connected at the processor root port, the 9-series PCH root port quirks are also not relevant.

---

http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Peetipablo

Member

Registered: 2015-06-26

Posts: 16

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Hi Alex,

Thanks for the reply.  I was reading your how-to series last night and was (attempting) try different things, the vfio-bind script is from your writeup.

Apologies, but I did not check to ensure my GPU has a UEFI-supporting ROM.  To be honest, I just assumed since it was a newer card it would support UEFI.  I'm not really sure how to check.

Edit:

it looks like reference 290 cards have only legacy bios roms.  I'm attempting to find and flash a UEFI rom so we'll see if that makes a difference.  Thanks again for the help, much appreciated.

Edit2:

I flashed a UEFI rom on my card and changed a few things around.  I removed pci-stub and the vfio-pci script to bind, and followed the guide to load vfio-pci directly in the kernel.  Still using linux-mainline 4.1 kernel.

/etc/default/grub:

GRUB_CMDLINE_LINUX_DEFAULT="intel_iommu=on iommu=pt"
GRUB_PRELOAD_MODULES="vfio-pci part_gpt part_msdos"

/etc/mkinitcpio.conf:

MODULES="nls_cp437 xhci_hcd vfio vfio_iommu_type1 vfio_pci vfio_virqfd"
FILES="/etc/modprobe.d/radeon_blacklist.conf /etc/modprobe.d/vfio-options.conf"

/etc/modprobe.d/radeon_blacklist.conf:

blacklist radeon

/etc/modprobe.d/vfio-options.conf:

options vfio-pci ids=1002:ffffffff:ffffffff:ffffffff:00030000:ffff00ff,1002:ffffffff:ffffffff:ffffffff:00040300:ffffffff,10de:ffffffff:ffffffff:ffffffff:00030000:ffff00ff,10de:ffffffff:ffffffff:ffffffff:00040300:ffffffff

lsmod:

[peetipablo@donbot ~]$ lsmod|less
[peetipablo@donbot ~]$ lsmod|grep vfio
vfio_pci               36864  0 
vfio_virqfd            16384  1 vfio_pci
vfio_iommu_type1       20480  0 
vfio                   24576  2 vfio_iommu_type1,vfio_pci

vfio_pci bound:

[peetipablo@donbot ~]$ sudo tree /sys/bus/pci/devices/0000\:01\:00.0/driver
[sudo] password for peetipablo: 
/sys/bus/pci/devices/0000:01:00.0/driver
├── 0000:01:00.0 -> ../../../../devices/pci0000:00/0000:00:01.0/0000:01:00.0
├── 0000:01:00.1 -> ../../../../devices/pci0000:00/0000:00:01.0/0000:01:00.1
├── bind
├── module -> ../../../../module/vfio_pci
├── new_id
├── remove_id
├── uevent
└── unbind

Before I flashed UEFI bios I just wouldn't get any output on my DVI(dedicated GPU) with 01:00.0 and 01:00.1 bound to the VM, now I don't get any output AND my system freezes and I have to hard reboot   It seems I'm making progress in the wrong direction?

Anyway, if anyone has any idea or could point me in the right direction it'd be extremely appreciated.  I'm extremely new to Arch, I jumped in headfirst for the first time and built my system last weekend and I've learned so much in the short week I've been using it.  I feel like I've spent dozens of hours trying to get this to work and I'm not sure what else to do.

Last edited by Peetipablo (2015-06-26 18:51:27)

aw

Member

Registered: 2013-10-04

Posts: 921

Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

@Peetipablo

I'm not sure what you were doing with pci-stub and vfio-pci previously, but it seemed like you ended up with the device attached to vfio-pci.  If that's true, then the biggest change is the GPU ROM, so that's what I would be suspicious of first.  Can you flash it back to the original ROM and get the original problem?  Did the ROM come from the GPU card vendor?

---

http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Peetipablo

Member

Registered: 2015-06-26

Posts: 16

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

AW,

I flashed the backup I made of the rom, and am not freezing anymore.  Still no output from discreet GPU video.  The ROM was from MSI (gpu vendor).

aw

Member

Registered: 2013-10-04

Posts: 921

Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Guess you're stuck with VGA-mode then

---

http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Peetipablo

Member

Registered: 2015-06-26

Posts: 16

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Guess you're right I just tried VGA mode and I was able to pass the screen to DVI-D.

Thanks a bunch for your help, I'm sure you get a million people asking for help w/ this setup every day.

Denso

Member

Registered: 2014-08-30

Posts: 179

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Any update on the modprobe.d vfio-pci ids not loaded on boot ?

I'm starting to suspect that compiling the kernel with GCC 5.1 is causing that , just like compiling the qemu package using GCC 5.1 caused all the bridging issues in the past week for me .

Let's hope that Arch would push 4.1 to the Core repo. soon

walkindude

Member

Registered: 2012-03-04

Posts: 29

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

windows 10 build 10130
all running well on sabertooth 990fx r2.0, fx 8320, 8 gb ram, r9 270
except from time to time I got freezes (vm and host also).. but sometimes, if I'm listening for the music on host sound continues or loops (like 50/50)
in /var/log/messages I got this

Jun 27 03:35:50 localhost kernel: [ 4038.876277] general protection fault: 0000 [#1] SMP 
Jun 27 03:35:50 localhost kernel: [ 4038.877219] Modules linked in: hidp vhost_net vhost macvtap macvlan ecb tun act_police cls_basic cls_flow cls_fw cls_u32 sch_tbf sch_prio sch_htb sch_hfsc sch_ingress sch_sfq xt_CHECKSUM ipt_rpfilter xt_statistic xt_CT xt_connlimit xt_realm xt_addrtype xt_comment xt_recent xt_nat ipt_REJECT nf_reject_ipv4 ipt_MASQUERADE nf_nat_masquerade_ipv4 ipt_ECN ipt_CLUSTERIP ipt_ah xt_set ip_set xt_NFLOG nfnetlink_log xt_LOG nf_log_ipv4 nf_log_common nf_nat_tftp nf_nat_snmp_basic nf_conntrack_snmp nf_nat_sip nf_nat_pptp nf_nat_proto_gre nf_nat_irc nf_nat_h323 nf_nat_ftp nf_nat_amanda ts_kmp nf_conntrack_amanda nf_conntrack_sane nf_conntrack_tftp nf_conntrack_sip nf_conntrack_proto_udplite nf_conntrack_proto_sctp nf_conntrack_pptp nf_conntrack_proto_gre nf_conntrack_netlink nf_conntrack_netbios_ns nf_conntrack_broadcast nf_conntrack_irc nf_conntrack_h323 nf_conntrack_ftp xt_TPROXY nf_defrag_ipv6 xt_time xt_TCPMSS xt_tcpmss xt_sctp xt_pkttype xt_physdev br_netfilter bridge stp llc xt_owner xt_NFQUEUE xt_multiport xt_mark xt_mac xt_limit xt_length xt_iprange xt_helper xt_hashlimit xt_DSCP xt_dscp xt_dccp xt_conntrack xt_connmark xt_CLASSIFY xt_tcpudp xt_state iptable_raw iptable_nat nf_nat_ipv4 nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_conntrack iptable_mangle nfnetlink rfcomm fuse ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter arc4 mousedev joydev btusb bluetooth hid_generic psmouse serio_raw atkbd libps2 ath9k_htc evdev led_class ath9k_common ath9k_hw ath kvm_amd mac80211 amdkfd amd_iommu_v2 kvm radeon crct10dif_pclmul crc32_pclmul crc32c_intel cfg80211 ghash_clmulni_intel snd_hda_codec_realtek aesni_intel aes_x86_64 glue_helper snd_hda_codec_generic lrw gf128mul ablk_helper cryptd rfkill snd_hda_codec_hdmi i2c_algo_bit pcspkr fam15h_power snd_hda_intel drm_kms_helper k10temp snd_hda_controller usbhid ttm r8169 hid snd_hda_codec drm snd_hwdep snd_pcm sp5100_tco mii snd_timer i2c_piix4 snd i2c_core soundcore shpchp tpm_infineon tpm_tis i8042 serio tpm acpi_cpufreq processor button sch_fq_codel nls_iso8859_1 nls_cp437 vfat fat usblp it87 hwmon_vid hwmon nfsd nfs auth_rpcgss oid_registry nfs_acl lockd grace sunrpc fscache ip_tables x_tables ext4 crc16 mbcache jbd2 dm_mod sd_mod ohci_pci ahci xhci_pci ohci_hcd ehci_pci libahci xhci_hcd ehci_hcd libata usbcore usb_common scsi_mod
Jun 27 03:35:50 localhost kernel: [ 4038.892624] CPU: 3 PID: 3715 Comm: qemu-system-x86 Tainted: G       A        4.0.6-1-uksm-ck #1
Jun 27 03:35:50 localhost kernel: [ 4038.894141] Hardware name: To be filled by O.E.M. To be filled by O.E.M./SABERTOOTH 990FX R2.0, BIOS 2501 04/08/2014
Jun 27 03:35:50 localhost kernel: [ 4038.895671] task: ffff8801eee5ee40 ti: ffff8801e736c000 task.ti: ffff8801e736c000
Jun 27 03:35:50 localhost kernel: [ 4038.897240] RIP: 0010:[<ffffffff8107138d>]  [<ffffffff8107138d>] finish_wait+0x48/0x64
Jun 27 03:35:50 localhost kernel: [ 4038.898807] RSP: 0018:ffff8801e736fcb8  EFLAGS: 00010046
Jun 27 03:35:50 localhost kernel: [ 4038.900365] RAX: 0000000000000292 RBX: ffff8801e736fce8 RCX: 08ca6ab8cb26e9a3
Jun 27 03:35:50 localhost kernel: [ 4038.901924] RDX: a0369728737bf719 RSI: 0000000000000292 RDI: ffff8801e7370080
Jun 27 03:35:50 localhost kernel: [ 4038.903521] RBP: ffff8801e736fcd8 R08: 0000000000000292 R09: ffff88044ecca060
Jun 27 03:35:50 localhost kernel: [ 4038.905098] R10: 0000000000000000 R11: 000003ac41530809 R12: ffff8801e7370080
Jun 27 03:35:50 localhost kernel: [ 4038.906690] R13: ffff8801e736fd00 R14: 000003ac13433b48 R15: 0000000000000000
Jun 27 03:35:50 localhost kernel: [ 4038.908312] FS:  00007f91014f2700(0000) GS:ffff88044ecc0000(0000) knlGS:00000000ec2b6b40
Jun 27 03:35:50 localhost kernel: [ 4038.909911] CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b
Jun 27 03:35:50 localhost kernel: [ 4038.911512] CR2: 0000000000000000 CR3: 00000002370b7000 CR4: 00000000000407e0
Jun 27 03:35:50 localhost kernel: [ 4038.913149] Stack:
Jun 27 03:35:50 localhost kernel: [ 4038.914759]  ffff8801eee5ee40 ffff8801e7370000 0000000000000001 ffff8801e7370080
Jun 27 03:35:50 localhost kernel: [ 4038.916406]  ffff8801e736fd38 ffffffffa0653e15 0000000000000000 ffff8801eee5ee40
Jun 27 03:35:50 localhost kernel: [ 4038.918069]  ffffffff8107174e ffff8801e736fd00 ffff8801e736fd00 00000000f45e99a5
Jun 27 03:35:50 localhost kernel: [ 4038.919722] Call Trace:
Jun 27 03:35:50 localhost kernel: [ 4038.921373]  [<ffffffffa0653e15>] kvm_vcpu_block+0xea/0x13f [kvm]
Jun 27 03:35:50 localhost kernel: [ 4038.923064]  [<ffffffff8107174e>] ? wait_woken+0x76/0x76
Jun 27 03:35:50 localhost kernel: [ 4038.924731]  [<ffffffffa0755ff3>] ? svm_scale_tsc+0x49/0x49 [kvm_amd]
Jun 27 03:35:50 localhost kernel: [ 4038.926417]  [<ffffffffa0667f24>] kvm_arch_vcpu_ioctl_run+0x730/0xfa6 [kvm]
Jun 27 03:35:50 localhost kernel: [ 4038.928127]  [<ffffffffa0662abe>] ? kvm_arch_vcpu_load+0x13e/0x148 [kvm]
Jun 27 03:35:50 localhost kernel: [ 4038.929818]  [<ffffffffa0654c3f>] kvm_vcpu_ioctl+0x185/0x51a [kvm]
Jun 27 03:35:50 localhost kernel: [ 4038.931517]  [<ffffffff810985c3>] ? do_futex+0xe2/0x843
Jun 27 03:35:50 localhost kernel: [ 4038.933188]  [<ffffffff810b3632>] ? acct_update_integrals+0x1c/0x1e
Jun 27 03:35:50 localhost kernel: [ 4038.934878]  [<ffffffff81211c96>] ? __next_cpu+0x1a/0x29
Jun 27 03:35:50 localhost kernel: [ 4038.936584]  [<ffffffff8106c670>] ? check_smt_siblings+0x3f/0xa4
Jun 27 03:35:50 localhost kernel: [ 4038.938283]  [<ffffffff81159f9e>] do_vfs_ioctl+0x3a0/0x42a
Jun 27 03:35:50 localhost kernel: [ 4038.940008]  [<ffffffff8115a082>] SyS_ioctl+0x5a/0x7f
Jun 27 03:35:50 localhost kernel: [ 4038.941700]  [<ffffffff8141e649>] system_call_fastpath+0x12/0x17
Jun 27 03:35:50 localhost kernel: [ 4038.943388] Code: 00 00 4c 8d 6e 18 4c 3b 6e 18 75 06 4c 3b 6e 20 74 2d 48 89 f3 49 89 fc e8 4c cb 3a 00 48 8b 4b 18 48 89 c6 48 8b 53 20 4c 89 e7 <48> 89 51 08 48 89 0a 4c 89 6b 18 4c 89 6b 20 e8 ab cc 3a 00 58 
Jun 27 03:35:50 localhost kernel: [ 4038.948810]  RSP <ffff8801e736fcb8>
Jun 27 03:35:50 localhost kernel: [ 4038.958710] ---[ end trace 74a2ffa0673c39e8 ]---

tried with bfs and cfs, voluntary preemtion, 1000 hz and 300 hz, problem remains

Duelist

Member

Registered: 2014-09-22

Posts: 358

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

01:00.0 VGA compatible controller: Advanced Micro Devices, Inc. [AMD/ATI] Cape Verde PRO [Radeon HD 7750/8740 / R7 250E]
01:00.1 Audio device: Advanced Micro Devices, Inc. [AMD/ATI] Cape Verde/Pitcairn HDMI Audio [Radeon HD 7700/7800 Series]
02:00.0 VGA compatible controller: Advanced Micro Devices, Inc. [AMD/ATI] Cape Verde PRO [Radeon HD 7750/8740 / R7 250E]
02:00.1 Audio device: Advanced Micro Devices, Inc. [AMD/ATI] Cape Verde/Pitcairn HDMI Audio [Radeon HD 7700/7800 Series]
03:05.0 Multimedia audio controller: Ensoniq 5880B [AudioPCI] (rev 02)
04:00.0 VGA compatible controller: NVIDIA Corporation GF119 [GeForce GT 610] (rev a1)
04:00.1 Audio device: NVIDIA Corporation GF119 HDMI Audio Controller (rev a1)
05:00.0 USB controller: ASMedia Technology Inc. ASM1042 SuperSpeed USB Host Controller
06:00.0 Ethernet controller: Realtek Semiconductor Co., Ltd. RTL8111/8168/8411 PCI Express Gigabit Ethernet Controller (rev 09)
07:00.0 SATA controller: Silicon Image, Inc. SiI 3132 Serial ATA Raid II Controller (rev 01)

You might say something like "that guy is outright insane", but..
The third card is coming somewhere on monday. Yeah.

(sorry for the big image)

So the riser works. The motherboard is F2A55 by asus, so there's only 2 PCI slots left free. Push it to the limit.
So far crossfire on two cards works, built-in audio is passed-through to the VM, hardware mixering still WIP, but everything works despite common sense.

After the third card the next step to insanity would be buying something with pci-e 3.0 and an APU to swap that GT610 with a fourth card or maybe try passing through the APU's graphics to achieve so called Dual Graphics, a special type of XDMA crossfire.

---

The forum rules prohibit requesting support for distributions other than arch.
I gave up. It was too late.
What I was trying to do.
The reference about VFIO and KVM VGA passthrough.

Denso

Member

Registered: 2014-08-30

Posts: 179

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

@Duelist :

I wish there were high-end single-slot GPUs . I want to virtualize all of the house's PCs , but I have no PCI-E slots remaining , because both of my GPUs are dual-slot .

Oh and I wish there were PCI-E lanes splitters or something . That would allow you to split 16x into 2 8x . and 8x into 2 4x .

Is that a Fractal Design Core 1000 ? That case used to huse my gaming rig before I virtualize it .

Last edited by Denso (2015-06-27 16:30:11)

mostlyharmless

Member

Registered: 2014-01-16

Posts: 72

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

@Duelist

FYI Tried 1-3-2 and everything still works, the same. I've got host-passthrough, but Windows still thinks I have a Xeon.  I assume that's a Windows 7/OVMF/UEFI thing.  Not that it matters.

Last edited by mostlyharmless (2015-06-27 16:52:17)

aw

Member

Registered: 2013-10-04

Posts: 921

Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

@Denso

Quadro K4200 has pretty impressive specs for a single slot card, but at a huge premium.  On the plus side, they actually support VM usage models on it.

---

http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Denso

Member

Registered: 2014-08-30

Posts: 179

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

@aw :

Aren't Quadros optimized for rendering and workstation workloads rather than gaming ?

Hmmm , $600 for a used one seems reasonable enough ! Now let's see how it handles gaming ... Off to find some reviews

Oh , it seems that the newer Quadros are dual-slot only . (5000 & 6000) .

Last edited by Denso (2015-06-27 19:07:12)

aw

Member

Registered: 2013-10-04

Posts: 921

Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

@aw :

Aren't Quadros optimized for rendering and workstation workloads rather than gaming ?

Hmmm , $600 for a used one seems reasonable enough ! Now let's see how it handles gaming ... Off to find some reviews

Oh , it seems that the newer Quadros are dual-slot only . (5000 & 6000) .

@Denso

The 4th digit of the Quadro numbers are equivalent to the 2nd digit of Geforce numbers, bigger means more cores.  The 3rd digit represents the generation.  So a K4200 is newer than a K5000, but all of the Quadros in the Kx200 generation got a big bump in core count, so a K4200 may be more equivalent in performance to a K5000 than to a K4000.  IOW, K5000 and K6000 aren't newer than K4200, they're the higher core count versions of the previous generation.

For gaming, I don't know how the Geforce drivers compare to the Quadro drivers.  I wouldn't expect to see much difference.  Quadros typically aren't clocked as high as Geforce, so aim your sights a little lower for equivalent generation and core count.  You might also lose Geforce Experience, automatic game tuning and the other features it provides.

EDIT: If you're expecting to find a single slot Titan, it's not going to happen.  Even Quadro goes to dual-slot after the K4xxx size, but they pack more into a single slot than anything else.

Last edited by aw (2015-06-27 19:41:50)

---

http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users