Arch Linux

morat

Member

Registered: 2015-07-07

Posts: 38

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

2. Is there a better alternative than Synergy for gaming? Hitting the edges is a problem, and it seems to handle direct pointer captures very badly, so besides Steam Inhouse Streaming, is there a viable way to play D3D games?

Thanks for all your help getting me this far, and if anyone with similar issues to mine needs any help, I'd be more than happy.

If what you're referring to by "seems to handle direct pointer captures very badly" is what I think, ensure you have 'relativeMouseMoves = true' in your server config, bind a button to lockCursorToScreen(toggle), and then lock the cursor to the vm screen when you're playing, should solve that issue and also your edge problem.

Worked like a charm, thanks!

Just when I thought I was out, they pull me back in.

The most luck I've had with audio so far is Windows 8 recognising a "High Definition Audio Controller", which believes it's outputting sound, but I hear nothing. I have used vnc_allow_host_audio = 1, and tried setting the qemu user to my own, but that only causes permission issues with the pci. libvirt is set to use ich9.

Edit: I should also point out that I'm not using, nor have I been able to test, the nVidia HDMI sound, as the financial situ at the moment prohibits good monitors.

Last edited by morat (2015-07-19 08:41:17)

Schlunze

Member

Registered: 2013-10-03

Posts: 53

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

The most luck I've had with audio so far is Windows 8 recognising a "High Definition Audio Controller", which believes it's outputting sound, but I hear nothing. I have used vnc_allow_host_audio = 1, and tried setting the qemu user to my own, but that only causes permission issues with the pci. libvirt is set to use ich9.

Maybe you use libvirt / virtmanager and the guest window is closed then you get no sound output.

Col_Panic

Member

Registered: 2015-07-19

Posts: 4

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Hey guys! Long time reader here, first time poster. I hope someone can help me out!
I've had passthrough working for a while now, on my old GPU; I'm running kernel 4.1.2, and binding the GPU to vfio on boot.
I'm also loading the modules:
amd_iommu
vfio
vfio_iommu_type1
vfio_pci
vfio_virqfd
I don't have any other special configuration on either my host or guest.

After switching out my primary GPU for the old KVM one, and adding a brand shiny new secondary GPU, my setup is now:

CPU: AMD FX8350
MOBO: Asus M5A97 LE R2.0 (with latest BIOS)
GPU 1: Radeon HD 7770
GPU 2: Radeon R9 270x

Passthrough still works for the 7770 if I use that one as secondary instead, but trying to pass through my new GPU doesn't seem to work. My Windows guest starts, then my host locks up as the guest tries to load the GPU.
This is what it dumps out:

[  645.924773] qemu-system-x86: sending ioctl 5326 to a partition!
[  645.924784] qemu-system-x86: sending ioctl 80200204 to a partition!
[  648.526720] vfio-pci 0000:00:13.2: enabling device (0000 -> 0002)
[  648.558718] vfio_cap_init: 0000:00:13.2 hiding cap 0xa
[  648.674350] AMD-Vi: Completion-Wait loop timed out
[  648.787568] AMD-Vi: Completion-Wait loop timed out
[  648.900662] AMD-Vi: Completion-Wait loop timed out
[  648.936352] AMD-Vi: Completion-Wait loop timed out
[  649.012717] AMD-Vi: Completion-Wait loop timed out
[  649.123771] AMD-Vi: Completion-Wait loop timed out
[  649.234736] AMD-Vi: Completion-Wait loop timed out
[  649.349347] AMD-Vi: Completion-Wait loop timed out
[  649.464152] AMD-Vi: Completion-Wait loop timed out
[  649.478740] vfio-pci 0000:05:00.0: enabling device (0000 -> 0003)
[  649.478995] vfio_ecap_init: 0000:05:00.0 hiding ecap 0x19@0x270
[  649.479008] vfio_ecap_init: 0000:05:00.0 hiding ecap 0x1b@0x2d0
[  649.498706] vfio-pci 0000:05:00.1: enabling device (0000 -> 0002)
[  649.678811] AMD-Vi: Event logged [IOTLB_INV_TIMEOUT device=05:00.0 address=0x0000000536f2f810]
[  649.678823] AMD-Vi: Event logged [IOTLB_INV_TIMEOUT device=05:00.0 address=0x0000000536f2f830]
[  649.678829] AMD-Vi: Event logged [IOTLB_INV_TIMEOUT device=05:00.0 address=0x0000000536f2f850]
[  650.797361] AMD-Vi: Event logged [IOTLB_INV_TIMEOUT device=05:00.0 address=0x0000000536f2f8a0]
[  650.808317] AMD-Vi: Completion-Wait loop timed out
[  650.808320] ------------[ cut here ]------------
[  650.808326] WARNING: CPU: 6 PID: 3851 at /home/kernel/COD/linux/drivers/iommu/amd_iommu.c:1199 __domain_flush_pages+0x162/0x170()
[  650.808328] Modules linked in: vhost_net vhost macvtap macvlan xt_CHECKSUM iptable_mangle ipt_REJECT nf_reject_ipv4 xt_tcpudp ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter ip_tables x_tables cmac rfcomm bnep cfg80211 bridge stp llc nls_iso8859_1 btusb btbcm btintel eeepc_wmi bluetooth asus_wmi sparse_keymap video crct10dif_pclmul crc32_pclmul ghash_clmulni_intel snd_usb_audio aesni_intel joydev aes_x86_64 snd_usbmidi_lib snd_hda_codec_realtek lrw snd_hda_codec_generic snd_hda_intel gf128mul glue_helper snd_hda_controller ablk_helper edac_core snd_hda_codec cryptd snd_hda_core snd_hwdep serio_raw edac_mce_amd 8250_fintek tpm_infineon fam15h_power i2c_piix4 k10temp shpchp mac_hid snd_aloop snd_pcm snd_seq_midi snd_seq_midi_event snd_rawmidi snd_seq snd_seq_device snd_timer snd soundcore
[  650.808364]  kvm_amd kvm vfio_iommu_type1 parport_pc ppdev lp parport autofs4 hid_generic usbhid hid amdkfd amd_iommu_v2 radeon i2c_algo_bit ttm drm_kms_helper drm mxm_wmi psmouse vfio_pci ahci r8169 vfio_virqfd libahci mii vfio wmi
[  650.808379] CPU: 6 PID: 3851 Comm: qemu-system-x86 Not tainted 4.1.2-040102-generic #201507101335
[  650.808380] Hardware name: To be filled by O.E.M. To be filled by O.E.M./M5A97 LE R2.0, BIOS 2601 03/24/2015
[  650.808382]  ffffffff81b124e0 ffff88048e27bb88 ffffffff817d076c 0000000000000000
[  650.808385]  0000000000000000 ffff88048e27bbc8 ffffffff81079b0a 00000000fffffffb
[  650.808387]  ffff880533c53610 00000000fffffffb 0000000000000000 7fffffffffffffff
[  650.808389] Call Trace:
[  650.808394]  [<ffffffff817d076c>] dump_stack+0x45/0x57
[  650.808398]  [<ffffffff81079b0a>] warn_slowpath_common+0x8a/0xc0
[  650.808400]  [<ffffffff81079bfa>] warn_slowpath_null+0x1a/0x20
[  650.808401]  [<ffffffff814f4662>] __domain_flush_pages+0x162/0x170
[  650.808404]  [<ffffffff81013673>] ? __switch_to+0x1e3/0x580
[  650.808406]  [<ffffffff814f5259>] amd_iommu_unmap+0x119/0x140
[  650.808409]  [<ffffffff814eec79>] iommu_unmap+0xc9/0x1d0
[  650.808411]  [<ffffffff814f242f>] ? amd_iommu_iova_to_phys+0x2f/0x80
[  650.808417]  [<ffffffffc01575b7>] vfio_remove_dma+0x107/0x1f0 [vfio_iommu_type1]
[  650.808421]  [<ffffffffc0157d35>] vfio_iommu_type1_ioctl+0x3e5/0xa40 [vfio_iommu_type1]
[  650.808437]  [<ffffffffc01c674a>] ? kvm_set_memory_region+0x3a/0x50 [kvm]
[  650.808446]  [<ffffffffc01c6cb5>] ? kvm_vm_ioctl+0x555/0x790 [kvm]
[  650.808452]  [<ffffffffc000a6f9>] vfio_fops_unl_ioctl+0x79/0x2b0 [vfio]
[  650.808455]  [<ffffffff8120c4f8>] do_vfs_ioctl+0x2f8/0x510
[  650.808459]  [<ffffffff81066f16>] ? __do_page_fault+0x1b6/0x450
[  650.808461]  [<ffffffff8120c791>] SyS_ioctl+0x81/0xa0
[  650.808463]  [<ffffffff817d8272>] system_call_fastpath+0x16/0x75
[  650.808465] ---[ end trace 29b4fb9fbe5211cb ]---
[  650.926817] AMD-Vi: Completion-Wait loop timed out
[  650.926826] ------------[ cut here ]------------
[  650.926841] WARNING: CPU: 5 PID: 3851 at /home/kernel/COD/linux/drivers/iommu/amd_iommu.c:1199 __domain_flush_pages+0x162/0x170()
[  650.926844] Modules linked in: vhost_net vhost macvtap macvlan xt_CHECKSUM iptable_mangle ipt_REJECT nf_reject_ipv4 xt_tcpudp ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter ip_tables x_tables cmac rfcomm bnep cfg80211 bridge stp llc nls_iso8859_1 btusb btbcm btintel eeepc_wmi bluetooth asus_wmi sparse_keymap video crct10dif_pclmul crc32_pclmul ghash_clmulni_intel snd_usb_audio aesni_intel joydev aes_x86_64 snd_usbmidi_lib snd_hda_codec_realtek lrw snd_hda_codec_generic snd_hda_intel gf128mul glue_helper snd_hda_controller ablk_helper edac_core snd_hda_codec cryptd snd_hda_core snd_hwdep serio_raw edac_mce_amd 8250_fintek tpm_infineon fam15h_power i2c_piix4 k10temp shpchp mac_hid snd_aloop snd_pcm snd_seq_midi snd_seq_midi_event snd_rawmidi snd_seq snd_seq_device snd_timer snd soundcore
[  650.926923]  kvm_amd kvm vfio_iommu_type1 parport_pc ppdev lp parport autofs4 hid_generic usbhid hid amdkfd amd_iommu_v2 radeon i2c_algo_bit ttm drm_kms_helper drm mxm_wmi psmouse vfio_pci ahci r8169 vfio_virqfd libahci mii vfio wmi
[  650.926957] CPU: 5 PID: 3851 Comm: qemu-system-x86 Tainted: G        W       4.1.2-040102-generic #201507101335
[  650.926961] Hardware name: To be filled by O.E.M. To be filled by O.E.M./M5A97 LE R2.0, BIOS 2601 03/24/2015
[  650.926965]  ffffffff81b124e0 ffff88048e27bb88 ffffffff817d076c 0000000000000000
[  650.926971]  0000000000000000 ffff88048e27bbc8 ffffffff81079b0a 00000000fffffffb
[  650.926976]  ffff880533c53610 00000000fffffffb 0000000000000000 7fffffffffffffff
[  650.926982] Call Trace:
[  650.926991]  [<ffffffff817d076c>] dump_stack+0x45/0x57
[  650.926998]  [<ffffffff81079b0a>] warn_slowpath_common+0x8a/0xc0
[  650.927003]  [<ffffffff81079bfa>] warn_slowpath_null+0x1a/0x20
[  650.927007]  [<ffffffff814f4662>] __domain_flush_pages+0x162/0x170
[  650.927014]  [<ffffffff81013673>] ? __switch_to+0x1e3/0x580
[  650.927019]  [<ffffffff814f5259>] amd_iommu_unmap+0x119/0x140
[  650.927025]  [<ffffffff814eec79>] iommu_unmap+0xc9/0x1d0
[  650.927029]  [<ffffffff814f242f>] ? amd_iommu_iova_to_phys+0x2f/0x80
[  650.927041]  [<ffffffffc01575b7>] vfio_remove_dma+0x107/0x1f0 [vfio_iommu_type1]
[  650.927053]  [<ffffffffc0157d35>] vfio_iommu_type1_ioctl+0x3e5/0xa40 [vfio_iommu_type1]
[  650.927083]  [<ffffffffc01c674a>] ? kvm_set_memory_region+0x3a/0x50 [kvm]
[  650.927105]  [<ffffffffc01c6cb5>] ? kvm_vm_ioctl+0x555/0x790 [kvm]
[  650.927119]  [<ffffffffc000a6f9>] vfio_fops_unl_ioctl+0x79/0x2b0 [vfio]
[  650.927127]  [<ffffffff8120c4f8>] do_vfs_ioctl+0x2f8/0x510
[  650.927133]  [<ffffffff81066f16>] ? __do_page_fault+0x1b6/0x450
[  650.927138]  [<ffffffff8120c791>] SyS_ioctl+0x81/0xa0
[  650.927144]  [<ffffffff817d8272>] system_call_fastpath+0x16/0x75
[  650.927149] ---[ end trace 29b4fb9fbe5211cc ]---
[  651.058621] AMD-Vi: Completion-Wait loop timed out
[  651.058626] ------------[ cut here ]------------
[  651.058632] WARNING: CPU: 4 PID: 3851 at /home/kernel/COD/linux/drivers/iommu/amd_iommu.c:1199 __domain_flush_pages+0x162/0x170()
[  651.058633] Modules linked in: vhost_net vhost macvtap macvlan xt_CHECKSUM iptable_mangle ipt_REJECT nf_reject_ipv4 xt_tcpudp ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter ip_tables x_tables cmac rfcomm bnep cfg80211 bridge stp llc nls_iso8859_1 btusb btbcm btintel eeepc_wmi bluetooth asus_wmi sparse_keymap video crct10dif_pclmul crc32_pclmul ghash_clmulni_intel snd_usb_audio aesni_intel joydev aes_x86_64 snd_usbmidi_lib snd_hda_codec_realtek lrw snd_hda_codec_generic snd_hda_intel gf128mul glue_helper snd_hda_controller ablk_helper edac_core snd_hda_codec cryptd snd_hda_core snd_hwdep serio_raw edac_mce_amd 8250_fintek tpm_infineon fam15h_power i2c_piix4 k10temp shpchp mac_hid snd_aloop snd_pcm snd_seq_midi snd_seq_midi_event snd_rawmidi snd_seq snd_seq_device snd_timer snd soundcore
[  651.058669]  kvm_amd kvm vfio_iommu_type1 parport_pc ppdev lp parport autofs4 hid_generic usbhid hid amdkfd amd_iommu_v2 radeon i2c_algo_bit ttm drm_kms_helper drm mxm_wmi psmouse vfio_pci ahci r8169 vfio_virqfd libahci mii vfio wmi
[  651.058684] CPU: 4 PID: 3851 Comm: qemu-system-x86 Tainted: G        W       4.1.2-040102-generic #201507101335
[  651.058686] Hardware name: To be filled by O.E.M. To be filled by O.E.M./M5A97 LE R2.0, BIOS 2601 03/24/2015
[  651.058687]  ffffffff81b124e0 ffff88048e27bb88 ffffffff817d076c 0000000000000000
[  651.058689]  0000000000000000 ffff88048e27bbc8 ffffffff81079b0a 00000000fffffffb
[  651.058691]  ffff880533c53610 00000000fffffffb 0000000000000000 7fffffffffffffff
[  651.058694] Call Trace:
[  651.058699]  [<ffffffff817d076c>] dump_stack+0x45/0x57
[  651.058702]  [<ffffffff81079b0a>] warn_slowpath_common+0x8a/0xc0
[  651.058704]  [<ffffffff81079bfa>] warn_slowpath_null+0x1a/0x20
[  651.058705]  [<ffffffff814f4662>] __domain_flush_pages+0x162/0x170
[  651.058707]  [<ffffffff814f5259>] amd_iommu_unmap+0x119/0x140
[  651.058710]  [<ffffffff8118fb1e>] ? compound_unlock_irqrestore+0xe/0x20
[  651.058713]  [<ffffffff814eec79>] iommu_unmap+0xc9/0x1d0
[  651.058715]  [<ffffffff814f242f>] ? amd_iommu_iova_to_phys+0x2f/0x80
[  651.058720]  [<ffffffffc01575b7>] vfio_remove_dma+0x107/0x1f0 [vfio_iommu_type1]
[  651.058725]  [<ffffffffc0157d35>] vfio_iommu_type1_ioctl+0x3e5/0xa40 [vfio_iommu_type1]
[  651.058742]  [<ffffffffc01c674a>] ? kvm_set_memory_region+0x3a/0x50 [kvm]
[  651.058750]  [<ffffffffc01c6cb5>] ? kvm_vm_ioctl+0x555/0x790 [kvm]
[  651.058756]  [<ffffffffc000a6f9>] vfio_fops_unl_ioctl+0x79/0x2b0 [vfio]
[  651.058784]  [<ffffffff8120c4f8>] do_vfs_ioctl+0x2f8/0x510
[  651.058788]  [<ffffffff81066f16>] ? __do_page_fault+0x1b6/0x450
[  651.058800]  [<ffffffff8120c791>] SyS_ioctl+0x81/0xa0
[  651.058818]  [<ffffffff817d8272>] system_call_fastpath+0x16/0x75
[  651.058829] ---[ end trace 29b4fb9fbe5211cd ]---
[  651.183912] AMD-Vi: Completion-Wait loop timed out
[  651.183917] ------------[ cut here ]------------
[  651.183922] WARNING: CPU: 4 PID: 3851 at /home/kernel/COD/linux/drivers/iommu/amd_iommu.c:1199 __domain_flush_pages+0x162/0x170()
[  651.183924] Modules linked in: vhost_net vhost macvtap macvlan xt_CHECKSUM iptable_mangle ipt_REJECT nf_reject_ipv4 xt_tcpudp ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter ip_tables x_tables cmac rfcomm bnep cfg80211 bridge stp llc nls_iso8859_1 btusb btbcm btintel eeepc_wmi bluetooth asus_wmi sparse_keymap video crct10dif_pclmul crc32_pclmul ghash_clmulni_intel snd_usb_audio aesni_intel joydev aes_x86_64 snd_usbmidi_lib snd_hda_codec_realtek lrw snd_hda_codec_generic snd_hda_intel gf128mul glue_helper snd_hda_controller ablk_helper edac_core snd_hda_codec cryptd snd_hda_core snd_hwdep serio_raw edac_mce_amd 8250_fintek tpm_infineon fam15h_power i2c_piix4 k10temp shpchp mac_hid snd_aloop snd_pcm snd_seq_midi snd_seq_midi_event snd_rawmidi snd_seq snd_seq_device snd_timer snd soundcore
[  651.183956]  kvm_amd kvm vfio_iommu_type1 parport_pc ppdev lp parport autofs4 hid_generic usbhid hid amdkfd amd_iommu_v2 radeon i2c_algo_bit ttm drm_kms_helper drm mxm_wmi psmouse vfio_pci ahci r8169 vfio_virqfd libahci mii vfio wmi
[  651.183970] CPU: 4 PID: 3851 Comm: qemu-system-x86 Tainted: G        W       4.1.2-040102-generic #201507101335
[  651.183971] Hardware name: To be filled by O.E.M. To be filled by O.E.M./M5A97 LE R2.0, BIOS 2601 03/24/2015
[  651.183973]  ffffffff81b124e0 ffff88048e27bb88 ffffffff817d076c 0000000000000000
[  651.183975]  0000000000000000 ffff88048e27bbc8 ffffffff81079b0a 00000000fffffffb
[  651.183977]  ffff880533c53610 00000000fffffffb 0000000000000000 7fffffffffffffff
[  651.183979] Call Trace:
[  651.183984]  [<ffffffff817d076c>] dump_stack+0x45/0x57
[  651.183987]  [<ffffffff81079b0a>] warn_slowpath_common+0x8a/0xc0
[  651.183989]  [<ffffffff81079bfa>] warn_slowpath_null+0x1a/0x20
[  651.183990]  [<ffffffff814f4662>] __domain_flush_pages+0x162/0x170
[  651.183993]  [<ffffffff814f5259>] amd_iommu_unmap+0x119/0x140
[  651.183996]  [<ffffffff81190156>] ? put_compound_page+0xd6/0x1b0
[  651.183999]  [<ffffffff814eec79>] iommu_unmap+0xc9/0x1d0
[  651.184000]  [<ffffffff814f242f>] ? amd_iommu_iova_to_phys+0x2f/0x80
[  651.184006]  [<ffffffffc01575b7>] vfio_remove_dma+0x107/0x1f0 [vfio_iommu_type1]
[  651.184010]  [<ffffffffc0157d35>] vfio_iommu_type1_ioctl+0x3e5/0xa40 [vfio_iommu_type1]
[  651.184027]  [<ffffffffc01c674a>] ? kvm_set_memory_region+0x3a/0x50 [kvm]
[  651.184035]  [<ffffffffc01c6cb5>] ? kvm_vm_ioctl+0x555/0x790 [kvm]
[  651.184041]  [<ffffffffc000a6f9>] vfio_fops_unl_ioctl+0x79/0x2b0 [vfio]
[  651.184045]  [<ffffffff8120c4f8>] do_vfs_ioctl+0x2f8/0x510
[  651.184047]  [<ffffffff81066f16>] ? __do_page_fault+0x1b6/0x450
[  651.184050]  [<ffffffff8120c791>] SyS_ioctl+0x81/0xa0
[  651.184052]  [<ffffffff817d8272>] system_call_fastpath+0x16/0x75
[  651.184054] ---[ end trace 29b4fb9fbe5211ce ]---
[  651.311628] AMD-Vi: Completion-Wait loop timed out
[  651.311633] ------------[ cut here ]------------
[  651.311640] WARNING: CPU: 4 PID: 3851 at /home/kernel/COD/linux/drivers/iommu/amd_iommu.c:1199 __domain_flush_pages+0x162/0x170()
[  651.311641] Modules linked in: vhost_net vhost macvtap macvlan xt_CHECKSUM iptable_mangle ipt_REJECT nf_reject_ipv4 xt_tcpudp ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter ip_tables x_tables cmac rfcomm bnep cfg80211 bridge stp llc nls_iso8859_1 btusb btbcm btintel eeepc_wmi bluetooth asus_wmi sparse_keymap video crct10dif_pclmul crc32_pclmul ghash_clmulni_intel snd_usb_audio aesni_intel joydev aes_x86_64 snd_usbmidi_lib snd_hda_codec_realtek lrw snd_hda_codec_generic snd_hda_intel gf128mul glue_helper snd_hda_controller ablk_helper edac_core snd_hda_codec cryptd snd_hda_core snd_hwdep serio_raw edac_mce_amd 8250_fintek tpm_infineon fam15h_power i2c_piix4 k10temp shpchp mac_hid snd_aloop snd_pcm snd_seq_midi snd_seq_midi_event snd_rawmidi snd_seq snd_seq_device snd_timer snd soundcore
[  651.311682]  kvm_amd kvm vfio_iommu_type1 parport_pc ppdev lp parport autofs4 hid_generic usbhid hid amdkfd amd_iommu_v2 radeon i2c_algo_bit ttm drm_kms_helper drm mxm_wmi psmouse vfio_pci ahci r8169 vfio_virqfd libahci mii vfio wmi
[  651.311699] CPU: 4 PID: 3851 Comm: qemu-system-x86 Tainted: G        W       4.1.2-040102-generic #201507101335
[  651.311701] Hardware name: To be filled by O.E.M. To be filled by O.E.M./M5A97 LE R2.0, BIOS 2601 03/24/2015
[  651.311703]  ffffffff81b124e0 ffff88048e27bb88 ffffffff817d076c 0000000000000000
[  651.311706]  0000000000000000 ffff88048e27bbc8 ffffffff81079b0a 00000000fffffffb
[  651.311708]  ffff880533c53610 00000000fffffffb 0000000000000000 7fffffffffffffff
[  651.311711] Call Trace:
[  651.311716]  [<ffffffff817d076c>] dump_stack+0x45/0x57
[  651.311720]  [<ffffffff81079b0a>] warn_slowpath_common+0x8a/0xc0
[  651.311722]  [<ffffffff81079bfa>] warn_slowpath_null+0x1a/0x20
[  651.311724]  [<ffffffff814f4662>] __domain_flush_pages+0x162/0x170
[  651.311727]  [<ffffffff814f5259>] amd_iommu_unmap+0x119/0x140
[  651.311730]  [<ffffffff814eec79>] iommu_unmap+0xc9/0x1d0
[  651.311732]  [<ffffffff814f242f>] ? amd_iommu_iova_to_phys+0x2f/0x80
[  651.311739]  [<ffffffffc01575b7>] vfio_remove_dma+0x107/0x1f0 [vfio_iommu_type1]
[  651.311744]  [<ffffffffc0157d35>] vfio_iommu_type1_ioctl+0x3e5/0xa40 [vfio_iommu_type1]
[  651.311762]  [<ffffffffc01c674a>] ? kvm_set_memory_region+0x3a/0x50 [kvm]
[  651.311772]  [<ffffffffc01c6cb5>] ? kvm_vm_ioctl+0x555/0x790 [kvm]
[  651.311779]  [<ffffffffc000a6f9>] vfio_fops_unl_ioctl+0x79/0x2b0 [vfio]
[  651.311783]  [<ffffffff8120c4f8>] do_vfs_ioctl+0x2f8/0x510
[  651.311786]  [<ffffffff81066f16>] ? __do_page_fault+0x1b6/0x450
[  651.311789]  [<ffffffff8120c791>] SyS_ioctl+0x81/0xa0
[  651.311792]  [<ffffffff817d8272>] system_call_fastpath+0x16/0x75
[  651.311794] ---[ end trace 29b4fb9fbe5211cf ]---
[  651.437788] AMD-Vi: Completion-Wait loop timed out
[  651.437793] ------------[ cut here ]------------
[  651.437798] WARNING: CPU: 5 PID: 3851 at /home/kernel/COD/linux/drivers/iommu/amd_iommu.c:1199 __domain_flush_pages+0x162/0x170()
[  651.437799] Modules linked in: vhost_net vhost macvtap macvlan xt_CHECKSUM iptable_mangle ipt_REJECT nf_reject_ipv4 xt_tcpudp ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter ip_tables x_tables cmac rfcomm bnep cfg80211 bridge stp llc nls_iso8859_1 btusb btbcm btintel eeepc_wmi bluetooth asus_wmi sparse_keymap video crct10dif_pclmul crc32_pclmul ghash_clmulni_intel snd_usb_audio aesni_intel joydev aes_x86_64 snd_usbmidi_lib snd_hda_codec_realtek lrw snd_hda_codec_generic snd_hda_intel gf128mul glue_helper snd_hda_controller ablk_helper edac_core snd_hda_codec cryptd snd_hda_core snd_hwdep serio_raw edac_mce_amd 8250_fintek tpm_infineon fam15h_power i2c_piix4 k10temp shpchp mac_hid snd_aloop snd_pcm snd_seq_midi snd_seq_midi_event snd_rawmidi snd_seq snd_seq_device snd_timer snd soundcore
[  651.437837]  kvm_amd kvm vfio_iommu_type1 parport_pc ppdev lp parport autofs4 hid_generic usbhid hid amdkfd amd_iommu_v2 radeon i2c_algo_bit ttm drm_kms_helper drm mxm_wmi psmouse vfio_pci ahci r8169 vfio_virqfd libahci mii vfio wmi
[  651.437853] CPU: 5 PID: 3851 Comm: qemu-system-x86 Tainted: G        W       4.1.2-040102-generic #201507101335
[  651.437854] Hardware name: To be filled by O.E.M. To be filled by O.E.M./M5A97 LE R2.0, BIOS 2601 03/24/2015
[  651.437856]  ffffffff81b124e0 ffff88048e27bb88 ffffffff817d076c 0000000000000000
[  651.437858]  0000000000000000 ffff88048e27bbc8 ffffffff81079b0a 00000000fffffffb
[  651.437860]  ffff880533c53610 00000000fffffffb 0000000000000000 7fffffffffffffff
[  651.437862] Call Trace:
[  651.437867]  [<ffffffff817d076c>] dump_stack+0x45/0x57
[  651.437870]  [<ffffffff81079b0a>] warn_slowpath_common+0x8a/0xc0
[  651.437872]  [<ffffffff81079bfa>] warn_slowpath_null+0x1a/0x20
[  651.437873]  [<ffffffff814f4662>] __domain_flush_pages+0x162/0x170
[  651.437876]  [<ffffffff81013673>] ? __switch_to+0x1e3/0x580
[  651.437878]  [<ffffffff814f5259>] amd_iommu_unmap+0x119/0x140
[  651.437881]  [<ffffffff814eec79>] iommu_unmap+0xc9/0x1d0
[  651.437883]  [<ffffffff814f242f>] ? amd_iommu_iova_to_phys+0x2f/0x80
[  651.437888]  [<ffffffffc01575b7>] vfio_remove_dma+0x107/0x1f0 [vfio_iommu_type1]
[  651.437893]  [<ffffffffc0157d35>] vfio_iommu_type1_ioctl+0x3e5/0xa40 [vfio_iommu_type1]
[  651.437909]  [<ffffffffc01c674a>] ? kvm_set_memory_region+0x3a/0x50 [kvm]
[  651.437917]  [<ffffffffc01c6cb5>] ? kvm_vm_ioctl+0x555/0x790 [kvm]
[  651.437923]  [<ffffffffc000a6f9>] vfio_fops_unl_ioctl+0x79/0x2b0 [vfio]
[  651.437927]  [<ffffffff8120c4f8>] do_vfs_ioctl+0x2f8/0x510
[  651.437930]  [<ffffffff81066f16>] ? __do_page_fault+0x1b6/0x450
[  651.437932]  [<ffffffff8120c791>] SyS_ioctl+0x81/0xa0
[  651.437935]  [<ffffffff817d8272>] system_call_fastpath+0x16/0x75
[  651.437936] ---[ end trace 29b4fb9fbe5211d0 ]---
[  651.562996] AMD-Vi: Completion-Wait loop timed out
[  651.563001] ------------[ cut here ]------------
[  651.563007] WARNING: CPU: 4 PID: 3851 at /home/kernel/COD/linux/drivers/iommu/amd_iommu.c:1199 __domain_flush_pages+0x162/0x170()
[  651.563008] Modules linked in: vhost_net vhost macvtap macvlan xt_CHECKSUM iptable_mangle ipt_REJECT nf_reject_ipv4 xt_tcpudp ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter ip_tables x_tables cmac rfcomm bnep cfg80211 bridge stp llc nls_iso8859_1 btusb btbcm btintel eeepc_wmi bluetooth asus_wmi sparse_keymap video crct10dif_pclmul crc32_pclmul ghash_clmulni_intel snd_usb_audio aesni_intel joydev aes_x86_64 snd_usbmidi_lib snd_hda_codec_realtek lrw snd_hda_codec_generic snd_hda_intel gf128mul glue_helper snd_hda_controller ablk_helper edac_core snd_hda_codec cryptd snd_hda_core snd_hwdep serio_raw edac_mce_amd 8250_fintek tpm_infineon fam15h_power i2c_piix4 k10temp shpchp mac_hid snd_aloop snd_pcm snd_seq_midi snd_seq_midi_event snd_rawmidi snd_seq snd_seq_device snd_timer snd soundcore
[  651.563045]  kvm_amd kvm vfio_iommu_type1 parport_pc ppdev lp parport autofs4 hid_generic usbhid hid amdkfd amd_iommu_v2 radeon i2c_algo_bit ttm drm_kms_helper drm mxm_wmi psmouse vfio_pci ahci r8169 vfio_virqfd libahci mii vfio wmi
[  651.563061] CPU: 4 PID: 3851 Comm: qemu-system-x86 Tainted: G        W       4.1.2-040102-generic #201507101335
[  651.563062] Hardware name: To be filled by O.E.M. To be filled by O.E.M./M5A97 LE R2.0, BIOS 2601 03/24/2015
[  651.563064]  ffffffff81b124e0 ffff88048e27bb88 ffffffff817d076c 0000000000000000
[  651.563066]  0000000000000000 ffff88048e27bbc8 ffffffff81079b0a 00000000fffffffb
[  651.563068]  ffff880533c53610 00000000fffffffb 0000000000000000 7fffffffffffffff
[  651.563070] Call Trace:
[  651.563075]  [<ffffffff817d076c>] dump_stack+0x45/0x57
[  651.563078]  [<ffffffff81079b0a>] warn_slowpath_common+0x8a/0xc0
[  651.563080]  [<ffffffff81079bfa>] warn_slowpath_null+0x1a/0x20
[  651.563081]  [<ffffffff814f4662>] __domain_flush_pages+0x162/0x170
[  651.563083]  [<ffffffff814f5259>] amd_iommu_unmap+0x119/0x140
[  651.563087]  [<ffffffff8118fb1e>] ? compound_unlock_irqrestore+0xe/0x20
[  651.563090]  [<ffffffff814eec79>] iommu_unmap+0xc9/0x1d0
[  651.563091]  [<ffffffff814f242f>] ? amd_iommu_iova_to_phys+0x2f/0x80
[  651.563097]  [<ffffffffc01575b7>] vfio_remove_dma+0x107/0x1f0 [vfio_iommu_type1]
[  651.563101]  [<ffffffffc0157d35>] vfio_iommu_type1_ioctl+0x3e5/0xa40 [vfio_iommu_type1]
[  651.563117]  [<ffffffffc01c674a>] ? kvm_set_memory_region+0x3a/0x50 [kvm]
[  651.563126]  [<ffffffffc01c6cb5>] ? kvm_vm_ioctl+0x555/0x790 [kvm]
[  651.563132]  [<ffffffffc000a6f9>] vfio_fops_unl_ioctl+0x79/0x2b0 [vfio]
[  651.563135]  [<ffffffff8120c4f8>] do_vfs_ioctl+0x2f8/0x510
[  651.563138]  [<ffffffff81066f16>] ? __do_page_fault+0x1b6/0x450
[  651.563140]  [<ffffffff8120c791>] SyS_ioctl+0x81/0xa0
[  651.563143]  [<ffffffff817d8272>] system_call_fastpath+0x16/0x75
[  651.563145] ---[ end trace 29b4fb9fbe5211d1 ]---
[  651.654821] BUG: unable to handle kernel NULL pointer dereference at 0000000000000049
[  651.654855] IP: [<ffffffff810f745e>] drop_futex_key_refs.isra.14+0xe/0x70
[  651.654882] PGD 0 
[  651.654890] Oops: 0000 [#1] SMP 
[  651.654903] Modules linked in: vhost_net vhost macvtap macvlan xt_CHECKSUM iptable_mangle ipt_REJECT nf_reject_ipv4 xt_tcpudp ebtable_filter ebtables ip6table_filter ip6_tables iptable_filter ip_tables x_tables cmac rfcomm bnep cfg80211 bridge stp llc nls_iso8859_1 btusb btbcm btintel eeepc_wmi bluetooth asus_wmi sparse_keymap video crct10dif_pclmul crc32_pclmul ghash_clmulni_intel snd_usb_audio aesni_intel joydev aes_x86_64 snd_usbmidi_lib snd_hda_codec_realtek lrw snd_hda_codec_generic snd_hda_intel gf128mul glue_helper snd_hda_controller ablk_helper edac_core snd_hda_codec cryptd snd_hda_core snd_hwdep serio_raw edac_mce_amd 8250_fintek tpm_infineon fam15h_power i2c_piix4 k10temp shpchp mac_hid snd_aloop snd_pcm snd_seq_midi snd_seq_midi_event snd_rawmidi snd_seq snd_seq_device snd_timer snd soundcore
[  651.655166]  kvm_amd kvm vfio_iommu_type1 parport_pc ppdev lp parport autofs4 hid_generic usbhid hid amdkfd amd_iommu_v2 radeon i2c_algo_bit ttm drm_kms_helper drm mxm_wmi psmouse vfio_pci ahci r8169 vfio_virqfd libahci mii vfio wmi
[  651.655244] CPU: 4 PID: 3848 Comm: qemu-system-x86 Tainted: G        W       4.1.2-040102-generic #201507101335
[  651.655275] Hardware name: To be filled by O.E.M. To be filled by O.E.M./M5A97 LE R2.0, BIOS 2601 03/24/2015
[  651.655305] task: ffff88048e2764a0 ti: ffff8804bc08c000 task.ti: ffff8804bc08c000
[  651.655350] RIP: 0010:[<ffffffff810f745e>]  [<ffffffff810f745e>] drop_futex_key_refs.isra.14+0xe/0x70
[  651.655380] RSP: 0018:ffff8804bc08fcc8  EFLAGS: 00010286
[  651.655396] RAX: ffff88048e2764a0 RBX: 0000000000000001 RCX: 00000000c0000100
[  651.655419] RDX: ffff8804bc090000 RSI: 0000000000000049 RDI: ffff8805313eda00
[  651.655440] RBP: ffff8804bc08fcc8 R08: ffff8804bc08c000 R09: 0000000000000000
[  651.655461] R10: 0000000000000004 R11: 0000000000000005 R12: 0000000000000000
[  651.655482] R13: 0000000000000000 R14: 0000000000000000 R15: ffff88048e2764a0
[  651.655504] FS:  00007f24f8e3cb00(0000) GS:ffff88054ed00000(0000) knlGS:0000000000000000
[  651.655528] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[  651.655545] CR2: 0000000000000049 CR3: 00000004bc371000 CR4: 00000000000407e0
[  651.655566] Stack:
[  651.655574]  ffff8804bc08fe28 ffffffff810f9122 ffff8804bc11bc00 ffffffff00000019
[  651.655598]  0000000000000000 ffff8804bc08fd60 0000000000000002 000055989a1a1ac0
[  651.655622]  dead000000200200 ffffc90003567780 0000000000000064 ffff8804bc08fd20
[  651.655646] Call Trace:
[  651.655657]  [<ffffffff810f9122>] futex_wait+0x1a2/0x280
[  651.655676]  [<ffffffff8120d160>] ? poll_select_copy_remaining+0x130/0x130
[  651.655698]  [<ffffffff8120d160>] ? poll_select_copy_remaining+0x130/0x130
[  651.655719]  [<ffffffff810fa4ee>] do_futex+0xde/0x5f0
[  651.655736]  [<ffffffff8101e2b9>] ? read_tsc+0x9/0x10
[  651.655753]  [<ffffffff810ebd8e>] ? ktime_get_ts64+0x4e/0xf0
[  651.655771]  [<ffffffff8120d131>] ? poll_select_copy_remaining+0x101/0x130
[  651.655793]  [<ffffffff810faa71>] SyS_futex+0x71/0x150
[  651.655811]  [<ffffffff817d8272>] system_call_fastpath+0x16/0x75
[  651.655830] Code: c3 0f 1f 40 00 55 48 89 e5 0f 1f 40 00 e8 0b cb 11 00 5d f3 c3 0f 1f 84 00 00 00 00 00 66 66 66 66 90 55 48 85 ff 48 89 e5 74 3a <8b> 06 83 e0 03 83 f8 01 74 10 83 f8 02 75 06 f0 ff 4f 4c 74 15 
[  651.655921] RIP  [<ffffffff810f745e>] drop_futex_key_refs.isra.14+0xe/0x70
[  651.655943]  RSP <ffff8804bc08fcc8>
[  651.655953] CR2: 0000000000000049
[  651.660818] ---[ end trace 29b4fb9fbe5211d2 ]---
[  651.916078] AMD-Vi: Event logged [IOTLB_INV_TIMEOUT device=05:00.0 address=0x0000000536f2f920]
[  651.916087] AMD-Vi: Event logged [IOTLB_INV_TIMEOUT device=05:00.0 address=0x0000000536f2f940]
[  653.034592] AMD-Vi: Event logged [IOTLB_INV_TIMEOUT device=05:00.0 address=0x0000000536f2fdb0]
[  653.034601] AMD-Vi: Event logged [IOTLB_INV_TIMEOUT device=05:00.0 address=0x0000000536f2f9d0]
[  653.034605] AMD-Vi: Event logged [IOTLB_INV_TIMEOUT device=05:00.0 address=0x0000000536f2f9f0]
[  653.034609] AMD-Vi: Event logged [IOTLB_INV_TIMEOUT device=05:00.0 address=0x0000000536f2fa10]
[  653.034613] AMD-Vi: Event logged [IOTLB_INV_TIMEOUT device=05:00.0 address=0x0000000536f2fa30]
[  653.034616] AMD-Vi: Event logged [IOTLB_INV_TIMEOUT device=05:00.0 address=0x0000000536f2fa50]
[  653.034619] AMD-Vi: Event logged [IOTLB_INV_TIMEOUT device=05:00.0 address=0x0000000536f2fa70]
[  653.034622] AMD-Vi: Event logged [IOTLB_INV_TIMEOUT device=05:00.0 address=0x0000000536f2fa90]
[  653.034626] AMD-Vi: Event logged [IOTLB_INV_TIMEOUT device=05:00.0 address=0x0000000536f2fab0]
[  653.034629] AMD-Vi: Event logged [IOTLB_INV_TIMEOUT device=05:00.0 address=0x0000000536f2fad0]
[  653.034632] AMD-Vi: Event logged [IOTLB_INV_TIMEOUT device=05:00.0 address=0x0000000536f2faf0]
[  653.034635] AMD-Vi: Event logged [IOTLB_INV_TIMEOUT device=05:00.0 address=0x0000000536f2fb10]
[  653.034639] AMD-Vi: Event logged [IOTLB_INV_TIMEOUT device=05:00.0 address=0x0000000536f2fb30]
[  653.034642] AMD-Vi: Event logged [IOTLB_INV_TIMEOUT device=05:00.0 address=0x0000000536f2fb50]
[  653.034645] AMD-Vi: Event logged [IOTLB_INV_TIMEOUT device=05:00.0 address=0x0000000536f2fb70]
[  653.034648] AMD-Vi: Event logged [IOTLB_INV_TIMEOUT device=05:00.0 address=0x0000000536f2fb90]
[  653.034651] AMD-Vi: Event logged [IOTLB_INV_TIMEOUT device=05:00.0 address=0x0000000536f2fbb0]
[  653.034654] AMD-Vi: Event logged [IOTLB_INV_TIMEOUT device=05:00.0 address=0x0000000536f2fbd0]
[  653.034658] AMD-Vi: Event logged [IOTLB_INV_TIMEOUT device=05:00.0 address=0x0000000536f2fbf0]
[  653.034661] AMD-Vi: Event logged [IOTLB_INV_TIMEOUT device=05:00.0 address=0x0000000536f2fc10]
[  653.034664] AMD-Vi: Event logged [IOTLB_INV_TIMEOUT device=05:00.0 address=0x0000000536f2fc30]
[  653.034667] AMD-Vi: Event logged [IOTLB_INV_TIMEOUT device=05:00.0 address=0x0000000536f2fc50]
[  653.034670] AMD-Vi: Event logged [IOTLB_INV_TIMEOUT device=05:00.0 address=0x0000000536f2fc70]
[  653.034674] AMD-Vi: Event logged [IOTLB_INV_TIMEOUT device=05:00.0 address=0x0000000536f2fc90]
[  653.034677] AMD-Vi: Event logged [IOTLB_INV_TIMEOUT device=05:00.0 address=0x0000000536f2fcb0]
[  653.034680] AMD-Vi: Event logged [IOTLB_INV_TIMEOUT device=05:00.0 address=0x0000000536f2fcd0]
[  653.034683] AMD-Vi: Event logged [IOTLB_INV_TIMEOUT device=05:00.0 address=0x0000000536f2fcf0]
[  653.034686] AMD-Vi: Event logged [IOTLB_INV_TIMEOUT device=05:00.0 address=0x0000000536f2fd10]
[  653.034689] AMD-Vi: Event logged [IOTLB_INV_TIMEOUT device=05:00.0 address=0x0000000536f2fd30]
[  653.034693] AMD-Vi: Event logged [IOTLB_INV_TIMEOUT device=05:00.0 address=0x0000000536f2fd50]
[  653.034696] AMD-Vi: Event logged [IOTLB_INV_TIMEOUT device=05:00.0 address=0x0000000536f2fd70]

...followed by a lot more IOTLB_INV_TIMEOUT messages.

I also get a set of "AMD-Vi: Completion-Wait loop timed out" messages on boot if I bind the 270x card (otherwise, it's the same as the other card), but afterwards, lspci shows the card properly bound to vfio.

I'm not sure what to do here, Has anyone seen this issue before, who might be able to point me in the right direction?

morat

Member

Registered: 2015-07-07

Posts: 38

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

@Col_Panic
I was getting inexplicable kernel panics for the longest time, before it was pointed out to me that one of the PCI ids was mistyped. Check and double check, and if it still hangs, pastebin your dmesg and command line/domain xml; and perhaps someone wiser than I could lend a hand.

aw

Member

Registered: 2013-10-04

Posts: 921

Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

@Col_Panic

modprobe.d:
options vfio_iommu_type1 disable_hugepages=1

That's my default answer for anyone getting strange AMD-Vi problems.

BTW, amd_iommu cannot be built as a module.

---

http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

morat

Member

Registered: 2015-07-07

Posts: 38

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

The most luck I've had with audio so far is Windows 8 recognising a "High Definition Audio Controller", which believes it's outputting sound, but I hear nothing. I have used vnc_allow_host_audio = 1, and tried setting the qemu user to my own, but that only causes permission issues with the pci. libvirt is set to use ich9.

Maybe you use libvirt / virtmanager and the guest window is closed then you get no sound output.

So then, is it impossible to have sound passed through the host without a libvirt VNC session?

Col_Panic

Member

Registered: 2015-07-19

Posts: 4

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

@Col_Panic

modprobe.d:
options vfio_iommu_type1 disable_hugepages=1

That's my default answer for anyone getting strange AMD-Vi problems.

BTW, amd_iommu cannot be built as a module.

Thanks! This seems to have gotten me somewhere,
I'm still getting the same iommu_unmap, etc messages, followed by IOTLB_INV_TIMEOUT. But instead of crashing the host after IOTLB_INV_TIMEOUT, it's constantly looping back to the first error.
Passthrough isn't working yet, but at least one problem is solved!

impulse_255

Member

Registered: 2015-06-29

Posts: 22

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

No success whatsoever with the pciid spoofing. I attempted to spoof my Titan X as an M6000 for the purpose of using Hyper-V enlightenments. I patched Qemu 2.3.0 with the patch posted earlier. It needed some modifications to work with 2.3.0. The file was moved to vfio/pci.c instead of misc/vfio.c and one of the structs was now VFIOPCIDevice instead of VFIODevice.

Once I started Qemu with the new pci id, the VM GPU would not show a picture. I let it boot the OS but it would remain blank. Of course I did a sanity check to see if it would work with the alteration enabled but with the original pci id. That worked. However, passing any rom with the romfile option produced a black screen, even with the actual rom dumped with GPU-Z in guest Windows. In summary:

1) altered PCIIDs to M6000 (x-vid=0x10DE,x-did=0x17f0,x-ss-vid=0x10DE,x-ss-did=0x1129) => black screen
2) altered PCIIDs to M6000, supplied original rom => black screen
3) altered PCIIDS to original (x-vid=0x10DE,x-did=0x17c2,x-ss-vid=0x10DE,x-ss-did=0x1132) => works
4) altered PCIIDS to original, supplied original rom => black screen

#4 puzzles me the most. Another thing worth mentioning is that I got a BSOD when I tried to boot without any trickery after attempting all of the above. After a host reboot, I could boot the guest again, suggesting that the GPU was left in a broken state before the reboot. I don't know if supplying an M6000 rom, which I don't have, would do any good since even the card's own rom gives me a black screen. I wonder if Nvidia has added more obstacles to the cards.

Could someone else with a Maxwell card try this?

EDIT: For clarification, I also tried #4 after a host reboot to make sure the GPU was not left in a messed up state. Black screen. I forgot to mention that I briefly tried SeaBIOS instead of OVMF with a more minimalistic qemu command line and #1 but I still couldn't get a picture out of the GPU.

Here's an example of my configuration for the GPU passthrough. I removed the hostdev configuration and replaced it with a custom qemu command line configuration:

  <qemu:commandline>
    <qemu:arg value='-device'/>
    <qemu:arg value='vfio-pci,host=01:00.0,id=hostdevvga,bus=pci.0,addr=0x8,x-vid=0x10DE,x-did=0x17c2,x-ss-vid=0x10DE,x-ss-did=0x1132,romfile=GM200.rom'/>
  </qemu:commandline>

where the values are according to #4.

Last edited by impulse_255 (2015-07-19 22:54:09)

slis

Member

Registered: 2014-06-02

Posts: 127

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Thx impulse_255 for trying it out....

This is how I start my vm as grid k2....

<qemu:arg value='-device'/>
    <qemu:arg value='vfio-pci,host=01:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on,x-vid=0x10DE,x-did=0x11BF,x-ss-vid=0x10DE,x-ss-did=0x0965''/>

impulse_255

Member

Registered: 2015-06-29

Posts: 22

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Ok, I got it working. First of all, the rom dump I took from Windows was different to one I took from the host with the GPU in an uninitialised state (resumed from S3 suspend and had not started the VM after that). Once I repeated #4 with that dump, it worked.

Next I read the eevblog forum posts on hardware pci id modding (euphemism for fiddling with your pricey graphics cards with a soldering iron) and rom flashing. Damn, we have it easy. Everything can be safely done in software. I took the working original rom and changed the pci ids with a hex editor. VGA roms have a rudimentary checksum which can be fooled by changing some non-critical plaintext strings present in the rom to compensate for any relevant changes: http://www.eevblog.com/forum/chat/hacki … #msg208495

It works! In installed the Quadro driver and enabled hyper-v enlightenments. I did not run benchmarks yet but I can't see any significant difference, though. Crysis Warhead which runs noticeably worse than on bare metal is still as cpu limited as ever. Well, it was an exiting journey anyway, lol.

slis

Member

Registered: 2014-06-02

Posts: 127

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Nice, does it work with geforce driver? not quadro one? but I guess its same... now i can go maxwell haha
edit: you got i7 2600? about crysis... did you try run different cpu config like cores=4/8 threads=1? and what about cpu governor is it set to performance? I'm using it like this http://justpaste.it/mgsj

edit2: maybe you should post it to eevlog, so that guys with vt-d don't have to solder

Last edited by slis (2015-07-20 11:50:42)

impulse_255

Member

Registered: 2015-06-29

Posts: 22

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

The quadro driver is a separate package. After booting with the changes, Windows was in standard vga mode and I had to install the other driver. It seems to work just the same, including the nv control panel.

I have tried various cpu settings such as cores=4,threads=2 which will give the guest all the throughput it can get. Performance governor too. Now I use cores=3,threads=2 and keep host processes on the remaining core to keep latency and stutter down. I'm not that concerned with how Crysis runs. I noticed its framerate issues and I have since used it as a benchmark for a game that does not play nicely in a VM (for me at least). It would be good to iron out such exceptional issues. Other games such as Witcher 3 run just fine with everything cranked up.

BTW, a disadvantage of having a card appear as Quadro: MSI Afterburner no longer provides voltage controls for overclocking. Maybe it could be fooled somehow as well. It's the exact same card and bios except for the ids after all. But that does not belong here.

slis

Member

Registered: 2014-06-02

Posts: 127

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

For me using threads=1 with cores=4/8 instead threads=2 cores=2/4 is much faster in cpu dependent games / dota2.

impulse_255

Member

Registered: 2015-06-29

Posts: 22

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

For me using threads=1 with cores=4/8 instead threads=2 cores=2/4 is much faster in cpu dependent games / dota2.

Hmm. Someone on reddit claimed the exact opposite. Do you know if Windows actually treats logical (HT) processors differently than physical or does threads=1 vs. 2 change hypervisor behaviour?

slis

Member

Registered: 2014-06-02

Posts: 127

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Well, I don't know, you should try it, with smth similar to what I posted on that pasteit link, real cores are 0-3.

impulse_255

Member

Registered: 2015-06-29

Posts: 22

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Well, I don't know, you should try it, with smth similar to what I posted on that pasteit link, real cores are 0-3.

All right, thanks. I'll try that.

SXX

Member

Registered: 2014-03-31

Posts: 15

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Just wonder is it possible to pass 2nd GPU or two GPUs to same VM without ACS patch?
I just trying to pass HD6950 (work alone just fine) and GTX260 into same VM, but it's just don't work.
When I try to only pass GTX260 (that use 2nd PCIe port) it's fail with "KVM internal error. Suberror: 1".

Is that would work if I'm going to apply ACS? And if so is same patch would work for 4.1 kernel?

PS: Also a bit off-topic question, but anyway. Is there way to make QEMU emulate USB mouse with more than 3 buttons? I already using "-device usb-mouse", but on host I have mouse with 7 buttons while QEMU only grabbing 3 first.

Last edited by SXX (2015-07-20 20:57:45)

impulse_255

Member

Registered: 2015-06-29

Posts: 22

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Nngh. The Quadro solution isn't problem free either. If hyper-v enlightenments are enabled, the driver does work but runs in some weird crippled mode. There are some missing configuration options such as display rotation, digital audio and custom resolutions. It also would not let me use refresh rates over 85 Hz which is very annoying since I could go up to 144 Hz. Now I'm torn between making the most out of my monitor in less demanding games or getting some extra performance in more demanding ones. To make matters worse I can't simply switch them on or off between VM restarts because in Quadro mode and hyper-v disabled fullscreen games show a blank screen. WTF? Switching between "graphics cards" needs a driver reinstall every time. I'm starting to believe nvidia's explanation that it was indeed incompetence, not malice, that broke hyper-v.

As a bonus, I ran GTA 5's built-in benchmark with Quadro mod + HV and Titan X without HV. The difference is small but there were annoying stutters that occurred without HV. It could be placebo, too. Here are the results:

HV enabled:
Frames Per Second (Higher is better) Min, Max, Avg
Pass 0, 39.046333, 92.392410, 58.477955
Pass 1, 20.063923, 80.518539, 48.624252
Pass 2, 36.416340, 88.159317, 61.158989
Pass 3, 35.666138, 93.538368, 69.098633
Pass 4, 15.514916, 103.371994, 58.484459

HV disabled:
Frames Per Second (Higher is better) Min, Max, Avg
Pass 0, 24.502665, 69.663994, 57.547050
Pass 1, 27.271063, 61.661015, 47.173367
Pass 2, 36.337238, 79.242561, 59.791538
Pass 3, 24.399113, 84.966293, 67.116913
Pass 4, 15.372684, 90.449654, 55.312733

I think I'll stick without HV at least for now because of the myriad issues.

The_Moves

Member

Registered: 2015-01-06

Posts: 59

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

No success whatsoever with the pciid spoofing. I attempted to spoof my Titan X as an M6000 for the purpose of using Hyper-V enlightenments. I patched Qemu 2.3.0 with the patch posted earlier. It needed some modifications to work with 2.3.0. The file was moved to vfio/pci.c instead of misc/vfio.c and one of the structs was now VFIOPCIDevice instead of VFIODevice.

Impulse,

Do you think you could link your updated patch for the stock pci.c in QEMU 2.3.0? I've tried modifying the patch but i haven't been able to get it to work

Last edited by The_Moves (2015-07-21 00:20:06)

The_Moves

Member

Registered: 2015-01-06

Posts: 59

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

I think I might have gotten it, but now i'm getting this:

XML document failed to validate against schema: Unable to validate doc against/usr/share/libvirt/schemas/domain.rng
Element domain has extra content: qemu:command line

EDIT: Never mind, I must not have gotten it patched correctly...more research from my end on how to patch packages. I was able to get a Kernel patched, but this...

Last edited by The_Moves (2015-07-21 02:27:14)

Col_Panic

Member

Registered: 2015-07-19

Posts: 4

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

So I'm still having no luck getting past my problem for this GPU. I've been experimenting with different boot options, and I'm still stuck at where I was after aw's suggestion; instead of locking the host, I'm just getting an endless loop of iommu page flushes, and a guest that complains if I try to destroy it.
I thought I was getting somewhere with nofullflush, when I was able to shut down the guest, but it turns out that was a fluke.
I came across the same symptoms on the KVM dev list, and it was concluded in that thread that it was probably a BIOS problem, rather than a driver issue.
Looks like it's time to upgrade my discount-store MOBO

slis

Member

Registered: 2014-06-02

Posts: 127

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

@the_moves, you need to change first line of xml to make it take qemu args,..
edit:

<domain type='kvm' id='5' xmlns:qemu='http://libvirt.org/schemas/domain/qemu/1.0'> 

@impulse_255 thx for that, it shows good  boost of min/ max fps...  well if its detected like quadro it would show nv control panel with less options then as geforce, but you maybe could use nview for resolution and stuff

Last edited by slis (2015-07-21 05:03:14)

impulse_255

Member

Registered: 2015-06-29

Posts: 22

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

@The_Moves: Here is the updated spoofing patch for Qemu 2.3.0: http://pastebin.com/vFpN06LX

@slis: The control panel shows all the options when hyper-v is disabled. Only when hyper-v is enabled, stuff goes missing and refresh rates are limited, so it doesn't seem like it's meant to be some quadro specific limitation. In fact, there is a quadro specific "workstation" category which also goes missing with HV. I have only so far tried with all four hv_* options enabled or none at all but I'm assuming it works by detecting the Microsoft Hv string just like the Geforce driver.

I really wonder what makes Nvidia drivers so allergic to it.

EDIT: But of course if you don't need 100+ Hz refresh rates or all the dials and buttons, you probably won't have any problems. Even G-Sync works in the crippled mode. I'm just being my obsessive perfectionist self.

Last edited by impulse_255 (2015-07-21 08:51:04)

dwe11er

Member

Registered: 2014-03-18

Posts: 73

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Just wonder is it possible to pass 2nd GPU or two GPUs to same VM without ACS patch?
I just trying to pass HD6950 (work alone just fine) and GTX260 into same VM, but it's just don't work.
When I try to only pass GTX260 (that use 2nd PCIe port) it's fail with "KVM internal error. Suberror: 1".

Is that would work if I'm going to apply ACS? And if so is same patch would work for 4.1 kernel?

PS: Also a bit off-topic question, but anyway. Is there way to make QEMU emulate USB mouse with more than 3 buttons? I already using "-device usb-mouse", but on host I have mouse with 7 buttons while QEMU only grabbing 3 first.

Get second mouse for host (wireless for example) and passthrough main one
or
passthrough your mouse to VM and use Synergy (host as client, VM as server).

impulse_255

Member

Registered: 2015-06-29

Posts: 22

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Just wonder is it possible to pass 2nd GPU or two GPUs to same VM without ACS patch?
I just trying to pass HD6950 (work alone just fine) and GTX260 into same VM, but it's just don't work.
When I try to only pass GTX260 (that use 2nd PCIe port) it's fail with "KVM internal error. Suberror: 1".

Is that would work if I'm going to apply ACS? And if so is same patch would work for 4.1 kernel?

PS: Also a bit off-topic question, but anyway. Is there way to make QEMU emulate USB mouse with more than 3 buttons? I already using "-device usb-mouse", but on host I have mouse with 7 buttons while QEMU only grabbing 3 first.

Get second mouse for host (wireless for example) and passthrough main one
or
passthrough your mouse to VM and use Synergy (host as client, VM as server).

No need for a second mouse and kb, a kvm switch (the other kvm) is handy if a software solution is not good enough.