Arch Linux

aw

Member

Registered: 2013-10-04

Posts: 921

Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Borderlands folks, I think this is the problem you're trying to describe:

This vantage point in the training area seems to give me a consistently low FPS and high latency and seems invariant to the lighting.  The VM shows about 1/3rd the FPS and 3x the latency.  What is that latency actually measuring anyway?  These are separate win7x64 installs on the same Haswell/GTX660 hardware with Nvidia Experience optimizing the game in both (highest settings for all).

I don't have any solutions yet, but it's good to understand the problem.  I did run the VM with vfio debugging turned on and I'm not seeing any excessive hits to the card, which means we're not bouncing out to userspace regularly as a result of the assigned devices (GPU + audio + NIC in this case).  I see a write to 0x88154 at a rate of about 10/s.  This is an area we must trap because it's an mmio mirror of PCI config space.  That happens regardless of what's going on in the game and also happens at a basic desktop, so if it was causing any trouble I'd expect it would be across the board and not select games.

I'm suspicious of USB passthrough, but my only evidence is that hitting the escape key in the game is not as responsive as bare metal and often requires a second hit.  Assignment of the USB controller might be an option, but didn't just work in my first attempt, maybe an issue with windows drivers.

EDIT: game reported latency appears to just be the inverse of FPS, so it's SPF (seconds per frame)

EDIT2: Further analysis (special thanks to Paolo Bonzini) shows two issues that are hurting performance here.  First is a known issue with Windows that it pounds on the time source, whether it's ACPI PM timer or HPET.  Code just recently went upstream (kernel and qemu) that enables a Hyper-V enlightened timer that can help to reduce this.  To enable it add "hv-time" to your -cpu option, ex. -cpu Haswell,hv-time  This gives me about 20% more FPS and is likely to help performance of any game running in a Windows guest.  The second problem is that we see extensive use of the processor debug registers.  Each move to or from these registers traps into the hypvervisor and they appear to be batched into a context switch function, so we have a burst of them all at once.  I've submitted a support ticket to gearbox software to see if they have any suggestions to avoid these registers.

Hello performance!  Patches were just posted upstream to do lazy tracking of the debug registers.  With "-cpu Haswell,hv-time" and these patches, this is what I can get now (better than 80% of bare metal):

This initial implementation is Intel-only, but hopefully we can do the same for AMD.

---

http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

redger

Member

Registered: 2014-02-26

Posts: 15

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

thanks kaeptnb !  Reverting to to qemu-1.7.0 and Seabios-1.7.3.1 overcame that problem
then the following "Could not initialize SDL" error was addressed by setting the -nographic option

now the test passes (text appears on the GPU ) ..... though for some strange reason my mouse now insists on constantly moving rapidly left .... which is making life entertaining

thanks again

Val532

Member

Registered: 2013-11-13

Posts: 35

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Hi all (again ^^)

@aw

I saw a new set of patch for qemu to use Intel IGD as a passthrough device but only with Xen, do you think with this new information you can add Intel IGD to vfio ?

aw

Member

Registered: 2013-10-04

Posts: 921

Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Hi all (again ^^)

@aw

I saw a new set of patch for qemu to use Intel IGD as a passthrough device but only with Xen, do you think with this new information you can add Intel IGD to vfio ?

There's already someone in the community working on IGD passthrough.

---

http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Val532

Member

Registered: 2013-11-13

Posts: 35

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Hi all (again ^^)

@aw

I saw a new set of patch for qemu to use Intel IGD as a passthrough device but only with Xen, do you think with this new information you can add Intel IGD to vfio ?

There's already someone in the community working on IGD passthrough.

Really ? Where can i find some information about this ?

tuoni

Member

Registered: 2013-12-19

Posts: 7

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

The other major issue i'm having is trouble getting networking working in any configuration other than the default user mode (as it is in my commandline).  All attempts to use tap/tun result in various "device busy" or "device can't be accessed" messages.  This wouldn't be a big deal, but it's clear that tap/tun should provide better performance, and also I need the guest to be able to talk to the host so I can use synergy, so it's a bit of an issue.

I don't specify a networking switch and have synergy working just fine by giving it the host's IP address (192.168.0.X) to the Win7 guest.  Saying that, I'm having similar problems getting tap/tun working, but I'm still getting plenty good enough throughput on the guest.

neona

Member

Registered: 2014-02-25

Posts: 6

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

The other major issue i'm having is trouble getting networking working in any configuration other than the default user mode (as it is in my commandline).  All attempts to use tap/tun result in various "device busy" or "device can't be accessed" messages.  This wouldn't be a big deal, but it's clear that tap/tun should provide better performance, and also I need the guest to be able to talk to the host so I can use synergy, so it's a bit of an issue.

I don't specify a networking switch and have synergy working just fine by giving it the host's IP address (192.168.0.X) to the Win7 guest.  Saying that, I'm having similar problems getting tap/tun working, but I'm still getting plenty good enough throughput on the guest.

The reason that synergy isn't working for me is that I'm trying to run the server on the guest (which is behind NAT by default) so I can usb passthrough my kb/m, both for ease of use and to expose all features on them (particularly a razer naga).

Of course, you've reminded me that the opposite setup is an option for the time being, so I should probably just do that until I can properly configure networking.  I'm considering just being lazy and sticking in an extra NIC to pass through though, that may offer better performance than any virtual option anyway.

neona

Member

Registered: 2014-02-25

Posts: 6

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

For some reason, qemu only claims to support oss, wav, and spice output, so sound won't work.  I double checked the PKGBUILD for qemu, and it's clearly being compiled with pulse and alsa support (or should be, anyway), so I have no clue what's causing this, and some googling around hasn't found anything.  I've tried various ways of wrapping oss to pulse/alsa, and none of that has worked either.  This is rather inconvienient, but at least if I can get networking to behave then I think it's possible to have windows output to pulse if nothing else works.  I might also get a usb soundcard, but I'd rather avoid spending money if I can.

Turns out I'm an idiot, and installed a different qemu at some point that was earlier in my path than the one from qemu-git.  Cleaned that out and sound works fine now.

Flyser

Member

Registered: 2013-12-19

Posts: 29

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Before I spend hours trying ... is it possible to pass the intel onboard sound card to the guest with vfio?

anickname

Member

Registered: 2014-01-07

Posts: 23

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Before I spend hours trying ... is it possible to pass the intel onboard sound card to the guest with vfio?

On my motherboard "Asrock Z77 Professional Fatal1ty" worked without any problems.
The sound is like on the real hardware.

Val532

Member

Registered: 2013-11-13

Posts: 35

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Is anybody have a working audio on HDMI for AMD card ?
For me, i have no audio on all of my VM.

andy123

Member

Registered: 2011-11-04

Posts: 169

Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Is anybody have a working audio on HDMI for AMD card ?
For me, i have no audio on all of my VM.

I actually used to have my sound setup like this "AMD guest gpu HDMI → Display → audio cable output on Display → line in on onboard sound", which worked pretty well.

As far as my understanding of the stuff goes the "soundcard" on the GPU is the .1 function in most cases, so you need to make sure that it's passed through. What GPU are you using and do you have any error messages besides no output? 

Since qemu is running as root and I'm using pulseaudio I'm now doing "remote" pulseaudio over localhost.

---

i'm sorry for my poor english wirting skills…

Val532

Member

Registered: 2013-11-13

Posts: 35

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Is anybody have a working audio on HDMI for AMD card ?
For me, i have no audio on all of my VM.

I actually used to have my sound setup like this "AMD guest gpu HDMI → Display → audio cable output on Display → line in on onboard sound", which worked pretty well.

As far as my understanding of the stuff goes the "soundcard" on the GPU is the .1 function in most cases, so you need to make sure that it's passed through. What GPU are you using and do you have any error messages besides no output? 

Since qemu is running as root and I'm using pulseaudio I'm now doing "remote" pulseaudio over localhost.

So i start (one of) my VM with :

sudo qemu-system-x86_64 \
-M q35 -m 8G -enable-kvm -cpu host,hv-time -vga none -boot menu=on -no-fdc \
-smp 4,sockets=1,cores=4,threads=1 \
-device ioh3420,bus=pcie.0,id=root.1,port=1,chassis=1,addr=1c.0,multifunction=on \
-device vfio-pci,host=01:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on \
-device vfio-pci,host=01:00.1,bus=root.1 \
-device ahci,bus=pcie.0,id=ahci \
-drive file=/home/val/Image\ HDD/Windows-7-Test.img,id=disk,format=qcow2 \
-device ide-hd,bus=ahci.0,drive=disk \
-net nic,model=e1000 -net tap,ifname=tap0 \
-usb -usbdevice host:05e3:0607 -usbdevice host:1532:002f -usbdevice host:046d:c229 -usbdevice host:046d:c228

And my graphique card is an AMD HD5570.

I see an audio card in the device manager of Windows, but with error code 10.

About another subject, i try hv-time and i do not understand why but with -cpu host,hv-time does not work with Windows 8.1 Entreprise, but work with Windows 7 it's ok. It works with -cpu Haswell,hv-time for both.

Last edited by Val532 (2014-02-27 20:05:48)

MichaelLong

Member

Registered: 2014-01-06

Posts: 3

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Borderlands folks, I think this is the problem you're trying to describe:

https://dl.dropboxusercontent.com/u/198 … emetal.jpg

https://dl.dropboxusercontent.com/u/198 … ds2-vm.jpg

This vantage point in the training area seems to give me a consistently low FPS and high latency and seems invariant to the lighting.  The VM shows about 1/3rd the FPS and 3x the latency.  What is that latency actually measuring anyway?  These are separate win7x64 installs on the same Haswell/GTX660 hardware with Nvidia Experience optimizing the game in both (highest settings for all).

I don't have any solutions yet, but it's good to understand the problem.  I did run the VM with vfio debugging turned on and I'm not seeing any excessive hits to the card, which means we're not bouncing out to userspace regularly as a result of the assigned devices (GPU + audio + NIC in this case).  I see a write to 0x88154 at a rate of about 10/s.  This is an area we must trap because it's an mmio mirror of PCI config space.  That happens regardless of what's going on in the game and also happens at a basic desktop, so if it was causing any trouble I'd expect it would be across the board and not select games.

I'm suspicious of USB passthrough, but my only evidence is that hitting the escape key in the game is not as responsive as bare metal and often requires a second hit.  Assignment of the USB controller might be an option, but didn't just work in my first attempt, maybe an issue with windows drivers.

EDIT: game reported latency appears to just be the inverse of FPS, so it's SPF (seconds per frame)

EDIT2: Further analysis (special thanks to Paolo Bonzini) shows two issues that are hurting performance here.  First is a known issue with Windows that it pounds on the time source, whether it's ACPI PM timer or HPET.  Code just recently went upstream (kernel and qemu) that enables a Hyper-V enlightened timer that can help to reduce this.  To enable it add "hv-time" to your -cpu option, ex. -cpu Haswell,hv-time  This gives me about 20% more FPS and is likely to help performance of any game running in a Windows guest.  The second problem is that we see extensive use of the processor debug registers.  Each move to or from these registers traps into the hypvervisor and they appear to be batched into a context switch function, so we have a burst of them all at once.  I've submitted a support ticket to gearbox software to see if they have any suggestions to avoid these registers.

Hello performance!  Patches were just posted upstream to do lazy tracking of the debug registers.  With "-cpu Haswell,hv-time" and these patches, this is what I can get now (better than 80% of bare metal):

https://dl.dropboxusercontent.com/u/198 … vm-new.jpg

This initial implementation is Intel-only, but hopefully we can do the same for AMD.

Great news. I applied those patches ontop of kernel 3.14-rc4 and I can report that they also fixed my performance problems in Battlefield 4! I couldn't do any intensive tests yet but after playing about 1h or so it is clear that the performance has nearly restored. Thx a lot.

Val532

Member

Registered: 2013-11-13

Posts: 35

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Is anybody tried to install Hyper-V on a Windows 8 or 2012 server VM ?

When i want add Hyper-V to my Windows 8 Entreprise (juste for testing purpose) i can because of an "hypervisor is already run" (something like that, because the message is in french for me ^^)

aw

Member

Registered: 2013-10-04

Posts: 921

Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Is anybody tried to install Hyper-V on a Windows 8 or 2012 server VM ?

When i want add Hyper-V to my Windows 8 Entreprise (juste for testing purpose) i can because of an "hypervisor is already run" (something like that, because the message is in french for me ^^)

You need to enabled nested hypervisor support for that to work... but it's not really relevant to this thread.

---

http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Kaidax

Member

Registered: 2014-02-09

Posts: 6

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Hello performance!  Patches were just posted upstream to do lazy tracking of the debug registers.  With "-cpu Haswell,hv-time" and these patches, this is what I can get now (better than 80% of bare metal):

https://dl.dropboxusercontent.com/u/198 … vm-new.jpg

This initial implementation is Intel-only, but hopefully we can do the same for AMD.

That does the trick! Except I have to disable hv-time to play; enabling it drops performance to 15 fps, introduces severe graphical corruption, and makes journald dump gigabytes of logs.

Val532

Member

Registered: 2013-11-13

Posts: 35

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Is anybody tried to install Hyper-V on a Windows 8 or 2012 server VM ?

When i want add Hyper-V to my Windows 8 Entreprise (juste for testing purpose) i can because of an "hypervisor is already run" (something like that, because the message is in french for me ^^)

You need to enabled nested hypervisor support for that to work... but it's not really relevant to this thread.

Yeah i know it seems not relevant to the thread, but if i want to try it, it because of GPU passtrough ^^.

I have a working setup with two GPU on the same VM and i want to try the remoteFX fonction ^^.

But could you explain to me how to enable nested fonction ? (i see nested:bool in kvm_intel module).

Thanks.

aw

Member

Registered: 2013-10-04

Posts: 921

Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Is anybody tried to install Hyper-V on a Windows 8 or 2012 server VM ?

When i want add Hyper-V to my Windows 8 Entreprise (juste for testing purpose) i can because of an "hypervisor is already run" (something like that, because the message is in french for me ^^)

You need to enabled nested hypervisor support for that to work... but it's not really relevant to this thread.

Yeah i know it seems not relevant to the thread, but if i want to try it, it because of GPU passtrough ^^.

I have a working setup with two GPU on the same VM and i want to try the remoteFX fonction ^^.

But could you explain to me how to enable nested fonction ? (i see nested:bool in kvm_intel module).

Thanks.

More than I know here - http://git.kernel.org/cgit/linux/kernel … ed-vmx.txt

---

http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Val532

Member

Registered: 2013-11-13

Posts: 35

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

You need to enabled nested hypervisor support for that to work... but it's not really relevant to this thread.

Yeah i know it seems not relevant to the thread, but if i want to try it, it because of GPU passtrough ^^.

I have a working setup with two GPU on the same VM and i want to try the remoteFX fonction ^^.

But could you explain to me how to enable nested fonction ? (i see nested:bool in kvm_intel module).

Thanks.

More than I know here - http://git.kernel.org/cgit/linux/kernel … ed-vmx.txt

Ok so this is what i already do.

But Windows 8.1 knows it's runing on the top of an hypervisor, so i does not let me install Hyper-V.

Probably it's because of some modification to some Hyper-V change on qemu ?

MacCoaster

Member

Registered: 2014-01-06

Posts: 10

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Sigh... my system was working fine with a R9 290X, even after the new ACS patch (no downstream). But now I'm getting:

[  310.018977] dmar: DRHD: handling fault status reg 3
[  310.033576] dmar: DMAR:[DMA Read] Request device [01:00.0] fault addr 1e9fae0000 
[  310.033576] DMAR:[fault reason 06] PTE Read access is not set
[  310.218996] dmar: DRHD: handling fault status reg 3
[  310.233609] dmar: DMAR:[DMA Read] Request device [01:00.0] fault addr 1e9fae0000 
[  310.233609] DMAR:[fault reason 06] PTE Read access is not set
[  310.468989] dmar: DRHD: handling fault status reg 3
[  310.483594] dmar: DMAR:[DMA Read] Request device [01:00.0] fault addr 1e9fae0000 
[  310.483594] DMAR:[fault reason 06] PTE Read access is not set

1:00.0 is my R9 290X. Screen goes crazy once the AMD drivers load.

lsgroup:

### Group 0 ###
    00:00.0 Host bridge: Intel Corporation Xeon E3-1200 v3 Processor DRAM Controller (rev 06)
### Group 1 ###
    00:01.0 PCI bridge: Intel Corporation Xeon E3-1200 v3/4th Gen Core Processor PCI Express x16 Controller (rev 06)
    01:00.0 VGA compatible controller: Advanced Micro Devices, Inc. [AMD/ATI] Hawaii XT [Radeon HD 8970]
    01:00.1 Audio device: Advanced Micro Devices, Inc. [AMD/ATI] Device aac8
### Group 2 ###
    00:02.0 VGA compatible controller: Intel Corporation Xeon E3-1200 v3 Processor Integrated Graphics Controller (rev 06)
### Group 3 ###
    00:03.0 Audio device: Intel Corporation Xeon E3-1200 v3/4th Gen Core Processor HD Audio Controller (rev 06)
### Group 4 ###
    00:14.0 USB controller: Intel Corporation 8 Series/C220 Series Chipset Family USB xHCI (rev 04)
### Group 5 ###
    00:16.0 Communication controller: Intel Corporation 8 Series/C220 Series Chipset Family MEI Controller #1 (rev 04)
### Group 6 ###
    00:1a.0 USB controller: Intel Corporation 8 Series/C220 Series Chipset Family USB EHCI #2 (rev 04)
### Group 7 ###
    00:1b.0 Audio device: Intel Corporation 8 Series/C220 Series Chipset High Definition Audio Controller (rev 04)
### Group 8 ###
    00:1c.0 PCI bridge: Intel Corporation 8 Series/C220 Series Chipset Family PCI Express Root Port #1 (rev d4)
### Group 9 ###
    00:1c.1 PCI bridge: Intel Corporation 82801 PCI Bridge (rev d4)
    03:00.0 PCI bridge: ASMedia Technology Inc. ASM1083/1085 PCIe to PCI Bridge (rev 03)
    04:02.0 FireWire (IEEE 1394): VIA Technologies, Inc. VT6306/7/8 [Fire II(M)] IEEE 1394 OHCI Controller (rev c0)
### Group 10 ###
    00:1c.2 PCI bridge: Intel Corporation 8 Series/C220 Series Chipset Family PCI Express Root Port #3 (rev d4)
### Group 11 ###
    00:1c.3 PCI bridge: Intel Corporation 8 Series/C220 Series Chipset Family PCI Express Root Port #4 (rev d4)
### Group 12 ###
    00:1c.4 PCI bridge: Intel Corporation 8 Series/C220 Series Chipset Family PCI Express Root Port #5 (rev d4)
### Group 13 ###
    00:1d.0 USB controller: Intel Corporation 8 Series/C220 Series Chipset Family USB EHCI #1 (rev 04)
### Group 14 ###
    00:1f.0 ISA bridge: Intel Corporation C226 Series Chipset Family Server Advanced SKU LPC Controller (rev 04)
    00:1f.2 SATA controller: Intel Corporation 8 Series/C220 Series Chipset Family 6-port SATA Controller 1 [AHCI mode] (rev 04)
    00:1f.3 SMBus: Intel Corporation 8 Series/C220 Series Chipset Family SMBus Controller (rev 04)
### Group 15 ###
    05:00.0 Ethernet controller: Intel Corporation I210 Gigabit Network Connection (rev 03)
### Group 16 ###
    06:00.0 Ethernet controller: Intel Corporation I210 Gigabit Network Connection (rev 03)
### Group 17 ###
    07:00.0 Serial Attached SCSI controller: LSI Logic / Symbios Logic SAS2008 PCI-Express Fusion-MPT SAS-2 [Falcon] (rev 03)

GRUB_CMDLINE_LINUX:

GRUB_CMDLINE_LINUX="cryptdevice=/dev/disk/by-uuid/442a2a16-e934-48d6-a043-ff00534c5dfe:sys vfio_iommu_type1.allow_unsafe_interrupts=1 intel_iommu=on pci-stub.ids=1002:67b0,1002:aac8 kvm_intel.emulate_invalid_guest_state=0 nohz=off"

Am I screwed?

Bronek

Member

From: London

Registered: 2014-02-14

Posts: 123

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Sigh... my system was working fine with a R9 290X, even after the new ACS patch (no downstream). But now I'm getting:

[  310.018977] dmar: DRHD: handling fault status reg 3
[  310.033576] dmar: DMAR:[DMA Read] Request device [01:00.0] fault addr 1e9fae0000 
[  310.033576] DMAR:[fault reason 06] PTE Read access is not set
[  310.218996] dmar: DRHD: handling fault status reg 3
[  310.233609] dmar: DMAR:[DMA Read] Request device [01:00.0] fault addr 1e9fae0000 
[  310.233609] DMAR:[fault reason 06] PTE Read access is not set
[  310.468989] dmar: DRHD: handling fault status reg 3
[  310.483594] dmar: DMAR:[DMA Read] Request device [01:00.0] fault addr 1e9fae0000 
[  310.483594] DMAR:[fault reason 06] PTE Read access is not set

1:00.0 is my R9 290X. Screen goes crazy once the AMD drivers load.

Am I screwed?

This is not your primary GPU now, is it? These things may change depending on your computer BIOS setup and PCIe slot location it is attached to.

Basically what I learned from reading aw posts above is that if you use pci-stub on a GPU device (as you should) this will prevent device driver from attaching to it, but will not prevent vgacon from using the same device. And vgacon is not smart enough to care about hypervisor arbitration etc, it will simply corrupt your device state when the GPU gets attached to VM.

So the trick here is either to ensure that some other card is primary on boot, or let the smarter driver (rather than vgacon) use the device prior to VM startup, by not assigning it to the pci-stub module on kernel boot.

At least, that's my understanding, but I'm very much beginner in such setup

Last edited by Bronek (2014-02-28 10:45:41)

aw

Member

Registered: 2013-10-04

Posts: 921

Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Sigh... my system was working fine with a R9 290X, even after the new ACS patch (no downstream). But now I'm getting:

[  310.018977] dmar: DRHD: handling fault status reg 3
[  310.033576] dmar: DMAR:[DMA Read] Request device [01:00.0] fault addr 1e9fae0000 
[  310.033576] DMAR:[fault reason 06] PTE Read access is not set
[  310.218996] dmar: DRHD: handling fault status reg 3
[  310.233609] dmar: DMAR:[DMA Read] Request device [01:00.0] fault addr 1e9fae0000 
[  310.233609] DMAR:[fault reason 06] PTE Read access is not set
[  310.468989] dmar: DRHD: handling fault status reg 3
[  310.483594] dmar: DMAR:[DMA Read] Request device [01:00.0] fault addr 1e9fae0000 
[  310.483594] DMAR:[fault reason 06] PTE Read access is not set

1:00.0 is my R9 290X. Screen goes crazy once the AMD drivers load.

Am I screwed?

This is not your primary GPU now, is it? These things may change depending on your computer BIOS setup and PCIe slot location it is attached to.

Basically what I learned from reading aw posts above is that if you use pci-stub on a GPU device (as you should) this will prevent device driver from attaching to it, but will not prevent vgacon from using the same device. And vgacon is not smart enough to care about hypervisor arbitration etc, it will simply corrupt your device state when the GPU gets attached to VM.

So the trick here is either to ensure that some other card is primary on boot, or let the smarter driver (rather than vgacon) use the device prior to VM startup, by not assigning it to the pci-stub module on kernel boot.

At least, that's my understanding, but I'm very much beginner in such setup

That's true, but I'm not sure it would cause this problem.  The VGA region is accessed by the CPU and therefore not subject to IOMMU translation.  The IOMMU only handles accesses initiated by the device.  So if vgacon is being used we'll get corrupted screens, as you indicate, but we won't get DMAR faults.

The DMAR faults indicate they are blocking the 8970 from reading memory just below 128GB.  It seems like a safe assumption that you don't have a 128GB VM, so the IOMMU is probably doing it's job by preventing these.  Does the card still work on bare metal?  It is possible that a failed memory controller on the card could be generating these accesses.  Does the address remotely resemble the host or guest address of the device?  (lspci in the host, info pci/info mtree in the qemu monitor)  If you haven't already, a hard power cycle might be a good idea.

---

http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

mbroemme

Member

From: Cologne

Registered: 2014-02-05

Posts: 40

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

The nvidia guys seem to have a problem with their setup: https://devtalk.nvidia.com/default/topi … 4/#4131164

We have a number on their internal bugtracker, so maybe we should try helping them with their issues, so they can reproduce it and fix it at some point.

I've told the guy at Nvidia already in a private message that I will follow-up next week as I was on a conference.

jonascj

Member

Registered: 2014-02-28

Posts: 20

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

I am new to passthrough with KVM and have read this thread from beginning to end with the intention of setting up a similar system but I have a some basic / "superficial" questions before I begin.

#1: How is the Windows guest visualized (what/where is its display)?
My only experience with "graphical access" to KVM guests is via VNC, but that will not do it for gaming. So how is the guest visualized / what is its display? As far as I can understand from the thread people simply have a monitor connected directly to the graphics card which is passed through to the guest, is this understood correctly? This directly connected monitor can then have multiple inputs which can be changed on the monitor itself or one could use a display switch box.

#2: Any alternatives to #1?
In the thread I see a couple of discussions about copying the frame buffer of one graphics card to another, but all of them end with the poster saying it is just speculation and that it should be theoretically possible. Anyone who have succeeded with getting the video output from the guest into a window on the host via whatever method?

#3: Are two powerful graphics cards needed?
Quite a few posters have two quite powerful graphics cards installed. Isn't one powerful graphics card enough (one to pass to the guest used for playing games) and a less powerful one for running the host displays?

#4: Is a dedicated sound card needed?
Do I need a dedicated sound card or can the host and guest share the onboard sound card?

#5: Requirements for CPU and mother board?
Is it (more or less) sufficient to select a CPU with Intel VT-d or AMD-Vi, and a matching mother board (supporting the CPU and Intel VT-d / AMD-Vi)?