Arch Linux

aw

Member

Registered: 2013-10-04

Posts: 921

Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

how do I fix that?

Enable the IOMMU!

---

http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

rabcor

Banned

Registered: 2013-02-09

Posts: 500

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

how do I fix that?

Enable the IOMMU!

Like this?

Kernel command line: BOOT_IMAGE=/vmlinuz-linux root=UUID=298f5fd5-65b3-4f84-9de7-046660a66597 rw quiet intel_iommu=1 pci-stub.ids=10de:1189,10de:0e0a

That isn't enough it would seem.
What part did I miss?

Last edited by rabcor (2014-04-28 21:56:07)

---

Making Logitech G510 Work on Linux!

aw

Member

Registered: 2013-10-04

Posts: 921

Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

how do I fix that?

Enable the IOMMU!

Like this?

Kernel command line: BOOT_IMAGE=/vmlinuz-linux root=UUID=298f5fd5-65b3-4f84-9de7-046660a66597 rw quiet intel_iommu=1 pci-stub.ids=10de:1189,10de:0e0a

That isn't enough it would seem.
What part did I miss?

intel_iommu=on and make sure it's enabled in your BIOS and supported by your processor (ark.intel.com)

---

http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

rabcor

Banned

Registered: 2013-02-09

Posts: 500

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Thanks! it was the "intel_iommu=on" part instead of 1 however:

qemu-system-x86_64: -device vfio-pci,host=02:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on: vfio: error, group 5 is not viable, please ensure all devices within the iommu_group are bound to their vfio bus driver.
qemu-system-x86_64: -device vfio-pci,host=02:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on: vfio: failed to get group 5
qemu-system-x86_64: -device vfio-pci,host=02:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on: Device initialization failed.
qemu-system-x86_64: -device vfio-pci,host=02:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on: Device 'vfio-pci' could not be initialized

I read a post (by you) saying I need the ACS override patch. I installed the linux mainline kernel from the OP which should have this patch, but still no luck. I am also using the nouveau drivers as is rather than the proprietary drivers. So it's not that I need to patch my gpu drivers I would think.

Last edited by rabcor (2014-04-28 23:13:09)

---

Making Logitech G510 Work on Linux!

aw

Member

Registered: 2013-10-04

Posts: 921

Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Thanks! it was the "intel_iommu=on" part instead of 1 however:

qemu-system-x86_64: -device vfio-pci,host=02:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on: vfio: error, group 5 is not viable, please ensure all devices within the iommu_group are bound to their vfio bus driver.
qemu-system-x86_64: -device vfio-pci,host=02:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on: vfio: failed to get group 5
qemu-system-x86_64: -device vfio-pci,host=02:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on: Device initialization failed.
qemu-system-x86_64: -device vfio-pci,host=02:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on: Device 'vfio-pci' could not be initialized

I read a post (by you) saying I need the ACS override patch. I installed the linux mainline kernel from the OP which should have this patch, but still no luck. I am also using the nouveau drivers as is rather than the proprietary drivers. So it's not that I need to patch my gpu drivers I would think.

And notice how the patch in that link requires kernel command line options to enable:

--- a/Documentation/kernel-parameters.txt
+++ b/Documentation/kernel-parameters.txt
@@ -2349,6 +2349,16 @@ bytes respectively. Such letter suffixes can also be entirely omitted.
 		nomsi	Do not use MSI for native PCIe PME signaling (this makes
 			all PCIe root ports use INTx for all services).
 
+	pcie_acs_override =
+			[PCIE] Override missing PCIe ACS support for:
+		downstream
+			All downstream ports - full ACS capabilties
+		multifunction
+			All multifunction devices - multifunction ACS subset
+		id:nnnn:nnnn
+			Specfic device - full ACS capabilities
+			Specified as vid:did (vendor/device ID) in hex
+

---

http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

rabcor

Banned

Registered: 2013-02-09

Posts: 500

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Ah, so essentially add "pcie_acs_override=downstream" to command line?  this seems to have worked, thanks

Qemu now opens as a blank window. However, the passthroughed card is giving no signals. (Monitor says "No Video Input")

---

Making Logitech G510 Work on Linux!

myweb

Member

Registered: 2013-07-13

Posts: 69

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Dear All,

Is it possible to fix:

...
Apr 29 07:11:48 dhcppc5 kernel: dmar: DRHD: handling fault status reg 3
Apr 29 07:11:48 dhcppc5 kernel: dmar: DMAR:[DMA Write] Request device [04:00.0] fault addr 7ff96000
                                DMAR:[fault reason 02] Present bit in context entry is clear
Apr 29 07:11:48 dhcppc5 kernel: dmar: DRHD: handling fault status reg 3
Apr 29 07:11:48 dhcppc5 kernel: dmar: DMAR:[DMA Write] Request device [04:00.0] fault addr 7ff96000
                                DMAR:[fault reason 02] Present bit in context entry is clear
...

The root of the issue is DVB card

04:02.0 Multimedia controller [0480]: Philips Semiconductors SAA7146 [1131:7146] (rev 01)

PCI tree:

-[0000:00]-+-00.0  Intel Corporation Xeon E3-1200 v2/3rd Gen Core processor DRAM Controller
           +-01.0-[01]--+-00.0  NVIDIA Corporation GK107 [GeForce GT 640]
           |            \-00.1  NVIDIA Corporation GK107 HDMI Audio Controller
           +-02.0  Intel Corporation Xeon E3-1200 v2/3rd Gen Core processor Graphics Controller
           +-14.0  Intel Corporation 7 Series/C210 Series Chipset Family USB xHCI Host Controller
           +-16.0  Intel Corporation 7 Series/C210 Series Chipset Family MEI Controller #1
           +-1a.0  Intel Corporation 7 Series/C210 Series Chipset Family USB Enhanced Host Controller #2
           +-1b.0  Intel Corporation 7 Series/C210 Series Chipset Family High Definition Audio Controller
           +-1c.0-[02]--+-00.0  Advanced Micro Devices, Inc. [AMD/ATI] Cape Verde PRO [Radeon HD 7750]
           |            \-00.1  Advanced Micro Devices, Inc. [AMD/ATI] Cape Verde/Pitcairn HDMI Audio [Radeon HD 7700/7800 Series]
           +-1c.4-[03-04]----00.0-[04]----02.0  Philips Semiconductors SAA7146

The messages aboves occurs if DVB card is passed to VM via:

-device vfio-pci,host=03:00.0,bus=pcie.0 \
-device vfio-pci,host=04:02.0,bus=pcie.0 \

If try to use DVB card on the host - host freezes.
Could you please help me to find a way how to use a dvb card (in VM or in Host - it does not matter)
Thank you in advance.

aw

Member

Registered: 2013-10-04

Posts: 921

Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Dear All,

Is it possible to fix:

...
Apr 29 07:11:48 dhcppc5 kernel: dmar: DRHD: handling fault status reg 3
Apr 29 07:11:48 dhcppc5 kernel: dmar: DMAR:[DMA Write] Request device [04:00.0] fault addr 7ff96000
                                DMAR:[fault reason 02] Present bit in context entry is clear
Apr 29 07:11:48 dhcppc5 kernel: dmar: DRHD: handling fault status reg 3
Apr 29 07:11:48 dhcppc5 kernel: dmar: DMAR:[DMA Write] Request device [04:00.0] fault addr 7ff96000
                                DMAR:[fault reason 02] Present bit in context entry is clear
...

The root of the issue is DVB card

04:02.0 Multimedia controller [0480]: Philips Semiconductors SAA7146 [1131:7146] (rev 01)

PCI tree:

-[0000:00]-+-00.0  Intel Corporation Xeon E3-1200 v2/3rd Gen Core processor DRAM Controller
           +-01.0-[01]--+-00.0  NVIDIA Corporation GK107 [GeForce GT 640]
           |            \-00.1  NVIDIA Corporation GK107 HDMI Audio Controller
           +-02.0  Intel Corporation Xeon E3-1200 v2/3rd Gen Core processor Graphics Controller
           +-14.0  Intel Corporation 7 Series/C210 Series Chipset Family USB xHCI Host Controller
           +-16.0  Intel Corporation 7 Series/C210 Series Chipset Family MEI Controller #1
           +-1a.0  Intel Corporation 7 Series/C210 Series Chipset Family USB Enhanced Host Controller #2
           +-1b.0  Intel Corporation 7 Series/C210 Series Chipset Family High Definition Audio Controller
           +-1c.0-[02]--+-00.0  Advanced Micro Devices, Inc. [AMD/ATI] Cape Verde PRO [Radeon HD 7750]
           |            \-00.1  Advanced Micro Devices, Inc. [AMD/ATI] Cape Verde/Pitcairn HDMI Audio [Radeon HD 7700/7800 Series]
           +-1c.4-[03-04]----00.0-[04]----02.0  Philips Semiconductors SAA7146

The messages aboves occurs if DVB card is passed to VM via:

-device vfio-pci,host=03:00.0,bus=pcie.0 \
-device vfio-pci,host=04:02.0,bus=pcie.0 \

If try to use DVB card on the host - host freezes.
Could you please help me to find a way how to use a dvb card (in VM or in Host - it does not matter)
Thank you in advance.

What's the bridge device at 3:00.0?  lspci -vvv -s 3:00.0

EDIT: Oh, you're passing 3:00.0 to the guest, never pass the bridge, this shouldn't even be allowed

Last edited by aw (2014-04-29 01:22:49)

---

http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

myweb

Member

Registered: 2013-07-13

Posts: 69

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

What's the bridge device at 3:00.0?  lspci -vvv -s 3:00.0

03:00.0 PCI bridge: ASMedia Technology Inc. ASM1083/1085 PCIe to PCI Bridge (rev 03) (prog-if 01 [Subtractive decode])
        Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
        Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
        Latency: 0, Cache Line Size: 64 bytes
        Bus: primary=03, secondary=04, subordinate=04, sec-latency=32
        I/O behind bridge: 0000f000-00000fff
        Memory behind bridge: f7900000-f79fffff
        Prefetchable memory behind bridge: 00000000fff00000-00000000000fffff
        Secondary status: 66MHz+ FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort+ <SERR- <PERR-
        BridgeCtl: Parity- SERR- NoISA- VGA- MAbort- >Reset- FastB2B-
                PriDiscTmr- SecDiscTmr- DiscTmrStat- DiscTmrSERREn-
        Capabilities: [c0] Subsystem: ASRock Incorporation Motherboard

rabcor

Banned

Registered: 2013-02-09

Posts: 500

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

I'm seeing someone had a similar problem to my own with unpatched nvidia drivers, I want to take a shot at patching the nvidia drivers instead of using nouveau, but I have no idea how to actually apply the patch to the drivers, can somebody point me in the right direction please?

Last edited by rabcor (2014-04-29 03:28:18)

---

Making Logitech G510 Work on Linux!

andy123

Member

Registered: 2011-11-04

Posts: 169

Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

I'm seeing someone had a similar problem to my own with unpatched nvidia drivers, I want to take a shot at patching the nvidia drivers instead of using nouveau, but I have no idea how to actually apply the patch to the drivers, can somebody point me in the right direction please?

I think I have a old gist somewhere with a pkgbuild… lets see. Here you go!.

---

i'm sorry for my poor english wirting skills…

rabcor

Banned

Registered: 2013-02-09

Posts: 500

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

So, download nvidia-dkms.install from here, and update pkgver to 337.12 and md5sum to 'c610d5bff8f7f758dc25fdd51b1b65eb' right? Thanks would be nice to have something similar setup for the kernel ACS patch, too bad I don't know how to set that up.

Last edited by rabcor (2014-04-29 04:30:53)

---

Making Logitech G510 Work on Linux!

adejour

Member

Registered: 2014-04-29

Posts: 6

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Hello,

I have the same error as gneville ( https://bbs.archlinux.org/viewtopic.php … 8#p1403248 ).
VGA Passthrough works when I launch windows but if I reboot or stop and then launch again, I have this message&BSOD (PAGE_FAULT_IN_NONPAGED_AREA). It is just after the loading of windows (black background) and just before the launch of the desktop. It is exactly at this moment that the screen resolution must change.

Résumé:
- If I launch windows with my VM then stop windows and the VM, then reboot the host, and launch the VM again, it works.
- If I launch windows with my VM then stop windows and the VM, then launch the VM again, there is this BSOD.

Any Idea?
have you find a solution gneville?

My last tried configuration is :

MB : ASUS H87M-PLUS
GPU : Radeon R7 260X
RAM : 8Go
CPU : i5 4570
Qemu : http://wiki.qemu-project.org/download/q … .0.tar.bz2
Kernel : https://www.kernel.org/pub/linux/kernel … 4.1.tar.xz
patch -Np1 -i override_for_missing_acs_capabilities.patch
patch -Np1 -i i915_314.patch

/usr/local/bin/qemu-system-x86_64 \
-enable-kvm -M q35 -m 4G -cpu host \
-balloon none  -rtc clock=host  -serial null  -parallel null  -monitor none  -display none \
-smp 4,sockets=1,cores=4,threads=1 \
-bios /usr/share/qemu/bios.bin -vga none \
-drive file=/home/ja/VM/windows2.img,id=disk,format=raw -device ide-hd,bus=ide.0,drive=disk \
-net nic -net tap,ifname=tap0,script=no,downscript=no \
-device ioh3420,bus=pcie.0,addr=1c.0,multifunction=on,port=1,chassis=1,id=root.1 \
-device vfio-pci,host=01:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on

I've just found something interesting with my setup.

I can get around the BSOD on Windows boot after installing the drivers, providing I do a fresh reload of the Host. I also had to remove the HDMI Audio from the ioh3420 device.

Edit: Confirmed this is the same for Win7 & Win8 as a Guest

But provided I reboot the Host everytime I boot the Windows Guest with the below qemu commands it will start up and use the AMD drivers correctly, if I then reboot the Guest or Shutdown and restart the Qemu again from Command line then Windows will BSOD on boot with the "PAGE_FAULT_IN_NONPAGED_AREA" error.

Any ideas what can be causing this please?

/usr/local/qemu-xen-4-4-git-patched/bin/qemu-system-x86_64 -name win8 -enable-kvm -M q35 -m 6144  \
-cpu Haswell,hv-time,hv_relaxed,hv_vapic,hv_spinlocks=0x1000 \
-boot order=dc \
-smp 2,sockets=1,cores=2,threads=1 \
-bios /home/xen/vfio/bios.bin -vga none \
-usb -usbdevice tablet \
-spice port=5902,disable-ticketing \
-vnc 0.0.0.0:0 \
-net nic \
-net tap,ifname=tap0,script=no,downscript=no \
-device ioh3420,bus=pcie.0,addr=1c.0,multifunction=on,port=1,chassis=1,id=root.1 \
-device vfio-pci,host=01:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on \
-device vfio-pci,host=01:00.1,bus=pcie.0 \
-device vfio-pci,host=00:14.0,bus=pcie.0 \
-device ahci,bus=pcie.0,id=ahci \
-drive file=/dev/mapper/vg1-winv1,media=disk,index=1,id=disk,cache=none,if=virtio \
-drive file=/dev/mapper/vg1-winv1data,media=disk,index=2,id=diskb,cache=none,if=virtio \
-drive file=/home/xen/iso/windows/en-gb_windows_8_1_enterprise_x64_dvd_2971910.iso,boot=on,media=cdrom,index=3,id=isocd  \
-device ide-cd,bus=ahci.1,drive=isocd \
-drive file=/root/virtio-win-0.1-74.iso,media=cdrom,index=4,id=virtio  \
-device ide-cd,bus=ahci.2,drive=virtio

Still not having much luck with vfio vga passthrough on my AMD R9 290

I've tried pretty much all the suggestions I've seen on here but no matter what I try the guest always crashes after installing video drivers.

I did swap from Win8 to Win7 and noticed that after installing the video drivers and rebooting when windows boots up it crashes with a BSOD saying

"PAGE_FAULT_IN_NONPAGED_AREA"

I've tried the following:
multiple linux kernels all with the patches rolled in.
multiple Qemu versions (1.7.0, 1.7.1, 2.0.0RC0,1 &2) and have applied the vga reset patches
Tried both machine types of q35 and i440fx, of which both successfully passthrough the vga card but crash after installing the drivers and rebooting windows
Tried just passing through the Video card leaving out the HDMI Audio.
Tried installing drivers without installing the Catalyst Control Centre as suggested by some other forums.
Tried both AMD drivers and Windows Provided drivers via Windows Updates.
Tried using RomFiles

I'm guessing the problem lies within Qemu somewhere.

On top of this I have just installed the OP linux-mainline (3.13.6-1-mainline),seabios (1.7.4.r1788.g8abc6a8-1) & qemu (2.0.r31714.g03d5142-1) versions (Just added reset patch and manually compiled (https://lists.gnu.org/archive/html/qemu … 0767.html))

and still no change

gneville

Member

Registered: 2014-04-02

Posts: 27

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Hi adejour,

Yes I managed to fix the bsods on reboots by following the instructions this guy had in his blog http://blog.ktz.me/?p=219 it works perfectly now.

Hello,

I have the same error as gneville ( https://bbs.archlinux.org/viewtopic.php … 8#p1403248 ).
VGA Passthrough works when I launch windows but if I reboot or stop and then launch again, I have this message&BSOD (PAGE_FAULT_IN_NONPAGED_AREA). It is just after the loading of windows (black background) and just before the launch of the desktop. It is exactly at this moment that the screen resolution must change.

Résumé:
- If I launch windows with my VM then stop windows and the VM, then reboot the host, and launch the VM again, it works.
- If I launch windows with my VM then stop windows and the VM, then launch the VM again, there is this BSOD.

Any Idea?
have you find a solution gneville?

My last tried configuration is :

MB : ASUS H87M-PLUS
GPU : Radeon R7 260X
RAM : 8Go
CPU : i5 4570
Qemu : http://wiki.qemu-project.org/download/q … .0.tar.bz2
Kernel : https://www.kernel.org/pub/linux/kernel … 4.1.tar.xz
patch -Np1 -i override_for_missing_acs_capabilities.patch
patch -Np1 -i i915_314.patch

/usr/local/bin/qemu-system-x86_64 \
-enable-kvm -M q35 -m 4G -cpu host \
-balloon none  -rtc clock=host  -serial null  -parallel null  -monitor none  -display none \
-smp 4,sockets=1,cores=4,threads=1 \
-bios /usr/share/qemu/bios.bin -vga none \
-drive file=/home/ja/VM/windows2.img,id=disk,format=raw -device ide-hd,bus=ide.0,drive=disk \
-net nic -net tap,ifname=tap0,script=no,downscript=no \
-device ioh3420,bus=pcie.0,addr=1c.0,multifunction=on,port=1,chassis=1,id=root.1 \
-device vfio-pci,host=01:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on

I've just found something interesting with my setup.

I can get around the BSOD on Windows boot after installing the drivers, providing I do a fresh reload of the Host. I also had to remove the HDMI Audio from the ioh3420 device.

Edit: Confirmed this is the same for Win7 & Win8 as a Guest

But provided I reboot the Host everytime I boot the Windows Guest with the below qemu commands it will start up and use the AMD drivers correctly, if I then reboot the Guest or Shutdown and restart the Qemu again from Command line then Windows will BSOD on boot with the "PAGE_FAULT_IN_NONPAGED_AREA" error.

Any ideas what can be causing this please?

/usr/local/qemu-xen-4-4-git-patched/bin/qemu-system-x86_64 -name win8 -enable-kvm -M q35 -m 6144  \
-cpu Haswell,hv-time,hv_relaxed,hv_vapic,hv_spinlocks=0x1000 \
-boot order=dc \
-smp 2,sockets=1,cores=2,threads=1 \
-bios /home/xen/vfio/bios.bin -vga none \
-usb -usbdevice tablet \
-spice port=5902,disable-ticketing \
-vnc 0.0.0.0:0 \
-net nic \
-net tap,ifname=tap0,script=no,downscript=no \
-device ioh3420,bus=pcie.0,addr=1c.0,multifunction=on,port=1,chassis=1,id=root.1 \
-device vfio-pci,host=01:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on \
-device vfio-pci,host=01:00.1,bus=pcie.0 \
-device vfio-pci,host=00:14.0,bus=pcie.0 \
-device ahci,bus=pcie.0,id=ahci \
-drive file=/dev/mapper/vg1-winv1,media=disk,index=1,id=disk,cache=none,if=virtio \
-drive file=/dev/mapper/vg1-winv1data,media=disk,index=2,id=diskb,cache=none,if=virtio \
-drive file=/home/xen/iso/windows/en-gb_windows_8_1_enterprise_x64_dvd_2971910.iso,boot=on,media=cdrom,index=3,id=isocd  \
-device ide-cd,bus=ahci.1,drive=isocd \
-drive file=/root/virtio-win-0.1-74.iso,media=cdrom,index=4,id=virtio  \
-device ide-cd,bus=ahci.2,drive=virtio

Still not having much luck with vfio vga passthrough on my AMD R9 290

I've tried pretty much all the suggestions I've seen on here but no matter what I try the guest always crashes after installing video drivers.

I did swap from Win8 to Win7 and noticed that after installing the video drivers and rebooting when windows boots up it crashes with a BSOD saying

"PAGE_FAULT_IN_NONPAGED_AREA"

I've tried the following:
multiple linux kernels all with the patches rolled in.
multiple Qemu versions (1.7.0, 1.7.1, 2.0.0RC0,1 &2) and have applied the vga reset patches
Tried both machine types of q35 and i440fx, of which both successfully passthrough the vga card but crash after installing the drivers and rebooting windows
Tried just passing through the Video card leaving out the HDMI Audio.
Tried installing drivers without installing the Catalyst Control Centre as suggested by some other forums.
Tried both AMD drivers and Windows Provided drivers via Windows Updates.
Tried using RomFiles

I'm guessing the problem lies within Qemu somewhere.

myweb

Member

Registered: 2013-07-13

Posts: 69

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

What's the bridge device at 3:00.0?  lspci -vvv -s 3:00.0

03:00.0 PCI bridge: ASMedia Technology Inc. ASM1083/1085 PCIe to PCI Bridge (rev 03) (prog-if 01 [Subtractive decode])
        Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
        Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
        Latency: 0, Cache Line Size: 64 bytes
        Bus: primary=03, secondary=04, subordinate=04, sec-latency=32
        I/O behind bridge: 0000f000-00000fff
        Memory behind bridge: f7900000-f79fffff
        Prefetchable memory behind bridge: 00000000fff00000-00000000000fffff
        Secondary status: 66MHz+ FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort+ <SERR- <PERR-
        BridgeCtl: Parity- SERR- NoISA- VGA- MAbort- >Reset- FastB2B-
                PriDiscTmr- SecDiscTmr- DiscTmrStat- DiscTmrSERREn-
        Capabilities: [c0] Subsystem: ASRock Incorporation Motherboard

Alex,
Is it correct that ASMedia pci bridge is root cause of "DMAR:[fault reason 02] Present bit in context entry is clear" issue?
Is it correct that it is a known issue and there is no way to fix it and it's impossible to use any PCI devices while IOMMU is turned on?

Thank you!

adejour

Member

Registered: 2014-04-29

Posts: 6

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Hi adejour,

Yes I managed to fix the bsods on reboots by following the instructions this guy had in his blog http://blog.ktz.me/?p=219 it works perfectly now.

Hello,

I have the same error as gneville ( https://bbs.archlinux.org/viewtopic.php … 8#p1403248 ).
VGA Passthrough works when I launch windows but if I reboot or stop and then launch again, I have this message&BSOD (PAGE_FAULT_IN_NONPAGED_AREA). It is just after the loading of windows (black background) and just before the launch of the desktop. It is exactly at this moment that the screen resolution must change.

Résumé:
- If I launch windows with my VM then stop windows and the VM, then reboot the host, and launch the VM again, it works.
- If I launch windows with my VM then stop windows and the VM, then launch the VM again, there is this BSOD.

Ok, it seems to be a good software workaround and I will try that. Thank you very much !

Anyway it doesn't explain the real problem. When the VM shutdown, the VM turns off but the screen still display the last image (windows blue background with "windows is shuting down" (or something like that, my windows is not in english)).
=> qemu doesn't switch off my GPU, and the screen stays on !

Do you know if it is a pbroblem with qemu? Or may be with the bios used with qemu?
Any idea to fix this ?!

Last edited by adejour (2014-04-30 10:02:21)

aw

Member

Registered: 2013-10-04

Posts: 921

Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

What's the bridge device at 3:00.0?  lspci -vvv -s 3:00.0

03:00.0 PCI bridge: ASMedia Technology Inc. ASM1083/1085 PCIe to PCI Bridge (rev 03) (prog-if 01 [Subtractive decode])
        Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
        Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
        Latency: 0, Cache Line Size: 64 bytes
        Bus: primary=03, secondary=04, subordinate=04, sec-latency=32
        I/O behind bridge: 0000f000-00000fff
        Memory behind bridge: f7900000-f79fffff
        Prefetchable memory behind bridge: 00000000fff00000-00000000000fffff
        Secondary status: 66MHz+ FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort+ <SERR- <PERR-
        BridgeCtl: Parity- SERR- NoISA- VGA- MAbort- >Reset- FastB2B-
                PriDiscTmr- SecDiscTmr- DiscTmrStat- DiscTmrSERREn-
        Capabilities: [c0] Subsystem: ASRock Incorporation Motherboard

Alex,
Is it correct that ASMedia pci bridge is root cause of "DMAR:[fault reason 02] Present bit in context entry is clear" issue?
Is it correct that it is a known issue and there is no way to fix it and it's impossible to use any PCI devices while IOMMU is turned on?

Thank you!

This particular bridge likely produces a WARN_ON when booting the kernel on a VT-d system because it lacks a PCIe capability (in violation of the PCIe spec).  As a result, I doubt that the IOMMU tables are getting setup correct for devices behind the bridge.  I've actually been working on trying to fix this for the last couple weeks.  In any case, assigning the bridge to the guest is always wrong, so I'd start with that.  I'll hopefully be posting patches to fix the situation with this bridge this week or next.

---

http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

adejour

Member

Registered: 2014-04-29

Posts: 6

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Gneville, this guy had exactly the same pb reported to a qemu developper : https://lists.gnu.org/archive/html/qemu … 00905.html

Patch the vfio.c file was the solution for him. He has the R9 290X and I'm using the R7 260X. I'll try.

I looked on the code, and it is may be linked to the Power Management Flag NoSoftRst (when you run lspci -vvv).
Can somebody who succeed in multiple reboots tell we what is the result of this command for him?

this part :

Capabilities: [70] Power Management version 3
                Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0-,D1-,D2-,D3hot+,D3cold-)
                Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=0 PME-

Thanks!

Ok, it seems to be a good software workaround and I will try that. Thank you very much !

Anyway it doesn't explain the real problem. When the VM shutdown, the VM turns off but the screen still display the last image (windows blue background with "windows is shuting down" (or something like that, my windows is not in english)).
=> qemu doesn't switch off my GPU, and the screen stays on !

Do you know if it is a pbroblem with qemu? Or may be with the bios used with qemu?
Any idea to fix this ?!

Hi adejour,

Yes I managed to fix the bsods on reboots by following the instructions this guy had in his blog http://blog.ktz.me/?p=219 it works perfectly now.

Hello,

I have the same error as gneville ( https://bbs.archlinux.org/viewtopic.php … 8#p1403248 ).
VGA Passthrough works when I launch windows but if I reboot or stop and then launch again, I have this message&BSOD (PAGE_FAULT_IN_NONPAGED_AREA). It is just after the loading of windows (black background) and just before the launch of the desktop. It is exactly at this moment that the screen resolution must change.

Résumé:
- If I launch windows with my VM then stop windows and the VM, then reboot the host, and launch the VM again, it works.
- If I launch windows with my VM then stop windows and the VM, then launch the VM again, there is this BSOD.

gneville

Member

Registered: 2014-04-02

Posts: 27

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

adejour, I did spot that a while back and tried it but it didn't fix the BSOD issue when you reboot a guest or shutdown the guest and power it back on. I do have it applied to my current version of Qemu I'm using.

From memory however, I believe it does kill the video output when the guest is killed so it may fix your issue there.

edit: FWIW this is my output of Power Management from "lspci -vvv" from my R290, with the guest having the card passed through running:

01:00.0 VGA compatible controller: Advanced Micro Devices, Inc. [AMD/ATI] Hawaii PRO (prog-if 00 [VGA controller])
        Capabilities: [50] Power Management version 3
                Flags: PMEClk- DSI- D1+ D2+ AuxCurrent=0mA PME(D0-,D1+,D2+,D3hot+,D3cold-)
                Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=0 PME-

Cheers

Gneville, this guy had exactly the same pb reported to a qemu developper : https://lists.gnu.org/archive/html/qemu … 00905.html

Patch the vfio.c file was the solution for him. He has the R9 290X and I'm using the R7 260X. I'll try.

I looked on the code, and it is may be linked to the Power Management Flag NoSoftRst (when you run lspci -vvv).
Can somebody who succeed in multiple reboots tell we what is the result of this command for him?

this part :

Capabilities: [70] Power Management version 3
                Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0-,D1-,D2-,D3hot+,D3cold-)
                Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=0 PME-

Thanks!

Last edited by gneville (2014-04-30 14:47:38)

apex8

Member

Registered: 2014-03-29

Posts: 60

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Hey together,

there`s something that bothers me and I hope somebody can give me an explaination for it:
My mainboard is a Asrock Z68 Pro3 Gen3.

lspci | grep USB                                                                                                                                                       ⏎
00:1a.0 USB controller: Intel Corporation 6 Series/C200 Series Chipset Family USB Enhanced Host Controller #2 (rev 05)
00:1d.0 USB controller: Intel Corporation 6 Series/C200 Series Chipset Family USB Enhanced Host Controller #1 (rev 05)
07:00.0 USB controller: Etron Technology, Inc. EJ168 USB 3.0 Host Controller (rev 01)

As you can see I have two usb 2.0 controllers. Initially I planned using a KVM switch to switch my keyboard/mouse between my VM an the host. So I added 00:1a.0 to pci-stub and binded it to vfio-pci, but I was still able to use the related ports on the host whenever I clicked the button on the KVM switch. Of course passthrough of 00:1a.0 to the VM didn't work either. Keyboard and mouse simply was usable on the host AND the VM (using parameter vga=std)

So I would like to know, if there is a reason why this controller behaves in this strange way. In the end, are these two controllers maybe the same device?
Anyway, I found Synergy being totally sufficient, but that issue still bugs me

BR
apex

maduraa

Member

Registered: 2014-05-01

Posts: 4

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Hello,

I followed the instructions with a stock kernel (3.14.1). When I run qemu(for testing using the commands given in the first post) I get no output from the passed through vga (nVidia 750GTX). But I do get a blank screen on qemu on the main display, I do not own another monitor so I ran the command and plugged the cable to the nVidia card to check whether it has the BIOS output as mentioned but that port is not on the monitor goes to standby when I plug it. I am using IGP as my primary VGA.

# ls -l /sys/bus/pci/drivers/vfio-pci/
total 0
lrwxrwxrwx 1 root root    0 May  1 15:31 0000:01:00.0 -> ../../../../devices/pci0000:00/0000:00:01.0/0000:01:00.0/
lrwxrwxrwx 1 root root    0 May  1 15:31 0000:01:00.1 -> ../../../../devices/pci0000:00/0000:00:01.0/0000:01:00.1/
--w------- 1 root root 4096 May  1 15:31 bind
lrwxrwxrwx 1 root root    0 May  1 15:31 module -> ../../../../module/vfio_pci/
--w------- 1 root root 4096 May  1 14:54 new_id
--w------- 1 root root 4096 May  1 15:31 remove_id
--w------- 1 root root 4096 May  1 14:54 uevent
--w------- 1 root root 4096 May  1 15:31 unbind

I installed Windows 7 on a VM and Windows is able to detect the card, I installed the drivers and then it got disabled by Windows(it says the devive didn't function as expected so it's disabled (code:43))

What can be the problem?

Thanks

Last edited by maduraa (2014-05-01 10:54:25)

maduraa

Member

Registered: 2014-05-01

Posts: 4

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

I've been trying adding acs to my config but still had no luck my guest works only as a basic video card, I get a code 43 in windows. I am gonna try a linux guest to get traces. When I try to load a vbios with the romfile option my guest has no outputs at all. Is someone has an idea of what else could I try ? My gtx 660 is not UEFI compatible and I start my computer in UEFI mode, could it be a problem ?

Are you using IGP and nvidia? I saw in an earlier post that the stock kernel wouldn't work this way, I'm currently building the linux-mainline kernel will be back with updates. I get the same result with stock kernel see above post by me.

EDIT: I also tried the given kernel, seabios-git, qemu-git I get nothing on the passthru'd VGA's output
EDIT2: I also tried with another monitor, I can confirm that the passthru'd card doesn't display anything.

Last edited by maduraa (2014-05-01 14:22:50)

adejour

Member

Registered: 2014-04-29

Posts: 6

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Hello,

I had the same problem.

If you have xHCI enabled in you BIOS, this controler is used for both usb 3 and usb 2 ports.
If you disable xHCI / USB3 in the BIOS, you can send one controler as you did before,
else you have to send the USB 3 conroler to the VM

I think you can also send the "device" with "-usb" as explained in the first post of this thread.

Hey together,

there`s something that bothers me and I hope somebody can give me an explaination for it:
My mainboard is a Asrock Z68 Pro3 Gen3.

lspci | grep USB                                                                                                                                                       ⏎
00:1a.0 USB controller: Intel Corporation 6 Series/C200 Series Chipset Family USB Enhanced Host Controller #2 (rev 05)
00:1d.0 USB controller: Intel Corporation 6 Series/C200 Series Chipset Family USB Enhanced Host Controller #1 (rev 05)
07:00.0 USB controller: Etron Technology, Inc. EJ168 USB 3.0 Host Controller (rev 01)

As you can see I have two usb 2.0 controllers. Initially I planned using a KVM switch to switch my keyboard/mouse between my VM an the host. So I added 00:1a.0 to pci-stub and binded it to vfio-pci, but I was still able to use the related ports on the host whenever I clicked the button on the KVM switch. Of course passthrough of 00:1a.0 to the VM didn't work either. Keyboard and mouse simply was usable on the host AND the VM (using parameter vga=std)

So I would like to know, if there is a reason why this controller behaves in this strange way. In the end, are these two controllers maybe the same device?
Anyway, I found Synergy being totally sufficient, but that issue still bugs me

BR
apex

myweb

Member

Registered: 2013-07-13

Posts: 69

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

This particular bridge likely produces a WARN_ON when booting the kernel on a VT-d system because it lacks a PCIe capability (in violation of the PCIe spec).  As a result, I doubt that the IOMMU tables are getting setup correct for devices behind the bridge.  I've actually been working on trying to fix this for the last couple weeks.  In any case, assigning the bridge to the guest is always wrong, so I'd start with that.  I'll hopefully be posting patches to fix the situation with this bridge this week or next.

This is amazing news!
Could you please specify where are you going to upload/write info about fix (twitter, github, ... - how to follow news)?

Dear All,
Could you please help to setup VM via libvirt? I have created the following xml:

<!--
WARNING: THIS IS AN AUTO-GENERATED FILE. CHANGES TO IT ARE LIKELY TO BE
OVERWRITTEN AND LOST. Changes to this xml configuration should be made using:
  virsh edit X
or other application using the libvirt API.
-->

<domain type='kvm'>
  <name>X</name>
  <uuid>9efa8a71-b193-c50b-c770-9ce6dad02390</uuid>
  <memory unit='KiB'>4194304</memory>
  <currentMemory unit='KiB'>4194304</currentMemory>
  <vcpu placement='static'>2</vcpu>
  <os>
    <type arch='x86_64' machine='pc-q35-2.0'>hvm</type>
    <boot dev='hd'/>
  </os>
  <features>
    <acpi/>
    <apic/>
    <pae/>
  </features>
  <clock offset='utc'/>
  <on_poweroff>destroy</on_poweroff>
  <on_reboot>restart</on_reboot>
  <on_crash>restart</on_crash>
  <devices>
    <emulator>/usr/sbin/qemu-system-x86_64</emulator>
    <disk type='block' device='disk'>
      <driver name='qemu' type='raw'/>
      <source dev='/dev/md1'/>
      <target dev='sda' bus='sata'/>
      <address type='drive' controller='0' bus='0' target='0' unit='0'/>
    </disk>
    <controller type='pci' index='0' model='pcie-root'/>
    <controller type='pci' index='1' model='dmi-to-pci-bridge'>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x1e' function='0x0'/>
    </controller>
    <controller type='pci' index='2' model='pci-bridge'>
      <address type='pci' domain='0x0000' bus='0x01' slot='0x01' function='0x0'/>
    </controller>
    <controller type='sata' index='0'>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x1f' function='0x2'/>
    </controller>
    <controller type='ide' index='0'/>
    <interface type='bridge'>
      <mac address='52:54:00:4b:be:6d'/>
      <source bridge='br0'/>
      <model type='virtio'/>
      <address type='pci' domain='0x0000' bus='0x02' slot='0x03' function='0x0'/>
    </interface>
    <memballoon model='virtio'>
      <address type='pci' domain='0x0000' bus='0x02' slot='0x05' function='0x0'/>
    </memballoon>
  </devices>
  <qemu:commandline>
        <qemu:arg value='-device'/>
        <qemu:arg value='ioh3420,bus=pcie.0,addr=1c.0,multifunction=on,port=1,chassis=1,id=root.1'/>
        <qemu:arg value='-device'/>
        <qemu:arg value='vfio-pci,host=01:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on'/>
        <qemu:arg value='-device'/>
        <qemu:arg value='vfio-pci,host=01:00.1,bus=pcie.0'/>
        <qemu:arg value='-device'/>
        <qemu:arg value='vfio-pci,host=00:1a.0,bus=pcie.0'/>
        <qemu:arg value='-device'/>
        <qemu:arg value='vfio-pci,host=00:1b.0,bus=pcie.0'/>
  </qemu:commandline>
</domain>

VM started, but there is no VGA.

I use libvirt-git.

aw

Member

Registered: 2013-10-04

Posts: 921

Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

This particular bridge likely produces a WARN_ON when booting the kernel on a VT-d system because it lacks a PCIe capability (in violation of the PCIe spec).  As a result, I doubt that the IOMMU tables are getting setup correct for devices behind the bridge.  I've actually been working on trying to fix this for the last couple weeks.  In any case, assigning the bridge to the guest is always wrong, so I'd start with that.  I'll hopefully be posting patches to fix the situation with this bridge this week or next.

This is amazing news!
Could you please specify where are you going to upload/write info about fix (twitter, github, ... - how to follow news)?

I posted the patches today, here's a link to the header with a description:

http://www.spinics.net/lists/linux-pci/msg30941.html

For your convenience, I've also pushed a branch out to github that is 3.15-rc3 + these changes, you'll need to add whatever other patches (i915/acs) on top:

git://github.com/awilliam/linux-vfio.git

branch: dma-alias

Let me know how it goes.

---

http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users