Arch Linux

abarahc

Member

Registered: 2010-04-30

Posts: 128

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Hi! sorry if all this questio below are already answered and i didnt get them, but!

Question - 01 - How can i check it?

Setting up vfio and kvm modules

If your board doesn't enable interrupt remapping, you need to add this to your grub cfg:

i've read this http://www.linux-kvm.org/page/How_to_as … T-d_in_KVM
and on step 3 we have this command

dmesg | grep -e DMAR -e IOMMU

which i think is the answer for my question! True?

Question - 02 - I got only one line on command's output
the command

dmesg | grep -e DMAR -e IOMMU

must have a positive returning like this

DMAR:DRHD base: 0x000000feb03000 flags: 0x0
IOMMU feb03000: ver 1:0 cap c9008020e30260 ecap 1000

but in my case i got this

[    0.000000] Intel-IOMMU: enabled

so this is mean i dont have remapping support? or is something else?

Question - 03 - Why vfio?

Binding a device to vfio-pci

Assuming the kernel, qemu and seabios are built and working, lets bind some devices.
You can use this script to make life easier:

#!/bin/bash

modprobe vfio-pci

for dev in "$@"; do
        vendor=$(cat /sys/bus/pci/devices/$dev/vendor)
        device=$(cat /sys/bus/pci/devices/$dev/device)
        if [ -e /sys/bus/pci/devices/$dev/driver ]; then
                echo $dev > /sys/bus/pci/devices/$dev/driver/unbind
        fi
        echo $vendor $device > /sys/bus/pci/drivers/vfio-pci/new_id
done

Save it as /usr/bin/vfio-bind

chmod 755 /usr/bin/vfio-bind

on the tutorial link is mentioned another path to make this procedures. so whats the real difference between pci-stub and vfio-stub?

on my case both solution do not work correctly. every time a try

echo XX:XXX:XX > /sys/bus/pci/drivers/vfio-pci/new_id

or

echo XX:XXX:XX >  /sys/bus/pci/drivers/pci-stub/new_id

i got permission issues.

those files are on write mode only! and even after i changing the modes to read/write or read/write/execute nothing change error message about i can write nothing

1. yes i was making this instructions as root
2. yes, i tried "su" and "su -". no changes

Question - 04 So whats the problem with this files?

Question - 05 - Where is this BUS id?

in a few lines of this tuto when you have to pass a devices to qemu you write the correct bus for them. But i really dont know where i can find this info.

sometimes is "bus=root.1" other times are "bus=pcie.0"

so how can i determine?

---

user@localhost $ grep -rnw "." -e "hacking"

aw

Member

Registered: 2013-10-04

Posts: 921

Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Hi! sorry if all this questio below are already answered and i didnt get them, but!

Question - 01 - How can i check it?

Setting up vfio and kvm modules

If your board doesn't enable interrupt remapping, you need to add this to your grub cfg:

i've read this http://www.linux-kvm.org/page/How_to_as … T-d_in_KVM
and on step 3 we have this command

dmesg | grep -e DMAR -e IOMMU

which i think is the answer for my question! True?

Interrupt remapping depends on IOMMU support, but having IOMMU support doesn't imply interrupt remapping.  You can look for something like "Enabled IRQ remapping in ...... mode", or you can just try it.  vfio won't let you assign a device without it or the option to bypass it, so if it doesn't work, look in dmesg and you'll see a message about how to enable it if that's the problem.

Question - 02 - I got only one line on command's output
the command

dmesg | grep -e DMAR -e IOMMU

must have a positive returning like this

DMAR:DRHD base: 0x000000feb03000 flags: 0x0
IOMMU feb03000: ver 1:0 cap c9008020e30260 ecap 1000

but in my case i got this

[    0.000000] Intel-IOMMU: enabled

so this is mean i dont have remapping support? or is something else?

It means you don't have VT-d support.  Make sure it's enabled in you BIOS and make sure you're booting with intel_iommu=on

Question - 03 - Why vfio?

Binding a device to vfio-pci

Assuming the kernel, qemu and seabios are built and working, lets bind some devices.
You can use this script to make life easier:

#!/bin/bash

modprobe vfio-pci

for dev in "$@"; do
        vendor=$(cat /sys/bus/pci/devices/$dev/vendor)
        device=$(cat /sys/bus/pci/devices/$dev/device)
        if [ -e /sys/bus/pci/devices/$dev/driver ]; then
                echo $dev > /sys/bus/pci/devices/$dev/driver/unbind
        fi
        echo $vendor $device > /sys/bus/pci/drivers/vfio-pci/new_id
done

Save it as /usr/bin/vfio-bind

chmod 755 /usr/bin/vfio-bind

on the tutorial link is mentioned another path to make this procedures. so whats the real difference between pci-stub and vfio-stub?

on my case both solution do not work correctly. every time a try

echo XX:XXX:XX > /sys/bus/pci/drivers/vfio-pci/new_id

or

echo XX:XXX:XX >  /sys/bus/pci/drivers/pci-stub/new_id

i got permission issues.

those files are on write mode only! and even after i changing the modes to read/write or read/write/execute nothing change error message about i can write nothing

1. yes i was making this instructions as root
2. yes, i tried "su" and "su -". no changes

Question - 04 So whats the problem with this files?

You're using them incorrectly.  Note that the script echos $vendor and $device to the new_id file while you're trying to use the (incomplete) PCI address.  You're getting permission denied because of what you're trying to echo, not because of file permissions.  As to why vfio-pci vs pci-stub, vfio is a new userspace driver interface where qemu is just a userspace driver using it for device assignment.  Legacy kvm device assignment with pci-stub is effectively deprecated.  Also, if you want to do VGA, vfio is the only device assignment interface with this support.

Question - 05 - Where is this BUS id?

in a few lines of this tuto when you have to pass a devices to qemu you write the correct bus for them. But i really dont know where i can find this info.

sometimes is "bus=root.1" other times are "bus=pcie.0"

so how can i determine?

bus= addresses are qemu defined identifiers.  When using the q35 model there is an implicit id=pcie.0 on the root bus.  That means any devices placed there using bus=pcie.0 will appear on the PCIe root complex.  When root ports are added to the topology they create a new bus.  You're able to name that bus with the id= option.  root.1 is just a name some people use, it could just as easily be named bob.  To place a device on that bus, use the bus=bob option.

When specifying a device to be assigned, it's the host= option.  This always takes the form of bus:device.function (ex. 01:00.0) or segment:bus:device.function (ex. 0000:01:00.0).  These match your physical device layout.

---

http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

abarahc

Member

Registered: 2010-04-30

Posts: 128

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

@aw

Thanks so much to reply so fast!

It means you don't have VT-d support.  Make sure it's enabled in you BIOS and make sure you're booting with intel_iommu=on

the "intel_iommu=on" is set as parameter on kernel boot line!

linux   /boot/vmlinuz-linux root=UUID=aaed6a49-c921-463c-bc9b-a22b74a6ee4e rw  quiet intel_iommu=on vfio_iommu_type1.allow_unsafe_interrupts=1 pci-stub.ids=10de:0de9,8086:1e20

but i cant find anything about this on my notebook BIOS! (Asus K45VM). This probably means i dont have support to remapping, right?

The pci.stub.ids are based on this output

lspci

00:00.0 Host bridge: Intel Corporation 3rd Gen Core processor DRAM Controller (rev 09)
00:01.0 PCI bridge: Intel Corporation Xeon E3-1200 v2/3rd Gen Core processor PCI Express Root Port (rev 09)
00:02.0 VGA compatible controller: Intel Corporation 3rd Gen Core processor Graphics Controller (rev 09)
00:14.0 USB controller: Intel Corporation 7 Series/C210 Series Chipset Family USB xHCI Host Controller (rev 04)
00:16.0 Communication controller: Intel Corporation 7 Series/C210 Series Chipset Family MEI Controller #1 (rev 04)
00:1a.0 USB controller: Intel Corporation 7 Series/C210 Series Chipset Family USB Enhanced Host Controller #2 (rev 04)
00:1b.0 Audio device: Intel Corporation 7 Series/C210 Series Chipset Family High Definition Audio Controller (rev 04)
00:1c.0 PCI bridge: Intel Corporation 7 Series/C210 Series Chipset Family PCI Express Root Port 1 (rev c4)
00:1c.1 PCI bridge: Intel Corporation 7 Series/C210 Series Chipset Family PCI Express Root Port 2 (rev c4)
00:1d.0 USB controller: Intel Corporation 7 Series/C210 Series Chipset Family USB Enhanced Host Controller #1 (rev 04)
00:1f.0 ISA bridge: Intel Corporation HM76 Express Chipset LPC Controller (rev 04)
00:1f.2 SATA controller: Intel Corporation 7 Series Chipset Family 6-port SATA Controller [AHCI mode] (rev 04)
00:1f.3 SMBus: Intel Corporation 7 Series/C210 Series Chipset Family SMBus Controller (rev 04)
01:00.0 VGA compatible controller: NVIDIA Corporation GF108M [GeForce GT 630M] (rev a1)
02:00.0 Ethernet controller: Realtek Semiconductor Co., Ltd. RTL8111/8168 PCI Express Gigabit Ethernet controller (rev 07)
03:00.0 Network controller: Qualcomm Atheros AR9485 Wireless Network Adapter (rev 01)

lspci -n

00:00.0 0600: 8086:0154 (rev 09)
00:01.0 0604: 8086:0151 (rev 09)
00:02.0 0300: 8086:0166 (rev 09)
00:14.0 0c03: 8086:1e31 (rev 04)
00:16.0 0780: 8086:1e3a (rev 04)
00:1a.0 0c03: 8086:1e2d (rev 04)
00:1b.0 0403: 8086:1e20 (rev 04)
00:1c.0 0604: 8086:1e10 (rev c4)
00:1c.1 0604: 8086:1e12 (rev c4)
00:1d.0 0c03: 8086:1e26 (rev 04)
00:1f.0 0601: 8086:1e59 (rev 04)
00:1f.2 0106: 8086:1e03 (rev 04)
00:1f.3 0c05: 8086:1e22 (rev 04)
01:00.0 0300: 10de:0de9 (rev a1)
02:00.0 0200: 10ec:8168 (rev 07)
03:00.0 0280: 168c:0032 (rev 01)

---

user@localhost $ grep -rnw "." -e "hacking"

aw

Member

Registered: 2013-10-04

Posts: 921

Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

@aw

Thanks so much to reply so fast!

It means you don't have VT-d support.  Make sure it's enabled in you BIOS and make sure you're booting with intel_iommu=on

the "intel_iommu=on" is set as parameter on kernel boot line!

linux   /boot/vmlinuz-linux root=UUID=aaed6a49-c921-463c-bc9b-a22b74a6ee4e rw  quiet intel_iommu=on vfio_iommu_type1.allow_unsafe_interrupts=1 pci-stub.ids=10de:0de9,8086:1e20

but i cant find anything about this on my notebook BIOS! (Asus K45VM). This probably means i dont have support to remapping, right?

According to Asus, the CPUs available for this laptop are:

Intel® Core™ i7 3610QM Processor
Intel® Core™ i5 3210M Processor
Intel® Core™ i3 3110M Processor

Cross referencing these to http://ark.intel.com, you do not have VT-d (assuming the rPGA version of 3210M).  Full stop, you can't do device assignment.  But the good news is that it probably would not have worked anyway.  Secondary graphics devices built into laptops are extremely customized to the device, often requiring vendor specific drivers.  I have not heard of any cases of these working when assigned to a VM.  Folks here that are having success are using fully independent discrete graphics cards.  Sorry.

---

http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

abarahc

Member

Registered: 2010-04-30

Posts: 128

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

@aw

Thanks so much to reply so fast!

It means you don't have VT-d support.  Make sure it's enabled in you BIOS and make sure you're booting with intel_iommu=on

the "intel_iommu=on" is set as parameter on kernel boot line!

linux   /boot/vmlinuz-linux root=UUID=aaed6a49-c921-463c-bc9b-a22b74a6ee4e rw  quiet intel_iommu=on vfio_iommu_type1.allow_unsafe_interrupts=1 pci-stub.ids=10de:0de9,8086:1e20

but i cant find anything about this on my notebook BIOS! (Asus K45VM). This probably means i dont have support to remapping, right?

According to Asus, the CPUs available for this laptop are:

Intel® Core™ i7 3610QM Processor
Intel® Core™ i5 3210M Processor
Intel® Core™ i3 3110M Processor

Cross referencing these to http://ark.intel.com, you do not have VT-d (assuming the rPGA version of 3210M).  Full stop, you can't do device assignment.  But the good news is that it probably would not have worked anyway.  Secondary graphics devices built into laptops are extremely customized to the device, often requiring vendor specific drivers.  I have not heard of any cases of these working when assigned to a VM.  Folks here that are having success are using fully independent discrete graphics cards.  Sorry.

Ok buddy :'/
Thank you very much for the explanatiion.

---

user@localhost $ grep -rnw "." -e "hacking"

paradexes

Member

Registered: 2013-11-30

Posts: 13

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

AW thank you for your replies. You actually answered one of my questions as well in this. I have also an Opteron system Tyan S8225 with dual 4234 CPUs on it. From what I have been reading it appears to support IOMMU (and the option also called SVVM) is enabled in the bios. I presume this would support VGA passthrough? Its obvious my intel setup doesnt :*( But least there is hope. Can you confirm if this is the case?

aw

Member

Registered: 2013-10-04

Posts: 921

Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

AW thank you for your replies. You actually answered one of my questions as well in this. I have also an Opteron system Tyan S8225 with dual 4234 CPUs on it. From what I have been reading it appears to support IOMMU (and the option also called SVVM) is enabled in the bios. I presume this would support VGA passthrough? Its obvious my intel setup doesnt :*( But least there is hope. Can you confirm if this is the case?

The DMAR and intel_iommu=on tips are Intel specific.  On AMD the IOMMU is described by the IVRS table in ACPI instead of the DMAR, AMD calls this AMD-Vi.  The equivalent boot option is amd_iommu=on.  SVM is the processor level virtualization technology, you can verify this with 'grep svm /proc/cpuinfo' (vmx is the equivalent Intel flag).  Unfortunately AMD doesn't have a convenient reference page for checking CPU and chipset features like Intel does with ark.

---

http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

ilya80

Member

Registered: 2013-11-12

Posts: 34

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Real GRID cards report the same Code 12 on the q35 chipset model, try 440fx instead.  I can't figure out what resources it's missing, nor have I had time to experiment.  Linux guests will work fine on the q35 model.  Again, you're not going to get pre-boot VGA output unless the VBIOS matches the hardware.  Also, Nvidia does not support GRID in a VGA mode, it's only supported as a secondary device to emulated VGA.  The professional version of the drivers detect that they're running on a VM and behave differently.  The quirks enabled with x-vga=on do not work correctly with those drivers.

Thanks for the suggestions!

I downloaded BIOS from GPU using GPU-Z, patched device ID, updated the CRC, added romfile=/path/to/rom/file to options for VGA portion of GPU, still no luck with output to monitors on VM start. All I get in kern.log is:

Dec  3 21:29:33 cave-lin kernel: [  286.128960] vfio_ecap_init: 0000:01:00.0 hiding ecap 0x19@0x900
Dec  3 21:30:10 cave-lin kernel: [  324.069324] kvm: zapping shadow pages for mmio generation wraparound
Dec  3 21:30:10 cave-lin kernel: [  324.106954] vfio-pci 0000:01:00.0: irq 57 for MSI/MSI-X

Also I have tried running VM with FX440 chipset, but it doesnt seem to have pcie bus, so no luck starting VM at all:

kalujny@cave-lin:~$ sudo qemu-system-x86_64 -enable-kvm -M pc -m 4096 -cpu SandyBridge -smp 3,sockets=1,cores=3,threads=1 -bios /opt/kalujny/kernel/seabios/out/bios.bin -device ioh3420,bus=pcie.0,addr=1c.0,multifunction=on,port=1,chassis=1,id=root.1 -device vfio-pci,host=01:00.0,bus=root.1,addr=00.0,multifunction=on,romfile=/opt/kalujny/thrash/PNYGTX6802GB_GRIDK2.rom -device vfio-pci,host=01:00.1,bus=root.1,addr=00.1 -device ahci,bus=pcie.0,id=ahci -drive file=/opt/kalujny/thrash/windows.img,id=disk,format=raw -device ide-hd,bus=ahci.0,drive=disk -usbdevice tablet                                                
qemu-system-x86_64: -device ioh3420,bus=pcie.0,addr=1c.0,multifunction=on,port=1,chassis=1,id=root.1: Bus 'pcie.0' not found 

aw

Member

Registered: 2013-10-04

Posts: 921

Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Real GRID cards report the same Code 12 on the q35 chipset model, try 440fx instead.  I can't figure out what resources it's missing, nor have I had time to experiment.  Linux guests will work fine on the q35 model.  Again, you're not going to get pre-boot VGA output unless the VBIOS matches the hardware.  Also, Nvidia does not support GRID in a VGA mode, it's only supported as a secondary device to emulated VGA.  The professional version of the drivers detect that they're running on a VM and behave differently.  The quirks enabled with x-vga=on do not work correctly with those drivers.

Thanks for the suggestions!

I downloaded BIOS from GPU using GPU-Z, patched device ID, updated the CRC, added romfile=/path/to/rom/file to options for VGA portion of GPU, still no luck with output to monitors on VM start. All I get in kern.log is:

Dec  3 21:29:33 cave-lin kernel: [  286.128960] vfio_ecap_init: 0000:01:00.0 hiding ecap 0x19@0x900
Dec  3 21:30:10 cave-lin kernel: [  324.069324] kvm: zapping shadow pages for mmio generation wraparound
Dec  3 21:30:10 cave-lin kernel: [  324.106954] vfio-pci 0000:01:00.0: irq 57 for MSI/MSI-X

Also I have tried running VM with FX440 chipset, but it doesnt seem to have pcie bus, so no luck starting VM at all:

kalujny@cave-lin:~$ sudo qemu-system-x86_64 -enable-kvm -M pc -m 4096 -cpu SandyBridge -smp 3,sockets=1,cores=3,threads=1 -bios /opt/kalujny/kernel/seabios/out/bios.bin -device ioh3420,bus=pcie.0,addr=1c.0,multifunction=on,port=1,chassis=1,id=root.1 -device vfio-pci,host=01:00.0,bus=root.1,addr=00.0,multifunction=on,romfile=/opt/kalujny/thrash/PNYGTX6802GB_GRIDK2.rom -device vfio-pci,host=01:00.1,bus=root.1,addr=00.1 -device ahci,bus=pcie.0,id=ahci -drive file=/opt/kalujny/thrash/windows.img,id=disk,format=raw -device ide-hd,bus=ahci.0,drive=disk -usbdevice tablet                                                
qemu-system-x86_64: -device ioh3420,bus=pcie.0,addr=1c.0,multifunction=on,port=1,chassis=1,id=root.1: Bus 'pcie.0' not found 

Root ports (ioh3420) are incompatible with 44fx, the device needs to be placed on the host bridge (pci.0) which is the default if not specified.  Try something like this:

sudo qemu-system-x86_64 -enable-kvm -M pc -m 4096 -cpu SandyBridge -smp 3,sockets=1,cores=3,threads=1 -bios /opt/kalujny/kernel/seabios/out/bios.bin \
-device vfio-pci,host=01:00.0,addr=07.0,multifunction=on,romfile=/opt/kalujny/thrash/PNYGTX6802GB_GRIDK2.rom -device vfio-pci,host=01:00.1,addr=07.1 \
-drive file=/opt/kalujny/thrash/windows.img,if=none,id=disk,format=raw -device ide-hd,bus=ide.0,unit=0,drive=disk -usbdevice tablet 

(I chose device address 7, move it if you want)

---

http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

paradexes

Member

Registered: 2013-11-30

Posts: 13

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

AW thank you for your replies. You actually answered one of my questions as well in this. I have also an Opteron system Tyan S8225 with dual 4234 CPUs on it. From what I have been reading it appears to support IOMMU (and the option also called SVVM) is enabled in the bios. I presume this would support VGA passthrough? Its obvious my intel setup doesnt :*( But least there is hope. Can you confirm if this is the case?

The DMAR and intel_iommu=on tips are Intel specific.  On AMD the IOMMU is described by the IVRS table in ACPI instead of the DMAR, AMD calls this AMD-Vi.  The equivalent boot option is amd_iommu=on.  SVM is the processor level virtualization technology, you can verify this with 'grep svm /proc/cpuinfo' (vmx is the equivalent Intel flag).  Unfortunately AMD doesn't have a convenient reference page for checking CPU and chipset features like Intel does with ark.

Well in regards to my intel server system you answered my questions so thats appreciated. I also have an AMD opteron setup as well. So at least I have hope to make VGA passthrough work on something lol.

No they don't but wikipedia is a decent alternative. Here is a page I found for Opterons. At least AMD-V is clearly supported on the valencias (4200 series) and my board does support IOMMU. I guess I will give it a go and see what happens. Thanks again.
https://en.wikipedia.org/wiki/List_of_A … 2832_nm.29

Last edited by paradexes (2013-12-03 17:49:15)

aw

Member

Registered: 2013-10-04

Posts: 921

Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

AW thank you for your replies. You actually answered one of my questions as well in this. I have also an Opteron system Tyan S8225 with dual 4234 CPUs on it. From what I have been reading it appears to support IOMMU (and the option also called SVVM) is enabled in the bios. I presume this would support VGA passthrough? Its obvious my intel setup doesnt :*( But least there is hope. Can you confirm if this is the case?

The DMAR and intel_iommu=on tips are Intel specific.  On AMD the IOMMU is described by the IVRS table in ACPI instead of the DMAR, AMD calls this AMD-Vi.  The equivalent boot option is amd_iommu=on.  SVM is the processor level virtualization technology, you can verify this with 'grep svm /proc/cpuinfo' (vmx is the equivalent Intel flag).  Unfortunately AMD doesn't have a convenient reference page for checking CPU and chipset features like Intel does with ark.

Well in regards to my intel server system you answered my questions so thats appreciated. I also have an AMD opteron setup as well. So at least I have hope to make VGA passthrough work on something lol.

No they don't but wikipedia is a decent alternative. Here is a page I found for Opterons. At least AMD-V is clearly supported on the valencias (4200 series) and my board does support IOMMU. I guess I will give it a go and see what happens. Thanks again.
https://en.wikipedia.org/wiki/List_of_A … 2832_nm.29

I'm sorry to say that I will be incredibly surprised if it has AMD IOMMU (AMD-Vi) support.  AMD-V (no i) is the CPU virtualization, aka SVM.  AFAIK the RD870 (maybe 890) was the first AMD-IOMMU chipset.  Most people using AMD for device assignment are using an 970/990.  Note that documentation and BIOS for AMD system can be very confusing because they'll refer to the onboard GART as an IOMMU.  While originally intended for graphics, it became a general purpose IOMMU for many system.  Unfortunately to support device assignment we need an IOMMU capable of both translation and isolation, the GART only has translation.  Furthermore, I believe it's a window based IOMMU rather than page table based, so even if you wanted to forego isolation, it doesn't even support the model we use for device assignment.

---

http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

paradexes

Member

Registered: 2013-11-30

Posts: 13

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

The DMAR and intel_iommu=on tips are Intel specific.  On AMD the IOMMU is described by the IVRS table in ACPI instead of the DMAR, AMD calls this AMD-Vi.  The equivalent boot option is amd_iommu=on.  SVM is the processor level virtualization technology, you can verify this with 'grep svm /proc/cpuinfo' (vmx is the equivalent Intel flag).  Unfortunately AMD doesn't have a convenient reference page for checking CPU and chipset features like Intel does with ark.

Well in regards to my intel server system you answered my questions so thats appreciated. I also have an AMD opteron setup as well. So at least I have hope to make VGA passthrough work on something lol.

No they don't but wikipedia is a decent alternative. Here is a page I found for Opterons. At least AMD-V is clearly supported on the valencias (4200 series) and my board does support IOMMU. I guess I will give it a go and see what happens. Thanks again.
https://en.wikipedia.org/wiki/List_of_A … 2832_nm.29

I'm sorry to say that I will be incredibly surprised if it has AMD IOMMU (AMD-Vi) support.  AMD-V (no i) is the CPU virtualization, aka SVM.  AFAIK the RD870 (maybe 890) was the first AMD-IOMMU chipset.  Most people using AMD for device assignment are using an 970/990.  Note that documentation and BIOS for AMD system can be very confusing because they'll refer to the onboard GART as an IOMMU.  While originally intended for graphics, it became a general purpose IOMMU for many system.  Unfortunately to support device assignment we need an IOMMU capable of both translation and isolation, the GART only has translation.  Furthermore, I believe it's a window based IOMMU rather than page table based, so even if you wanted to forego isolation, it doesn't even support the model we use for device assignment.

Well this is interesting. AMD-Vi IS IOMMU. Least according to this wiki, I think I might be good here.  The chipset for the board is reported to be working for VGA passthrough in xen. I would imagine the same would apply here.
http://en.wikipedia.org/wiki/X86_virtualization

aw

Member

Registered: 2013-10-04

Posts: 921

Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Well in regards to my intel server system you answered my questions so thats appreciated. I also have an AMD opteron setup as well. So at least I have hope to make VGA passthrough work on something lol.

No they don't but wikipedia is a decent alternative. Here is a page I found for Opterons. At least AMD-V is clearly supported on the valencias (4200 series) and my board does support IOMMU. I guess I will give it a go and see what happens. Thanks again.
https://en.wikipedia.org/wiki/List_of_A … 2832_nm.29

I'm sorry to say that I will be incredibly surprised if it has AMD IOMMU (AMD-Vi) support.  AMD-V (no i) is the CPU virtualization, aka SVM.  AFAIK the RD870 (maybe 890) was the first AMD-IOMMU chipset.  Most people using AMD for device assignment are using an 970/990.  Note that documentation and BIOS for AMD system can be very confusing because they'll refer to the onboard GART as an IOMMU.  While originally intended for graphics, it became a general purpose IOMMU for many system.  Unfortunately to support device assignment we need an IOMMU capable of both translation and isolation, the GART only has translation.  Furthermore, I believe it's a window based IOMMU rather than page table based, so even if you wanted to forego isolation, it doesn't even support the model we use for device assignment.

Well this is interesting. AMD-Vi IS IOMMU. Least according to this wiki, I think I might be good here.  The chipset for the board is reported to be working for VGA passthrough in xen. I would imagine the same would apply here.
http://en.wikipedia.org/wiki/X86_virtualization

Found this page - http://www.amd.com/us/products/server/p … tions.aspx

And here http://en.wikipedia.org/wiki/AMD_800_chipset_series indicates that the SR5690 is a rebranding of the RD890, so I agree, you should be good.

---

http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

nivs

Member

Registered: 2013-12-05

Posts: 1

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

I'd try to passthrough GTX780 with i7-4771 CPU and Z78 chipset, but got "Invalid ROM contents" in dmesg and 43 or 12 error in windows. Kernel and qemu from first post.

BOOT_IMAGE=/boot/vmlinuz-linux-mainline root=UUID=b011c3ab-e015-4d13-b5d8-afd54b708603 rw max_loop=64 intel_iommu=on pci-stub.ids=10de:1004,10de:0e1a vfio_iommu_type1.allow_unsafe_interrupts=1

# zcat /proc/config.gz| grep -i vfio                                                                       22:05 pts/2
CONFIG_VFIO_IOMMU_TYPE1=y
CONFIG_VFIO=y
CONFIG_VFIO_PCI=y
CONFIG_VFIO_PCI_VGA=y

Win 7 x64 and win 8 x64 were installed. Host GPU is Intel, but now it's off, cable connect to GTX 780, i see terminals. All work do in vncserver from notebook or SSH connection with "-nographic -monitor stdio". When i run this:

qemu-system-x86_64 -enable-kvm -M q35 -m 2048 -cpu host -smp 3,sockets=1,cores=3,threads=1 -bios /usr/share/qemu/bios.bin -device ioh3420,bus=pcie.0,addr=1c.0,multifunction=on,port=1,chassis=1,id=root.1 -device vfio-pci,host=0000:01:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on,rombar=0,romfile=/var/lib/libvirt/images/780.rom -device vfio-pci,host=0000:01:00.1,bus=root.1,addr=00.1 -device ahci,id=ahci0,bus=pcie.0,addr=0x5 -device virtio-serial-pci,id=virtio-serial0,bus=pcie.0,addr=0x6 -drive file=/var/lib/libvirt/images/windVirt7.img,if=none,id=drive-sata0-0-0,format=qcow2,cache=none -device ide-hd,bus=ahci0.0,drive=drive-sata0-0-0,id=sata0-0-0,bootindex=1 -drive file=/var/lib/libvirt/images/ru_7x64.iso,if=none,id=drive-ide0-1-0,readonly=on,format=raw -device ide-cd,bus=ide.1,unit=0,drive=drive-ide0-1-0,id=ide0-1-0,bootindex=2 -net nic -net bridge,br=xenbr0

in dmesg appers this:

[ 1175.295251] device tap0 entered promiscuous mode
[ 1175.295332] xenbr0: port 2(tap0) entered forwarding state
[ 1175.295335] xenbr0: port 2(tap0) entered forwarding state
[ 1175.919638] vfio_ecap_init: 0000:01:00.0 hiding ecap 0x19@0x900
[ 1175.919685] vfio-pci 0000:01:00.0: Invalid ROM contents
[ 1175.954591] vfio-pci 0000:01:00.0: BAR 3: can't reserve [mem 0xf0000000-0xf1ffffff 64bit pref]
[ 1179.035148] kvm: zapping shadow pages for mmio generation wraparound

With -vga none this: http://pastebin.com/MPJnUY3a
And in terminal:

qemu-system-x86_64: -device vfio-pci,host=01:00.0,addr=07.0,multifunction=on,romfile=/var/lib/libvirt/images/780.rom: Warning, device 0000:01:00.0 does not support reset
qemu-system-x86_64: -device vfio-pci,host=01:00.0,addr=07.0,multifunction=on,romfile=/var/lib/libvirt/images/780.rom: VFIO 0000:01:00.0 BAR 3 mmap unsupported. Performance may be slow
qemu-system-x86_64: -device vfio-pci,host=01:00.1,addr=07.1: Warning, device 0000:01:00.1 does not support reset

With

#!/bin/bash
qemu-system-x86_64 -enable-kvm -M pc -m 4096 -cpu SandyBridge -smp 3,sockets=1,cores=3,threads=1 -bios /usr/share/qemu/bios.bin \
-device vfio-pci,host=01:00.0,addr=07.0,multifunction=on,romfile=/var/lib/libvirt/images/780.rom -device vfio-pci,host=01:00.1,addr=07.1 \
-drive file=/var/lib/libvirt/images/virt.img,if=none,id=disk,format=raw -device ide-hd,bus=ide.0,unit=0,drive=disk -usbdevice tablet  -vga none

i also get this. What should I do to make it finally work? I'm doing it for 4 days, with Xen also didn't work. If i wait about 40 sec with -vga none, in monitor (GTX 780) appeard this http://f-bit.ru/uploads/297542.jpg. 40 sec is time to select "repair" in menu. MB's BIOS up to date, GTX's BIOS too.

5-10 minutes after the start in the terminal there was a lot of messages:

qemu-system-x86_64: vfio_bar_write(,0x12ffc, 0x0, 4) failed: Device or resource busy
qemu-system-x86_64: vfio_bar_write(,0x98, 0xbffa41, 4) failed: Device or resource busy
qemu-system-x86_64: vfio_bar_write(,0x9c, 0x0, 4) failed: Device or resource busy
qemu-system-x86_64: vfio_bar_write(,0x13000, 0xabcdabcd, 4) failed: Device or resource busy
qemu-system-x86_64: vfio_bar_write(,0x13004, 0xabcdabcd, 4) failed: Device or resource busy
qemu-system-x86_64: vfio_bar_write(,0x13008, 0xabcdabcd, 4) failed: Device or resource busy
qemu-system-x86_64: vfio_bar_write(,0x1300c, 0xabcdabcd, 4) failed: Device or resource busy

With qemu debug:

<4>[  108.175567] vfio-pci 0000:01:00.0: BAR 3: can't reserve [mem 0xf0000000-0xf1ffffff 64bit pref]
<2>[  110.177869] ERROR: DMA PTE for vPFN 0x0 already set (to 416bc3003 not 403400083)

or

<3>[   89.103901] irq 16: nobody cared (try booting with the "irqpoll" option)
<4>[   89.106576] CPU: 0 PID: 0 Comm: swapper/0 Tainted: G        WC   3.12.0-1-mainline #1
<4>[   89.106577] Hardware name: Gigabyte Technology Co., Ltd. Z87M-HD3/Z87M-HD3, BIOS F7 08/03/2013
<4>[   89.106578]  ffff88042841d68c ffff88043f203e48 ffffffff814f5a4b ffff88042841d600
<4>[   89.106580]  ffff88043f203e70 ffffffff810b8e22 ffff88042841d600 0000000000000010
<4>[   89.106581]  0000000000000000 ffff88043f203eb0 ffffffff810b9248 ffffffff813d4402
<4>[   89.106583] Call Trace:
<4>[   89.106584]  <IRQ>  [<ffffffff814f5a4b>] dump_stack+0x54/0x8d
<4>[   89.106591]  [<ffffffff810b8e22>] __report_bad_irq+0x32/0xd0
<4>[   89.106593]  [<ffffffff810b9248>] note_interrupt+0x138/0x1f0
<4>[   89.106596]  [<ffffffff813d4402>] ? cpuidle_enter_state+0x52/0xc0
<4>[   89.106598]  [<ffffffff810b6c39>] handle_irq_event_percpu+0xf9/0x250
<4>[   89.106599]  [<ffffffff810b6dcd>] handle_irq_event+0x3d/0x60
<4>[   89.106601]  [<ffffffff810b9dba>] handle_fasteoi_irq+0x5a/0x100
<4>[   89.106605]  [<ffffffff81016b0e>] handle_irq+0x1e/0x30
<4>[   89.106607]  [<ffffffff8150690d>] do_IRQ+0x4d/0xc0
<4>[   89.106609]  [<ffffffff814fcfed>] common_interrupt+0x6d/0x6d
<4>[   89.106610]  <EOI>  [<ffffffff813d4402>] ? cpuidle_enter_state+0x52/0xc0
<4>[   89.106613]  [<ffffffff813d43f8>] ? cpuidle_enter_state+0x48/0xc0
<4>[   89.106614]  [<ffffffff813d4537>] cpuidle_idle_call+0xc7/0x280
<4>[   89.106616]  [<ffffffff8101e30e>] arch_cpu_idle+0xe/0x30
<4>[   89.106617]  [<ffffffff810b6207>] cpu_startup_entry+0x267/0x2e0
<4>[   89.106620]  [<ffffffff814e8254>] rest_init+0x84/0x90
<4>[   89.106621]  [<ffffffff818e2f01>] start_kernel+0x414/0x420
<4>[   89.106622]  [<ffffffff818e28f6>] ? repair_env_string+0x5c/0x5c
<4>[   89.106624]  [<ffffffff818e2120>] ? early_idt_handlers+0x120/0x120
<4>[   89.106625]  [<ffffffff818e25de>] x86_64_start_reservations+0x2a/0x2c
<4>[   89.106626]  [<ffffffff818e26e8>] x86_64_start_kernel+0x108/0x117
<3>[   89.106627] handlers:
<3>[   89.107536] [<ffffffffa0056ae0>] usb_hcd_irq [usbcore]
<3>[   89.109648] [<ffffffff813b23f0>] vfio_intx_handler
<0>[   89.111623] Disabling IRQ #16

P.S.: sorry for my English.

Last edited by nivs (2013-12-05 23:40:43)

keatsfonam

Member

Registered: 2013-03-28

Posts: 3

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

EDIT:
Solved the error 43 with AW's current qemu-vfio.  Im having the reboot issue still, working on that.  Thanks for all the great info in this thread.

EDIT2:

All problems solved by avoiding letting vfio-pci take the device from pci-stub upon starting the vm with the script provided.  Thanks

------

Similiar situation as previous poster, can't get past error 43:

all my hardware supports VT-d
I am trying to run host on my intel integrated (ivy bridge) but I see that the i915 driver has caused problems.  I tried running the guest before X even and still couldn't get past the problem.

For the record I am trying to pass through an nvidia gtx 560

dmesg | grep -e IOMMU -e DMAR

[    0.000000] Warning: PCIe ACS overrides enabled; This may allow non-IOMMU protected peer-to-peer DMA
[    0.000000] ACPI: DMAR 00000000be0f4c60 000B8 (v01 INTEL      SNB  00000001 INTL 00000001)
[    0.000000] Intel-IOMMU: enabled
[    0.077767] dmar: IOMMU 0: reg_base_addr fed90000 ver 1:0 cap c0000020e60262 ecap f0101a
[    0.077771] dmar: IOMMU 1: reg_base_addr fed91000 ver 1:0 cap c9008020660262 ecap f0105a
[    0.077851] IOAPIC id 2 under DRHD base  0xfed91000 IOMMU 1
[    0.254226] DMAR: No ATSR found
[    0.254249] IOMMU 0 0xfed90000: using Queued invalidation
[    0.254250] IOMMU 1 0xfed91000: using Queued invalidation
[    0.254251] IOMMU: Setting RMRR:
[    0.254260] IOMMU: Setting identity map for device 0000:00:02.0 [0xbf800000 - 0xcf9fffff]
[    0.255421] IOMMU: Setting identity map for device 0000:00:1d.0 [0xbdb64000 - 0xbdb8efff]
[    0.255438] IOMMU: Setting identity map for device 0000:00:1a.0 [0xbdb64000 - 0xbdb8efff]
[    0.255451] IOMMU: Setting identity map for device 0000:00:14.0 [0xbdb64000 - 0xbdb8efff]
[    0.255459] IOMMU: Prepare 0-16MiB unity mapping for LPC
[    0.255465] IOMMU: Setting identity map for device 0000:00:1f.0 [0x0 - 0xffffff]

/proc/cmdline:

BOOT_IMAGE=../vmlinuz-linux-mainline root=UUID=0ff525e8-6238-4741-a5f3-dec5653ebd14 ro intel_iommu=on vfio_iommu_type1.allow_unsafe_interrupts=1 pci-stub.ids=10de:1201,10de:0e0c initrd=../initramfs-linux-mainline.img

and tried:

BOOT_IMAGE=../vmlinuz-linux-mainline root=UUID=0ff525e8-6238-4741-a5f3-dec5653ebd14 ro intel_iommu=on initrd=../initramfs-linux-mainline.img

uname -r

3.12.0-1-mainline

tried with the three packages on the first post with the same kconfig
tried both qemu-git and linux-mainline from aur, same problem.

I can boot into windows 7 or windows 8 guest but I get the error 43.  Additionally I cannot reboot or it will lock up the host system.  I tried the patch that aw said fixes the error 43, but it still doesn't seem to work.

qemu-system-x86_64 -enable-kvm -M q35 -m 2048 -cpu host -smp 6,sockets=1,cores=2,threads=1 -bios /usr/share/qemu/bios.bin -device ioh3420,bus=pcie.0,addr=1c.0,multifunction=on,port=1,chassis=1,id=root.0 -boot once=d,menu=on -drive file=/home/me/win7.iso,id=isocd -device ide-cd,bus=ide.1,drive=isocd -drive file=/home/me/images/win7.img,id=disk,format=raw -device ide-hd,bus=ide.0,drive=disk -device vfio-pci,host=01:00.0,bus=root.0,addr=00.0,multifunction=on,x-vga=on -vga none -usbdevice host:1532:010b -usbdevice host:046d:c00c

I have tried -cpu IvyBridge

dmesg:

 8.119274] [drm] Initialized drm 1.1.0 20060810
[    8.127061] [drm] Memory usable by graphics device = 2048M
[    8.127065] i915 0000:00:02.0: setting latency timer to 64
[    8.146428] i915 0000:00:02.0: irq 62 for MSI/MSI-X
[    8.146440] [drm] Supports vblank timestamp caching Rev 1 (10.10.2010).
[    8.146442] [drm] Driver supports precise vblank timestamp query.
[    8.167917] [drm] Wrong MCH_SSKPD value: 0x20100406
[    8.167920] [drm] This can cause pipe underruns and display issues.
[    8.167921] [drm] Please upgrade your BIOS to fix this.
[    8.266493] [drm] GMBUS [i915 gmbus dpc] timed out, falling back to bit banging on pin 4
[    8.277615] fbcon: inteldrmfb (fb0) is primary device
[    8.277752] Console: switching to colour frame buffer device 210x65
[    8.277762] i915 0000:00:02.0: fb0: inteldrmfb frame buffer device
[    8.277764] i915 0000:00:02.0: registered panic notifier
[    8.277768] vgaarb: device changed decodes: PCI:0000:00:02.0,olddecodes=io+mem,decodes=io:owns=io
[    8.282909] acpi device:43: registered as cooling_device4
[    8.283082] ACPI: Video Device [GFX0] (multi-head: yes  rom: no  post: no)
[    8.283130] input: Video Bus as /devices/LNXSYSTM:00/device:00/PNP0A08:00/LNXVIDEO:00/input/input8
[    8.283204] [drm] Initialized i915 1.6.0 20080730 for 0000:00:02.0 on minor 0
[    8.442675] vfio-pci 0000:01:00.0: Invalid ROM contents
[    9.775993] [drm] Enabling RC6 states: RC6 on, RC6p on, RC6pp off
[   19.912534] xenbr0: port 1(enp4s0) entered forwarding state

I think the invalid ROM contents is the key error here, but I might be wrong.
I'm not sure if MCH_SSKPD is relevant.
I dumped my rom and tried to load it and it gave me all sorts of errors about invalid commands and locked up my host system completely forcing a reboot.

dma: command 78 not supported
dma: command 78 not supported

And a slew of similiar command not supported messages until I kill the process

dma: command 8c not supported
dma: unregistered DMA channel used nchan=0 dma_pos=0 dma_len=64769
dma: unregistered DMA channel used nchan=0 dma_pos=0 dma_len=64769
dma: command 5c not supported
dma: unregistered DMA channel used nchan=0 dma_pos=0 dma_len=64769
dma: command 5c not supported
dma: unregistered DMA channel used nchan=0 dma_pos=0 dma_len=64769
dma: unregistered DMA channel used nchan=0 dma_pos=0 dma_len=64769
dma: command 58 not supported
dma: command 58 not supported
dma: command 84 not supported
dma: command 60 not supported
dma: unregistered DMA channel used nchan=0 dma_pos=0 dma_len=64769
dma: unregistered DMA channel used nchan=0 dma_pos=0 dma_len=64769
dma: command 4c not supported
dma: unregistered DMA channel used nchan=0 dma_pos=0 dma_len=64769
dma: unregistered DMA channel used nchan=0 dma_pos=0 dma_len=64769
dma: command 4c not supported
dma: unregistered DMA channel used nchan=0 dma_pos=0 dma_len=64769
dma: unregistered DMA channel used nchan=0 dma_pos=0 dma_len=64769

When I start the VM:

[ 1513.598793] vfio-pci 0000:01:00.0: enabling device (0000 -> 0003)
[ 1516.405951] usb 2-1.5: reset full-speed USB device number 3 using ehci-pci
[ 1516.745059] usb 3-2: reset low-speed USB device number 2 using xhci_hcd
[ 1516.758317] xhci_hcd 0000:00:14.0: xHCI xhci_drop_endpoint called with disabled ep ffff880417f98a00
[ 1516.758331] usb 3-2: ep 0x81 - rounding interval to 64 microframes, ep desc says 80 microframes

The usb and xhci_hcd are spammed 5 or 6 more times, but it seems like the vfio-pci has no error message. 
However I saw some other user's dmesgs that said

assign device 0:X:0.0

Which is missing from mine.  Does that mean the device isn't actually getting assigned?

I'm not sure if the problem stems for qemu or from linux-mainline, so I keep trying different combinations with no luck.
Any help is appreciated, I suspect I may be better off with a radeon card.

Last edited by keatsfonam (2013-12-07 09:29:02)

jackun

Member

Registered: 2012-11-07

Posts: 2

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

I'm dual booting and sometimes messing with passthrough and having a dual input monitor, it was getting tiresome to switch monitor's input. So one way to use a gfxcard for passthrough without setting another card to be primary is to force off any FB consoles with kernel argument 'vga=0' and then manually (create a systemd service or something) use fbset and con2fb. Default low res text-only bios console should be still working.

You can get con2fb source from http://www.dafyddcrosby.com/src/con2fb.c
So:

cat /proc/fb

to see that only one fb is setup (probably radeondrmfb or something).

Then with working kms driver and 1920x1080 monitor:

fbset /dev/fb0 -xres 1920 -vxres 1920 -yres 1080 -vyres 1080  #initializes framebuffer and wakes up monitor (??)
con2fb /dev/fb0 /dev/tty1 #map ttys to framebuffer. Even though only tty1 is passed, it seems all ttys are move to fb0

Now you can see bios etc. stuff still on your (probably) more powerful card and use it in windows and still use it with passthrough too.
Ofcourse pci-stub etc. the gfxcard you intend to use for passthrough.

Now the question, can you unbind kernel's VGA console without 'vga=0' ? Then 'fbcon=map:1' could be enough probably.

Last edited by jackun (2013-12-07 23:24:41)

aw

Member

Registered: 2013-10-04

Posts: 921

Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

EDIT2:

All problems solved by avoiding letting vfio-pci take the device from pci-stub upon starting the vm with the script provided.  Thanks

What exactly does that mean?

BTW, as of this morning everything except i915 vga arbiter stuff is in upstream, kernel and qemu.

---

http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Chetyre

Member

Registered: 2013-03-27

Posts: 34

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Does anyone know of a way to monitor CPU temperature from within the guest and output it to an OSD program like RTSS? Proper cpu statistics would also be really nice.

Damon

Member

From: Germany

Registered: 2009-04-09

Posts: 7

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

I'm trying to pass a SATA controller of my Supermicro X10SAE to the guest.

Kernel: 3.12.3 with acs-override patch.

/usr/bin/qemu-system-x86_64 --version
QEMU emulator version 1.7.50, Copyright (c) 2003-2008 Fabrice Bellard

$ dmesg | grep -e DMAR -e IOMMU
[    0.000000] Warning: PCIe ACS overrides enabled; This may allow non-IOMMU protected peer-to-peer DMA
[    0.000000] ACPI: DMAR 00000000d9508dd0 000B8 (v01 INTEL      HSW  00000001 INTL 00000001)
[    0.000000] Intel-IOMMU: enabled
[    0.025640] dmar: IOMMU 0: reg_base_addr fed90000 ver 1:0 cap c0000020660462 ecap f0101a
[    0.025644] dmar: IOMMU 1: reg_base_addr fed91000 ver 1:0 cap d2008020660462 ecap f010da
[    0.025710] IOAPIC id 2 under DRHD base  0xfed91000 IOMMU 1
[    0.430025] DMAR: No ATSR found
[    0.430050] IOMMU 0 0xfed90000: using Queued invalidation
[    0.430051] IOMMU 1 0xfed91000: using Queued invalidation
[    0.430053] IOMMU: Setting RMRR:
[    0.430062] IOMMU: Setting identity map for device 0000:00:02.0 [0xdc000000 - 0xde1fffff]
[    0.430254] IOMMU: Setting identity map for device 0000:00:1d.0 [0xd9ea8000 - 0xd9eb6fff]
[    0.430276] IOMMU: Setting identity map for device 0000:00:1a.0 [0xd9ea8000 - 0xd9eb6fff]
[    0.430295] IOMMU: Setting identity map for device 0000:00:14.0 [0xd9ea8000 - 0xd9eb6fff]
[    0.430309] IOMMU: Prepare 0-16MiB unity mapping for LPC
[    0.430315] IOMMU: Setting identity map for device 0000:00:1f.0 [0x0 - 0xffffff]

$ cat /proc/cmdline
BOOT_IMAGE=/@/boot/vmlinuz-3.12.3+ root=UUID=4a5ab603-c89c-42fb-93bb-ef541d20f32e ro rootflags=subvol=@ crashkernel=384M-2G:64M,2G-:128M splash quiet intel_iommu=on pcie_acs_override=downstream vt.handoff=7

$ ./lsgroup
### Group 0 ###
    00:00.0 Host bridge: Intel Corporation Xeon E3-1200 v3 Processor DRAM Controller (rev 06)
### Group 1 ###
    00:02.0 VGA compatible controller: Intel Corporation Xeon E3-1200 v3 Processor Integrated Graphics Controller (rev 06)
### Group 2 ###
    00:03.0 Audio device: Intel Corporation Xeon E3-1200 v3/4th Gen Core Processor HD Audio Controller (rev 06)
### Group 3 ###
    00:14.0 USB controller: Intel Corporation 8 Series/C220 Series Chipset Family USB xHCI (rev 05)
### Group 4 ###
    00:16.0 Communication controller: Intel Corporation 8 Series/C220 Series Chipset Family MEI Controller #1 (rev 04)
    00:16.3 Serial controller: Intel Corporation 8 Series/C220 Series Chipset Family KT Controller (rev 04)
### Group 5 ###
    00:19.0 Ethernet controller: Intel Corporation Ethernet Connection I217-LM (rev 05)
### Group 6 ###
    00:1a.0 USB controller: Intel Corporation 8 Series/C220 Series Chipset Family USB EHCI #2 (rev 05)
### Group 7 ###
    00:1b.0 Audio device: Intel Corporation 8 Series/C220 Series Chipset High Definition Audio Controller (rev 05)
### Group 8 ###
    00:1c.0 PCI bridge: Intel Corporation 8 Series/C220 Series Chipset Family PCI Express Root Port #1 (rev d5)
### Group 9 ###
    00:1c.3 PCI bridge: Intel Corporation 8 Series/C220 Series Chipset Family PCI Express Root Port #4 (rev d5)
### Group 10 ###
    00:1c.5 PCI bridge: Intel Corporation 8 Series/C220 Series Chipset Family PCI Express Root Port #6 (rev d5)
### Group 11 ###
    00:1c.6 PCI bridge: Intel Corporation 8 Series/C220 Series Chipset Family PCI Express Root Port #7 (rev d5)
### Group 12 ###
    00:1c.7 PCI bridge: Intel Corporation 8 Series/C220 Series Chipset Family PCI Express Root Port #8 (rev d5)
### Group 13 ###
    00:1d.0 USB controller: Intel Corporation 8 Series/C220 Series Chipset Family USB EHCI #1 (rev 05)
### Group 14 ###
    00:1f.0 ISA bridge: Intel Corporation C226 Series Chipset Family Server Advanced SKU LPC Controller (rev 05)
    00:1f.2 SATA controller: Intel Corporation 8 Series/C220 Series Chipset Family 6-port SATA Controller 1 [AHCI mode] (rev 05)
    00:1f.3 SMBus: Intel Corporation 8 Series/C220 Series Chipset Family SMBus Controller (rev 05)
    00:1f.6 Signal processing controller: Intel Corporation 8 Series Chipset Family Thermal Management Controller (rev 05)
### Group 15 ###
    02:00.0 Ethernet controller: Intel Corporation I210 Gigabit Network Connection (rev 03)
### Group 16 ###
    03:00.0 PCI bridge: Tundra Semiconductor Corp. Device 8113 (rev 01)
    04:03.0 FireWire (IEEE 1394): Texas Instruments TSB43AB22A IEEE-1394a-2000 Controller (PHY/Link) [iOHCI-Lynx]
### Group 17 ###
    05:00.0 USB controller: Renesas Technology Corp. uPD720202 USB 3.0 Host Controller (rev 02)
### Group 18 ###
    06:00.0 SATA controller: ASMedia Technology Inc. ASM1062 Serial ATA Controller (rev 01)

The device I want to pass is in the IOMMU-Group 14

00:1f.2 SATA controller: Intel Corporation 8 Series/C220 Series Chipset Family 6-port SATA Controller 1 [AHCI mode] (rev 05)

Therefore I bind this group to vfio-pci

./vfio-group 14
0000:00:1f.0
0000:00:1f.2
0x8086 0x8c56
0x8086 0x8c02
0x8086 0x8c22
0x8086 0x8c24

Finally I want to test if it works:

$ /usr/bin/qemu-system-x86_64 -enable-kvm -M q35 -m 1024 -cpu host -smp 6,sockets=1,cores=6,threads=1 -bios /usr/share/qemu/bios.bin -vga none -device ioh3420,bus=pcie.0,addr=1c.0,multifunction=on,port=1,chassis=1,id=root.1 -device vfio-pci,host=00:1f.2,bus=root.1,addr=00.0
qemu-system-x86_64: vfio: Cannot reset device 0000:00:1f.2, no available reset mechanism.
qemu-system-x86_64: vfio: Cannot reset device 0000:00:1f.2, no available reset mechanism.

I really don't know what I'm doing wrong or what I have missed to do. Initially I tried to pass the controller using pci-assign but that wasn't possible due to the lack of FLReset.

I would really appreciate any help.

nbhs

Member

From: Montevideo, Uruguay

Registered: 2013-05-02

Posts: 402

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

linux-mainline.tar.gz (3.13-rc3, acs override patch) Doesnt include i935 vga arbiter fixes
qemu-git.tar.gz Spice disabled due to build errors
seabios-git.tar.gz

Aur:

https://aur.archlinux.org/packages/qemu-git/
https://aur.archlinux.org/packages/linux-mainline/ (still -rc2 as of today)

Last edited by nbhs (2013-12-08 14:41:30)

alphahere

Member

Registered: 2013-10-12

Posts: 7

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

The the i915 patch does not apply cleanly to linux-3.13-rc3. This is the request when I attempted to patch linux-3.13-rc3.

patch -p1 -i ../vfio-previous/linux-mainline/i915_fixes.patch 
patching file drivers/gpu/drm/i915/i915_dma.c
patching file drivers/gpu/drm/i915/intel_display.c
Hunk #1 succeeded at 10585 (offset 544 lines).
Hunk #2 succeeded at 10601 with fuzz 1 (offset 544 lines).
Hunk #3 succeeded at 11105 (offset 543 lines).
patching file include/linux/vgaarb.h
patching file drivers/gpu/drm/i915/i915_dma.c
Hunk #1 succeeded at 1355 (offset 4 lines).
patching file drivers/gpu/drm/i915/intel_display.c
Hunk #1 succeeded at 10585 (offset 545 lines).
Hunk #2 succeeded at 10592 (offset 545 lines).
Hunk #3 succeeded at 10606 with fuzz 1 (offset 545 lines).
Hunk #4 succeeded at 10904 with fuzz 1 (offset 552 lines).
Hunk #5 succeeded at 11110 (offset 544 lines).
patching file drivers/gpu/drm/i915/intel_drv.h
Hunk #1 FAILED at 792.
1 out of 1 hunk FAILED -- saving rejects to file drivers/gpu/drm/i915/intel_drv.h.rej

I am  able to pass a usb 3.0 controller to the VM but totally unable to install a driver under windows 7. Complete BSOD when installing the driver. Anyone know of a working usb 3.0 or usb 2.0 controller? I even tried passing the intel motherboard usb devices. Still could not install a driver. If there is additional information please let me know. I have no problem passing a usb device but want to pass a controller.

I think it is the controller. So much for that Black Friday special ;-). At least it was cheap. I think I will try a via usb 3.0 controller. I am looking for pci express card. If anyone has one that works please list the model.

The card I was trying to use is a Vantec Model UGT-PC312

04:00.0 USB controller: Renesas Technology Corp. uPD720202 USB 3.0 Host Controller (rev 02) (prog-if 30 [XHCI])
        Subsystem: Renesas Technology Corp. uPD720202 USB 3.0 Host Controller
        Control: I/O- Mem+ BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
        Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
        Interrupt: pin A routed to IRQ 18
        Region 0: Memory at f7900000 (64-bit, non-prefetchable) [size=8K]
        Capabilities: [50] Power Management version 3
                Flags: PMEClk- DSI- D1- D2- AuxCurrent=375mA PME(D0+,D1-,D2-,D3hot+,D3cold+)
                Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=0 PME-
        Capabilities: [70] MSI: Enable- Count=1/8 Maskable- 64bit+
                Address: 0000000000000000  Data: 0000
        Capabilities: [90] MSI-X: Enable- Count=8 Masked-
                Vector table: BAR=0 offset=00001000
                PBA: BAR=0 offset=00001080
        Capabilities: [a0] Express (v2) Endpoint, MSI 00
                DevCap: MaxPayload 128 bytes, PhantFunc 0, Latency L0s unlimited, L1 unlimited
                        ExtTag- AttnBtn- AttnInd- PwrInd- RBE+ FLReset-
                DevCtl: Report errors: Correctable- Non-Fatal- Fatal- Unsupported-
                        RlxdOrd- ExtTag- PhantFunc- AuxPwr- NoSnoop+
                        MaxPayload 128 bytes, MaxReadReq 512 bytes
                DevSta: CorrErr- UncorrErr- FatalErr- UnsuppReq- AuxPwr+ TransPend-
                LnkCap: Port #0, Speed 5GT/s, Width x1, ASPM L0s L1, Latency L0 <4us, L1 unlimited
                        ClockPM+ Surprise- LLActRep- BwNot-
                LnkCtl: ASPM Disabled; RCB 64 bytes Disabled- Retrain- CommClk+
                        ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
                LnkSta: Speed 5GT/s, Width x1, TrErr- Train- SlotClk+ DLActive- BWMgmt- ABWMgmt-
                DevCap2: Completion Timeout: Not Supported, TimeoutDis+
                DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis-
                LnkCtl2: Target Link Speed: 5GT/s, EnterCompliance- SpeedDis-, Selectable De-emphasis: -6dB
                         Transmit Margin: Normal Operating Range, EnterModifiedCompliance- ComplianceSOS-
                         Compliance De-emphasis: -6dB
                LnkSta2: Current De-emphasis Level: -6dB, EqualizationComplete-, EqualizationPhase1-
                         EqualizationPhase2-, EqualizationPhase3-, LinkEqualizationRequest-
        Capabilities: [100 v1] Advanced Error Reporting
                UESta:  DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
                UEMsk:  DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
                UESvrt: DLP+ SDES+ TLP- FCP+ CmpltTO- CmpltAbrt- UnxCmplt- RxOF+ MalfTLP+ ECRC- UnsupReq- ACSViol-
                CESta:  RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr-
                CEMsk:  RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr-
                AERCap: First Error Pointer: 00, GenCap- CGenEn- ChkCap- ChkEn-
        Capabilities: [150 v1] Latency Tolerance Reporting
                Max snoop latency: 0ns
                Max no snoop latency: 0ns
        Kernel driver in use: vfio-pci

Thanks.

Last edited by alphahere (2013-12-09 03:18:26)

binaryplease

Member

Registered: 2012-04-11

Posts: 16

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

I cant get pass the step from unbinding step. Everytime i execute the script or do

echo "0000:02:00.0" > /sys/bus/pci/devices/0000:02:00.0/driver/unbind

my system freezes and i have to switch it of holding the power button.

I have a lenovo y500 with two geforce gt650m cards (SLI).
Im using the nouveau driver. Can anyone help?

dalingrin

Member

Registered: 2009-03-18

Posts: 128

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Has anyone tried passing through 2 cards in SLI?

teekay

Member

Registered: 2011-10-26

Posts: 271

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

I am  able to pass a usb 3.0 controller to the VM but totally unable to install a driver under windows 7. Complete BSOD when installing the driver. Anyone know of a working usb 3.0 or usb 2.0 controller? I even tried passing the intel motherboard usb devices. Still could not install a driver. If there is additional information please let me know. I have no problem passing a usb device but want to pass a controller.

I think it is the controller. So much for that Black Friday special ;-). At least it was cheap. I think I will try a via usb 3.0 controller. I am looking for pci express card. If anyone has one that works please list the model.

The card I was trying to use is a Vantec Model UGT-PC312

04:00.0 USB controller: Renesas Technology Corp. uPD720202 USB 3.0 Host Controller (rev 02) (prog-if 30 [XHCI])

I bought one (because my onboard Etron controllers cause BSOD in Win7 when trying to install the driver).
The one I got works absolutely perfect in both Win 7 and 8.1, it's a

05:00.0 USB controller: Renesas Technology Corp. uPD720201 USB 3.0 Host Controller (rev 03) (prog-if 30 [XHCI])

/me hides, giggeling.

Last edited by teekay (2013-12-09 18:38:04)

peaquino

Member

Registered: 2013-12-09

Posts: 5

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Hi all!

I think I read all the posts in this long thread, but unfortunatelly I'm still not able to passthrough my Radeon 7870XT to a Windows 7 VM. My other components are:
- AMD APU A10-5800K (with Radeon 7660D acting as primary VGA)
- Asrock A85 FM2A85X Extreme6
- 16G RAM

lspci

00:00.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Family 15h (Models 10h-1fh) Processor Root Complex
00:00.2 IOMMU: Advanced Micro Devices, Inc. [AMD] Family 15h (Models 10h-1fh) I/O Memory Management Unit
00:01.0 VGA compatible controller: Advanced Micro Devices, Inc. [AMD/ATI] Trinity [Radeon HD 7660D]
00:02.0 PCI bridge: Advanced Micro Devices, Inc. [AMD] Family 15h (Models 10h-1fh) Processor Root Port
00:03.0 PCI bridge: Advanced Micro Devices, Inc. [AMD] Family 15h (Models 10h-1fh) Processor Root Port
00:10.0 USB controller: Advanced Micro Devices, Inc. [AMD] FCH USB XHCI Controller (rev 03)
00:10.1 USB controller: Advanced Micro Devices, Inc. [AMD] FCH USB XHCI Controller (rev 03)
00:11.0 SATA controller: Advanced Micro Devices, Inc. [AMD] FCH SATA Controller [AHCI mode] (rev 40)
00:12.0 USB controller: Advanced Micro Devices, Inc. [AMD] FCH USB OHCI Controller (rev 11)
00:12.2 USB controller: Advanced Micro Devices, Inc. [AMD] FCH USB EHCI Controller (rev 11)
00:13.0 USB controller: Advanced Micro Devices, Inc. [AMD] FCH USB OHCI Controller (rev 11)
00:13.2 USB controller: Advanced Micro Devices, Inc. [AMD] FCH USB EHCI Controller (rev 11)
00:14.0 SMBus: Advanced Micro Devices, Inc. [AMD] FCH SMBus Controller (rev 14)
00:14.1 IDE interface: Advanced Micro Devices, Inc. [AMD] FCH IDE Controller
00:14.2 Audio device: Advanced Micro Devices, Inc. [AMD] FCH Azalia Controller (rev 01)
00:14.3 ISA bridge: Advanced Micro Devices, Inc. [AMD] FCH LPC Bridge (rev 11)
00:14.4 PCI bridge: Advanced Micro Devices, Inc. [AMD] FCH PCI Bridge (rev 40)
00:14.5 USB controller: Advanced Micro Devices, Inc. [AMD] FCH USB OHCI Controller (rev 11)
00:15.0 PCI bridge: Advanced Micro Devices, Inc. [AMD] Hudson PCI to PCI bridge (PCIE port 0)
00:15.2 PCI bridge: Advanced Micro Devices, Inc. [AMD] Hudson PCI to PCI bridge (PCIE port 2)
00:15.3 PCI bridge: Advanced Micro Devices, Inc. [AMD] Hudson PCI to PCI bridge (PCIE port 3)
00:18.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Family 15h (Models 10h-1fh) Processor Function 0
00:18.1 Host bridge: Advanced Micro Devices, Inc. [AMD] Family 15h (Models 10h-1fh) Processor Function 1
00:18.2 Host bridge: Advanced Micro Devices, Inc. [AMD] Family 15h (Models 10h-1fh) Processor Function 2
00:18.3 Host bridge: Advanced Micro Devices, Inc. [AMD] Family 15h (Models 10h-1fh) Processor Function 3
00:18.4 Host bridge: Advanced Micro Devices, Inc. [AMD] Family 15h (Models 10h-1fh) Processor Function 4
00:18.5 Host bridge: Advanced Micro Devices, Inc. [AMD] Family 15h (Models 10h-1fh) Processor Function 5
01:00.0 VGA compatible controller: Advanced Micro Devices, Inc. [AMD/ATI] Tahiti LE [Radeon HD 7870 XT]
01:00.1 Audio device: Advanced Micro Devices, Inc. [AMD/ATI] Tahiti XT HDMI Audio [Radeon HD 7970 Series]
02:00.0 USB controller: MosChip Semiconductor Technology Ltd. MCS9990 PCIe to 4‐Port USB 2.0 Host Controller
02:00.1 USB controller: MosChip Semiconductor Technology Ltd. MCS9990 PCIe to 4‐Port USB 2.0 Host Controller
02:00.2 USB controller: MosChip Semiconductor Technology Ltd. MCS9990 PCIe to 4‐Port USB 2.0 Host Controller
02:00.3 USB controller: MosChip Semiconductor Technology Ltd. MCS9990 PCIe to 4‐Port USB 2.0 Host Controller
02:00.4 USB controller: MosChip Semiconductor Technology Ltd. MCS9990 PCIe to 4‐Port USB 2.0 Host Controller
02:00.5 USB controller: MosChip Semiconductor Technology Ltd. MCS9990 PCIe to 4‐Port USB 2.0 Host Controller
02:00.6 USB controller: MosChip Semiconductor Technology Ltd. MCS9990 PCIe to 4‐Port USB 2.0 Host Controller
02:00.7 USB controller: MosChip Semiconductor Technology Ltd. MCS9990 PCIe to 4‐Port USB 2.0 Host Controller
05:00.0 USB controller: ASMedia Technology Inc. ASM1042 SuperSpeed USB Host Controller
06:00.0 Ethernet controller: Realtek Semiconductor Co., Ltd. RTL8111/8168 PCI Express Gigabit Ethernet controller (rev 06)

I'm using linux-mainline, qemu and seabios from the first page of the thread.

According to dmesg, the board properly activates IOMMU, and the 7870 is properly bound to pci-stub:

[    0.000000] Command line: BOOT_IMAGE=/arch/vmlinuz-linux-mainline root=UUID=c25d5a87-40b9-4a3d-89db-3e263ff8b839 rw quiet pci-stub.ids=1002:679e,1002:aaa0 radeon.dpm=1 radeon.fastfb=1 iommu=pt

dmesg | grep AMD-Vi

[    1.139471] AMD-Vi: Found IOMMU at 0000:00:00.2 cap 0x40
[    1.139472] AMD-Vi:  Extended features:  PreF PPR GT IA
[    1.139475] AMD-Vi: Interrupt remapping enabled
[    1.139647] AMD-Vi: Initialized for Passthrough Mode

dmesg | grep pci-stub

[    0.000000] Command line: BOOT_IMAGE=/arch/vmlinuz-linux-mainline root=UUID=c25d5a87-40b9-4a3d-89db-3e263ff8b839 rw quiet pci-stub.ids=1002:679e,1002:aaa0 radeon.dpm=1 radeon.fastfb=1 iommu=pt
[    0.000000] Kernel command line: BOOT_IMAGE=/arch/vmlinuz-linux-mainline root=UUID=c25d5a87-40b9-4a3d-89db-3e263ff8b839 rw quiet pci-stub.ids=1002:679e,1002:aaa0 radeon.dpm=1 radeon.fastfb=1 iommu=pt
[    1.175368] pci-stub: add 1002:679E sub=FFFFFFFF:FFFFFFFF cls=00000000/00000000
[    1.175380] pci-stub 0000:01:00.0: claimed by stub
[    1.175389] pci-stub: add 1002:AAA0 sub=FFFFFFFF:FFFFFFFF cls=00000000/00000000
[    1.175395] pci-stub 0000:01:00.1: claimed by stub

I'm trying to run the VM with the following script:

#!/bin/bash
QEMU_PA_SAMPLES=128 QEMU_AUDIO_DRV=pa qemu-system-x86_64 -enable-kvm -cpu qemu64 -M q35 -m 8192 \
-smp 2,sockets=1,cores=2,threads=1 \
-bios /usr/share/qemu/bios.bin \
-vga none \
-monitor stdio \
-nographic \
-device ahci,bus=pcie.0,id=ahci \
-drive file=/dev/bcache/by-uuid/d79f9dd0-c884-42f7-b5a9-79db832ea9d0,id=disk,format=raw \
-device ide-hd,bus=ahci.0,drive=disk \
-device ioh3420,bus=pcie.0,addr=1c.0,multifunction=on,port=1,chassis=1,id=root.1 \
-device vfio-pci,host=01:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on,rombar=0,romfile=/home/myuser/Tahiti.rom \
-device vfio-pci,host=01:00.1,bus=pcie.0,multifunction=on \
-device ich9-intel-hda,bus=pcie.0,addr=1b.0,id=sound0 \
-device hda-duplex,id=sound0-codec0,bus=sound0.0,cad=0 \
-usb -usbdevice host:04fc:05d8 \
-boot menu=on

I can see the BIOS output and Windows starting animation on the monitor connected to the Radeon, but as soon as Catalyst tries to initialize the card, I get a BSOD with atikmpag.sys 116 error.
Additionally, after Seabios initializes the 7870, the color palette in Gnome running on the 7660D is totally messed up (I can to post a photo of the screen, if that could be any help). Qemu monitor is not showing any errors, but I get
the follwing entries in journal:

[ 2204.807168] vfio-pci 0000:01:00.0: enabling device (0000 -> 0003)
[ 2204.831486] vfio_ecap_init: 0000:01:00.0 hiding ecap 0x19@0x270
[ 2204.831500] vfio_ecap_init: 0000:01:00.0 hiding ecap 0x1b@0x2d0
[ 2204.873898] vfio-pci 0000:01:00.1: enabling device (0000 -> 0002)
[ 2246.082840] kvm: zapping shadow pages for mmio generation wraparound
[ 2248.441789] AMD-Vi: Event logged [IO_PAGE_FAULT device=01:00.1 domain=0x0002 address=0x00000000857fc000 flags=0x0010]
[ 2248.441798] AMD-Vi: Event logged [IO_PAGE_FAULT device=01:00.1 domain=0x0002 address=0x00000000857fc800 flags=0x0010]
[ 2248.441801] AMD-Vi: Event logged [IO_PAGE_FAULT device=01:00.1 domain=0x0002 address=0x00000000857fc800 flags=0x0030]
[ 2248.441803] AMD-Vi: Event logged [IO_PAGE_FAULT device=01:00.1 domain=0x0002 address=0x00000000857fc800 flags=0x0010]
[ 2248.441805] AMD-Vi: Event logged [IO_PAGE_FAULT device=01:00.1 domain=0x0002 address=0x00000000857fc810 flags=0x0030]
[ 2248.441807] AMD-Vi: Event logged [IO_PAGE_FAULT device=01:00.1 domain=0x0002 address=0x00000000857fc800 flags=0x0010]
[ 2248.441809] AMD-Vi: Event logged [IO_PAGE_FAULT device=01:00.1 domain=0x0002 address=0x00000000857fc810 flags=0x0030]

I tried every workaround I could find, nothing seems to work.
About 7 months ago I was able to passthrough on the same hardware using Xen 4.1 (but I abandoned this setup beacause of problems with card resets and host hangs on Windows reboots, I wasn't aware of the dev eject hack then).

I hope I'm just being stupid and overlooked something obvious. If not, what motherboard and processor would you recommend me to replace my APU setup?

Thanks for all pointers!