Arch Linux

aw

Member

Registered: 2013-10-04

Posts: 921

Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Thanks for the info, it's doing exactly what it's supposed to...

On my AsRock Z87 extrem 6 :
Without patch

### Group 8 ###
    00:1c.0 PCI bridge: Intel Corporation 8 Series/C220 Series Chipset Family PCI Express Root Port #1 (rev d4)
    00:1c.2 PCI bridge: Intel Corporation 8 Series/C220 Series Chipset Family PCI Express Root Port #3 (rev d4)
    00:1c.3 PCI bridge: Intel Corporation 8 Series/C220 Series Chipset Family PCI Express Root Port #4 (rev d4)
    00:1c.4 PCI bridge: Intel Corporation 8 Series/C220 Series Chipset Family PCI Express Root Port #5 (rev d4)
    05:00.0 SATA controller: ASMedia Technology Inc. ASM1062 Serial ATA Controller (rev 01)
    06:00.0 Ethernet controller: Intel Corporation I211 Gigabit Network Connection (rev 03)
    07:00.0 SATA controller: ASMedia Technology Inc. ASM1062 Serial ATA Controller (rev 01)

I've trimmed this down to just the problem caused by the PCH root ports

With old ACS patch :

Doesn't matter, trimmed

Whit the new ACS patch :

### Group 8 ###
    00:1c.0 PCI bridge: Intel Corporation 8 Series/C220 Series Chipset Family PCI Express Root Port #1 (rev d4)
### Group 9 ###
    00:1c.2 PCI bridge: Intel Corporation 8 Series/C220 Series Chipset Family PCI Express Root Port #3 (rev d4)
### Group 10 ###
    00:1c.3 PCI bridge: Intel Corporation 8 Series/C220 Series Chipset Family PCI Express Root Port #4 (rev d4)
### Group 11 ###
    00:1c.4 PCI bridge: Intel Corporation 8 Series/C220 Series Chipset Family PCI Express Root Port #5 (rev d4)
### Group 14 ###
    05:00.0 SATA controller: ASMedia Technology Inc. ASM1062 Serial ATA Controller (rev 01)
### Group 15 ###
    06:00.0 Ethernet controller: Intel Corporation I211 Gigabit Network Connection (rev 03)
### Group 16 ###
    07:00.0 SATA controller: ASMedia Technology Inc. ASM1062 Serial ATA Controller (rev 01)

So all of the problems caused by lack of _PCH_ root more ACS are gone, the devices are properly separated.

On my GigaByte X58 .... :

Doesn't matter, X58 is ICH, not PCH.

So you seem to be expecting this patch to address lack of ACS on every known Intel root port.  As stated in the patch, this only supports PCH root ports.  As of yet, I cannot confirm whether it's safe to use the ACS override patch on other root ports, but I will continue to push Intel for solutions.

---

http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Val532

Member

Registered: 2013-11-13

Posts: 35

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Thanks for the info, it's doing exactly what it's supposed to...

But no, or i do not understand, for the X58 ok it just for compare (and my report is not good).
When i use the new acs patch on my Z87 hardware my two VGA card are on the same iommu group ?!

### Group 1 ###
    00:01.0 PCI bridge: Intel Corporation Xeon E3-1200 v3/4th Gen Core Processor PCI Express x16 Controller (rev 06)
    00:01.1 PCI bridge: Intel Corporation Xeon E3-1200 v3/4th Gen Core Processor PCI Express x8 Controller (rev 06)
    00:01.2 PCI bridge: Intel Corporation Xeon E3-1200 v3/4th Gen Core Processor PCI Express x4 Controller (rev 06)
    01:00.0 VGA compatible controller: Advanced Micro Devices, Inc. [AMD/ATI] Cape Verde XT [Radeon HD 7770 GHz Edition]
    01:00.1 Audio device: Advanced Micro Devices, Inc. [AMD/ATI] Cape Verde/Pitcairn HDMI Audio [Radeon HD 7700/7800 Series]
    02:00.0 VGA compatible controller: Advanced Micro Devices, Inc. [AMD/ATI] Redwood PRO [Radeon HD 5550/5570/5630/6510/6610/7570]
    02:00.1 Audio device: Advanced Micro Devices, Inc. [AMD/ATI] Redwood HDMI Audio [Radeon HD 5000 Series]

But why with the old patch i have that :

### Group 1 ###
    00:01.0 PCI bridge: Intel Corporation Xeon E3-1200 v3/4th Gen Core Processor PCI Express x16 Controller (rev 06)
### Group 2 ###
    00:01.1 PCI bridge: Intel Corporation Xeon E3-1200 v3/4th Gen Core Processor PCI Express x8 Controller (rev 06)
### Group 3 ###
    00:01.2 PCI bridge: Intel Corporation Xeon E3-1200 v3/4th Gen Core Processor PCI Express x4 Controller (rev 06)
### Group 9 ###
    00:1b.0 Audio device: Intel Corporation 8 Series/C220 Series Chipset High Definition Audio Controller (rev 04)
### Group 10 ###
    00:1c.0 PCI bridge: Intel Corporation 8 Series/C220 Series Chipset Family PCI Express Root Port #1 (rev d4)
### Group 11 ###
    00:1c.2 PCI bridge: Intel Corporation 8 Series/C220 Series Chipset Family PCI Express Root Port #3 (rev d4)
### Group 12 ###
    00:1c.3 PCI bridge: Intel Corporation 8 Series/C220 Series Chipset Family PCI Express Root Port #4 (rev d4)
### Group 13 ###
    00:1c.4 PCI bridge: Intel Corporation 8 Series/C220 Series Chipset Family PCI Express Root Port #5 (rev d4)
### Group 16 ###
    01:00.0 VGA compatible controller: Advanced Micro Devices, Inc. [AMD/ATI] Cape Verde XT [Radeon HD 7770 GHz Edition]
    01:00.1 Audio device: Advanced Micro Devices, Inc. [AMD/ATI] Cape Verde/Pitcairn HDMI Audio [Radeon HD 7700/7800 Series]
### Group 17 ###
    02:00.0 VGA compatible controller: Advanced Micro Devices, Inc. [AMD/ATI] Redwood PRO [Radeon HD 5550/5570/5630/6510/6610/7570]
    02:00.1 Audio device: Advanced Micro Devices, Inc. [AMD/ATI] Redwood HDMI Audio [Radeon HD 5000 Series]

If i understand corectly, the root port on the CPU is to "ACS" enable ?

Last edited by Val532 (2014-02-17 22:12:17)

aw

Member

Registered: 2013-10-04

Posts: 921

Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Thanks for the info, it's doing exactly what it's supposed to...

But no, or i do not understand, for the X58 ok it just for compare (and my report is not good).
When i use the new acs patch on my Z87 hardware my two VGA card are on the same iommu group ?!

See this diagram:

The line coming out of the top left of the processor goes to a set of boxes representing the possible root port configuration for processor-based root ports.  You have the last option, which provides 3 slots, however those root ports are sourced from the processor, not the PCH, so are unaffected by this patch that's currently going upstream.

### Group 1 ###
    00:01.0 PCI bridge: Intel Corporation Xeon E3-1200 v3/4th Gen Core Processor PCI Express x16 Controller (rev 06)
    00:01.1 PCI bridge: Intel Corporation Xeon E3-1200 v3/4th Gen Core Processor PCI Express x8 Controller (rev 06)
    00:01.2 PCI bridge: Intel Corporation Xeon E3-1200 v3/4th Gen Core Processor PCI Express x4 Controller (rev 06)
    01:00.0 VGA compatible controller: Advanced Micro Devices, Inc. [AMD/ATI] Cape Verde XT [Radeon HD 7770 GHz Edition]
    01:00.1 Audio device: Advanced Micro Devices, Inc. [AMD/ATI] Cape Verde/Pitcairn HDMI Audio [Radeon HD 7700/7800 Series]
    02:00.0 VGA compatible controller: Advanced Micro Devices, Inc. [AMD/ATI] Redwood PRO [Radeon HD 5550/5570/5630/6510/6610/7570]
    02:00.1 Audio device: Advanced Micro Devices, Inc. [AMD/ATI] Redwood HDMI Audio [Radeon HD 5000 Series]

But why with the old patch i have that :

### Group 1 ###
    00:01.0 PCI bridge: Intel Corporation Xeon E3-1200 v3/4th Gen Core Processor PCI Express x16 Controller (rev 06)
### Group 2 ###
    00:01.1 PCI bridge: Intel Corporation Xeon E3-1200 v3/4th Gen Core Processor PCI Express x8 Controller (rev 06)
### Group 3 ###
    00:01.2 PCI bridge: Intel Corporation Xeon E3-1200 v3/4th Gen Core Processor PCI Express x4 Controller (rev 06)
### Group 9 ###
    00:1b.0 Audio device: Intel Corporation 8 Series/C220 Series Chipset High Definition Audio Controller (rev 04)
### Group 10 ###
    00:1c.0 PCI bridge: Intel Corporation 8 Series/C220 Series Chipset Family PCI Express Root Port #1 (rev d4)
### Group 11 ###
    00:1c.2 PCI bridge: Intel Corporation 8 Series/C220 Series Chipset Family PCI Express Root Port #3 (rev d4)
### Group 12 ###
    00:1c.3 PCI bridge: Intel Corporation 8 Series/C220 Series Chipset Family PCI Express Root Port #4 (rev d4)
### Group 13 ###
    00:1c.4 PCI bridge: Intel Corporation 8 Series/C220 Series Chipset Family PCI Express Root Port #5 (rev d4)
### Group 16 ###
    01:00.0 VGA compatible controller: Advanced Micro Devices, Inc. [AMD/ATI] Cape Verde XT [Radeon HD 7770 GHz Edition]
    01:00.1 Audio device: Advanced Micro Devices, Inc. [AMD/ATI] Cape Verde/Pitcairn HDMI Audio [Radeon HD 7700/7800 Series]
### Group 17 ###
    02:00.0 VGA compatible controller: Advanced Micro Devices, Inc. [AMD/ATI] Redwood PRO [Radeon HD 5550/5570/5630/6510/6610/7570]
    02:00.1 Audio device: Advanced Micro Devices, Inc. [AMD/ATI] Redwood HDMI Audio [Radeon HD 5000 Series]

This is likely because you're using the "downstream" option of the patch.  Regardless, we don't have confirmation from Intel that it's safe to do this for processor-based root ports or what configuration bits we can tweak to make it safe.  So for the time being, we've only been able to create a quirk for PCH-based root ports, which are the ones coming out of the top left box of the Z87 chipset box in the diagram above.  Unfortunately, that's often where lower speed devices are attached, but it's all we've got at the moment.  You'll need to continue to use the ACS override patch and take the risk that there could be non-IOMMU translated DMA between devices.

---

http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Val532

Member

Registered: 2013-11-13

Posts: 35

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

This is likely because you're using the "downstream" option of the patch.  Regardless, we don't have confirmation from Intel that it's safe to do this for processor-based root ports or what configuration bits we can tweak to make it safe.  So for the time being, we've only been able to create a quirk for PCH-based root ports, which are the ones coming out of the top left box of the Z87 chipset box in the diagram above.  Unfortunately, that's often where lower speed devices are attached, but it's all we've got at the moment.  You'll need to continue to use the ACS override patch and take the risk that there could be non-IOMMU translated DMA between devices.

Ok so for the moment i have to use the old patch not the new even if it's probably not a safe patch.

Ok ^^.

Are your planning to make an adition to the new patch ? That add the processor-based root ports a not safe patch but just for people like me with multiple VGA passthrough ^^.

Last edited by Val532 (2014-02-17 22:46:26)

aw

Member

Registered: 2013-10-04

Posts: 921

Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

This is likely because you're using the "downstream" option of the patch.  Regardless, we don't have confirmation from Intel that it's safe to do this for processor-based root ports or what configuration bits we can tweak to make it safe.  So for the time being, we've only been able to create a quirk for PCH-based root ports, which are the ones coming out of the top left box of the Z87 chipset box in the diagram above.  Unfortunately, that's often where lower speed devices are attached, but it's all we've got at the moment.  You'll need to continue to use the ACS override patch and take the risk that there could be non-IOMMU translated DMA between devices.

Ok so for the moment i have to use the old patch not the new because of probably not a safe patch.

Ok ^^.

Are your planning to make an adition to the new patch ? That add the processor-based root ports a not safe patch but just for people like me with multiple VGA passthrough ^^.

No, I proposed the ACS override patch upstream and it was rejected for fear that it once again places a dangerous tool in the arsenal that users are quick to jump on for functionality without fully understanding the implications (as evidenced by this forum ).  The kernel will only accept quirks that the vendors identify as safe.  I expect that as a result of this, we'll see a resurgence of ACS support in Intel chipsets, but getting confirmation and code in place for existing devices is a tedious process.

---

http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Val532

Member

Registered: 2013-11-13

Posts: 35

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

No, I proposed the ACS override patch upstream and it was rejected for fear that it once again places a dangerous tool in the arsenal that users are quick to jump on for functionality without fully understanding the implications (as evidenced by this forum ).  The kernel will only accept quirks that the vendors identify as safe.  I expect that as a result of this, we'll see a resurgence of ACS support in Intel chipsets, but getting confirmation and code in place for existing devices is a tedious process.

Ok i understand, but it's a fun fonction ^^. I understand the acs is a tools that can make some problem for some people (like me ^^), but for the problem is just i was to fast and not completly read the comment in the patch ^^, so it's my fault.

For the moment the old acs is sufficient for the use of people like my (with multiple vga card and multiple guest ^^) but i think a proper implementation of the stuf in the i915 patch is needed, and probably the correction of some bug on iommu for intel (like the AsMedia or marvell sata controler and probably other). And by the way any news for this problem ?

Thanks for the time you spent to answering to me.

PS : If some body is intersted for a Mac OSX 10.x VM with VGA passthrough i can help to make this, because i have a "working" VM with Max OSX 10.8.

AKSN74

Member

From: Taichung, Taiwan

Registered: 2014-02-18

Posts: 52

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Hello everyone, I'm trying to do this these days, finally works, only have one problem.
And...if this question unsuitable to post here, please help me to delete this reply.
Also, my English is bad, so apologize for my bad English.

Post my PC spec first:

CPU: Intel Core i7 3820
MB: ASUS Rampage IV Gene (Intel X79)
RAM: 4GB*2
VGA1: AMD Radeon R9 270 (Host)
VGA2: NVIDIA GeForce GTX770 (Guest)

But I'm not use Archlinux to do this, I'm using Ubuntu 12.04.4 x86_64.

And here is my installation steps

1.  Install VFIO-patched kernel as module and iommu_vfio

1.1  Add settings to kernel (CONFIG_VIFO*=y)

1.2  Add "intel_iommu=on" and "pci-stub.ids=(HID)" into /etc/default/grub

1.3  Set nouveau to blacklist

1.4. Add pci-stub to /etc/modules

When I finished in here, I checked dmesg, IOMMU and pci-stub is work.
Then.

2.  Reboot. And make vfio-bind according from 1st post, copy to /usr/bin and set permission.

2.1  Make service according from 1st post into /etc/systemd/system

2.2  Put GTX770 pci number into /etc/vfio-pci.cfg

3.  Reboot again, now bind GTX770 pci number into VFIO (need root)

4.  Start VM with this command:

# qemu-system-x86_64 -enable-kvm -M q35 -m 4096 -cpu host \
-smp 4,sockets=1,cores=4,threads=1 \
-bios /usr/share/qemu/bios.bin -vga none \
-device ioh3420,bus=pcie.0,addr=1c.0,multifunction=on,port=1,chassis=1,id=root.1 \
-device vfio-pci,host=03:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on \
-device vfio-pci,host=03:00.1,bus=root.1,addr=00.1 \
-device ahci,bus=pcie.0,id=ahci \
-drive file=/home/ak74/test.img,id=disk,format=raw -device ide-hd,bus=ahci.0,drive=disk \
-drive file=/home/ak74/win7.iso,id=isocd -device ide-cd,bus=ahci.1,drive=isocd \

Now a problem is, it can run without errors. But when I typed start command, it only shows

"VNC server started at 127.0.0.1:5900"

And didn't show any VNC screen, just like nothing happen.
But if I plug DVI cable on GTX770 it can shows VM screen (as I saw is WIndows 7 setup.)

After that, I installed TigerVNC and try to connect VM's VNC server, but when I connected, it only show QEMU command.

Maybe I'll try Archlinux to do this, but how can I use VNC to show GTX770's screen?
What step that I doing wrong or I forget to do?

Many thanks.

aw

Member

Registered: 2013-10-04

Posts: 921

Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

# qemu-system-x86_64 -enable-kvm -M q35 -m 4096 -cpu host \
-smp 4,sockets=1,cores=4,threads=1 \
-bios /usr/share/qemu/bios.bin -vga none \
-device ioh3420,bus=pcie.0,addr=1c.0,multifunction=on,port=1,chassis=1,id=root.1 \
-device vfio-pci,host=03:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on \
-device vfio-pci,host=03:00.1,bus=root.1,addr=00.1 \
-device ahci,bus=pcie.0,id=ahci \
-drive file=/home/ak74/test.img,id=disk,format=raw -device ide-hd,bus=ahci.0,drive=disk \
-drive file=/home/ak74/win7.iso,id=isocd -device ide-cd,bus=ahci.1,drive=isocd \

Now a problem is, it can run without errors. But when I typed start command, it only shows

"VNC server started at 127.0.0.1:5900"

When using an assigned VGA, the VNC server doesn't have anything to connect to.  You can use the -nographic option to prevent it from being started.  If you need remote access to the accelerated desktop, you'll need to get it from inside the VM, just like you would if it was a separate physical system (ex. guest-based VNC server).

---

http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

AKSN74

Member

From: Taichung, Taiwan

Registered: 2014-02-18

Posts: 52

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

When using an assigned VGA, the VNC server doesn't have anything to connect to.  You can use the -nographic option to prevent it from being started.  If you need remote access to the accelerated desktop, you'll need to get it from inside the VM, just like you would if it was a separate physical system (ex. guest-based VNC server).

Thanks for your reply.

So let me make a example, if I want to install Windows first, I need to use virtual graphic at beginning
After that, install VNC server that support guest OS into Windows, then disable virtual VGA and virtual graphic, and connect to guest OS when VM is already get into Windows.

Will it work?

aw

Member

Registered: 2013-10-04

Posts: 921

Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

When using an assigned VGA, the VNC server doesn't have anything to connect to.  You can use the -nographic option to prevent it from being started.  If you need remote access to the accelerated desktop, you'll need to get it from inside the VM, just like you would if it was a separate physical system (ex. guest-based VNC server).

Thanks for your reply.

So let me make a example, if I want to install Windows first, I need to use virtual graphic at beginning
After that, install VNC server that support guest OS into Windows, then disable virtual VGA and virtual graphic, and connect to guest OS when VM is already get into Windows.

Will it work?

Yes, if you can't use a physical monitor attached to the card, that's precisely what you'll need to do.

---

http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

zzz3000

Member

Registered: 2014-02-16

Posts: 12

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Video passthrought had worked for me with reboot problem when I used pci-stub metod and started VM via virt-manager.

Reboot  problem dissapeared  for me when I used vfio method and started via console, but host intel graphics becomes spoiled.

After I had installed Radeon Driver it stopped working at all, and host begun to crash.

AKSN74

Member

From: Taichung, Taiwan

Registered: 2014-02-18

Posts: 52

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Yes, if you can't use a physical monitor attached to the card, that's precisely what you'll need to do.

Actually, I can. My monitor support HDMI and DVI, so I can use DVI to see GTX770's VM screen.
The problem is I can't assign my wireless keyboard and mouse into VM, even I assigned USB 2.0 controllor into VFIO.

Also I tried:

-usb -usbdevice host:046d:c534

but got error

qemu: could not add USB device 'host:046d:c534'

anickname

Member

Registered: 2014-01-07

Posts: 23

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Also I tried:

-usb -usbdevice host:046d:c534

but got error

qemu: could not add USB device 'host:046d:c534'

You need to prefix them with 0x (those numbers are in hexadecimal).

anickname

Member

Registered: 2014-01-07

Posts: 23

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

What is the cause of this crash ?

[    0.387396] ------------[ cut here ]------------
[    0.387401] WARNING: CPU: 0 PID: 1 at drivers/pci/search.c:46 pci_find_upstream_pcie_bridge+0x77/0x90()
[    0.387402] Modules linked in:
[    0.387405] CPU: 0 PID: 1 Comm: swapper/0 Not tainted 3.13.0-1-mainline #1
[    0.387406] Hardware name: To Be Filled By O.E.M. To Be Filled By O.E.M./Z77 Professional, BIOS P1.70B 11/12/2013
[    0.387407]  0000000000000009 ffff880803469d20 ffffffff814f5cc4 0000000000000000
[    0.387410]  ffff880803469d58 ffffffff81061abd ffff8808032b3098 ffff8808032b3000
[    0.387412]  ffff8808032b3098 0000000000000000 ffff88080e8097e0 ffff880803469d68
[    0.387413] Call Trace:
[    0.387418]  [<ffffffff814f5cc4>] dump_stack+0x4d/0x6f
[    0.387422]  [<ffffffff81061abd>] warn_slowpath_common+0x7d/0xa0
[    0.387424]  [<ffffffff81061b9a>] warn_slowpath_null+0x1a/0x20
[    0.387426]  [<ffffffff812c9c07>] pci_find_upstream_pcie_bridge+0x77/0x90
[    0.387429]  [<ffffffff813e429d>] intel_iommu_add_device+0x4d/0x220
[    0.387431]  [<ffffffff813da620>] ? bus_set_iommu+0x50/0x50
[    0.387433]  [<ffffffff813da64a>] add_iommu_group+0x2a/0x50
[    0.387436]  [<ffffffff8138c633>] bus_for_each_dev+0x63/0xa0
[    0.387438]  [<ffffffff813da618>] bus_set_iommu+0x48/0x50
[    0.387441]  [<ffffffff8192d0cb>] intel_iommu_init+0xaad/0xba2
[    0.387443]  [<ffffffff81052355>] ? free_init_pages+0x75/0xa0
[    0.387446]  [<ffffffff818e61c5>] ? maybe_link.part.1+0x10e/0x10e
[    0.387449]  [<ffffffff818eed6a>] ? memblock_find_dma_reserve+0x124/0x124
[    0.387451]  [<ffffffff818eed7c>] pci_iommu_init+0x12/0x3c
[    0.387453]  [<ffffffff8100214a>] do_one_initcall+0xfa/0x1b0
[    0.387456]  [<ffffffff810818b5>] ? parse_args+0x225/0x3f0
[    0.387459]  [<ffffffff818e50da>] kernel_init_freeable+0x17b/0x200
[    0.387461]  [<ffffffff818e48d0>] ? do_early_param+0x88/0x88
[    0.387464]  [<ffffffff814e8bf0>] ? rest_init+0x90/0x90
[    0.387466]  [<ffffffff814e8bfe>] kernel_init+0xe/0x130
[    0.387467]  [<ffffffff8150407c>] ret_from_fork+0x7c/0xb0
[    0.387469]  [<ffffffff814e8bf0>] ? rest_init+0x90/0x90
[    0.387474] ---[ end trace a2c46857dc8df419 ]---

nbhs

Member

From: Montevideo, Uruguay

Registered: 2013-05-02

Posts: 402

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Video passthrought had worked for me with reboot problem when I used pci-stub metod and started VM via virt-manager.

Reboot  problem dissapeared  for me when I used vfio method and started via console, but host intel graphics becomes spoiled.

After I had installed Radeon Driver it stopped working at all, and host begun to crash.

You'll probably need to use the i915 patch

aw

Member

Registered: 2013-10-04

Posts: 921

Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

What is the cause of this crash ?

[    0.387396] ------------[ cut here ]------------
[    0.387401] WARNING: CPU: 0 PID: 1 at drivers/pci/search.c:46 pci_find_upstream_pcie_bridge+0x77/0x90()

A trivial search would find https://bugzilla.kernel.org/show_bug.cgi?id=44881

The short answer is broken hardware.  A number of PCIe-to-PCI bridges do not include a PCIe capability, in direct violation of the PCIe spec, confusing the code that tries to find the upstream PCIe bridge and suddenly finds itself on a non-bridge PCIe device.

---

http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

anickname

Member

Registered: 2014-01-07

Posts: 23

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Thanks.

Is it possible to find the device which causes the crash in order to disable it ?

00:00.0 Host bridge: Intel Corporation 2nd Generation Core Processor Family DRAM Controller (rev 09)
00:01.0 PCI bridge: Intel Corporation Xeon E3-1200/2nd Generation Core Processor Family PCI Express Root Port (rev 09)
00:02.0 VGA compatible controller: Intel Corporation 2nd Generation Core Processor Family Integrated Graphics Controller (rev 09)
00:14.0 USB controller: Intel Corporation 7 Series/C210 Series Chipset Family USB xHCI Host Controller (rev 04)
00:16.0 Communication controller: Intel Corporation 7 Series/C210 Series Chipset Family MEI Controller #1 (rev 04)
00:1a.0 USB controller: Intel Corporation 7 Series/C210 Series Chipset Family USB Enhanced Host Controller #2 (rev 04)
00:1b.0 Audio device: Intel Corporation 7 Series/C210 Series Chipset Family High Definition Audio Controller (rev 04)
00:1c.0 PCI bridge: Intel Corporation 7 Series/C210 Series Chipset Family PCI Express Root Port 1 (rev c4)
00:1c.7 PCI bridge: Intel Corporation 7 Series/C210 Series Chipset Family PCI Express Root Port 8 (rev c4)
00:1d.0 USB controller: Intel Corporation 7 Series/C210 Series Chipset Family USB Enhanced Host Controller #1 (rev 04)
00:1f.0 ISA bridge: Intel Corporation Z77 Express Chipset LPC Controller (rev 04)
00:1f.2 SATA controller: Intel Corporation 7 Series/C210 Series Chipset Family 6-port SATA Controller [AHCI mode] (rev 04)
00:1f.3 SMBus: Intel Corporation 7 Series/C210 Series Chipset Family SMBus Controller (rev 04)
01:00.0 VGA compatible controller: Advanced Micro Devices, Inc. [AMD/ATI] Bonaire XT [Radeon HD 7790/8770]
01:00.1 Audio device: Advanced Micro Devices, Inc. [AMD/ATI] Device 0002
02:00.0 SCSI storage controller: LSI Logic / Symbios Logic SAS1068E PCI-Express Fusion-MPT SAS (rev 08)
03:00.0 PCI bridge: PLX Technology, Inc. PEX 8608 8-lane, 8-Port PCI Express Gen 2 (5.0 GT/s) Switch (rev ba)
04:01.0 PCI bridge: PLX Technology, Inc. PEX 8608 8-lane, 8-Port PCI Express Gen 2 (5.0 GT/s) Switch (rev ba)
04:04.0 PCI bridge: PLX Technology, Inc. PEX 8608 8-lane, 8-Port PCI Express Gen 2 (5.0 GT/s) Switch (rev ba)
04:05.0 PCI bridge: PLX Technology, Inc. PEX 8608 8-lane, 8-Port PCI Express Gen 2 (5.0 GT/s) Switch (rev ba)
04:06.0 PCI bridge: PLX Technology, Inc. PEX 8608 8-lane, 8-Port PCI Express Gen 2 (5.0 GT/s) Switch (rev ba)
04:07.0 PCI bridge: PLX Technology, Inc. PEX 8608 8-lane, 8-Port PCI Express Gen 2 (5.0 GT/s) Switch (rev ba)
04:08.0 PCI bridge: PLX Technology, Inc. PEX 8608 8-lane, 8-Port PCI Express Gen 2 (5.0 GT/s) Switch (rev ba)
04:09.0 PCI bridge: PLX Technology, Inc. PEX 8608 8-lane, 8-Port PCI Express Gen 2 (5.0 GT/s) Switch (rev ba)
05:00.0 SATA controller: ASMedia Technology Inc. ASM1062 Serial ATA Controller (rev 01)
06:00.0 PCI bridge: ASMedia Technology Inc. ASM1083/1085 PCIe to PCI Bridge (rev 01)
08:00.0 SATA controller: ASMedia Technology Inc. ASM1062 Serial ATA Controller (rev 01)
09:00.0 Ethernet controller: Broadcom Corporation NetLink BCM57781 Gigabit Ethernet PCIe (rev 10)
0a:00.0 USB controller: Etron Technology, Inc. EJ188/EJ198 USB 3.0 Host Controller
0b:00.0 Ethernet controller: Broadcom Corporation NetLink BCM57781 Gigabit Ethernet PCIe (rev 10)
0c:00.0 FireWire (IEEE 1394): VIA Technologies, Inc. VT6315 Series Firewire Controller
0c:00.1 IDE interface: VIA Technologies, Inc. VT6415 PATA IDE Host Controller (rev a0)

-[0000:00]-+-00.0  Intel Corporation 2nd Generation Core Processor Family DRAM Controller
           +-01.0-[01]--+-00.0  Advanced Micro Devices, Inc. [AMD/ATI] Bonaire XT [Radeon HD 7790/8770]
           |            \-00.1  Advanced Micro Devices, Inc. [AMD/ATI] Device 0002
           +-02.0  Intel Corporation 2nd Generation Core Processor Family Integrated Graphics Controller
           +-14.0  Intel Corporation 7 Series/C210 Series Chipset Family USB xHCI Host Controller
           +-16.0  Intel Corporation 7 Series/C210 Series Chipset Family MEI Controller #1
           +-1a.0  Intel Corporation 7 Series/C210 Series Chipset Family USB Enhanced Host Controller #2
           +-1b.0  Intel Corporation 7 Series/C210 Series Chipset Family High Definition Audio Controller
           +-1c.0-[02]----00.0  LSI Logic / Symbios Logic SAS1068E PCI-Express Fusion-MPT SAS
           +-1c.7-[03-0c]----00.0-[04-0c]--+-01.0-[05]----00.0  ASMedia Technology Inc. ASM1062 Serial ATA Controller
           |                               +-04.0-[06-07]----00.0-[07]--
           |                               +-05.0-[08]----00.0  ASMedia Technology Inc. ASM1062 Serial ATA Controller
           |                               +-06.0-[09]----00.0  Broadcom Corporation NetLink BCM57781 Gigabit Ethernet PCIe
           |                               +-07.0-[0a]----00.0  Etron Technology, Inc. EJ188/EJ198 USB 3.0 Host Controller
           |                               +-08.0-[0b]----00.0  Broadcom Corporation NetLink BCM57781 Gigabit Ethernet PCIe
           |                               \-09.0-[0c]--+-00.0  VIA Technologies, Inc. VT6315 Series Firewire Controller
           |                                            \-00.1  VIA Technologies, Inc. VT6415 PATA IDE Host Controller
           +-1d.0  Intel Corporation 7 Series/C210 Series Chipset Family USB Enhanced Host Controller #1
           +-1f.0  Intel Corporation Z77 Express Chipset LPC Controller
           +-1f.2  Intel Corporation 7 Series/C210 Series Chipset Family 6-port SATA Controller [AHCI mode]
           \-1f.3  Intel Corporation 7 Series/C210 Series Chipset Family SMBus Controller

aw

Member

Registered: 2013-10-04

Posts: 921

Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Thanks.

Is it possible to find the device which causes the crash in order to disable it ?

Probably not unless you're very good with a soldering iron.

06:00.0 PCI bridge: ASMedia Technology Inc. ASM1083/1085 PCIe to PCI Bridge (rev 01)

This is a known offender.

           |                               +-04.0-[06-07]----00.0-[07]--

You don't have anything behind it anyway, so can't you ignore it?  It's not actually a kernel crash, it's just a warning and stack trace.

---

http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

aw

Member

Registered: 2013-10-04

Posts: 921

Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Borderlands folks, I think this is the problem you're trying to describe:

This vantage point in the training area seems to give me a consistently low FPS and high latency and seems invariant to the lighting.  The VM shows about 1/3rd the FPS and 3x the latency.  What is that latency actually measuring anyway?  These are separate win7x64 installs on the same Haswell/GTX660 hardware with Nvidia Experience optimizing the game in both (highest settings for all).

I don't have any solutions yet, but it's good to understand the problem.  I did run the VM with vfio debugging turned on and I'm not seeing any excessive hits to the card, which means we're not bouncing out to userspace regularly as a result of the assigned devices (GPU + audio + NIC in this case).  I see a write to 0x88154 at a rate of about 10/s.  This is an area we must trap because it's an mmio mirror of PCI config space.  That happens regardless of what's going on in the game and also happens at a basic desktop, so if it was causing any trouble I'd expect it would be across the board and not select games.

I'm suspicious of USB passthrough, but my only evidence is that hitting the escape key in the game is not as responsive as bare metal and often requires a second hit.  Assignment of the USB controller might be an option, but didn't just work in my first attempt, maybe an issue with windows drivers.

EDIT: game reported latency appears to just be the inverse of FPS, so it's SPF (seconds per frame)

EDIT2: Further analysis (special thanks to Paolo Bonzini) shows two issues that are hurting performance here.  First is a known issue with Windows that it pounds on the time source, whether it's ACPI PM timer or HPET.  Code just recently went upstream (kernel and qemu) that enables a Hyper-V enlightened timer that can help to reduce this.  To enable it add "hv-time" to your -cpu option, ex. -cpu Haswell,hv-time  This gives me about 20% more FPS and is likely to help performance of any game running in a Windows guest.  The second problem is that we see extensive use of the processor debug registers.  Each move to or from these registers traps into the hypvervisor and they appear to be batched into a context switch function, so we have a burst of them all at once.  I've submitted a support ticket to gearbox software to see if they have any suggestions to avoid these registers.

Last edited by aw (2014-02-19 17:23:21)

---

http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

peedeeX21

Member

Registered: 2014-02-18

Posts: 2

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

hi ppl,

tying to set up a virual machine and passing throug my gt610m.

CPU: Intel Core i5 3317
RAM: 4GB*2
VGA1: Intel HD4000
VGA2: Nvidia GeForce GT 610M
OS: Linux Mint 16

I set up everything like GizmoChicken posted.

I am using a Kernel build with the options:

CONFIG_VFIO_IOMMU_TYPE1=y
CONFIG_VFIO=y
CONFIG_VFIO_PCI=y
CONFIG_VFIO_PCI_VGA=y
CONFIG_INTEL_IOMMU_DEFAULT_ON=y

(stock kernel 1.11.0-12 does also not work...)

The problem ist now, that here are no iommu_group links in /sys/bus/pci/devices/xxxx:xx:xx.x.

Things i did / checked:

• In BIOS the Intel Virtualisation Technology is enable

• intel_iommu=1 is set in /etc/default/grub

• echo "options vfio_iommu_type1 allow_unsafe_interrupts=1" > /etc/modprobe.d/vfio_iommu_type1.conf

Has anybody an idea why this links are missing? thanks.

mostlyharmless

Member

Registered: 2014-01-16

Posts: 72

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

What motherboard are you using?  Intel Virtualization technology just means Vt-x not Vt-d.

peedeeX21

Member

Registered: 2014-02-18

Posts: 2

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

What motherboard are you using?  Intel Virtualization technology just means Vt-x not Vt-d.

oh... i missed that. i got a Lenovo motherboard with Mobile Intel HM77 Express Chipset on it, which does not support vt-d.

thanks for your support.

EDIT:

but how is it possible

 dmesg | grep IOMMU 

tells me

 Intel-IOMMU: enabled 

?

Last edited by peedeeX21 (2014-02-18 21:25:14)

mostlyharmless

Member

Registered: 2014-01-16

Posts: 72

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

I believe that is the kernel acknowledging your intel_iommu=on

AKSN74

Member

From: Taichung, Taiwan

Registered: 2014-02-18

Posts: 52

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

After some test, now I can use this VM functionally.

I tried to use TigerVNC server, but I don't know why if it set to service mode, client application can't connect.
And use user-mode server, it can connect, but after less 10 sec, server will get crashes.

Finally, I use TeamViewer to my temporary VNC. Looks like plug another monitor is the best choice, or does it have better solutions to this?

Also, sound is another problem, too.
I don't know why my QEMU doesn't have ALSA or pa.
When I type -audio-help, it only shows oss and wav. And oss can't use.
How can I find other modules and how to install it?

Sorry for my 'newbie' questions.

DanaGoyette

Member

Registered: 2014-01-03

Posts: 46

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

I just installed a second of the same video card (to forward to the same VM), and no matter what I try, qemu always aborts with an assertion failure.

With EFI and piix4 (since EFI has never worked with q35):

ASSERT /home/dana/Downloads/edk2/PcAtChipsetPkg/PciHostBridgeDxe/PciHostBridge.c(394): ((BOOLEAN)(0==1))

With BIOS mode and q35, I've tried the same settings that somewhat work with a single card:

qemu-system-x86_64: hw/pci/pci.c:232: pcibus_reset: Assertion `bus->irq_count[i] == 0' failed.

With a single video card, I still get this assertion failure, but only when I try to disable the Radeon in Device Manager.

I'd love to get the second video card working, but that's not possible, I'll have to return it.

EDIT: Hmm, after some more wrestling, I may have gotten it working.  Anyway, the "irq count" assertion seems like there must be some broken logic somewhere.

Last edit for the night: I got it to actually, totally, work (aside from likely crashing if I disable a device).  I had to make sure both video cards had the dual-ROM switch set to the GOP mode.  It would still be good to get that assertion fixed, though.

Last edited by DanaGoyette (2014-02-20 07:19:43)