Arch Linux

BulliteShot

Member

Registered: 2013-07-28

Posts: 27

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Driver installed is 12.104.0.0 along with Catalyst 13.4 and I'm using Windows 7 in the VM. I'll mess around with the drivers and see if it likes the BETA or 12.10 versions.

Windows boots fine with the standard driver, but thats not much better than the standard VGA in Qemu to be honest

Try catalyst 12.10, works for me

I've tried 12.10, 13.4 and 13.6 BETA... no luck. Tried passing the VGA BIOS through too, but no luck there either

Edit: We really need a list of hardware combinations which is confirmed to be working.

Last edited by BulliteShot (2013-07-31 11:22:17)

Diggo

Member

Registered: 2013-05-29

Posts: 13

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

I updated my system, which resulted in a nonworking DomU
Guess that´s the point where i throw away my old approach with pci-stub and the unpatched system. I am going to try vfio-vga and report my story :-)

@Bullite: So you´re setup was already working? I´d like to take your mobo into my list of vt-d hardware (*edit: sorry i missed the part with ati bsod*). But i am interested in collecting working combinations also.

Last edited by Diggo (2013-07-31 11:53:59)

BulliteShot

Member

Registered: 2013-07-28

Posts: 27

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

It "works" using the old pci-attach without vfio. I seem to experience random BSOD of the VM due to the ATI driver... and then because it crashed without being able to eject the graphics card, the reset crashes the entire system.

That's why I would like to upgrade to hardware that works using this new vfio method.

My hardware is listed a few packages back.

Diggo

Member

Registered: 2013-05-29

Posts: 13

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Ok i noted it here (http://www.techdudes.de/752/iommu-und-vt-d-taugliche-hardware/ - it´s german *g* so maybe it is not that useful for you). I haven´t read a lot in the last moths on vfio, what´s the major advantage over pci-attach?

Last edited by Diggo (2013-07-31 12:05:14)

BulliteShot

Member

Registered: 2013-07-28

Posts: 27

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Ok i noted it here (http://www.techdudes.de/752/iommu-und-vt-d-taugliche-hardware/ - it´s german *g* so maybe it is not that useful for you). I haven´t read a lot in the last moths on vfio, what´s the major advantage over pci-attach?

The vfio-reset patch and BIOS-level access from the VGA are the only advantages I am aware of.

If there was a similar fix for pci-attach then i'd be happy

mukiex

Member

Registered: 2013-07-27

Posts: 18

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Try catalyst 12.10, works for me

Thanks for the heads up on blacklisting the Intel driver, it worked!

I'm getting a blue screen on windows loads up like many other people. Tried catalyst 12.10. Gonna try 12-6 as it's still after my card's release date, but I don't have high points. 'bout to knock out, I'll update you guys when it's done. Thankfully, being on ZFS makes rolling back to a pre-driver Windows install pretty trivial.

Edit note: Yep, bluescreen on atikmpag.sys , same on Catalyst 12.10 or 12.6

Last edited by mukiex (2013-07-31 17:26:45)

BulliteShot

Member

Registered: 2013-07-28

Posts: 27

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Try catalyst 12.10, works for me

Thanks for the heads up on blacklisting the Intel driver, it worked!

I'm getting a blue screen on windows loads up like many other people. Tried catalyst 12.10. Gonna try 12-6 as it's still after my card's release date, but I don't have high points. 'bout to knock out, I'll update you guys when it's done. Thankfully, being on ZFS makes rolling back to a pre-driver Windows install pretty trivial.

Edit note: Yep, bluescreen on atikmpag.sys , same on Catalyst 12.10 or 12.6

I (was) pretty sure this was because we were using Intel Graphics on the host... but i've disabled Intel Graphics in the BIOS and tried with a 2nd radeon card and I'm seeing the same BSOD. Now I have no idea what the problem could be.

nbhs

Member

From: Montevideo, Uruguay

Registered: 2013-05-02

Posts: 402

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

have you guys tried installing only the driver without the hdmi driver and amd control panel? you could also try playing with the intel_iommu= options

Last edited by nbhs (2013-07-31 19:12:06)

mukiex

Member

Registered: 2013-07-27

Posts: 18

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Hey nbhs. Just to make sure, are there any drivers we need to install that would improve things before doing Catalyst? Is there a specific method?

Right now I'm installing catalyst by letting the installer extract (and closing it), and then right-clicking the VGA device, and pointing "update driver" to that installation folder. Is there a more in-depth method?

Quick note for searchability: It's the issue with atikmpag.sys ending with timestamp 51ae5d89

I'll let you guys know if I find a solution.

Last edited by mukiex (2013-08-01 13:24:53)

nbhs

Member

From: Montevideo, Uruguay

Registered: 2013-05-02

Posts: 402

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Hey nbhs. Just to make sure, are there any drivers we need to install that would improve things before doing Catalyst? Is there a specific method?

Right now I'm installing catalyst by letting the installer extract (and closing it), and then right-clicking the VGA device, and pointing "update driver" to that installation folder. Is there a more in-depth method?

Quick note for searchability: It's the issue with atikmpag.sys ending with timestamp 51ae5d89

I'll let you guys know if I find a solution.

Nope, no driver needed, all i did was run the catalyst installer then rebooted

EDIT: have you tried playing with the intel_iommu= options like i mentioned? for example intel_iommu=pt,igfx_off

Last edited by nbhs (2013-08-01 13:38:07)

mukiex

Member

Registered: 2013-07-27

Posts: 18

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Haven't tried intel_iommu=pt, but I've tried both igfx_ options. I'll let you know what my results are.

Result: With intel_iommu=pt , iommu_group disappears

cannot access /sys/bus/pci/devices/0000:01:00.0/iommu_group/devices: No such file or directory

Last edited by mukiex (2013-08-01 14:04:46)

BulliteShot

Member

Registered: 2013-07-28

Posts: 27

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

It seems that only "locked" Intel processors support IOMMU which means no overclocking.

Can AMD processors overclock aswell as enabling IOMMU?

mukiex

Member

Registered: 2013-07-27

Posts: 18

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Hey BulliteShot: Indeed, that is the case. I own a 2nd i5 because of that ^_^

nbhs

Member

From: Montevideo, Uruguay

Registered: 2013-05-02

Posts: 402

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

It seems that only "locked" Intel processors support IOMMU which means no overclocking.

Can AMD processors overclock aswell as enabling IOMMU?

Yeah IOMMU support is only required on the mobo on amd systems, i own a bulldozer 8120 black box with unlocked multiplier, i can get it to 4.6ghz stable on a closed loop water system

pcworld

Member

From: Germany

Registered: 2013-07-28

Posts: 6

Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Like you, my ASUS motherboard has a broken IVRS table (with precisely the same wrong values as yours, apparently). In addition to overriding it in the kernel, …

How to do that? I'm considering buying an "ASUS M5A97 Evo R2.0", however I would still like to be able to use IOMMU if that board suffers from the same issue.

BulliteShot

Member

Registered: 2013-07-28

Posts: 27

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Hey BulliteShot: Indeed, that is the case. I own a 2nd i5 because of that ^_^

It seems that only "locked" Intel processors support IOMMU which means no overclocking.

Can AMD processors overclock aswell as enabling IOMMU?

Yeah IOMMU support is only required on the mobo on amd systems, i own a bulldozer 8120 black box with unlocked multiplier, i can get it to 4.6ghz stable on a closed loop water system

*sigh* I for one am sick of Intel pushing these limitations.

Goodbye Intel, hello AMD!

nbhs

Member

From: Montevideo, Uruguay

Registered: 2013-05-02

Posts: 402

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Like you, my ASUS motherboard has a broken IVRS table (with precisely the same wrong values as yours, apparently). In addition to overriding it in the kernel, …

How to do that? I'm considering buying an "ASUS M5A97 Evo R2.0", however I would still like to be able to use IOMMU if that board suffers from the same issue.

there's a "ivrs_override=" kernel parameter, IOMMU should work even without it, the workaround is to enable interrupt remapping support on these boards

Last edited by nbhs (2013-08-01 16:10:34)

cmdr

Member

Registered: 2013-08-01

Posts: 8

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Hi,

thank you very much for your guide.
I'm running into some problems when I try to passthrough a gpu to a guest.

Hardware:

• Xeon E3-1230v2

• ASRock Fatal1ty Z77 professional

• NVIDIA GTX-470 (Host)

• ATI Radeon 5450 (Guest)

# cat /proc/cmdline 
BOOT_IMAGE=/vmlinuz-linux-mainline root=/dev/mapper/desktop-root ro pci-stub.ids=1002:68f9,1002:aa68 intel_iommu=on cryptdevice=/dev/sda2:main quiet

# lspci -tv
-[0000:00]-+-00.0  Intel Corporation Xeon E3-1200 v2/Ivy Bridge DRAM Controller
           +-01.0-[01]--+-00.0  NVIDIA Corporation GF100 [GeForce GTX 470]
           |            \-00.1  NVIDIA Corporation GF100 High Definition Audio Controller
           +-01.1-[02]--+-00.0  Advanced Micro Devices, Inc. [AMD/ATI] Cedar [Radeon HD 5000/6000/7350/8350 Series]
           |            \-00.1  Advanced Micro Devices, Inc. [AMD/ATI] Cedar HDMI Audio [Radeon HD 5400/6300 Series]
           +-14.0  Intel Corporation 7 Series/C210 Series Chipset Family USB xHCI Host Controller
           +-16.0  Intel Corporation 7 Series/C210 Series Chipset Family MEI Controller #1
           +-1a.0  Intel Corporation 7 Series/C210 Series Chipset Family USB Enhanced Host Controller #2
           +-1b.0  Intel Corporation 7 Series/C210 Series Chipset Family High Definition Audio Controller
           +-1c.0-[03]--
           +-1c.7-[04-0d]----00.0-[05-0d]--+-01.0-[06]----00.0  ASMedia Technology Inc. ASM1062 Serial ATA Controller
           |                               +-04.0-[07-08]----00.0-[08]--
           |                               +-05.0-[09]----00.0  ASMedia Technology Inc. ASM1062 Serial ATA Controller
           |                               +-06.0-[0a]----00.0  Broadcom Corporation NetLink BCM57781 Gigabit Ethernet PCIe
           |                               +-07.0-[0b]----00.0  Etron Technology, Inc. EJ188/EJ198 USB 3.0 Host Controller
           |                               +-08.0-[0c]----00.0  Broadcom Corporation NetLink BCM57781 Gigabit Ethernet PCIe
           |                               \-09.0-[0d]--+-00.0  VIA Technologies, Inc. VT6315 Series Firewire Controller
           |                                            \-00.1  VIA Technologies, Inc. VT6415 PATA IDE Host Controller
           +-1d.0  Intel Corporation 7 Series/C210 Series Chipset Family USB Enhanced Host Controller #1
           +-1f.0  Intel Corporation Z77 Express Chipset LPC Controller
           +-1f.2  Intel Corporation 7 Series/C210 Series Chipset Family 6-port SATA Controller [AHCI mode]
           \-1f.3  Intel Corporation 7 Series/C210 Series Chipset Family SMBus Controller

Using the second pcie slot, both gpus are in the same iommu_group (thats why i can't bind all devices in the group to vfio).

# readlink /sys/bus/pci/devices/0000\:02\:00.0/iommu_group
../../../../kernel/iommu_groups/1

# ls -l /sys/kernel/iommu_groups/1/devices/
total 0
lrwxrwxrwx 1 root root 0 Aug  1 17:40 0000:00:01.0 -> ../../../../devices/pci0000:00/0000:00:01.0 <- pci-bridge
lrwxrwxrwx 1 root root 0 Aug  1 17:40 0000:00:01.1 -> ../../../../devices/pci0000:00/0000:00:01.1 <- pci-bridge
lrwxrwxrwx 1 root root 0 Aug  1 17:40 0000:01:00.0 -> ../../../../devices/pci0000:00/0000:00:01.0/0000:01:00.0
lrwxrwxrwx 1 root root 0 Aug  1 17:40 0000:01:00.1 -> ../../../../devices/pci0000:00/0000:00:01.0/0000:01:00.1
lrwxrwxrwx 1 root root 0 Aug  1 17:40 0000:02:00.0 -> ../../../../devices/pci0000:00/0000:00:01.1/0000:02:00.0
lrwxrwxrwx 1 root root 0 Aug  1 17:40 0000:02:00.1 -> ../../../../devices/pci0000:00/0000:00:01.1/0000:02:00.1

Using the third pcie slot, the second gpu is in the same iommu_group with the NIC's and stuff.

# lspci | grep ATI
02:00.0 VGA compatible controller: Advanced Micro Devices, Inc. [AMD/ATI] Cedar [Radeon HD 5000/6000/7350/8350 Series]
02:00.1 Audio device: Advanced Micro Devices, Inc. [AMD/ATI] Cedar HDMI Audio [Radeon HD 5400/6300 Series]

# readlink /sys/bus/pci/devices/0000\:02\:00.0/iommu_group
../../../../kernel/iommu_groups/6

# ls -l /sys/kernel/iommu_groups/6/devices/
total 0
lrwxrwxrwx 1 root root 0 Aug  1 18:00 0000:00:1c.0 -> ../../../../devices/pci0000:00/0000:00:1c.0
lrwxrwxrwx 1 root root 0 Aug  1 18:00 0000:00:1c.7 -> ../../../../devices/pci0000:00/0000:00:1c.7
lrwxrwxrwx 1 root root 0 Aug  1 18:00 0000:02:00.0 -> ../../../../devices/pci0000:00/0000:00:1c.0/0000:02:00.0
lrwxrwxrwx 1 root root 0 Aug  1 18:00 0000:02:00.1 -> ../../../../devices/pci0000:00/0000:00:1c.0/0000:02:00.1
lrwxrwxrwx 1 root root 0 Aug  1 18:00 0000:03:00.0 -> ../../../../devices/pci0000:00/0000:00:1c.7/0000:03:00.0
lrwxrwxrwx 1 root root 0 Aug  1 18:00 0000:04:01.0 -> ../../../../devices/pci0000:00/0000:00:1c.7/0000:03:00.0/0000:04:01.0
lrwxrwxrwx 1 root root 0 Aug  1 18:00 0000:04:04.0 -> ../../../../devices/pci0000:00/0000:00:1c.7/0000:03:00.0/0000:04:04.0
lrwxrwxrwx 1 root root 0 Aug  1 18:00 0000:04:05.0 -> ../../../../devices/pci0000:00/0000:00:1c.7/0000:03:00.0/0000:04:05.0
lrwxrwxrwx 1 root root 0 Aug  1 18:00 0000:04:06.0 -> ../../../../devices/pci0000:00/0000:00:1c.7/0000:03:00.0/0000:04:06.0
lrwxrwxrwx 1 root root 0 Aug  1 18:00 0000:04:07.0 -> ../../../../devices/pci0000:00/0000:00:1c.7/0000:03:00.0/0000:04:07.0
lrwxrwxrwx 1 root root 0 Aug  1 18:00 0000:04:08.0 -> ../../../../devices/pci0000:00/0000:00:1c.7/0000:03:00.0/0000:04:08.0
lrwxrwxrwx 1 root root 0 Aug  1 18:00 0000:04:09.0 -> ../../../../devices/pci0000:00/0000:00:1c.7/0000:03:00.0/0000:04:09.0
lrwxrwxrwx 1 root root 0 Aug  1 18:00 0000:05:00.0 -> ../../../../devices/pci0000:00/0000:00:1c.7/0000:03:00.0/0000:04:01.0/0000:05:00.0
lrwxrwxrwx 1 root root 0 Aug  1 18:00 0000:06:00.0 -> ../../../../devices/pci0000:00/0000:00:1c.7/0000:03:00.0/0000:04:04.0/0000:06:00.0
lrwxrwxrwx 1 root root 0 Aug  1 18:00 0000:08:00.0 -> ../../../../devices/pci0000:00/0000:00:1c.7/0000:03:00.0/0000:04:05.0/0000:08:00.0
lrwxrwxrwx 1 root root 0 Aug  1 18:00 0000:09:00.0 -> ../../../../devices/pci0000:00/0000:00:1c.7/0000:03:00.0/0000:04:06.0/0000:09:00.0
lrwxrwxrwx 1 root root 0 Aug  1 18:00 0000:0a:00.0 -> ../../../../devices/pci0000:00/0000:00:1c.7/0000:03:00.0/0000:04:07.0/0000:0a:00.0
lrwxrwxrwx 1 root root 0 Aug  1 18:00 0000:0b:00.0 -> ../../../../devices/pci0000:00/0000:00:1c.7/0000:03:00.0/0000:04:08.0/0000:0b:00.0
lrwxrwxrwx 1 root root 0 Aug  1 18:00 0000:0c:00.0 -> ../../../../devices/pci0000:00/0000:00:1c.7/0000:03:00.0/0000:04:09.0/0000:0c:00.0
lrwxrwxrwx 1 root root 0 Aug  1 18:00 0000:0c:00.1 -> ../../../../devices/pci0000:00/0000:00:1c.7/0000:03:00.0/0000:04:09.0/0000:0c:00.1

Question:
Is there a workaround/fix for this or should i get a new mobo?

Thank you!

nbhs

Member

From: Montevideo, Uruguay

Registered: 2013-05-02

Posts: 402

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Hi,

thank you very much for your guide.
I'm running into some problems when I try to passthrough a gpu to a guest.

Hardware:

• Xeon E3-1230v2

• ASRock Fatal1ty Z77 professional

• NVIDIA GTX-470 (Host)

• ATI Radeon 5450 (Guest)

# cat /proc/cmdline 
BOOT_IMAGE=/vmlinuz-linux-mainline root=/dev/mapper/desktop-root ro pci-stub.ids=1002:68f9,1002:aa68 intel_iommu=on cryptdevice=/dev/sda2:main quiet

# lspci -tv
-[0000:00]-+-00.0  Intel Corporation Xeon E3-1200 v2/Ivy Bridge DRAM Controller
           +-01.0-[01]--+-00.0  NVIDIA Corporation GF100 [GeForce GTX 470]
           |            \-00.1  NVIDIA Corporation GF100 High Definition Audio Controller
           +-01.1-[02]--+-00.0  Advanced Micro Devices, Inc. [AMD/ATI] Cedar [Radeon HD 5000/6000/7350/8350 Series]
           |            \-00.1  Advanced Micro Devices, Inc. [AMD/ATI] Cedar HDMI Audio [Radeon HD 5400/6300 Series]
           +-14.0  Intel Corporation 7 Series/C210 Series Chipset Family USB xHCI Host Controller
           +-16.0  Intel Corporation 7 Series/C210 Series Chipset Family MEI Controller #1
           +-1a.0  Intel Corporation 7 Series/C210 Series Chipset Family USB Enhanced Host Controller #2
           +-1b.0  Intel Corporation 7 Series/C210 Series Chipset Family High Definition Audio Controller
           +-1c.0-[03]--
           +-1c.7-[04-0d]----00.0-[05-0d]--+-01.0-[06]----00.0  ASMedia Technology Inc. ASM1062 Serial ATA Controller
           |                               +-04.0-[07-08]----00.0-[08]--
           |                               +-05.0-[09]----00.0  ASMedia Technology Inc. ASM1062 Serial ATA Controller
           |                               +-06.0-[0a]----00.0  Broadcom Corporation NetLink BCM57781 Gigabit Ethernet PCIe
           |                               +-07.0-[0b]----00.0  Etron Technology, Inc. EJ188/EJ198 USB 3.0 Host Controller
           |                               +-08.0-[0c]----00.0  Broadcom Corporation NetLink BCM57781 Gigabit Ethernet PCIe
           |                               \-09.0-[0d]--+-00.0  VIA Technologies, Inc. VT6315 Series Firewire Controller
           |                                            \-00.1  VIA Technologies, Inc. VT6415 PATA IDE Host Controller
           +-1d.0  Intel Corporation 7 Series/C210 Series Chipset Family USB Enhanced Host Controller #1
           +-1f.0  Intel Corporation Z77 Express Chipset LPC Controller
           +-1f.2  Intel Corporation 7 Series/C210 Series Chipset Family 6-port SATA Controller [AHCI mode]
           \-1f.3  Intel Corporation 7 Series/C210 Series Chipset Family SMBus Controller

Using the second pcie slot, both gpus are in the same iommu_group (thats why i can't bind all devices in the group to vfio).

# readlink /sys/bus/pci/devices/0000\:02\:00.0/iommu_group
../../../../kernel/iommu_groups/1

# ls -l /sys/kernel/iommu_groups/1/devices/
total 0
lrwxrwxrwx 1 root root 0 Aug  1 17:40 0000:00:01.0 -> ../../../../devices/pci0000:00/0000:00:01.0 <- pci-bridge
lrwxrwxrwx 1 root root 0 Aug  1 17:40 0000:00:01.1 -> ../../../../devices/pci0000:00/0000:00:01.1 <- pci-bridge
lrwxrwxrwx 1 root root 0 Aug  1 17:40 0000:01:00.0 -> ../../../../devices/pci0000:00/0000:00:01.0/0000:01:00.0
lrwxrwxrwx 1 root root 0 Aug  1 17:40 0000:01:00.1 -> ../../../../devices/pci0000:00/0000:00:01.0/0000:01:00.1
lrwxrwxrwx 1 root root 0 Aug  1 17:40 0000:02:00.0 -> ../../../../devices/pci0000:00/0000:00:01.1/0000:02:00.0
lrwxrwxrwx 1 root root 0 Aug  1 17:40 0000:02:00.1 -> ../../../../devices/pci0000:00/0000:00:01.1/0000:02:00.1

Using the third pcie slot, the second gpu is in the same iommu_group with the NIC's and stuff.

# lspci | grep ATI
02:00.0 VGA compatible controller: Advanced Micro Devices, Inc. [AMD/ATI] Cedar [Radeon HD 5000/6000/7350/8350 Series]
02:00.1 Audio device: Advanced Micro Devices, Inc. [AMD/ATI] Cedar HDMI Audio [Radeon HD 5400/6300 Series]

# readlink /sys/bus/pci/devices/0000\:02\:00.0/iommu_group
../../../../kernel/iommu_groups/6

# ls -l /sys/kernel/iommu_groups/6/devices/
total 0
lrwxrwxrwx 1 root root 0 Aug  1 18:00 0000:00:1c.0 -> ../../../../devices/pci0000:00/0000:00:1c.0
lrwxrwxrwx 1 root root 0 Aug  1 18:00 0000:00:1c.7 -> ../../../../devices/pci0000:00/0000:00:1c.7
lrwxrwxrwx 1 root root 0 Aug  1 18:00 0000:02:00.0 -> ../../../../devices/pci0000:00/0000:00:1c.0/0000:02:00.0
lrwxrwxrwx 1 root root 0 Aug  1 18:00 0000:02:00.1 -> ../../../../devices/pci0000:00/0000:00:1c.0/0000:02:00.1
lrwxrwxrwx 1 root root 0 Aug  1 18:00 0000:03:00.0 -> ../../../../devices/pci0000:00/0000:00:1c.7/0000:03:00.0
lrwxrwxrwx 1 root root 0 Aug  1 18:00 0000:04:01.0 -> ../../../../devices/pci0000:00/0000:00:1c.7/0000:03:00.0/0000:04:01.0
lrwxrwxrwx 1 root root 0 Aug  1 18:00 0000:04:04.0 -> ../../../../devices/pci0000:00/0000:00:1c.7/0000:03:00.0/0000:04:04.0
lrwxrwxrwx 1 root root 0 Aug  1 18:00 0000:04:05.0 -> ../../../../devices/pci0000:00/0000:00:1c.7/0000:03:00.0/0000:04:05.0
lrwxrwxrwx 1 root root 0 Aug  1 18:00 0000:04:06.0 -> ../../../../devices/pci0000:00/0000:00:1c.7/0000:03:00.0/0000:04:06.0
lrwxrwxrwx 1 root root 0 Aug  1 18:00 0000:04:07.0 -> ../../../../devices/pci0000:00/0000:00:1c.7/0000:03:00.0/0000:04:07.0
lrwxrwxrwx 1 root root 0 Aug  1 18:00 0000:04:08.0 -> ../../../../devices/pci0000:00/0000:00:1c.7/0000:03:00.0/0000:04:08.0
lrwxrwxrwx 1 root root 0 Aug  1 18:00 0000:04:09.0 -> ../../../../devices/pci0000:00/0000:00:1c.7/0000:03:00.0/0000:04:09.0
lrwxrwxrwx 1 root root 0 Aug  1 18:00 0000:05:00.0 -> ../../../../devices/pci0000:00/0000:00:1c.7/0000:03:00.0/0000:04:01.0/0000:05:00.0
lrwxrwxrwx 1 root root 0 Aug  1 18:00 0000:06:00.0 -> ../../../../devices/pci0000:00/0000:00:1c.7/0000:03:00.0/0000:04:04.0/0000:06:00.0
lrwxrwxrwx 1 root root 0 Aug  1 18:00 0000:08:00.0 -> ../../../../devices/pci0000:00/0000:00:1c.7/0000:03:00.0/0000:04:05.0/0000:08:00.0
lrwxrwxrwx 1 root root 0 Aug  1 18:00 0000:09:00.0 -> ../../../../devices/pci0000:00/0000:00:1c.7/0000:03:00.0/0000:04:06.0/0000:09:00.0
lrwxrwxrwx 1 root root 0 Aug  1 18:00 0000:0a:00.0 -> ../../../../devices/pci0000:00/0000:00:1c.7/0000:03:00.0/0000:04:07.0/0000:0a:00.0
lrwxrwxrwx 1 root root 0 Aug  1 18:00 0000:0b:00.0 -> ../../../../devices/pci0000:00/0000:00:1c.7/0000:03:00.0/0000:04:08.0/0000:0b:00.0
lrwxrwxrwx 1 root root 0 Aug  1 18:00 0000:0c:00.0 -> ../../../../devices/pci0000:00/0000:00:1c.7/0000:03:00.0/0000:04:09.0/0000:0c:00.0
lrwxrwxrwx 1 root root 0 Aug  1 18:00 0000:0c:00.1 -> ../../../../devices/pci0000:00/0000:00:1c.7/0000:03:00.0/0000:04:09.0/0000:0c:00.1

Question:
Is there a workaround/fix for this or should i get a new mobo?

Thank you!

Some people were having this same problem with intel boards, take a look at this post on the kvm mailing list, im not sure this is the solution though: http://marc.info/?l=linux-kernel&m=136993923507507&w=2

Last edited by nbhs (2013-08-01 16:26:11)

nbhs

Member

From: Montevideo, Uruguay

Registered: 2013-05-02

Posts: 402

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Hey BulliteShot: Indeed, that is the case. I own a 2nd i5 because of that ^_^

It seems that only "locked" Intel processors support IOMMU which means no overclocking.

Can AMD processors overclock aswell as enabling IOMMU?

Yeah IOMMU support is only required on the mobo on amd systems, i own a bulldozer 8120 black box with unlocked multiplier, i can get it to 4.6ghz stable on a closed loop water system

*sigh* I for one am sick of Intel pushing these limitations.

Goodbye Intel, hello AMD!

I think instead of spending money perhaps you could try posting you issue on the kvm and/or qemu mailing list, remember this is still an experimental feature being worked on

Last edited by nbhs (2013-08-01 16:49:15)

cmdr

Member

Registered: 2013-08-01

Posts: 8

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

-snip-

Some people were having this same problem with intel boards, take a look at this post on the kvm mailing list, im not sure this is the solution though: http://marc.info/?l=linux-kernel&m=136993923507507&w=2

Hi,
thanks for your fast reply. I applied the patch, changed my command line and now the devices have their own groups (second gpu together with pci-bridge).

# cat /proc/cmdline
BOOT_IMAGE=/vmlinuz-linux-mainline root=/dev/mapper/desktop-root ro pci-stub.ids=1002:68f9,1002:aa68 intel_iommu=on pcie_acs_override=id:8086:1e1e cryptdevice=/dev/sda2:main quiet

# ls -l /sys/bus/pci/devices/0000\:02\:00.0/iommu_group/devices/
total 0
lrwxrwxrwx 1 root root 0 Aug  2 02:27 0000:00:1c.0 -> ../../../../devices/pci0000:00/0000:00:1c.0
lrwxrwxrwx 1 root root 0 Aug  2 02:27 0000:02:00.0 -> ../../../../devices/pci0000:00/0000:00:1c.0/0000:02:00.0
lrwxrwxrwx 1 root root 0 Aug  2 02:27 0000:02:00.1 -> ../../../../devices/pci0000:00/0000:00:1c.0/0000:02:00.1

[   53.014493] vfio-pci 0000:02:00.0: enabling device (0000 -> 0003)

I bind the gpu manually to vfio and qemu starts without errors but the attached screen stays black.

Edit: I blacklisted nouveau, nvidia, radeon and tried it without X but i had no success.

qemu-system-x86_64 -enable-kvm -M q35 -m 1024 -cpu host \
-smp 6,sockets=1,cores=6,threads=1 \
-bios /usr/share/qemu/bios.bin -vga none \
-device ioh3420,bus=pcie.0,addr=1c.0,multifunction=on,port=1,chassis=1,id=root.1 \
-device vfio-pci,host=02:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on \
-device vfio-pci,host=02:00.1,bus=root.1,addr=00.1

Last edited by cmdr (2013-08-02 01:53:54)

nbhs

Member

From: Montevideo, Uruguay

Registered: 2013-05-02

Posts: 402

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

-snip-

Some people were having this same problem with intel boards, take a look at this post on the kvm mailing list, im not sure this is the solution though: http://marc.info/?l=linux-kernel&m=136993923507507&w=2

Hi,
thanks for your fast reply. I applied the patch, changed my command line and now the devices have their own groups (second gpu together with pci-bridge).

# cat /proc/cmdline
BOOT_IMAGE=/vmlinuz-linux-mainline root=/dev/mapper/desktop-root ro pci-stub.ids=1002:68f9,1002:aa68 intel_iommu=on pcie_acs_override=id:8086:1e1e cryptdevice=/dev/sda2:main quiet

# ls -l /sys/bus/pci/devices/0000\:02\:00.0/iommu_group/devices/
total 0
lrwxrwxrwx 1 root root 0 Aug  2 02:27 0000:00:1c.0 -> ../../../../devices/pci0000:00/0000:00:1c.0
lrwxrwxrwx 1 root root 0 Aug  2 02:27 0000:02:00.0 -> ../../../../devices/pci0000:00/0000:00:1c.0/0000:02:00.0
lrwxrwxrwx 1 root root 0 Aug  2 02:27 0000:02:00.1 -> ../../../../devices/pci0000:00/0000:00:1c.0/0000:02:00.1

[   53.014493] vfio-pci 0000:02:00.0: enabling device (0000 -> 0003)

I bind the gpu manually to vfio and qemu starts without errors but the attached screen stays black.

Edit: I blacklisted nouveau, nvidia, radeon and tried it without X but i had no success.

qemu-system-x86_64 -enable-kvm -M q35 -m 1024 -cpu host \
-smp 6,sockets=1,cores=6,threads=1 \
-bios /usr/share/qemu/bios.bin -vga none \
-device ioh3420,bus=pcie.0,addr=1c.0,multifunction=on,port=1,chassis=1,id=root.1 \
-device vfio-pci,host=02:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on \
-device vfio-pci,host=02:00.1,bus=root.1,addr=00.1

Are you using the patched seabios? have you tried swapping cards? perhaps using pcie_acs_override=downstream or mutlifucion (i have really no idea here)?

Last edited by nbhs (2013-08-02 02:46:52)

reintseri

Member

Registered: 2013-07-16

Posts: 6

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Ok I was meant to post my specs earlier, but summer vacation got in the way .

So. I'm using a 2009 Mac Pro with OS X and Arch Linux dual boot as a basis. 2 Xeon processors (8 cores) and 20 gigs of ram . Modified kernel, QEMU and Seabios - downloaded from this thread.

Host graphics card - nVidia GeForce 9500 GT (Mac Edition) (Mac EFI boot loader needs this in the first PCIe slot to boot unfortunately.) Tested graphics cards to work in the second 16x PCIe slot: Radeon HD 5450 and Radeon HD 7750.

So on their own they work fine. But I've also tried a 3-graphics-cards-setup to have two VMs. At this point it's not so stable anymore. I can have one Windows 8 VM with the HD 7750 running together with Ubuntu 12.04 with HD5450, but as soon as I install FGLRX or catalyst driver for it- it crashes and won't boot anymore. Also various error messages from the kernel can be seen during start and shutdown. Windows installation fails even earlier. So at this setup the HD 5450 is on a 4x PCIe slot.

Now the problem might be similar to what was posted earlier about the iommu_groups, the HD 7750 (on the 2nd 16x slot) shows only:

0000:06:00.0  0000:06:00.1

But the HD 5450 (on one of the free 4x slots) shows:

0000:02:04.0  0000:04:00.0  0000:04:00.1

So it includes one of the pcie switches:

02:04.0 PCI bridge: Integrated Device Technology, Inc. [IDT] PES12T3G2 PCI Express Gen2 Switch (rev 01)

Could this be the source of the problem? And could I change the iommu_groups somehow.

Thanks again,
Teemu Kämäräinen

cmdr

Member

Registered: 2013-08-01

Posts: 8

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Some people were having this same problem with intel boards, take a look at this post on the kvm mailing list, im not sure this is the solution though: http://marc.info/?l=linux-kernel&m=136993923507507&w=2

Hi,
thanks for your fast reply. I applied the patch, changed my command line and now the devices have their own groups (second gpu together with pci-bridge).

# cat /proc/cmdline
BOOT_IMAGE=/vmlinuz-linux-mainline root=/dev/mapper/desktop-root ro pci-stub.ids=1002:68f9,1002:aa68 intel_iommu=on pcie_acs_override=id:8086:1e1e cryptdevice=/dev/sda2:main quiet

# ls -l /sys/bus/pci/devices/0000\:02\:00.0/iommu_group/devices/
total 0
lrwxrwxrwx 1 root root 0 Aug  2 02:27 0000:00:1c.0 -> ../../../../devices/pci0000:00/0000:00:1c.0
lrwxrwxrwx 1 root root 0 Aug  2 02:27 0000:02:00.0 -> ../../../../devices/pci0000:00/0000:00:1c.0/0000:02:00.0
lrwxrwxrwx 1 root root 0 Aug  2 02:27 0000:02:00.1 -> ../../../../devices/pci0000:00/0000:00:1c.0/0000:02:00.1

[   53.014493] vfio-pci 0000:02:00.0: enabling device (0000 -> 0003)

I bind the gpu manually to vfio and qemu starts without errors but the attached screen stays black.

Edit: I blacklisted nouveau, nvidia, radeon and tried it without X but i had no success.

qemu-system-x86_64 -enable-kvm -M q35 -m 1024 -cpu host \
-smp 6,sockets=1,cores=6,threads=1 \
-bios /usr/share/qemu/bios.bin -vga none \
-device ioh3420,bus=pcie.0,addr=1c.0,multifunction=on,port=1,chassis=1,id=root.1 \
-device vfio-pci,host=02:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on \
-device vfio-pci,host=02:00.1,bus=root.1,addr=00.1

Are you using the patched seabios? have you tried swapping cards? perhaps using pcie_acs_override=downstream or mutlifucion (i have really no idea here)?

Some minutes ago i applied this patch (http://lists.gnu.org/archive/html/qemu- … 04300.html) to the nvidia source and build it (i am using PKGBUILD from nvidia-all). Now i can see the boot messages on the attached screen but i have no boot media to install and test windows .
What i have done to get it to this point:

• clean arch install

• kernel package from the first post with this patch (http://www.spinics.net/lists/kvm/msg92163.html)

• qemu & seabios package from the first post

• nvidia-all (https://aur.archlinux.org/packages/nvidia-all/) with this patch (http://lists.gnu.org/archive/html/qemu- … 04300.html)

• manually attached devices to vfio (like in the usage examples http://lxr.free-electrons.com/source/Do … /vfio.txt)

$ cat /proc/cmdline 
BOOT_IMAGE=/vmlinuz-linux-mainline root=/dev/mapper/desktop-root ro pci-stub.ids=1002:68f9,1002:aa68 intel_iommu=on pcie_acs_override=id:8086:1e1e cryptdevice=/dev/sda2:main quiet

I think this works now, i'll check it later that day with a windows installation. Thanks for your ideas!

nbhs

Member

From: Montevideo, Uruguay

Registered: 2013-05-02

Posts: 402

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Hi,
thanks for your fast reply. I applied the patch, changed my command line and now the devices have their own groups (second gpu together with pci-bridge).

I bind the gpu manually to vfio and qemu starts without errors but the attached screen stays black.

Edit: I blacklisted nouveau, nvidia, radeon and tried it without X but i had no success.

Are you using the patched seabios? have you tried swapping cards? perhaps using pcie_acs_override=downstream or mutlifucion (i have really no idea here)?

Some minutes ago i applied this patch (http://lists.gnu.org/archive/html/qemu- … 04300.html) to the nvidia source and build it (i am using PKGBUILD from nvidia-all). Now i can see the boot messages on the attached screen but i have no boot media to install and test windows .
What i have done to get it to this point:

• clean arch install

• kernel package from the first post with this patch (http://www.spinics.net/lists/kvm/msg92163.html)

• qemu & seabios package from the first post

• nvidia-all (https://aur.archlinux.org/packages/nvidia-all/) with this patch (http://lists.gnu.org/archive/html/qemu- … 04300.html)

• manually attached devices to vfio (like in the usage examples http://lxr.free-electrons.com/source/Do … /vfio.txt)

$ cat /proc/cmdline 
BOOT_IMAGE=/vmlinuz-linux-mainline root=/dev/mapper/desktop-root ro pci-stub.ids=1002:68f9,1002:aa68 intel_iommu=on pcie_acs_override=id:8086:1e1e cryptdevice=/dev/sda2:main quiet

I think this works now, i'll check it later that day with a windows installation. Thanks for your ideas!

Interesting, thanks a lot for your report