You are not logged in.

#376 2013-07-31 09:21:07

BulliteShot
Member
Registered: 2013-07-28
Posts: 26

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

nbhs wrote:
BulliteShot wrote:

Driver installed is 12.104.0.0 along with Catalyst 13.4 and I'm using Windows 7 in the VM. I'll mess around with the drivers and see if it likes the BETA or 12.10 versions.

Windows boots fine with the standard driver, but thats not much better than the standard VGA in Qemu to be honest tongue

Try catalyst 12.10, works for me

I've tried 12.10, 13.4 and 13.6 BETA... no luck. Tried passing the VGA BIOS through too, but no luck there either hmm

Edit: We really need a list of hardware combinations which is confirmed to be working.

Last edited by BulliteShot (2013-07-31 11:22:17)

Offline

#377 2013-07-31 11:53:21

Diggo
Member
Registered: 2013-05-29
Posts: 13

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

I updated my system, which resulted in a nonworking DomU hmm
Guess that´s the point where i throw away my old approach with pci-stub and the unpatched system. I am going to try vfio-vga and report my story :-)

@Bullite: So you´re setup was already working? I´d like to take your mobo into my list of vt-d hardware (*edit: sorry i missed the part with ati bsod*). But i am interested in collecting working combinations also.

Last edited by Diggo (2013-07-31 11:53:59)

Offline

#378 2013-07-31 11:59:44

BulliteShot
Member
Registered: 2013-07-28
Posts: 26

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

It "works" using the old pci-attach without vfio. I seem to experience random BSOD of the VM due to the ATI driver... and then because it crashed without being able to eject the graphics card, the reset crashes the entire system.

That's why I would like to upgrade to hardware that works using this new vfio method.

My hardware is listed a few packages back.

Offline

#379 2013-07-31 12:02:37

Diggo
Member
Registered: 2013-05-29
Posts: 13

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Ok i noted it here (http://www.techdudes.de/752/iommu-und-vt-d-taugliche-hardware/ - it´s german *g* so maybe it is not that useful for you). I haven´t read a lot in the last moths on vfio, what´s the major advantage over pci-attach?

Last edited by Diggo (2013-07-31 12:05:14)

Offline

#380 2013-07-31 12:53:01

BulliteShot
Member
Registered: 2013-07-28
Posts: 26

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Diggo wrote:

Ok i noted it here (http://www.techdudes.de/752/iommu-und-vt-d-taugliche-hardware/ - it´s german *g* so maybe it is not that useful for you). I haven´t read a lot in the last moths on vfio, what´s the major advantage over pci-attach?

The vfio-reset patch and BIOS-level access from the VGA are the only advantages I am aware of.

If there was a similar fix for pci-attach then i'd be happy

Offline

#381 2013-07-31 17:18:30

mukiex
Member
Registered: 2013-07-27
Posts: 18

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

nbhs wrote:

Try catalyst 12.10, works for me

Thanks for the heads up on blacklisting the Intel driver, it worked!

I'm getting a blue screen on windows loads up like many other people. Tried catalyst 12.10. Gonna try 12-6 as it's still after my card's release date, but I don't have high points. 'bout to knock out, I'll update you guys when it's done. Thankfully, being on ZFS makes rolling back to a pre-driver Windows install pretty trivial.

Edit note: Yep, bluescreen on atikmpag.sys , same on Catalyst 12.10 or 12.6

Last edited by mukiex (2013-07-31 17:26:45)

Offline

#382 2013-07-31 18:36:45

BulliteShot
Member
Registered: 2013-07-28
Posts: 26

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

mukiex wrote:
nbhs wrote:

Try catalyst 12.10, works for me

Thanks for the heads up on blacklisting the Intel driver, it worked!

I'm getting a blue screen on windows loads up like many other people. Tried catalyst 12.10. Gonna try 12-6 as it's still after my card's release date, but I don't have high points. 'bout to knock out, I'll update you guys when it's done. Thankfully, being on ZFS makes rolling back to a pre-driver Windows install pretty trivial.

Edit note: Yep, bluescreen on atikmpag.sys , same on Catalyst 12.10 or 12.6

I (was) pretty sure this was because we were using Intel Graphics on the host... but i've disabled Intel Graphics in the BIOS and tried with a 2nd radeon card and I'm seeing the same BSOD. Now I have no idea what the problem could be.

Offline

#383 2013-07-31 19:10:25

nbhs
Member
From: Montevideo, Uruguay
Registered: 2013-05-02
Posts: 402

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

have you guys tried installing only the driver without the hdmi driver and amd control panel? you could also try playing with the intel_iommu= options

Last edited by nbhs (2013-07-31 19:12:06)

Offline

#384 2013-08-01 10:59:30

mukiex
Member
Registered: 2013-07-27
Posts: 18

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Hey nbhs. Just to make sure, are there any drivers we need to install that would improve things before doing Catalyst? Is there a specific method?

Right now I'm installing catalyst by letting the installer extract (and closing it), and then right-clicking the VGA device, and pointing "update driver" to that installation folder. Is there a more in-depth method?

Quick note for searchability: It's the issue with atikmpag.sys ending with timestamp 51ae5d89

I'll let you guys know if I find a solution.

Last edited by mukiex (2013-08-01 13:24:53)

Offline

#385 2013-08-01 13:33:45

nbhs
Member
From: Montevideo, Uruguay
Registered: 2013-05-02
Posts: 402

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

mukiex wrote:

Hey nbhs. Just to make sure, are there any drivers we need to install that would improve things before doing Catalyst? Is there a specific method?

Right now I'm installing catalyst by letting the installer extract (and closing it), and then right-clicking the VGA device, and pointing "update driver" to that installation folder. Is there a more in-depth method?

Quick note for searchability: It's the issue with atikmpag.sys ending with timestamp 51ae5d89

I'll let you guys know if I find a solution.

Nope, no driver needed, all i did was run the catalyst installer then rebooted

EDIT: have you tried playing with the intel_iommu= options like i mentioned? for example intel_iommu=pt,igfx_off

Last edited by nbhs (2013-08-01 13:38:07)

Offline

#386 2013-08-01 13:55:12

mukiex
Member
Registered: 2013-07-27
Posts: 18

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Haven't tried intel_iommu=pt, but I've tried both igfx_ options. I'll let you know what my results are.

Result: With intel_iommu=pt , iommu_group disappears

cannot access /sys/bus/pci/devices/0000:01:00.0/iommu_group/devices: No such file or directory

Last edited by mukiex (2013-08-01 14:04:46)

Offline

#387 2013-08-01 14:05:06

BulliteShot
Member
Registered: 2013-07-28
Posts: 26

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

It seems that only "locked" Intel processors support IOMMU which means no overclocking.

Can AMD processors overclock aswell as enabling IOMMU?

Offline

#388 2013-08-01 14:22:43

mukiex
Member
Registered: 2013-07-27
Posts: 18

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Hey BulliteShot: Indeed, that is the case. I own a 2nd i5 because of that ^_^

Offline

#389 2013-08-01 15:16:16

nbhs
Member
From: Montevideo, Uruguay
Registered: 2013-05-02
Posts: 402

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

BulliteShot wrote:

It seems that only "locked" Intel processors support IOMMU which means no overclocking.

Can AMD processors overclock aswell as enabling IOMMU?

Yeah IOMMU support is only required on the mobo on amd systems, i own a bulldozer 8120 black box with unlocked multiplier, i can get it to 4.6ghz stable on a closed loop water system

Offline

#390 2013-08-01 15:39:08

pcworld
Member
From: Germany
Registered: 2013-07-28
Posts: 6
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

jgott wrote:

Like you, my ASUS motherboard has a broken IVRS table (with precisely the same wrong values as yours, apparently). In addition to overriding it in the kernel, …

How to do that? I'm considering buying an "ASUS M5A97 Evo R2.0", however I would still like to be able to use IOMMU if that board suffers from the same issue.

Offline

#391 2013-08-01 16:00:49

BulliteShot
Member
Registered: 2013-07-28
Posts: 26

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

mukiex wrote:

Hey BulliteShot: Indeed, that is the case. I own a 2nd i5 because of that ^_^

nbhs wrote:
BulliteShot wrote:

It seems that only "locked" Intel processors support IOMMU which means no overclocking.

Can AMD processors overclock aswell as enabling IOMMU?

Yeah IOMMU support is only required on the mobo on amd systems, i own a bulldozer 8120 black box with unlocked multiplier, i can get it to 4.6ghz stable on a closed loop water system

*sigh* I for one am sick of Intel pushing these limitations.

Goodbye Intel, hello AMD!

Offline

#392 2013-08-01 16:06:11

nbhs
Member
From: Montevideo, Uruguay
Registered: 2013-05-02
Posts: 402

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

pcworld wrote:
jgott wrote:

Like you, my ASUS motherboard has a broken IVRS table (with precisely the same wrong values as yours, apparently). In addition to overriding it in the kernel, …

How to do that? I'm considering buying an "ASUS M5A97 Evo R2.0", however I would still like to be able to use IOMMU if that board suffers from the same issue.

there's a "ivrs_override=" kernel parameter, IOMMU should work even without it, the workaround is to enable interrupt remapping support on these boards

Last edited by nbhs (2013-08-01 16:10:34)

Offline

#393 2013-08-01 16:11:22

cmdr
Member
Registered: 2013-08-01
Posts: 8

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Hi,

thank you very much for your guide.
I'm running into some problems when I try to passthrough a gpu to a guest.

Hardware:

  • Xeon E3-1230v2

  • ASRock Fatal1ty Z77 professional

  • NVIDIA GTX-470 (Host)

  • ATI Radeon 5450 (Guest)

# cat /proc/cmdline 
BOOT_IMAGE=/vmlinuz-linux-mainline root=/dev/mapper/desktop-root ro pci-stub.ids=1002:68f9,1002:aa68 intel_iommu=on cryptdevice=/dev/sda2:main quiet
# lspci -tv
-[0000:00]-+-00.0  Intel Corporation Xeon E3-1200 v2/Ivy Bridge DRAM Controller
           +-01.0-[01]--+-00.0  NVIDIA Corporation GF100 [GeForce GTX 470]
           |            \-00.1  NVIDIA Corporation GF100 High Definition Audio Controller
           +-01.1-[02]--+-00.0  Advanced Micro Devices, Inc. [AMD/ATI] Cedar [Radeon HD 5000/6000/7350/8350 Series]
           |            \-00.1  Advanced Micro Devices, Inc. [AMD/ATI] Cedar HDMI Audio [Radeon HD 5400/6300 Series]
           +-14.0  Intel Corporation 7 Series/C210 Series Chipset Family USB xHCI Host Controller
           +-16.0  Intel Corporation 7 Series/C210 Series Chipset Family MEI Controller #1
           +-1a.0  Intel Corporation 7 Series/C210 Series Chipset Family USB Enhanced Host Controller #2
           +-1b.0  Intel Corporation 7 Series/C210 Series Chipset Family High Definition Audio Controller
           +-1c.0-[03]--
           +-1c.7-[04-0d]----00.0-[05-0d]--+-01.0-[06]----00.0  ASMedia Technology Inc. ASM1062 Serial ATA Controller
           |                               +-04.0-[07-08]----00.0-[08]--
           |                               +-05.0-[09]----00.0  ASMedia Technology Inc. ASM1062 Serial ATA Controller
           |                               +-06.0-[0a]----00.0  Broadcom Corporation NetLink BCM57781 Gigabit Ethernet PCIe
           |                               +-07.0-[0b]----00.0  Etron Technology, Inc. EJ188/EJ198 USB 3.0 Host Controller
           |                               +-08.0-[0c]----00.0  Broadcom Corporation NetLink BCM57781 Gigabit Ethernet PCIe
           |                               \-09.0-[0d]--+-00.0  VIA Technologies, Inc. VT6315 Series Firewire Controller
           |                                            \-00.1  VIA Technologies, Inc. VT6415 PATA IDE Host Controller
           +-1d.0  Intel Corporation 7 Series/C210 Series Chipset Family USB Enhanced Host Controller #1
           +-1f.0  Intel Corporation Z77 Express Chipset LPC Controller
           +-1f.2  Intel Corporation 7 Series/C210 Series Chipset Family 6-port SATA Controller [AHCI mode]
           \-1f.3  Intel Corporation 7 Series/C210 Series Chipset Family SMBus Controller

Using the second pcie slot, both gpus are in the same iommu_group (thats why i can't bind all devices in the group to vfio).

# readlink /sys/bus/pci/devices/0000\:02\:00.0/iommu_group
../../../../kernel/iommu_groups/1

# ls -l /sys/kernel/iommu_groups/1/devices/
total 0
lrwxrwxrwx 1 root root 0 Aug  1 17:40 0000:00:01.0 -> ../../../../devices/pci0000:00/0000:00:01.0 <- pci-bridge
lrwxrwxrwx 1 root root 0 Aug  1 17:40 0000:00:01.1 -> ../../../../devices/pci0000:00/0000:00:01.1 <- pci-bridge
lrwxrwxrwx 1 root root 0 Aug  1 17:40 0000:01:00.0 -> ../../../../devices/pci0000:00/0000:00:01.0/0000:01:00.0
lrwxrwxrwx 1 root root 0 Aug  1 17:40 0000:01:00.1 -> ../../../../devices/pci0000:00/0000:00:01.0/0000:01:00.1
lrwxrwxrwx 1 root root 0 Aug  1 17:40 0000:02:00.0 -> ../../../../devices/pci0000:00/0000:00:01.1/0000:02:00.0
lrwxrwxrwx 1 root root 0 Aug  1 17:40 0000:02:00.1 -> ../../../../devices/pci0000:00/0000:00:01.1/0000:02:00.1

Using the third pcie slot, the second gpu is in the same iommu_group with the NIC's and stuff.

# lspci | grep ATI
02:00.0 VGA compatible controller: Advanced Micro Devices, Inc. [AMD/ATI] Cedar [Radeon HD 5000/6000/7350/8350 Series]
02:00.1 Audio device: Advanced Micro Devices, Inc. [AMD/ATI] Cedar HDMI Audio [Radeon HD 5400/6300 Series]

# readlink /sys/bus/pci/devices/0000\:02\:00.0/iommu_group
../../../../kernel/iommu_groups/6

# ls -l /sys/kernel/iommu_groups/6/devices/
total 0
lrwxrwxrwx 1 root root 0 Aug  1 18:00 0000:00:1c.0 -> ../../../../devices/pci0000:00/0000:00:1c.0
lrwxrwxrwx 1 root root 0 Aug  1 18:00 0000:00:1c.7 -> ../../../../devices/pci0000:00/0000:00:1c.7
lrwxrwxrwx 1 root root 0 Aug  1 18:00 0000:02:00.0 -> ../../../../devices/pci0000:00/0000:00:1c.0/0000:02:00.0
lrwxrwxrwx 1 root root 0 Aug  1 18:00 0000:02:00.1 -> ../../../../devices/pci0000:00/0000:00:1c.0/0000:02:00.1
lrwxrwxrwx 1 root root 0 Aug  1 18:00 0000:03:00.0 -> ../../../../devices/pci0000:00/0000:00:1c.7/0000:03:00.0
lrwxrwxrwx 1 root root 0 Aug  1 18:00 0000:04:01.0 -> ../../../../devices/pci0000:00/0000:00:1c.7/0000:03:00.0/0000:04:01.0
lrwxrwxrwx 1 root root 0 Aug  1 18:00 0000:04:04.0 -> ../../../../devices/pci0000:00/0000:00:1c.7/0000:03:00.0/0000:04:04.0
lrwxrwxrwx 1 root root 0 Aug  1 18:00 0000:04:05.0 -> ../../../../devices/pci0000:00/0000:00:1c.7/0000:03:00.0/0000:04:05.0
lrwxrwxrwx 1 root root 0 Aug  1 18:00 0000:04:06.0 -> ../../../../devices/pci0000:00/0000:00:1c.7/0000:03:00.0/0000:04:06.0
lrwxrwxrwx 1 root root 0 Aug  1 18:00 0000:04:07.0 -> ../../../../devices/pci0000:00/0000:00:1c.7/0000:03:00.0/0000:04:07.0
lrwxrwxrwx 1 root root 0 Aug  1 18:00 0000:04:08.0 -> ../../../../devices/pci0000:00/0000:00:1c.7/0000:03:00.0/0000:04:08.0
lrwxrwxrwx 1 root root 0 Aug  1 18:00 0000:04:09.0 -> ../../../../devices/pci0000:00/0000:00:1c.7/0000:03:00.0/0000:04:09.0
lrwxrwxrwx 1 root root 0 Aug  1 18:00 0000:05:00.0 -> ../../../../devices/pci0000:00/0000:00:1c.7/0000:03:00.0/0000:04:01.0/0000:05:00.0
lrwxrwxrwx 1 root root 0 Aug  1 18:00 0000:06:00.0 -> ../../../../devices/pci0000:00/0000:00:1c.7/0000:03:00.0/0000:04:04.0/0000:06:00.0
lrwxrwxrwx 1 root root 0 Aug  1 18:00 0000:08:00.0 -> ../../../../devices/pci0000:00/0000:00:1c.7/0000:03:00.0/0000:04:05.0/0000:08:00.0
lrwxrwxrwx 1 root root 0 Aug  1 18:00 0000:09:00.0 -> ../../../../devices/pci0000:00/0000:00:1c.7/0000:03:00.0/0000:04:06.0/0000:09:00.0
lrwxrwxrwx 1 root root 0 Aug  1 18:00 0000:0a:00.0 -> ../../../../devices/pci0000:00/0000:00:1c.7/0000:03:00.0/0000:04:07.0/0000:0a:00.0
lrwxrwxrwx 1 root root 0 Aug  1 18:00 0000:0b:00.0 -> ../../../../devices/pci0000:00/0000:00:1c.7/0000:03:00.0/0000:04:08.0/0000:0b:00.0
lrwxrwxrwx 1 root root 0 Aug  1 18:00 0000:0c:00.0 -> ../../../../devices/pci0000:00/0000:00:1c.7/0000:03:00.0/0000:04:09.0/0000:0c:00.0
lrwxrwxrwx 1 root root 0 Aug  1 18:00 0000:0c:00.1 -> ../../../../devices/pci0000:00/0000:00:1c.7/0000:03:00.0/0000:04:09.0/0000:0c:00.1

Question:
Is there a workaround/fix for this or should i get a new mobo? big_smile

Thank you!

Offline

#394 2013-08-01 16:24:03

nbhs
Member
From: Montevideo, Uruguay
Registered: 2013-05-02
Posts: 402

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

cmdr wrote:

Hi,

thank you very much for your guide.
I'm running into some problems when I try to passthrough a gpu to a guest.

Hardware:

  • Xeon E3-1230v2

  • ASRock Fatal1ty Z77 professional

  • NVIDIA GTX-470 (Host)

  • ATI Radeon 5450 (Guest)

# cat /proc/cmdline 
BOOT_IMAGE=/vmlinuz-linux-mainline root=/dev/mapper/desktop-root ro pci-stub.ids=1002:68f9,1002:aa68 intel_iommu=on cryptdevice=/dev/sda2:main quiet
# lspci -tv
-[0000:00]-+-00.0  Intel Corporation Xeon E3-1200 v2/Ivy Bridge DRAM Controller
           +-01.0-[01]--+-00.0  NVIDIA Corporation GF100 [GeForce GTX 470]
           |            \-00.1  NVIDIA Corporation GF100 High Definition Audio Controller
           +-01.1-[02]--+-00.0  Advanced Micro Devices, Inc. [AMD/ATI] Cedar [Radeon HD 5000/6000/7350/8350 Series]
           |            \-00.1  Advanced Micro Devices, Inc. [AMD/ATI] Cedar HDMI Audio [Radeon HD 5400/6300 Series]
           +-14.0  Intel Corporation 7 Series/C210 Series Chipset Family USB xHCI Host Controller
           +-16.0  Intel Corporation 7 Series/C210 Series Chipset Family MEI Controller #1
           +-1a.0  Intel Corporation 7 Series/C210 Series Chipset Family USB Enhanced Host Controller #2
           +-1b.0  Intel Corporation 7 Series/C210 Series Chipset Family High Definition Audio Controller
           +-1c.0-[03]--
           +-1c.7-[04-0d]----00.0-[05-0d]--+-01.0-[06]----00.0  ASMedia Technology Inc. ASM1062 Serial ATA Controller
           |                               +-04.0-[07-08]----00.0-[08]--
           |                               +-05.0-[09]----00.0  ASMedia Technology Inc. ASM1062 Serial ATA Controller
           |                               +-06.0-[0a]----00.0  Broadcom Corporation NetLink BCM57781 Gigabit Ethernet PCIe
           |                               +-07.0-[0b]----00.0  Etron Technology, Inc. EJ188/EJ198 USB 3.0 Host Controller
           |                               +-08.0-[0c]----00.0  Broadcom Corporation NetLink BCM57781 Gigabit Ethernet PCIe
           |                               \-09.0-[0d]--+-00.0  VIA Technologies, Inc. VT6315 Series Firewire Controller
           |                                            \-00.1  VIA Technologies, Inc. VT6415 PATA IDE Host Controller
           +-1d.0  Intel Corporation 7 Series/C210 Series Chipset Family USB Enhanced Host Controller #1
           +-1f.0  Intel Corporation Z77 Express Chipset LPC Controller
           +-1f.2  Intel Corporation 7 Series/C210 Series Chipset Family 6-port SATA Controller [AHCI mode]
           \-1f.3  Intel Corporation 7 Series/C210 Series Chipset Family SMBus Controller

Using the second pcie slot, both gpus are in the same iommu_group (thats why i can't bind all devices in the group to vfio).

# readlink /sys/bus/pci/devices/0000\:02\:00.0/iommu_group
../../../../kernel/iommu_groups/1

# ls -l /sys/kernel/iommu_groups/1/devices/
total 0
lrwxrwxrwx 1 root root 0 Aug  1 17:40 0000:00:01.0 -> ../../../../devices/pci0000:00/0000:00:01.0 <- pci-bridge
lrwxrwxrwx 1 root root 0 Aug  1 17:40 0000:00:01.1 -> ../../../../devices/pci0000:00/0000:00:01.1 <- pci-bridge
lrwxrwxrwx 1 root root 0 Aug  1 17:40 0000:01:00.0 -> ../../../../devices/pci0000:00/0000:00:01.0/0000:01:00.0
lrwxrwxrwx 1 root root 0 Aug  1 17:40 0000:01:00.1 -> ../../../../devices/pci0000:00/0000:00:01.0/0000:01:00.1
lrwxrwxrwx 1 root root 0 Aug  1 17:40 0000:02:00.0 -> ../../../../devices/pci0000:00/0000:00:01.1/0000:02:00.0
lrwxrwxrwx 1 root root 0 Aug  1 17:40 0000:02:00.1 -> ../../../../devices/pci0000:00/0000:00:01.1/0000:02:00.1

Using the third pcie slot, the second gpu is in the same iommu_group with the NIC's and stuff.

# lspci | grep ATI
02:00.0 VGA compatible controller: Advanced Micro Devices, Inc. [AMD/ATI] Cedar [Radeon HD 5000/6000/7350/8350 Series]
02:00.1 Audio device: Advanced Micro Devices, Inc. [AMD/ATI] Cedar HDMI Audio [Radeon HD 5400/6300 Series]

# readlink /sys/bus/pci/devices/0000\:02\:00.0/iommu_group
../../../../kernel/iommu_groups/6

# ls -l /sys/kernel/iommu_groups/6/devices/
total 0
lrwxrwxrwx 1 root root 0 Aug  1 18:00 0000:00:1c.0 -> ../../../../devices/pci0000:00/0000:00:1c.0
lrwxrwxrwx 1 root root 0 Aug  1 18:00 0000:00:1c.7 -> ../../../../devices/pci0000:00/0000:00:1c.7
lrwxrwxrwx 1 root root 0 Aug  1 18:00 0000:02:00.0 -> ../../../../devices/pci0000:00/0000:00:1c.0/0000:02:00.0
lrwxrwxrwx 1 root root 0 Aug  1 18:00 0000:02:00.1 -> ../../../../devices/pci0000:00/0000:00:1c.0/0000:02:00.1
lrwxrwxrwx 1 root root 0 Aug  1 18:00 0000:03:00.0 -> ../../../../devices/pci0000:00/0000:00:1c.7/0000:03:00.0
lrwxrwxrwx 1 root root 0 Aug  1 18:00 0000:04:01.0 -> ../../../../devices/pci0000:00/0000:00:1c.7/0000:03:00.0/0000:04:01.0
lrwxrwxrwx 1 root root 0 Aug  1 18:00 0000:04:04.0 -> ../../../../devices/pci0000:00/0000:00:1c.7/0000:03:00.0/0000:04:04.0
lrwxrwxrwx 1 root root 0 Aug  1 18:00 0000:04:05.0 -> ../../../../devices/pci0000:00/0000:00:1c.7/0000:03:00.0/0000:04:05.0
lrwxrwxrwx 1 root root 0 Aug  1 18:00 0000:04:06.0 -> ../../../../devices/pci0000:00/0000:00:1c.7/0000:03:00.0/0000:04:06.0
lrwxrwxrwx 1 root root 0 Aug  1 18:00 0000:04:07.0 -> ../../../../devices/pci0000:00/0000:00:1c.7/0000:03:00.0/0000:04:07.0
lrwxrwxrwx 1 root root 0 Aug  1 18:00 0000:04:08.0 -> ../../../../devices/pci0000:00/0000:00:1c.7/0000:03:00.0/0000:04:08.0
lrwxrwxrwx 1 root root 0 Aug  1 18:00 0000:04:09.0 -> ../../../../devices/pci0000:00/0000:00:1c.7/0000:03:00.0/0000:04:09.0
lrwxrwxrwx 1 root root 0 Aug  1 18:00 0000:05:00.0 -> ../../../../devices/pci0000:00/0000:00:1c.7/0000:03:00.0/0000:04:01.0/0000:05:00.0
lrwxrwxrwx 1 root root 0 Aug  1 18:00 0000:06:00.0 -> ../../../../devices/pci0000:00/0000:00:1c.7/0000:03:00.0/0000:04:04.0/0000:06:00.0
lrwxrwxrwx 1 root root 0 Aug  1 18:00 0000:08:00.0 -> ../../../../devices/pci0000:00/0000:00:1c.7/0000:03:00.0/0000:04:05.0/0000:08:00.0
lrwxrwxrwx 1 root root 0 Aug  1 18:00 0000:09:00.0 -> ../../../../devices/pci0000:00/0000:00:1c.7/0000:03:00.0/0000:04:06.0/0000:09:00.0
lrwxrwxrwx 1 root root 0 Aug  1 18:00 0000:0a:00.0 -> ../../../../devices/pci0000:00/0000:00:1c.7/0000:03:00.0/0000:04:07.0/0000:0a:00.0
lrwxrwxrwx 1 root root 0 Aug  1 18:00 0000:0b:00.0 -> ../../../../devices/pci0000:00/0000:00:1c.7/0000:03:00.0/0000:04:08.0/0000:0b:00.0
lrwxrwxrwx 1 root root 0 Aug  1 18:00 0000:0c:00.0 -> ../../../../devices/pci0000:00/0000:00:1c.7/0000:03:00.0/0000:04:09.0/0000:0c:00.0
lrwxrwxrwx 1 root root 0 Aug  1 18:00 0000:0c:00.1 -> ../../../../devices/pci0000:00/0000:00:1c.7/0000:03:00.0/0000:04:09.0/0000:0c:00.1

Question:
Is there a workaround/fix for this or should i get a new mobo? big_smile

Thank you!

Some people were having this same problem with intel boards, take a look at this post on the kvm mailing list, im not sure this is the solution though: http://marc.info/?l=linux-kernel&m=136993923507507&w=2

Last edited by nbhs (2013-08-01 16:26:11)

Offline

#395 2013-08-01 16:47:27

nbhs
Member
From: Montevideo, Uruguay
Registered: 2013-05-02
Posts: 402

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

BulliteShot wrote:
mukiex wrote:

Hey BulliteShot: Indeed, that is the case. I own a 2nd i5 because of that ^_^

nbhs wrote:
BulliteShot wrote:

It seems that only "locked" Intel processors support IOMMU which means no overclocking.

Can AMD processors overclock aswell as enabling IOMMU?

Yeah IOMMU support is only required on the mobo on amd systems, i own a bulldozer 8120 black box with unlocked multiplier, i can get it to 4.6ghz stable on a closed loop water system

*sigh* I for one am sick of Intel pushing these limitations.

Goodbye Intel, hello AMD!

I think instead of spending money perhaps you could try posting you issue on the kvm and/or qemu mailing list, remember this is still an experimental feature being worked on smile

Last edited by nbhs (2013-08-01 16:49:15)

Offline

#396 2013-08-02 00:31:47

cmdr
Member
Registered: 2013-08-01
Posts: 8

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

nbhs wrote:
cmdr wrote:

-snip-

Some people were having this same problem with intel boards, take a look at this post on the kvm mailing list, im not sure this is the solution though: http://marc.info/?l=linux-kernel&m=136993923507507&w=2

Hi,
thanks for your fast reply. I applied the patch, changed my command line and now the devices have their own groups (second gpu together with pci-bridge).

# cat /proc/cmdline
BOOT_IMAGE=/vmlinuz-linux-mainline root=/dev/mapper/desktop-root ro pci-stub.ids=1002:68f9,1002:aa68 intel_iommu=on pcie_acs_override=id:8086:1e1e cryptdevice=/dev/sda2:main quiet

# ls -l /sys/bus/pci/devices/0000\:02\:00.0/iommu_group/devices/
total 0
lrwxrwxrwx 1 root root 0 Aug  2 02:27 0000:00:1c.0 -> ../../../../devices/pci0000:00/0000:00:1c.0
lrwxrwxrwx 1 root root 0 Aug  2 02:27 0000:02:00.0 -> ../../../../devices/pci0000:00/0000:00:1c.0/0000:02:00.0
lrwxrwxrwx 1 root root 0 Aug  2 02:27 0000:02:00.1 -> ../../../../devices/pci0000:00/0000:00:1c.0/0000:02:00.1

[   53.014493] vfio-pci 0000:02:00.0: enabling device (0000 -> 0003)

I bind the gpu manually to vfio and qemu starts without errors but the attached screen stays black.

Edit: I blacklisted nouveau, nvidia, radeon and tried it without X but i had no success.

qemu-system-x86_64 -enable-kvm -M q35 -m 1024 -cpu host \
-smp 6,sockets=1,cores=6,threads=1 \
-bios /usr/share/qemu/bios.bin -vga none \
-device ioh3420,bus=pcie.0,addr=1c.0,multifunction=on,port=1,chassis=1,id=root.1 \
-device vfio-pci,host=02:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on \
-device vfio-pci,host=02:00.1,bus=root.1,addr=00.1

Last edited by cmdr (2013-08-02 01:53:54)

Offline

#397 2013-08-02 02:34:43

nbhs
Member
From: Montevideo, Uruguay
Registered: 2013-05-02
Posts: 402

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

cmdr wrote:
nbhs wrote:
cmdr wrote:

-snip-

Some people were having this same problem with intel boards, take a look at this post on the kvm mailing list, im not sure this is the solution though: http://marc.info/?l=linux-kernel&m=136993923507507&w=2

Hi,
thanks for your fast reply. I applied the patch, changed my command line and now the devices have their own groups (second gpu together with pci-bridge).

# cat /proc/cmdline
BOOT_IMAGE=/vmlinuz-linux-mainline root=/dev/mapper/desktop-root ro pci-stub.ids=1002:68f9,1002:aa68 intel_iommu=on pcie_acs_override=id:8086:1e1e cryptdevice=/dev/sda2:main quiet

# ls -l /sys/bus/pci/devices/0000\:02\:00.0/iommu_group/devices/
total 0
lrwxrwxrwx 1 root root 0 Aug  2 02:27 0000:00:1c.0 -> ../../../../devices/pci0000:00/0000:00:1c.0
lrwxrwxrwx 1 root root 0 Aug  2 02:27 0000:02:00.0 -> ../../../../devices/pci0000:00/0000:00:1c.0/0000:02:00.0
lrwxrwxrwx 1 root root 0 Aug  2 02:27 0000:02:00.1 -> ../../../../devices/pci0000:00/0000:00:1c.0/0000:02:00.1

[   53.014493] vfio-pci 0000:02:00.0: enabling device (0000 -> 0003)

I bind the gpu manually to vfio and qemu starts without errors but the attached screen stays black.

Edit: I blacklisted nouveau, nvidia, radeon and tried it without X but i had no success.

qemu-system-x86_64 -enable-kvm -M q35 -m 1024 -cpu host \
-smp 6,sockets=1,cores=6,threads=1 \
-bios /usr/share/qemu/bios.bin -vga none \
-device ioh3420,bus=pcie.0,addr=1c.0,multifunction=on,port=1,chassis=1,id=root.1 \
-device vfio-pci,host=02:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on \
-device vfio-pci,host=02:00.1,bus=root.1,addr=00.1

Are you using the patched seabios? have you tried swapping cards? perhaps using pcie_acs_override=downstream or mutlifucion (i have really no idea here)?

Last edited by nbhs (2013-08-02 02:46:52)

Offline

#398 2013-08-02 10:32:46

reintseri
Member
Registered: 2013-07-16
Posts: 6

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Ok I was meant to post my specs earlier, but summer vacation got in the way smile.

So. I'm using a 2009 Mac Pro with OS X and Arch Linux dual boot as a basis. 2 Xeon processors (8 cores) and 20 gigs of ram smile. Modified kernel, QEMU and Seabios - downloaded from this thread.

Host graphics card - nVidia GeForce 9500 GT (Mac Edition) (Mac EFI boot loader needs this in the first PCIe slot to boot unfortunately.) Tested graphics cards to work in the second 16x PCIe slot: Radeon HD 5450 and Radeon HD 7750.

So on their own they work fine. But I've also tried a 3-graphics-cards-setup to have two VMs. At this point it's not so stable anymore. I can have one Windows 8 VM with the HD 7750 running together with Ubuntu 12.04 with HD5450, but as soon as I install FGLRX or catalyst driver for it- it crashes and won't boot anymore. Also various error messages from the kernel can be seen during start and shutdown. Windows installation fails even earlier. So at this setup the HD 5450 is on a 4x PCIe slot.

Now the problem might be similar to what was posted earlier about the iommu_groups, the HD 7750 (on the 2nd 16x slot) shows only:

0000:06:00.0  0000:06:00.1

But the HD 5450 (on one of the free 4x slots) shows:

0000:02:04.0  0000:04:00.0  0000:04:00.1

So it includes one of the pcie switches:

02:04.0 PCI bridge: Integrated Device Technology, Inc. [IDT] PES12T3G2 PCI Express Gen2 Switch (rev 01)

Could this be the source of the problem? And could I change the iommu_groups somehow.

Thanks again,
Teemu Kämäräinen

Offline

#399 2013-08-02 12:07:29

cmdr
Member
Registered: 2013-08-01
Posts: 8

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

nbhs wrote:
cmdr wrote:
nbhs wrote:

Some people were having this same problem with intel boards, take a look at this post on the kvm mailing list, im not sure this is the solution though: http://marc.info/?l=linux-kernel&m=136993923507507&w=2

Hi,
thanks for your fast reply. I applied the patch, changed my command line and now the devices have their own groups (second gpu together with pci-bridge).

# cat /proc/cmdline
BOOT_IMAGE=/vmlinuz-linux-mainline root=/dev/mapper/desktop-root ro pci-stub.ids=1002:68f9,1002:aa68 intel_iommu=on pcie_acs_override=id:8086:1e1e cryptdevice=/dev/sda2:main quiet

# ls -l /sys/bus/pci/devices/0000\:02\:00.0/iommu_group/devices/
total 0
lrwxrwxrwx 1 root root 0 Aug  2 02:27 0000:00:1c.0 -> ../../../../devices/pci0000:00/0000:00:1c.0
lrwxrwxrwx 1 root root 0 Aug  2 02:27 0000:02:00.0 -> ../../../../devices/pci0000:00/0000:00:1c.0/0000:02:00.0
lrwxrwxrwx 1 root root 0 Aug  2 02:27 0000:02:00.1 -> ../../../../devices/pci0000:00/0000:00:1c.0/0000:02:00.1

[   53.014493] vfio-pci 0000:02:00.0: enabling device (0000 -> 0003)

I bind the gpu manually to vfio and qemu starts without errors but the attached screen stays black.

Edit: I blacklisted nouveau, nvidia, radeon and tried it without X but i had no success.

qemu-system-x86_64 -enable-kvm -M q35 -m 1024 -cpu host \
-smp 6,sockets=1,cores=6,threads=1 \
-bios /usr/share/qemu/bios.bin -vga none \
-device ioh3420,bus=pcie.0,addr=1c.0,multifunction=on,port=1,chassis=1,id=root.1 \
-device vfio-pci,host=02:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on \
-device vfio-pci,host=02:00.1,bus=root.1,addr=00.1

Are you using the patched seabios? have you tried swapping cards? perhaps using pcie_acs_override=downstream or mutlifucion (i have really no idea here)?

Some minutes ago i applied this patch (http://lists.gnu.org/archive/html/qemu- … 04300.html) to the nvidia source and build it (i am using PKGBUILD from nvidia-all). Now i can see the boot messages on the attached screen but i have no boot media to install and test windows tongue .
What i have done to get it to this point:

$ cat /proc/cmdline 
BOOT_IMAGE=/vmlinuz-linux-mainline root=/dev/mapper/desktop-root ro pci-stub.ids=1002:68f9,1002:aa68 intel_iommu=on pcie_acs_override=id:8086:1e1e cryptdevice=/dev/sda2:main quiet

I think this works now, i'll check it later that day with a windows installation. Thanks for your ideas!

Offline

#400 2013-08-02 12:11:16

nbhs
Member
From: Montevideo, Uruguay
Registered: 2013-05-02
Posts: 402

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

cmdr wrote:
nbhs wrote:
cmdr wrote:

Hi,
thanks for your fast reply. I applied the patch, changed my command line and now the devices have their own groups (second gpu together with pci-bridge).


I bind the gpu manually to vfio and qemu starts without errors but the attached screen stays black.

Edit: I blacklisted nouveau, nvidia, radeon and tried it without X but i had no success.

Are you using the patched seabios? have you tried swapping cards? perhaps using pcie_acs_override=downstream or mutlifucion (i have really no idea here)?

Some minutes ago i applied this patch (http://lists.gnu.org/archive/html/qemu- … 04300.html) to the nvidia source and build it (i am using PKGBUILD from nvidia-all). Now i can see the boot messages on the attached screen but i have no boot media to install and test windows tongue .
What i have done to get it to this point:

$ cat /proc/cmdline 
BOOT_IMAGE=/vmlinuz-linux-mainline root=/dev/mapper/desktop-root ro pci-stub.ids=1002:68f9,1002:aa68 intel_iommu=on pcie_acs_override=id:8086:1e1e cryptdevice=/dev/sda2:main quiet

I think this works now, i'll check it later that day with a windows installation. Thanks for your ideas!

Interesting, thanks a lot for your report

Offline

Board footer

Powered by FluxBB