Arch Linux

Denso

Member

Registered: 2014-08-30

Posts: 179

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

AMD Bonaire and Hawaii users with reset issues, please try this QEMU patch (against v2.3.0-rc4): http://fpaste.org/214053/  Report back your GPU device ID and whether resets are resolved.  As noted in the patch, I expect the ultimate resolution will be a device specific reset in the kernel, this is just a prototype of that.  If you do not have an AMD GPU affected by reset problems (ie. one guest boot per host boot), please don't bother with this patch.

Anything for Nvidia's GPUs with reset issues ? XD

I'm not aware of any widespread reset issues on nvidia.

Well . It only happens after I install the nvidia's drivers . Using the Microsoft Basic one is fine . When the drivers are installed , rebooting the VM works , I can see the OVMF screen + Windows logo , then just before it shows the desktop it goes black . The VM doesn't crash nor the host . No dmesg errors . One guest boot per host boot . GT610 .

lspci -vvv :
02:00.0 VGA compatible controller: NVIDIA Corporation GF119 [GeForce GT 610] (rev a1) (prog-if 00 [VGA controller])
        Subsystem: Gigabyte Technology Co., Ltd Device 3549
        Physical Slot: 6
        Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR+ FastB2B- DisINTx-
        Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
        Latency: 0, Cache Line Size: 32 bytes
        Interrupt: pin A routed to IRQ 29
        Region 0: Memory at f8000000 (32-bit, non-prefetchable) [size=16M]
        Region 1: Memory at c0000000 (64-bit, prefetchable) [size=128M]
        Region 3: Memory at c8000000 (64-bit, prefetchable) [size=32M]
        Region 5: I/O ports at d000 [size=128]
        Expansion ROM at f9000000 [disabled] [size=512K]
        Capabilities: [60] Power Management version 3
                Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0-,D1-,D2-,D3hot-,D3cold-)
                Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=0 PME-
        Capabilities: [68] MSI: Enable- Count=1/1 Maskable- 64bit+
                Address: 0000000000000000  Data: 0000
        Capabilities: [78] Express (v2) Endpoint, MSI 00
                DevCap: MaxPayload 128 bytes, PhantFunc 0, Latency L0s unlimited, L1 <64us
                        ExtTag+ AttnBtn- AttnInd- PwrInd- RBE+ FLReset-
                DevCtl: Report errors: Correctable- Non-Fatal- Fatal- Unsupported-
                        RlxdOrd+ ExtTag+ PhantFunc- AuxPwr- NoSnoop+
                        MaxPayload 128 bytes, MaxReadReq 512 bytes
DevSta: CorrErr- UncorrErr- FatalErr- UnsuppReq- AuxPwr- TransPend-
                LnkCap: Port #0, Speed 2.5GT/s, Width x16, ASPM L0s L1, Exit Latency L0s <256ns, L1 <4us
                        ClockPM+ Surprise- LLActRep- BwNot- ASPMOptComp-
                LnkCtl: ASPM Disabled; RCB 64 bytes Disabled- CommClk+
                        ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
                LnkSta: Speed 2.5GT/s, Width x16, TrErr- Train- SlotClk+ DLActive- BWMgmt- ABWMgmt-
                DevCap2: Completion Timeout: Not Supported, TimeoutDis+, LTR-, OBFF Not Supported
                DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis-, LTR-, OBFF Disabled
                LnkCtl2: Target Link Speed: 2.5GT/s, EnterCompliance- SpeedDis-
                         Transmit Margin: Normal Operating Range, EnterModifiedCompliance- ComplianceSOS-
                         Compliance De-emphasis: -6dB
                LnkSta2: Current De-emphasis Level: -6dB, EqualizationComplete-, EqualizationPhase1-
                         EqualizationPhase2-, EqualizationPhase3-, LinkEqualizationRequest-
        Capabilities: [b4] Vendor Specific Information: Len=14 <?>
        Capabilities: [100 v1] Virtual Channel
                Caps:   LPEVC=0 RefClk=100ns PATEntryBits=1
                Arb:    Fixed- WRR32- WRR64- WRR128-
                Ctrl:   ArbSelect=Fixed
                Status: InProgress-
                VC0:    Caps:   PATOffset=00 MaxTimeSlots=1 RejSnoopTrans-
                        Arb:    Fixed- WRR32- WRR64- WRR128- TWRR128- WRR256-
                        Ctrl:   Enable+ ID=0 ArbSelect=Fixed TC/VC=01
                        Status: NegoPending- InProgress-
Capabilities: [128 v1] Power Budgeting <?>
        Capabilities: [600 v1] Vendor Specific Information: ID=0001 Rev=1 Len=024 <?>
        Kernel driver in use: vfio-pci
        Kernel modules: nouveau

aw

Member

Registered: 2013-10-04

Posts: 921

Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Anything for Nvidia's GPUs with reset issues ? XD

I'm not aware of any widespread reset issues on nvidia.

Well . It only happens after I install the nvidia's drivers . Using the Microsoft Basic one is fine . When the drivers are installed , rebooting the VM works , I can see the OVMF screen + Windows logo , then just before it shows the desktop it goes black . The VM doesn't crash nor the host . No dmesg errors . One guest boot per host boot . GT610 .

widespread...

Sorry, one user having trouble with a low-end Fermi-based card does not make for a class problem.

---

http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Parzival

Member

Registered: 2015-04-23

Posts: 3

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

I have a problem with passing through Intel HD 5000 on Intel NUC. I've followed the guide on the first pinned page of the thread:

• Installed linux-vfio kernel from aur

• loaded pci-stub on boot

• added the right boot options

This is the device:

00:02.0 VGA compatible controller: Intel Corporation Haswell-ULT Integrated Graphics Controller (rev 09)
	DeviceName:  CPU1
	Subsystem: Intel Corporation Device 2054
	Kernel driver in use: pci-stub
	Kernel modules: i915

The resulting fault message:

qemu-system-i386: -device vfio-pci,host=0000:00:02.0: vfio: failed to set iommu for container: Operation not permitted
qemu-system-i386: -device vfio-pci,host=0000:00:02.0: vfio: failed to setup container for group 1
qemu-system-i386: -device vfio-pci,host=0000:00:02.0: vfio: failed to get group 1
qemu-system-i386: -device vfio-pci,host=0000:00:02.0: Device initialization failed.
qemu-system-i386: -device vfio-pci,host=0000:00:02.0: Device 'vfio-pci' could not be initialized

dmesg

vfio-pci 0000:00:02.0: Device is ineligible for IOMMU domain attach due to platform RMRR requirement.  Contact your platform vendor.

So, I'm interested to know :

• whether anyone had the same problem and was able to solve it

• what/how should I ask the vendor (Intel I suppose) so I can resolve the above issue

Thanks

aw

Member

Registered: 2013-10-04

Posts: 921

Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Well . It only happens after I install the nvidia's drivers . Using the Microsoft Basic one is fine . When the drivers are installed , rebooting the VM works , I can see the OVMF screen + Windows logo , then just before it shows the desktop it goes black . The VM doesn't crash nor the host . No dmesg errors . One guest boot per host boot . GT610 .

        Capabilities: [100 v1] Virtual Channel
                Caps:   LPEVC=0 RefClk=100ns PATEntryBits=1
                Arb:    Fixed- WRR32- WRR64- WRR128-
                Ctrl:   ArbSelect=Fixed
                Status: InProgress-
                VC0:    Caps:   PATOffset=00 MaxTimeSlots=1 RejSnoopTrans-
                        Arb:    Fixed- WRR32- WRR64- WRR128- TWRR128- WRR256-
                        Ctrl:   Enable+ ID=0 ArbSelect=Fixed TC/VC=01
                        Status: NegoPending- InProgress-

Does the TC/VC map reported in the Virtual Channel capability (01 above) change from the first host boot to after a guest failure?  I had to add VC save/restore for platforms that use non-default TC/VC maps.  Kernels 3.14 and newer should already include this though.

---

http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

aw

Member

Registered: 2013-10-04

Posts: 921

Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

I have a problem with passing through Intel HD 5000 on Intel NUC.

IGD assignment does not work.

---

http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Parzival

Member

Registered: 2015-04-23

Posts: 3

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

I have a problem with passing through Intel HD 5000 on Intel NUC.

IGD assignment does not work.

Can you explain why? Will it be supported in the near future?

aw

Member

Registered: 2013-10-04

Posts: 921

Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

I have a problem with passing through Intel HD 5000 on Intel NUC.

IGD assignment does not work.

Can you explain why? Will it be supported in the near future?

Because not only is IGD not a discrete device in the sense that it's integrated, but also it's current driver requirements extend beyond the individual PCI device.  It requires various operation regions outside of the device to be poked through, it requires registers on the host bridge to be assigned, it requires specific PCI IDs matching the host chipset on the host bridge and PCH device, etc.  There are proof-of-concept level patches available, but I wouldn't recommend them unless you plan to contribute to their development.  Likely the only way we'll see progress is in a new driver from Intel that confines the platform requirements for IGD to the PCI device itself.

---

http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Denso

Member

Registered: 2014-08-30

Posts: 179

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Does the TC/VC map reported in the Virtual Channel capability (01 above) change from the first host boot to after a guest failure?  I had to add VC save/restore for platforms that use non-default TC/VC maps.  Kernels 3.14 and newer should already include this though.

No , nothing in the GPU or the integrated HDMI audio devices changes when the guest got rebooted .

On a side note , ASUS released a new BIOS update for X99-Deluxe . After applying it , I no longer recieve the "AER: Multiple Corrected error" in dmesg anymore

It seems to be a widespread issue with X99 . This platform's stability is not good compared to my old Z77 .

EDIT :

I am passing through the Intel Audio device to the same VM . This device gets TC/VC changes after guest reboot :

Before VM reboot :

VC0:    Caps:   PATOffset=00 MaxTimeSlots=1 RejSnoopTrans-
                        Arb:    Fixed- WRR32- WRR64- WRR128- TWRR128- WRR256-
                        Ctrl:   Enable+ ID=0 ArbSelect=Fixed TC/VC=01
                        Status: NegoPending- InProgress-
                VC1:    Caps:   PATOffset=00 MaxTimeSlots=1 RejSnoopTrans-
                        Arb:    Fixed- WRR32- WRR64- WRR128- TWRR128- WRR256-
                        Ctrl:   Enable+ ID=2 ArbSelect=Fixed TC/VC=04
                        Status: NegoPending- InProgress-
        Kernel driver in use: vfio-pci
        Kernel modules: snd_hda_intel

After VM reboot :

 VC0:    Caps:   PATOffset=00 MaxTimeSlots=1 RejSnoopTrans-
                        Arb:    Fixed- WRR32- WRR64- WRR128- TWRR128- WRR256-
                        Ctrl:   Enable+ ID=0 ArbSelect=Fixed TC/VC=ff
                        Status: NegoPending- InProgress-
                VC1:    Caps:   PATOffset=00 MaxTimeSlots=1 RejSnoopTrans-
                        Arb:    Fixed- WRR32- WRR64- WRR128- TWRR128- WRR256-
                        Ctrl:   Enable+ ID=2 ArbSelect=Fixed TC/VC=00
                        Status: NegoPending- InProgress-
        Kernel driver in use: vfio-pci
        Kernel modules: snd_hda_intel

  

I'll reboot the host & VM without attaching it , and will see if it is to blame .

Last edited by Denso (2015-04-23 14:46:27)

Parzival

Member

Registered: 2015-04-23

Posts: 3

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

IGD assignment does not work.

Can you explain why? Will it be supported in the near future?

Because not only is IGD not a discrete device in the sense that it's integrated, but also it's current driver requirements extend beyond the individual PCI device.  It requires various operation regions outside of the device to be poked through, it requires registers on the host bridge to be assigned, it requires specific PCI IDs matching the host chipset on the host bridge and PCH device, etc.  There are proof-of-concept level patches available, but I wouldn't recommend them unless you plan to contribute to their development.  Likely the only way we'll see progress is in a new driver from Intel that confines the platform requirements for IGD to the PCI device itself.

Thanks.
I know that this could be a broad answer, but still, I'll appreciate if you can explain why I do able to pass through IGD using Xen? What's the difference?

aw

Member

Registered: 2013-10-04

Posts: 921

Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Can you explain why? Will it be supported in the near future?

Because not only is IGD not a discrete device in the sense that it's integrated, but also it's current driver requirements extend beyond the individual PCI device.  It requires various operation regions outside of the device to be poked through, it requires registers on the host bridge to be assigned, it requires specific PCI IDs matching the host chipset on the host bridge and PCH device, etc.  There are proof-of-concept level patches available, but I wouldn't recommend them unless you plan to contribute to their development.  Likely the only way we'll see progress is in a new driver from Intel that confines the platform requirements for IGD to the PCI device itself.

Thanks.
I know that this could be a broad answer, but still, I'll appreciate if you can explain why I do able to pass through IGD using Xen? What's the difference?

Because Xen is willing to hack their VM to bits, regardless of the support-ability of the solution in order to make something work.  When they tried to merge those hacks into upstream QEMU is when we started talking to Intel about how to do this correctly.

---

http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Tyrewt

Member

Registered: 2014-09-13

Posts: 14

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

AMD Bonaire and Hawaii users with reset issues, please try this QEMU patch (against v2.3.0-rc4): http://fpaste.org/214053/  Report back your GPU device ID and whether resets are resolved.  As noted in the patch, I expect the ultimate resolution will be a device specific reset in the kernel, this is just a prototype of that.  If you do not have an AMD GPU affected by reset problems (ie. one guest boot per host boot), please don't bother with this patch.

I'm willing to test, but Ubuntu 14.04 LTS is at qemu-kvm 2.0.0+dfsg-2ubuntu1.9, would this patch still be applicable?

Duelist

Member

Registered: 2014-09-22

Posts: 358

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

I have a problem with passing through Intel HD 5000 on Intel NUC.

IGD assignment does not work.

Can you explain why? Will it be supported in the near future?

You can try looking into KVMGT... As i understood their talk and presentation - they're willing to allow VMs access intel's IGD and stuff. But there's only a prototype yet.

---

The forum rules prohibit requesting support for distributions other than arch.
I gave up. It was too late.
What I was trying to do.
The reference about VFIO and KVM VGA passthrough.

zir_blazer

Member

Registered: 2013-12-12

Posts: 35

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

This a random comment, but I suppose it may be useful for someone. I have -finally- discovered what the Intel VT-d enabled Chipsets (Q and C Series) are useful for when compared to the non VT-d ones.

This is my lspci output, with IOMMU Group data manually added:

00:00.0 Host bridge: Intel Corporation Xeon E3-1200 v3 Processor DRAM Controller (rev 06)                                             -- GROUP 0
00:01.0 PCI bridge: Intel Corporation Xeon E3-1200 v3/4th Gen Core Processor PCI Express x16 Controller (rev 06)                     -- GROUP 1
00:02.0 VGA compatible controller: Intel Corporation Xeon E3-1200 v3 Processor Integrated Graphics Controller (rev 06)               -- GROUP 2
00:03.0 Audio device: Intel Corporation Xeon E3-1200 v3/4th Gen Core Processor HD Audio Controller (rev 06)                          -- GROUP 3
00:14.0 USB controller: Intel Corporation 8 Series/C220 Series Chipset Family USB xHCI (rev 05)                                       -- GROUP 4
00:16.0 Communication controller: Intel Corporation 8 Series/C220 Series Chipset Family MEI Controller #1 (rev 04)                   -- GROUP 5
00:19.0 Ethernet controller: Intel Corporation Ethernet Connection I217-LM (rev 05)                                                      -- GROUP 6
00:1a.0 USB controller: Intel Corporation 8 Series/C220 Series Chipset Family USB EHCI #2 (rev 05)                                     -- GROUP 7
00:1b.0 Audio device: Intel Corporation 8 Series/C220 Series Chipset High Definition Audio Controller (rev 05)                         -- GROUP 8
00:1c.0 PCI bridge: Intel Corporation 8 Series/C220 Series Chipset Family PCI Express Root Port #1 (rev d5)                             -- GROUP 9
00:1c.1 PCI bridge: Intel Corporation 8 Series/C220 Series Chipset Family PCI Express Root Port #2 (rev d5)                             -- GROUP 10
00:1c.3 PCI bridge: Intel Corporation 8 Series/C220 Series Chipset Family PCI Express Root Port #4 (rev d5)                             -- GROUP 11
00:1c.4 PCI bridge: Intel Corporation 8 Series/C220 Series Chipset Family PCI Express Root Port #5 (rev d5)                            -- GROUP 12
00:1d.0 USB controller: Intel Corporation 8 Series/C220 Series Chipset Family USB EHCI #1 (rev 05)                                      -- GROUP 13
00:1f.0 ISA bridge: Intel Corporation C226 Series Chipset Family Server Advanced SKU LPC Controller (rev 05)                          -- GROUP 14
00:1f.2 SATA controller: Intel Corporation 8 Series/C220 Series Chipset Family 6-port SATA Controller 1 [AHCI mode] (rev 05)       -- GROUP 14
00:1f.3 SMBus: Intel Corporation 8 Series/C220 Series Chipset Family SMBus Controller (rev 05)                                         -- GROUP 14
00:1f.6 Signal processing controller: Intel Corporation 8 Series Chipset Family Thermal Management Controller (rev 05)              -- GROUP 14
01:00.0 VGA compatible controller: Advanced Micro Devices, Inc. [AMD/ATI] Juniper XT [Radeon HD 5770]                               -- GROUP 1
01:00.1 Audio device: Advanced Micro Devices, Inc. [AMD/ATI] Juniper HDMI Audio [Radeon HD 5700 Series]                           -- GROUP 1
02:00.0 SATA controller: ASMedia Technology Inc. ASM1062 Serial ATA Controller (rev 01)                                              -- GROUP 15
03:00.0 PCI bridge: PLX Technology, Inc. PEX 8606 6 Lane, 6 Port PCI Express Gen 2 (5.0 GT/s) Switch (rev ba)                      -- GROUP 16
04:01.0 PCI bridge: PLX Technology, Inc. PEX 8606 6 Lane, 6 Port PCI Express Gen 2 (5.0 GT/s) Switch (rev ba)                      -- GROUP 17
04:04.0 PCI bridge: PLX Technology, Inc. PEX 8606 6 Lane, 6 Port PCI Express Gen 2 (5.0 GT/s) Switch (rev ba)                      -- GROUP 18
04:05.0 PCI bridge: PLX Technology, Inc. PEX 8606 6 Lane, 6 Port PCI Express Gen 2 (5.0 GT/s) Switch (rev ba)                      -- GROUP 19
04:07.0 PCI bridge: PLX Technology, Inc. PEX 8606 6 Lane, 6 Port PCI Express Gen 2 (5.0 GT/s) Switch (rev ba)                      -- GROUP 20
04:09.0 PCI bridge: PLX Technology, Inc. PEX 8606 6 Lane, 6 Port PCI Express Gen 2 (5.0 GT/s) Switch (rev ba)                      -- GROUP 21
08:00.0 USB controller: Renesas Technology Corp. uPD720201 USB 3.0 Host Controller (rev 03)                                          -- GROUP 22
09:00.0 PCI bridge: Texas Instruments XIO2213A/B/XIO2221 PCI Express to PCI Bridge [Cheetah Express] (rev 01)                   -- GROUP 23
0a:00.0 FireWire (IEEE 1394): Texas Instruments XIO2213A/B/XIO2221 IEEE-1394b OHCI Controller [Cheetah Express] (rev 01)       -- GROUP 23
0b:00.0 Ethernet controller: Intel Corporation I210 Gigabit Network Connection (rev 03)                                                    -- GROUP 24

This is how a Z97 Chipset IOMMU Grouping looks like:

### Group 0 ###
    00:00.0 Host bridge: Intel Corporation 4th Gen Core Processor DRAM Controller (rev 06)
### Group 1 ###
    00:01.0 PCI bridge: Intel Corporation Xeon E3-1200 v3/4th Gen Core Processor PCI Express x16 Controller (rev 06)
    01:00.0 VGA compatible controller: NVIDIA Corporation GK106 [GeForce GTX 650 OEM] (rev a1)
    01:00.1 Audio device: NVIDIA Corporation GK106 HDMI Audio Controller (rev a1)
### Group 2 ###
    00:02.0 VGA compatible controller: Intel Corporation Xeon E3-1200 v3/4th Gen Core Processor Integrated Graphics Controller (rev 06)
### Group 3 ###
    00:03.0 Audio device: Intel Corporation Xeon E3-1200 v3/4th Gen Core Processor HD Audio Controller (rev 06)
### Group 4 ###
    00:14.0 USB controller: Intel Corporation 9 Series Chipset Family USB xHCI Controller
### Group 5 ###
    00:16.0 Communication controller: Intel Corporation 9 Series Chipset Family ME Interface #1
### Group 6 ###
    00:19.0 Ethernet controller: Intel Corporation Ethernet Connection (2) I218-V
### Group 7 ###
    00:1a.0 USB controller: Intel Corporation 9 Series Chipset Family USB EHCI Controller #2
### Group 8 ###
    00:1b.0 Audio device: Intel Corporation 9 Series Chipset Family HD Audio Controller
### Group 9 ###
    00:1c.0 PCI bridge: Intel Corporation 9 Series Chipset Family PCI Express Root Port 1 (rev d0)
    00:1c.3 PCI bridge: Intel Corporation 9 Series Chipset Family PCI Express Root Port 4 (rev d0)
    00:1c.6 PCI bridge: Intel Corporation 9 Series Chipset Family PCI Express Root Port 7 (rev d0)
    03:00.0 PCI bridge: ASMedia Technology Inc. Device 1184
    04:01.0 PCI bridge: ASMedia Technology Inc. Device 1184
    04:03.0 PCI bridge: ASMedia Technology Inc. Device 1184
    04:05.0 PCI bridge: ASMedia Technology Inc. Device 1184
    04:07.0 PCI bridge: ASMedia Technology Inc. Device 1184
    06:00.0 SATA controller: ASMedia Technology Inc. ASM1062 Serial ATA Controller (rev 02)
    08:00.0 SATA controller: ASMedia Technology Inc. ASM1062 Serial ATA Controller (rev 02)
    09:00.0 USB controller: ASMedia Technology Inc. ASM1042A USB 3.0 Host Controller
### Group 10 ###
    00:1d.0 USB controller: Intel Corporation 9 Series Chipset Family USB EHCI Controller #1
### Group 11 ###
    00:1f.0 ISA bridge: Intel Corporation 9 Series Chipset Family Z97 LPC Controller
    00:1f.2 SATA controller: Intel Corporation 9 Series Chipset Family SATA Controller [AHCI Mode]
    00:1f.3 SMBus: Intel Corporation 9 Series Chipset Family SMBus Controller

Notice Group 9. Everything that is attached to the Chipset PCIe Lanes is one stupid big group, while I have nearly everything in its own Group. So for anyone that is interesed in using proper Server Motherboards, you will have more isolating capabilities.

aw

Member

Registered: 2013-10-04

Posts: 921

Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Notice Group 9. Everything that is attached to the Chipset PCIe Lanes is one stupid big group, while I have nearly everything in its own Group. So for anyone that is interesed in using proper Server Motherboards, you will have more isolating capabilities.

Update your kernel, quirks for Lynxpoint PCH root ports were added in 3.15.

---

http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

zir_blazer

Member

Registered: 2013-12-12

Posts: 35

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Notice Group 9. Everything that is attached to the Chipset PCIe Lanes is one stupid big group, while I have nearly everything in its own Group. So for anyone that is interesed in using proper Server Motherboards, you will have more isolating capabilities.

Update your kernel, quirks for Lynxpoint PCH root ports were added in 3.15.

My computer is the first one with a C226, the other is from here:
http://ubuntuforums.org/showthread.php? … st13237568

I'm using Kernel 3.17.1, the other guy says he has 3.16. I wasn't aware that the grouping was related to a quirck. Got a more recent IOMMU Group list on a non-Q or C Chipset too see it fixed?
So far, I was unable to find what makes the VT-d support on these Chipsets useful for. I through than the IOMMU Grouping difference made sense.

Last edited by zir_blazer (2015-04-23 18:27:28)

aw

Member

Registered: 2013-10-04

Posts: 921

Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Notice Group 9. Everything that is attached to the Chipset PCIe Lanes is one stupid big group, while I have nearly everything in its own Group. So for anyone that is interesed in using proper Server Motherboards, you will have more isolating capabilities.

Update your kernel, quirks for Lynxpoint PCH root ports were added in 3.15.

My computer is the first one with a C226, the other is from here:
http://ubuntuforums.org/showthread.php? … st13237568

I'm using Kernel 3.17.1, the other guy says he has 3.16. I wasn't aware that the grouping was related to a quirck. Got a more recent IOMMU Group list on a non-Q or C Chipset too see it fixed?
So far, I was unable to find what makes the VT-d support on these Chipsets useful for. I through than the IOMMU Grouping difference made sense.

Ah, sorry I keyed on the 8-series chipset, that one is quirked and I'm currently harassing Intel about the 9-series.  TBH, I have no idea what you're talking about with Q vs C chipsets.  The reason that anything without native ACS allows smaller IOMMU groups is because we have quirks in the kernel that enable non-standard isolation within the chipset/device.

---

http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

zir_blazer

Member

Registered: 2013-12-12

Posts: 35

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

I'm talking about this: http://ark.intel.com/compare/75522,7501 … 0711,27215

Intel says that for LGA 1150, only the Q87 and C222, C224 and C226 Chipsets supports VT-d. The point is, what you actually need VT-d Chipset support for? Before I builded my current system (Xeon E3-1245V3 and a Supermicro X10SAT), I spend around six months researching if you needed Chipset support or not, since there was a lot of contradicting info. Some people said and proved that with only Processor VT-d support you could get Passthrough working, but you had manufacturers like ASUS that instead claimed that they didn't supported VT-d on typical H or Z-Chipsets based Motherboard since a VT-d Chipset was required too. I decided to play safe and go Server.
What I found out by looking at a few lists of IOMMU Groups like the one I linked, is that on consumer Motherboards, they usually have a single huge group for the Chipset's PCIe Lanes. Instead, on mine, nearly each device has its own group. Keep in mind that this is using only intel_iommu=on on the Boot Loader config file, there is no Kernel patch or anything else applied, its pretty much a very dull Arch Linux install, since I am using it as Dom0 for Xen, but I'm looking at KVM in case I get a new GeForce. Because during an entire year I found no other explanation with a practical example, I concluded that Intel calls VT-d support on the Chipset, one that implements IOMMU for the devices connected to its PCIe Controller. I think I saw some people that applied the ACS override patch and their IOMMU Group list looked like mine, but not out of the box. But I didn't paid attention to other details like the Kernel, or quircks like the one that you mention, so I don't know if my theory is true or not.

aw

Member

Registered: 2013-10-04

Posts: 921

Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

I'm talking about this: http://ark.intel.com/compare/75522,7501 … 0711,27215

Intel says that for LGA 1150, only the Q87 and C222, C224 and C226 Chipsets supports VT-d. The point is, what you actually need VT-d Chipset support for? Before I builded my current system (Xeon E3-1245V3 and a Supermicro X10SAT), I spend around six months researching if you needed Chipset support or not, since there was a lot of contradicting info. Some people said and proved that with only Processor VT-d support you could get Passthrough working, but you had manufacturers like ASUS that instead claimed that they didn't supported VT-d on typical H or Z-Chipsets based Motherboard since a VT-d Chipset was required too. I decided to play safe and go Server.
What I found out by looking at a few lists of IOMMU Groups like the one I linked, is that on consumer Motherboards, they usually have a single huge group for the Chipset's PCIe Lanes. Instead, on mine, nearly each device has its own group. Keep in mind that this is using only intel_iommu=on on the Boot Loader config file, there is no Kernel patch or anything else applied, its pretty much a very dull Arch Linux install, since I am using it as Dom0 for Xen, but I'm looking at KVM in case I get a new GeForce. Because during an entire year I found no other explanation with a practical example, I concluded that Intel calls VT-d support on the Chipset, one that implements IOMMU for the devices connected to its PCIe Controller. I think I saw some people that applied the ACS override patch and their IOMMU Group list looked like mine, but not out of the box. But I didn't paid attention to other details like the Kernel, or quircks like the one that you mention, so I don't know if my theory is true or not.

I guess I should stop using VT-d on my H67 desktop then, Intel® Virtualization Technology for Directed I/O (VT-d):    No

It is a good idea to use server-class hardware, but you really need to go up to Xeon E5 or better to make any difference or make Intel actually care about supporting native ACS.  In the early days of VT-d, back when there was an MCH (Memory Controller HUB) and IOH (I/O Hub), VT-d support in the chipset was required.  The processor wasn't involved in the IOMMU.  The IOMMU needed to live between I/O and memory, which the processor was not.  However, when the memory controller was pulled into the CPU, along with (sometimes) integrated graphics, and high speed root ports, VT-d also needed to move into the processor.  I don't really know what Intel is trying to say with that entry in ARK, whether it's any indication of actual support, or just a glitch in their database, but I'm pretty sure the difference between your 8-series having separate IOMMU groups and the 9-series not is software, not hardware.  Maybe Intel will be less willing to provide us with quirks if there's no server versions of a given chipset.

---

http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Duelist

Member

Registered: 2014-09-22

Posts: 358

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

It is a good idea to use server-class hardware, but you really need to go up to Xeon E5 or better to make any difference or make Intel actually care about supporting native ACS.

So we're back to basics, eh?
"If you want stability and even some support from the manufacturer - use expensive server-class hardware."
Cheapest Xeon E5 costs more that my whole PC.
Are you sure that this breakage and lack of isolation and more quirks coming from newer hardware isn't "accidental" like it was with nvidia?

---

The forum rules prohibit requesting support for distributions other than arch.
I gave up. It was too late.
What I was trying to do.
The reference about VFIO and KVM VGA passthrough.

aw

Member

Registered: 2013-10-04

Posts: 921

Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

It is a good idea to use server-class hardware, but you really need to go up to Xeon E5 or better to make any difference or make Intel actually care about supporting native ACS.

So we're back to basics, eh?
"If you want stability and even some support from the manufacturer - use expensive server-class hardware."
Cheapest Xeon E5 costs more that my whole PC.
Are you sure that this breakage and lack of isolation and more quirks coming from newer hardware isn't "accidental" like it was with nvidia?

Oh come on, is invoking Nvidia the new Godwin's law for this thread?  ACS and device isolation is a high-end feature.  I think it's caught Intel by surprise that yes, they do need to advertise the isolation of devices and no, VT-d isn't much use without it.  We've also been quirking Intel 10G NICs to expose isolation, along with workstation and server class chipsets like X79 and X99.  How does that fit into your conspiracy theory?  If we've sparked change in the consumer class products, it may be a couple years before those products start leaving the fab.  In the meantime, we're getting pretty good cooperation.  Maybe there are technical reasons why we're not getting quirks for client CPUs.  Maybe the thing that enables SLI/Crossfire is lack of isolation.  I don't know.

Furthermore, the cheapest E5 Xeons I see on newegg are around $200US, so if your PC is that crappy, maybe you could use an upgrade.

Last edited by aw (2015-04-23 22:22:48)

---

http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Duelist

Member

Registered: 2014-09-22

Posts: 358

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

It is a good idea to use server-class hardware, but you really need to go up to Xeon E5 or better to make any difference or make Intel actually care about supporting native ACS.

So we're back to basics, eh?
"If you want stability and even some support from the manufacturer - use expensive server-class hardware."
Cheapest Xeon E5 costs more that my whole PC.
Are you sure that this breakage and lack of isolation and more quirks coming from newer hardware isn't "accidental" like it was with nvidia?

Oh come on, is invoking Nvidia the new Godwin's law for this thread?  ACS and device isolation is a high-end feature.  I think it's caught Intel by surprise that yes, they do need to advertise the isolation of devices and no, VT-d isn't much use without it.  We've also been quirking Intel 10G NICs to expose isolation, along with workstation and server class chipsets like X79 and X99.  How does that fit into your conspiracy theory?  If we've sparked change in the consumer class products, it may be a couple years before those products start leaving the fab.  In the meantime, we're getting pretty good cooperation.  Maybe there are technical reasons why we're not getting quirks for client CPUs.  Maybe the thing that enables SLI/Crossfire is lack of isolation.  I don't know.

Furthermore, the cheapest E5 Xeons I see on newegg are around $200US, so if your PC is that crappy, maybe you could use an upgrade.

I just find it strange that AMD, being the main and the only "rival" of intel on HPC and desktop markets have no problems with this... right? I agree, sometimes they have different problems, like minor or major bugs or firmware support. Their problems are different.
If i recall correctly, IOMMU feature was present and was even working on athlon 64 x2 series of CPUs, that was, like, eight years ago. That sparkles my curiosity to try using vfio on these platforms and see how it works. (sadly, my local hardware mess-market was closed some time ago due to fire safety concerns)

I just can't conclude anything viable from this situation - all I see now is a lot of quirks and patches and cooperation related solely to intel.
Maybe their CPUs are more widespread?.. Or they are more "open" to the community(considering the open-source work they do)? Or, maybe, their hardware is more troublesome? Or this ACS isn't even present on AMD systems, and nobody noticed it?

P.S.
Cheapest modern Xeon E5 in my areacountry costs around 1000$.
P.P.S.
Godwin's law, heh, I see what you did there.

---

The forum rules prohibit requesting support for distributions other than arch.
I gave up. It was too late.
What I was trying to do.
The reference about VFIO and KVM VGA passthrough.

aw

Member

Registered: 2013-10-04

Posts: 921

Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

I just find it strange that AMD, being the main and the only "rival" of intel on HPC and desktop markets have no problems with this... right? I agree, sometimes they have different problems, like minor or major bugs or firmware support. Their problems are different.

False, while AMD did manage to get ACS on root ports, we still have ACS quirks for other AMD devices.

If i recall correctly, IOMMU feature was present and was even working on athlon 64 x2 series of CPUs, that was, like, eight years ago. That sparkles my curiosity to try using vfio on these platforms and see how it works. (sadly, my local hardware mess-market was closed some time ago due to fire safety concerns)

False, while AMD-IOMMU specs were maybe out before VT-d, it was incredibly difficult to get actual AMD-Vi hardware until well after Intel.  You certainly won't find it on Athlon 64 hardware with AMD-Vi.

I just can't conclude anything viable from this situation - all I see now is a lot of quirks and patches and cooperation related solely to intel.
Maybe their CPUs are more widespread?.. Or they are more "open" to the community(considering the open-source work they do)? Or, maybe, their hardware is more troublesome? Or this ACS isn't even present on AMD systems, and nobody noticed it?

Intel is far more widespread than AMD and they make far more devices.  They not only have more chipsets, but they rev them more often.  Take for instance the AMD 900-series chipset that started shipping in 2011 and is still the latest chipset for non-APU AMD processors.  That was about the same time Intel was shipping the 6-series chipset.  Since then Intel also shipped the 7-series, 8-series, and 9-series as well as X79 and X99.  Not to mention all the peripherals.

---

http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

mqddb

Member

Registered: 2015-04-24

Posts: 5

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Hello everyone:
      Will this vga-passthrough can support spice protocol? currently, spice only support qxl vga.

zir_blazer

Member

Registered: 2013-12-12

Posts: 35

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

I don't really know what Intel is trying to say with that entry in ARK, whether it's any indication of actual support, or just a glitch in their database, but I'm pretty sure the difference between your 8-series having separate IOMMU groups and the 9-series not is software, not hardware.  Maybe Intel will be less willing to provide us with quirks if there's no server versions of a given chipset.

ACS and device isolation is a high-end feature.

That is my point. What I try to say is that is intentional that on consumer Chipsets you get a single group with the PCI Controller Ports and everything else below it. On C226 I got it working out of the box since its a Workstation/Server Chipset, I didn't had to apply patchs or anything to see nearly everything in its own group. It seems that what you're applying quircks to, is to force to enable it anyways (Similar to Intel only allowing you to use the Unlocked Multiplier of their K Series Processors on Z Chipset Motherboards, while some manufacturers discovered ways to bypass that to overclock in cheaper Chipsets: http://www.xbitlabs.com/news/mainboards … oards.html ). Since you were unaware that Intel sells VT-d support as an extra feature in some Chipsets at the first place, its probable that these quircks are related to that.
What I'm missing is having a lot of IOMMU Groups examples on different platforms. I know how mine in a C226 looks. I know the guy with the Z97 that I posted (Which I think he mentioned to not be using the ACS Override patch), and recall a few more consumer platforms with that same arrangement on other Desktop 8-Series. It is not in-depth enough to confirm this, but I'm inclined to believe than that is what VT-d support on the Chipset means for Intel. It seems that the IOMMU isolation capabilities are exclusive for the Chipset PCIe Controller instead of a global one.

along with workstation and server class chipsets like X79 and X99.

These are high end Desktop. Workstation and Server class are their same-silicon and more expensive C counterparts, C202, C204 and C206 for Sandy Bridge-E/Ivy Bridge-E, and C216 for Haswell-E.

If i recall correctly, IOMMU feature was present and was even working on athlon 64 x2 series of CPUs, that was, like, eight years ago. That sparkles my curiosity to try using vfio on these platforms and see how it works. (sadly, my local hardware mess-market was closed some time ago due to fire safety concerns)

AMD-Vi was first supported on Desktop Chipsets with the 890FX, that was during the Phenom II era. 990FX supports it too. However, Bulldozer and derivated processors also incorporate their own IOMMU, so you don't need Chipset (Similar to Intel).

Last edited by zir_blazer (2015-04-24 06:50:40)

aw

Member

Registered: 2013-10-04

Posts: 921

Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Hello everyone:
      Will this vga-passthrough can support spice protocol? currently, spice only support qxl vga.

No, an assigned GPU does not support any of the -display methods.  Remote displays with GPU assignment requires guest-based solutions.

---

http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users