Arch Linux

noobman

Member

Registered: 2013-12-13

Posts: 21

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

About not being able to passthrough the onboard usb controllers, I also had that situation also with a gigabyte mobo, where apparently i could not pass any of the onboard usb controllers. None at all. What i found to be working was to disable the xhci controller entirelly, from bios. And then the passthrough was possible (for me), with either of the other remaining controllers, like a charm. Playing around with bios setup options for xhci and ehci paid off for me. Worth a try imo.

Last edited by noobman (2014-09-29 18:10:26)

Duelist

Member

Registered: 2014-09-22

Posts: 358

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

dmesg | grep vgaarb

[    0.000000] Command line: BOOT_IMAGE=/boot/vmlinuz-linux-mainline root=UUID=e7fe3792-3cba-cf01-605e-37923cbacf01 rw quiet i915.enable_hd_vgaarb=1 pci-stub.ids=1002:6720,1002:aa88 intel_iommu=on vfio_iommu_type1.allow_unsafe_interrupts=1 kvm_intel.emulate_invalid_guest_state=0
[    0.000000] Kernel command line: BOOT_IMAGE=/boot/vmlinuz-linux-mainline root=UUID=e7fe3792-3cba-cf01-605e-37923cbacf01 rw quiet i915.enable_hd_vgaarb=1 pci-stub.ids=1002:6720,1002:aa88 intel_iommu=on vfio_iommu_type1.allow_unsafe_interrupts=1 kvm_intel.emulate_invalid_guest_state=0
[    0.504315] vgaarb: device added: PCI:0000:00:02.0,decodes=io+mem,owns=io+mem,locks=none
[    0.504320] vgaarb: device added: PCI:0000:01:00.0,decodes=io+mem,owns=none,locks=none
[    0.504322] vgaarb: loaded
[    0.504323] vgaarb: bridge control possible 0000:01:00.0
[    0.504324] vgaarb: no bridge control possible 0000:00:02.0
[    1.723418] vgaarb: device changed decodes: PCI:0000:00:02.0,olddecodes=io+mem,decodes=io:owns=io+mem

Any advice would be appreciated.

Do you have host nvidia? If so - apply a patch.
http://www.mail-archive.com/qemu-devel@ … 74066.html
https://bpaste.net/show/109185

---

The forum rules prohibit requesting support for distributions other than arch.
I gave up. It was too late.
What I was trying to do.
The reference about VFIO and KVM VGA passthrough.

et38

Member

Registered: 2014-09-29

Posts: 3

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

About not being able to passthrough the onboard usb controllers, I also had that situation also with a gigabyte mobo, where apparently i could not pass any of the onboard usb controllers. None at all. What i found to be working was to disable the xhci controller entirelly, from bios. And then the passthrough was possible (for me), with either of the other remaining controllers, like a charm. Playing around with bios setup options for xhci and ehci paid off for me. Worth a try imo.

I was looking at the passtrough setup with ga-z97x-u3dh. If it's working, why is the motherboard not recommended in the list?

DanaGoyette

Member

Registered: 2014-01-03

Posts: 46

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

If the hardware is lying about it's support for snoop control then then a patch like below should stop the error.

--- a/include/linux/intel-iommu.h
+++ b/include/linux/intel-iommu.h
@@ -126,7 +126,7 @@ static inline void dmar_writeq(void __iomem *addr, u64 val)
 #define ecap_ir_support(e)	((e >> 3) & 0x1)
 #define ecap_dev_iotlb_support(e)	(((e) >> 2) & 0x1)
 #define ecap_max_handle_mask(e) ((e >> 20) & 0xf)
-#define ecap_sc_support(e)	((e >> 7) & 0x1) /* Snooping Control */
+#define ecap_sc_support(e)	((e >> 7) & 0x0) /* Snooping Control */
 
 /* IOTLB_REG */
 #define DMA_TLB_FLUSH_GRANU_OFFSET  60

I've applied this patch, and now the device works without errors.   Sweet!
For reference:

01:00.0 VGA compatible controller [0300]: Advanced Micro Devices, Inc. [AMD/ATI] Whistler [Radeon E6760] [1002:6743] (prog-if 00 [VGA controller])
	Subsystem: Hightech Information System Ltd. Device [1787:2323]
	Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx+
	Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
	Latency: 0, Cache Line Size: 64 bytes
	Interrupt: pin A routed to IRQ 51
	Region 0: Memory at c0000000 (64-bit, prefetchable) [size=256M]
	Region 2: Memory at eee20000 (64-bit, non-prefetchable) [size=128K]
	Region 4: I/O ports at e000 [size=256]
	Expansion ROM at eee00000 [disabled] [size=128K]
	Capabilities: [50] Power Management version 3
		Flags: PMEClk- DSI- D1+ D2+ AuxCurrent=0mA PME(D0-,D1-,D2-,D3hot-,D3cold-)
		Status: D0 NoSoftRst- PME-Enable- DSel=0 DScale=0 PME-
	Capabilities: [58] Express (v2) Legacy Endpoint, MSI 00
		DevCap:	MaxPayload 256 bytes, PhantFunc 0, Latency L0s <4us, L1 unlimited
			ExtTag+ AttnBtn- AttnInd- PwrInd- RBE+ FLReset-
		DevCtl:	Report errors: Correctable+ Non-Fatal+ Fatal+ Unsupported-
			RlxdOrd- ExtTag+ PhantFunc- AuxPwr- NoSnoop+
			MaxPayload 256 bytes, MaxReadReq 512 bytes
		DevSta:	CorrErr- UncorrErr- FatalErr- UnsuppReq- AuxPwr- TransPend-
		LnkCap:	Port #0, Speed 5GT/s, Width x16, ASPM L0s L1, Exit Latency L0s <64ns, L1 <1us
			ClockPM- Surprise- LLActRep- BwNot-
		LnkCtl:	ASPM Disabled; RCB 64 bytes Disabled- CommClk+
			ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
		LnkSta:	Speed 5GT/s, Width x16, TrErr- Train- SlotClk+ DLActive- BWMgmt- ABWMgmt-
		DevCap2: Completion Timeout: Not Supported, TimeoutDis-, LTR-, OBFF Not Supported
		DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis-, LTR-, OBFF Disabled
		LnkCtl2: Target Link Speed: 5GT/s, EnterCompliance- SpeedDis-
			 Transmit Margin: Normal Operating Range, EnterModifiedCompliance- ComplianceSOS-
			 Compliance De-emphasis: -6dB
		LnkSta2: Current De-emphasis Level: -3.5dB, EqualizationComplete-, EqualizationPhase1-
			 EqualizationPhase2-, EqualizationPhase3-, LinkEqualizationRequest-
	Capabilities: [a0] MSI: Enable+ Count=1/1 Maskable- 64bit+
		Address: 00000000fee00418  Data: 0000
	Capabilities: [100 v1] Vendor Specific Information: ID=0001 Rev=1 Len=010 <?>
	Capabilities: [150 v1] Advanced Error Reporting
		UESta:	DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
		UEMsk:	DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
		UESvrt:	DLP+ SDES+ TLP- FCP+ CmpltTO- CmpltAbrt- UnxCmplt- RxOF+ MalfTLP+ ECRC- UnsupReq- ACSViol-
		CESta:	RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr+
		CEMsk:	RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr+
		AERCap:	First Error Pointer: 00, GenCap+ CGenEn- ChkCap+ ChkEn-
	Kernel driver in use: vfio-pci

EDIT: This may have also fixed passthrough of my ASMedia ASM1062 SATA controller, as well.

Last edited by DanaGoyette (2014-09-29 23:08:15)

Lauer

Member

Registered: 2014-09-11

Posts: 9

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

dmesg | grep vgaarb

[    0.000000] Command line: BOOT_IMAGE=/boot/vmlinuz-linux-mainline root=UUID=e7fe3792-3cba-cf01-605e-37923cbacf01 rw quiet i915.enable_hd_vgaarb=1 pci-stub.ids=1002:6720,1002:aa88 intel_iommu=on vfio_iommu_type1.allow_unsafe_interrupts=1 kvm_intel.emulate_invalid_guest_state=0
[    0.000000] Kernel command line: BOOT_IMAGE=/boot/vmlinuz-linux-mainline root=UUID=e7fe3792-3cba-cf01-605e-37923cbacf01 rw quiet i915.enable_hd_vgaarb=1 pci-stub.ids=1002:6720,1002:aa88 intel_iommu=on vfio_iommu_type1.allow_unsafe_interrupts=1 kvm_intel.emulate_invalid_guest_state=0
[    0.504315] vgaarb: device added: PCI:0000:00:02.0,decodes=io+mem,owns=io+mem,locks=none
[    0.504320] vgaarb: device added: PCI:0000:01:00.0,decodes=io+mem,owns=none,locks=none
[    0.504322] vgaarb: loaded
[    0.504323] vgaarb: bridge control possible 0000:01:00.0
[    0.504324] vgaarb: no bridge control possible 0000:00:02.0
[    1.723418] vgaarb: device changed decodes: PCI:0000:00:02.0,olddecodes=io+mem,decodes=io:owns=io+mem

Any advice would be appreciated.

Do you have host nvidia? If so - apply a patch.
http://www.mail-archive.com/qemu-devel@ … 74066.html
https://bpaste.net/show/109185

Intel IGP and AMD 6990M.

aw

Member

Registered: 2013-10-04

Posts: 921

Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

If the hardware is lying about it's support for snoop control then then a patch like below should stop the error.

--- a/include/linux/intel-iommu.h
+++ b/include/linux/intel-iommu.h
@@ -126,7 +126,7 @@ static inline void dmar_writeq(void __iomem *addr, u64 val)
 #define ecap_ir_support(e)	((e >> 3) & 0x1)
 #define ecap_dev_iotlb_support(e)	(((e) >> 2) & 0x1)
 #define ecap_max_handle_mask(e) ((e >> 20) & 0xf)
-#define ecap_sc_support(e)	((e >> 7) & 0x1) /* Snooping Control */
+#define ecap_sc_support(e)	((e >> 7) & 0x0) /* Snooping Control */
 
 /* IOTLB_REG */
 #define DMA_TLB_FLUSH_GRANU_OFFSET  60

I've applied this patch, and now the device works without errors.   Sweet!

Please post the resulting /tmp/dmar.dsl file from:

$ sudo iasl -p /tmp/dmar -d /sys/firmware/acpi/tables/DMAR

---

http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Denso

Member

Registered: 2014-08-30

Posts: 179

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Alex , did you check the log I posted about DMA errors when using VFIO ? It's in the previous page !  : )

aw

Member

Registered: 2013-10-04

Posts: 921

Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Alex , did you check the log I posted about DMA errors when using VFIO ? It's in the previous page !  : )

Looks like:

http://git.kernel.org/cgit/linux/kernel … fe8702b098

Are you running 3.14 or newer?

---

http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Denso

Member

Registered: 2014-08-30

Posts: 179

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Alex , did you check the log I posted about DMA errors when using VFIO ? It's in the previous page !  : )

Looks like:

http://git.kernel.org/cgit/linux/kernel … fe8702b098

Are you running 3.14 or newer?

Yes , 3.16.3 .

aw

Member

Registered: 2013-10-04

Posts: 921

Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Alex , did you check the log I posted about DMA errors when using VFIO ? It's in the previous page !  : )

Looks like:

http://git.kernel.org/cgit/linux/kernel … fe8702b098

Are you running 3.14 or newer?

Yes , 3.16.3 .

Still seems potentially related to large pages.  Does using either of these options avoid it?

a) boot arg: intel_iommu=on,sp_off

b) module arg: vfio_iommu_type1 disable_hugepages=1

---

http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

DanaGoyette

Member

Registered: 2014-01-03

Posts: 46

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Please post the resulting /tmp/dmar.dsl file from:

$ sudo iasl -p /tmp/dmar -d /sys/firmware/acpi/tables/DMAR

/*
 * Intel ACPI Component Architecture
 * AML Disassembler version 20140214-64 [Mar 29 2014]
 * Copyright (c) 2000 - 2014 Intel Corporation
 * 
 * Disassembly of /sys/firmware/acpi/tables/DMAR, Mon Sep 29 19:22:29 2014
 *
 * ACPI Data Table [DMAR]
 *
 * Format: [HexOffset DecimalOffset ByteLength]  FieldName : FieldValue
 */

[000h 0000   4]                    Signature : "DMAR"    [DMA Remapping table]
[004h 0004   4]                 Table Length : 000000A8
[008h 0008   1]                     Revision : 01
[009h 0009   1]                     Checksum : 57
[00Ah 0010   6]                       Oem ID : "INTEL "
[010h 0016   8]                 Oem Table ID : "BDW "
[018h 0024   4]                 Oem Revision : 00000001
[01Ch 0028   4]              Asl Compiler ID : "INTL"
[020h 0032   4]        Asl Compiler Revision : 00000001

[024h 0036   1]           Host Address Width : 26
[025h 0037   1]                        Flags : 03
[026h 0038  10]                     Reserved : 00 00 00 00 00 00 00 00 00 00

[030h 0048   2]                Subtable Type : 0000 [Hardware Unit Definition]
[032h 0050   2]                       Length : 0018

[034h 0052   1]                        Flags : 00
[035h 0053   1]                     Reserved : 00
[036h 0054   2]           PCI Segment Number : 0000
[038h 0056   8]        Register Base Address : 00000000FED90000

[040h 0064   1]      Device Scope Entry Type : 01
[041h 0065   1]                 Entry Length : 08
[042h 0066   2]                     Reserved : 0000
[044h 0068   1]               Enumeration ID : 00
[045h 0069   1]               PCI Bus Number : 00

[046h 0070   2]                     PCI Path : 02,00


[048h 0072   2]                Subtable Type : 0000 [Hardware Unit Definition]
[04Ah 0074   2]                       Length : 0020

[04Ch 0076   1]                        Flags : 01
[04Dh 0077   1]                     Reserved : 00
[04Eh 0078   2]           PCI Segment Number : 0000
[050h 0080   8]        Register Base Address : 00000000FED91000

[058h 0088   1]      Device Scope Entry Type : 03
[059h 0089   1]                 Entry Length : 08
[05Ah 0090   2]                     Reserved : 0000
[05Ch 0092   1]               Enumeration ID : 08
[05Dh 0093   1]               PCI Bus Number : F0

[05Eh 0094   2]                     PCI Path : 1F,00


[060h 0096   1]      Device Scope Entry Type : 04
[061h 0097   1]                 Entry Length : 08
[062h 0098   2]                     Reserved : 0000
[064h 0100   1]               Enumeration ID : 00
[065h 0101   1]               PCI Bus Number : F0

[066h 0102   2]                     PCI Path : 0F,00


[068h 0104   2]                Subtable Type : 0001 [Reserved Memory Region]
[06Ah 0106   2]                       Length : 0020

[06Ch 0108   2]                     Reserved : 0000
[06Eh 0110   2]           PCI Segment Number : 0000
[070h 0112   8]                 Base Address : 000000006DEA5000
[078h 0120   8]          End Address (limit) : 000000006DEB3FFF

[080h 0128   1]      Device Scope Entry Type : 01
[081h 0129   1]                 Entry Length : 08
[082h 0130   2]                     Reserved : 0000
[084h 0132   1]               Enumeration ID : 00
[085h 0133   1]               PCI Bus Number : 00

[086h 0134   2]                     PCI Path : 14,00


[088h 0136   2]                Subtable Type : 0001 [Reserved Memory Region]
[08Ah 0138   2]                       Length : 0020

[08Ch 0140   2]                     Reserved : 0000
[08Eh 0142   2]           PCI Segment Number : 0000
[090h 0144   8]                 Base Address : 000000006F000000
[098h 0152   8]          End Address (limit) : 000000007F1FFFFF

[0A0h 0160   1]      Device Scope Entry Type : 01
[0A1h 0161   1]                 Entry Length : 08
[0A2h 0162   2]                     Reserved : 0000
[0A4h 0164   1]               Enumeration ID : 00
[0A5h 0165   1]               PCI Bus Number : 00

[0A6h 0166   2]                     PCI Path : 02,00


Raw Table Data: Length 168 (0xA8)

  0000: 44 4D 41 52 A8 00 00 00 01 57 49 4E 54 45 4C 20  DMAR.....WINTEL 
  0010: 42 44 57 20 00 00 00 00 01 00 00 00 49 4E 54 4C  BDW ........INTL
  0020: 01 00 00 00 26 03 00 00 00 00 00 00 00 00 00 00  ....&...........
  0030: 00 00 18 00 00 00 00 00 00 00 D9 FE 00 00 00 00  ................
  0040: 01 08 00 00 00 00 02 00 00 00 20 00 01 00 00 00  .......... .....
  0050: 00 10 D9 FE 00 00 00 00 03 08 00 00 08 F0 1F 00  ................
  0060: 04 08 00 00 00 F0 0F 00 01 00 20 00 00 00 00 00  .......... .....
  0070: 00 50 EA 6D 00 00 00 00 FF 3F EB 6D 00 00 00 00  .P.m.....?.m....
  0080: 01 08 00 00 00 00 14 00 01 00 20 00 00 00 00 00  .......... .....
  0090: 00 00 00 6F 00 00 00 00 FF FF 1F 7F 00 00 00 00  ...o............
  00A0: 01 08 00 00 00 00 02 00                          ........

aw

Member

Registered: 2013-10-04

Posts: 921

Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Please post the resulting /tmp/dmar.dsl file from:

$ sudo iasl -p /tmp/dmar -d /sys/firmware/acpi/tables/DMAR

/*
[030h 0048   2]                Subtable Type : 0000 [Hardware Unit Definition]
[032h 0050   2]                       Length : 0018

[034h 0052   1]                        Flags : 00
[035h 0053   1]                     Reserved : 00
[036h 0054   2]           PCI Segment Number : 0000
[038h 0056   8]        Register Base Address : 00000000FED90000

[040h 0064   1]      Device Scope Entry Type : 01
[041h 0065   1]                 Entry Length : 08
[042h 0066   2]                     Reserved : 0000
[044h 0068   1]               Enumeration ID : 00
[045h 0069   1]               PCI Bus Number : 00

[046h 0070   2]                     PCI Path : 02,00


[048h 0072   2]                Subtable Type : 0000 [Hardware Unit Definition]
[04Ah 0074   2]                       Length : 0020

[04Ch 0076   1]                        Flags : 01
[04Dh 0077   1]                     Reserved : 00
[04Eh 0078   2]           PCI Segment Number : 0000
[050h 0080   8]        Register Base Address : 00000000FED91000

Ok, as expected you have two IOMMU hardware units, the first is exclusively for the IGD (00:02.0).  We can match that to dmesg by comparing the base address:

dmar: IOMMU 0: reg_base_addr fed90000 ver 1:0 cap c0000020660462 ecap f0101a

The 2nd IOMMU hardware unit has the INCLUDE_PCI_ALL flag bit set, so everything else is handled by this IOMMU, which we can also verify by base address:

dmar: IOMMU 1: reg_base_addr fed91000 ver 1:0 cap d2008c20660462 ecap f010da

So, unless you're trying to assign IGD, we know you're exclusively using IOMMU1, which has an extended capability register value of 0xf010da.  Bit 7 in that register indicates support for Snoop Control and you've already verified that ignoring this bit fixes the problem.  When SC is supported, we can program the IOMMU page tables with the SNP bit set which causes processor caches to be snooped regardless of the NoSnoop tag on the transaction.  This effectively makes all DMA cache coherent and KVM doesn't need to emulated WBINVD to maintain coherency.

In your case the hardware tells us that SC is supported, but the IOMMU page table hardware considers it a reserved bit and throws a fault.  What CPU model are you using?  Is it perhaps a prototype?  I believe some systems have a BIOS config option to toggle IOMMU caching support, if you have such an option, disable it.  I think you're looking at either a hardware or BIOS issue.

---

http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Denso

Member

Registered: 2014-08-30

Posts: 179

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Looks like:

http://git.kernel.org/cgit/linux/kernel … fe8702b098

Are you running 3.14 or newer?

Yes , 3.16.3 .

Still seems potentially related to large pages.  Does using either of these options avoid it?

a) boot arg: intel_iommu=on,sp_off

b) module arg: vfio_iommu_type1 disable_hugepages=1

YES !
adding : "vfio_iommu_type1.disable_hugepages=1" to boot parameters solved the issue . Now I'll see if I can reboot this VM without crashing the host !

EDIT : Aaaand , it reboots just fine !

Thank you Alex , we appreciate your kind efforts !

Last edited by Denso (2014-09-30 03:55:44)

DanaGoyette

Member

Registered: 2014-01-03

Posts: 46

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

So, unless you're trying to assign IGD, we know you're exclusively using IOMMU1, which has an extended capability register value of 0xf010da.  Bit 7 in that register indicates support for Snoop Control and you've already verified that ignoring this bit fixes the problem.  When SC is supported, we can program the IOMMU page tables with the SNP bit set which causes processor caches to be snooped regardless of the NoSnoop tag on the transaction.  This effectively makes all DMA cache coherent and KVM doesn't need to emulated WBINVD to maintain coherency.

In your case the hardware tells us that SC is supported, but the IOMMU page table hardware considers it a reserved bit and throws a fault.  What CPU model are you using?  Is it perhaps a prototype?  I believe some systems have a BIOS config option to toggle IOMMU caching support, if you have such an option, disable it.  I think you're looking at either a hardware or BIOS issue.

For reference, the CPU is a Xeon E3-1245v3.

I looked around in my BIOS, and there were no options for IOMMU caching -- but there WAS an option for X2APIC Opt-Out.
X2APIC Opt-Out was enabled, so I disabled it.  Now I've booted the non-patched kernel, and the errors are no longer occurring. 
Thanks for the suggestion of checking the BIOS settings.

[    0.067534] dmar: Host address width 39
[    0.067537] dmar: DRHD base: 0x000000fed90000 flags: 0x0
[    0.067544] dmar: IOMMU 0: reg_base_addr fed90000 ver 1:0 cap c0000020660462 ecap f0101a
[    0.067546] dmar: DRHD base: 0x000000fed91000 flags: 0x1
[    0.067551] dmar: IOMMU 1: reg_base_addr fed91000 ver 1:0 cap d2008c20660462 ecap f010da
[    0.067552] dmar: RMRR base: 0x0000006dea5000 end: 0x0000006deb3fff
[    0.067553] dmar: RMRR base: 0x0000006f000000 end: 0x0000007f1fffff
[    0.067618] IOAPIC id 8 under DRHD base  0xfed91000 IOMMU 1
[    0.067621] Queued invalidation will be enabled to support x2apic and Intr-remapping.
[    0.067731] Enabled IRQ remapping in x2apic mode
[    0.067733] Enabling x2apic
[    0.067734] Enabled x2apic
[    0.067738] Switched APIC routing to cluster x2apic.
[    0.629294] DMAR: No ATSR found
[    0.629318] IOMMU 0 0xfed90000: using Queued invalidation
[    0.629319] IOMMU 1 0xfed91000: using Queued invalidation
[    0.629321] IOMMU: Setting RMRR:
[    0.629330] IOMMU: Setting identity map for device 0000:00:02.0 [0x6f000000 - 0x7f1fffff]
[    0.630530] IOMMU: Setting identity map for device 0000:00:14.0 [0x6dea5000 - 0x6deb3fff]
[    0.630547] IOMMU: Prepare 0-16MiB unity mapping for LPC
[    0.630554] IOMMU: Setting identity map for device 0000:00:1f.0 [0x0 - 0xffffff]

Last edited by DanaGoyette (2014-09-30 05:23:06)

Denso

Member

Registered: 2014-08-30

Posts: 179

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Hey everyone .

So I'm trying to use OVMF instead of SeaBIOS , I used ovmf-svn package from AUR and added the option "-pflash /*path to ovmf_x64.bin*" to my VM aguments . Also I deleted "-vga none" AND "x-vga=on" .

It booted just fine , GPU works as primary no problem .

Except for one big issue :

It doesn't detect either my VM's HDD nor ISOs .

So it goes like this :

EFI Floppy
EFI Floppy 1
EFI Misc Device
EFI Network

then it drops me to the UEFI shell .

Any reason why it didn't pickup my HDD or ISO images to boot from ?

aw

Member

Registered: 2013-10-04

Posts: 921

Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Hey everyone .

So I'm trying to use OVMF instead of SeaBIOS , I used ovmf-svn package from AUR and added the option "-pflash /*path to ovmf_x64.bin*" to my VM aguments . Also I deleted "-vga none" AND "x-vga=on" .

It booted just fine , GPU works as primary no problem .

Except for one big issue :

It doesn't detect either my VM's HDD nor ISOs .

So it goes like this :

EFI Floppy
EFI Floppy 1
EFI Misc Device
EFI Network

then it drops me to the UEFI shell .

Any reason why it didn't pickup my HDD or ISO images to boot from ?

What interface are you using to expose them to the guest?  IDE?  AHCI?  virtio?

---

http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Denso

Member

Registered: 2014-08-30

Posts: 179

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Hey everyone .

So I'm trying to use OVMF instead of SeaBIOS , I used ovmf-svn package from AUR and added the option "-pflash /*path to ovmf_x64.bin*" to my VM aguments . Also I deleted "-vga none" AND "x-vga=on" .

It booted just fine , GPU works as primary no problem .

Except for one big issue :

It doesn't detect either my VM's HDD nor ISOs .

So it goes like this :

EFI Floppy
EFI Floppy 1
EFI Misc Device
EFI Network

then it drops me to the UEFI shell .

Any reason why it didn't pickup my HDD or ISO images to boot from ?

What interface are you using to expose them to the guest?  IDE?  AHCI?  virtio?

Virtio for the HDD and IDE for the ISOs ... This is the entire command :

qemu-system-x86_64 -name main -nographic \
-enable-kvm -M q35 -m 6000 -cpu Haswell,hv_time,hv_relaxed,hv_vapic,hv_spinlocks=0x1000,kvm=off -smp 2,sockets=1,cores=1,threads=2 \
-pflash /VMs/ovmf_x64.bin \
-device ioh3420,bus=pcie.0,addr=1c.0,multifunction=on,port=1,chassis=1,id=root.1 \
-device vfio-pci,host=05:00.0,bus=root.1,addr=00.0,multifunction=on \
-device vfio-pci,host=05:00.1,bus=root.1,addr=00.1 \
-device vfio-pci,host=00:1b.0,bus=pcie.0 \
-drive file=/VMs/Win_Main.img,if=virtio,cache=none \
-drive file=/VMs/Win8.iso,id=isocd -device ide-cd,bus=ide.1,drive=isocd \
-drive file=/VMs/virtio.iso,id=isocd2 -device ide-cd,bus=ide.2,drive=isocd2 \
-net nic,model=virtio,macaddr=1c:2c:44:11:99:2d -net bridge,br=br0 \
-usb -usbdevice host:15d9:0a4f -usbdevice host:045e:0745 \
-localtime \
-monitor unix:/tmp/vm_main,server,nowait &

Last edited by Denso (2014-09-30 15:48:15)

aw

Member

Registered: 2013-10-04

Posts: 921

Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Virtio for the HDD and IDE for the ISOs ... This is the entire command :

qemu-system-x86_64 -name main -nographic \
-enable-kvm -M q35 -m 6000 -cpu Haswell,hv_time,hv_relaxed,hv_vapic,hv_spinlocks=0x1000,kvm=off -smp 2,sockets=1,cores=1,threads=2 \
-pflash /VMs/ovmf_x64.bin \
-device ioh3420,bus=pcie.0,addr=1c.0,multifunction=on,port=1,chassis=1,id=root.1 \
-device vfio-pci,host=05:00.0,bus=root.1,addr=00.0,multifunction=on \
-device vfio-pci,host=05:00.1,bus=root.1,addr=00.1 \
-device vfio-pci,host=00:1b.0,bus=pcie.0 \
-drive file=/VMs/Win_Main.img,if=virtio,cache=none \
-drive file=/VMs/Win8.iso,id=isocd -device ide-cd,bus=ide.1,drive=isocd \
-drive file=/VMs/virtio.iso,id=isocd2 -device ide-cd,bus=ide.2,drive=isocd2 \
-net nic,model=virtio,macaddr=1c:2c:44:11:99:2d -net bridge,br=br0 \
-usb -usbdevice host:15d9:0a4f -usbdevice host:045e:0745 \
-localtime \
-monitor unix:/tmp/vm_main,server,nowait &

Any particular reason for using q35?  When you get an EFI shell, do you have any disks available?  EFI shell is just like DOS except the disk are named fs0:, fs1:, etc rather than C:, D:...  If you have disks you can try to find a a boot EFI file and execute it.  You can also try adding bootindex=# to each drive to try to force a boot order.  I haven't had time yet to try to attempt a q35+ovmf install and I have little reason to switch from 440fx other than curiosity.

---

http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Denso

Member

Registered: 2014-08-30

Posts: 179

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Virtio for the HDD and IDE for the ISOs ... This is the entire command :

qemu-system-x86_64 -name main -nographic \
-enable-kvm -M q35 -m 6000 -cpu Haswell,hv_time,hv_relaxed,hv_vapic,hv_spinlocks=0x1000,kvm=off -smp 2,sockets=1,cores=1,threads=2 \
-pflash /VMs/ovmf_x64.bin \
-device ioh3420,bus=pcie.0,addr=1c.0,multifunction=on,port=1,chassis=1,id=root.1 \
-device vfio-pci,host=05:00.0,bus=root.1,addr=00.0,multifunction=on \
-device vfio-pci,host=05:00.1,bus=root.1,addr=00.1 \
-device vfio-pci,host=00:1b.0,bus=pcie.0 \
-drive file=/VMs/Win_Main.img,if=virtio,cache=none \
-drive file=/VMs/Win8.iso,id=isocd -device ide-cd,bus=ide.1,drive=isocd \
-drive file=/VMs/virtio.iso,id=isocd2 -device ide-cd,bus=ide.2,drive=isocd2 \
-net nic,model=virtio,macaddr=1c:2c:44:11:99:2d -net bridge,br=br0 \
-usb -usbdevice host:15d9:0a4f -usbdevice host:045e:0745 \
-localtime \
-monitor unix:/tmp/vm_main,server,nowait &

Any particular reason for using q35?  When you get an EFI shell, do you have any disks available?  EFI shell is just like DOS except the disk are named fs0:, fs1:, etc rather than C:, D:...  If you have disks you can try to find a a boot EFI file and execute it.  You can also try adding bootindex=# to each drive to try to force a boot order.  I haven't had time yet to try to attempt a q35+ovmf install and I have little reason to switch from 440fx other than curiosity.

Tried 440fx , but it complains that it can't find "pcie.0" for the GPU . What am I missing ?

And no disks are found in UEFI (No fs# , just BLK# which are the floppies I think) .

aw

Member

Registered: 2013-10-04

Posts: 921

Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Tried 440fx , but it complains that it can't find "pcie.0" for the GPU . What am I missing ?

And no disks are found in UEFI (No fs# , just BLK# which are the floppies I think) .

This

-device ioh3420,bus=pcie.0,addr=1c.0,multifunction=on,port=1,chassis=1,id=root.1 \
-device vfio-pci,host=05:00.0,bus=root.1,addr=00.0,multifunction=on \
-device vfio-pci,host=05:00.1,bus=root.1,addr=00.1 \
-device vfio-pci,host=00:1b.0,bus=pcie.0 \

Becomes this with 440fx:

-device vfio-pci,host=05:00.0,addr=06.0,multifunction=on \
-device vfio-pci,host=05:00.1,addr=06.1 \
-device vfio-pci,host=00:1b.0,addr=7.0 \

I picked arbitrary addresses, feel free to change them.

Last edited by aw (2014-09-30 16:08:08)

---

http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Denso

Member

Registered: 2014-08-30

Posts: 179

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Tried 440fx , but it complains that it can't find "pcie.0" for the GPU . What am I missing ?

And no disks are found in UEFI (No fs# , just BLK# which are the floppies I think) .

This

-device ioh3420,bus=pcie.0,addr=1c.0,multifunction=on,port=1,chassis=1,id=root.1 \
-device vfio-pci,host=05:00.0,bus=root.1,addr=00.0,multifunction=on \
-device vfio-pci,host=05:00.1,bus=root.1,addr=00.1 \
-device vfio-pci,host=00:1b.0,bus=pcie.0 \

Becomes this with 440fx:

-device vfio-pci,host=05:00.0,addr=06.0,multifunction=on \
-device vfio-pci,host=05:00.1,addr=06.1 \
-device vfio-pci,host=00:1b.0,addr=7.0 \

I picked arbitrary addresses, feel free to change them.

OK , it booted fine , but still no disks . Host is running on legacy BIOS , perhaps that's the reason ?

noobman

Member

Registered: 2013-12-13

Posts: 21

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

About not being able to passthrough the onboard usb controllers, I also had that situation also with a gigabyte mobo, where apparently i could not pass any of the onboard usb controllers. None at all. What i found to be working was to disable the xhci controller entirelly, from bios. And then the passthrough was possible (for me), with either of the other remaining controllers, like a charm. Playing around with bios setup options for xhci and ehci paid off for me. Worth a try imo.

I was looking at the passtrough setup with ga-z97x-u3dh. If it's working, why is the motherboard not recommended in the list?

I had a bad experience with it, it has many issues. First off the one above, have to disable xhci entirelly from bios, from which point the 3.0 usb is not entirelly lost but become 2.0 instead, and then and only then one can passthrough other usb controllers (F7 bios). Not sure why its like that, i guess its probably due to a particular usb tree implementation. Also the latest bios i tried (F7 again) spilled some other errors so imo that bios is also bit broke. E.g. board gave "cannot find upstream pcie" an error like that, again i guess due to the pci arhitecture tree, but this last one got fixed i think in 3.17. Back few (1-2-3?) months ago i think stock kernels gave like 10+ different errors on it, researched and customized and patched one and got the number down. The uefi side threw some errors, and i disabled it also, but that is ok coz i just dont like uefi anyway. Also a number of strange errors when i iommu isolate MEI controller, something which i like to do and which should be fine but in this case wasnt. Dunno how it is now if there is any F8 bios or other fixes are in 3.17+ or not, because i made the system for a friend and its gone (so all comments are just from my memory), but otherwise its working with passthrough. Board on the overall also had many small issues which added up for me. Just to have an ideea, for example the warning for CPU max temp and fan are disabled as default settings (warnings or ctrl) so just by using bios default settings there is nothing that would prevent a catastrophe if the cpu gets to max temp and over it (e.g. if fan fails). For me those default settings show how much attention was given to the board. I think the board was among first z97 implementation and i think it simply got rushed over to launch. Quite some months ago i was looking for the latest, looked great on paper and specs, but a worse in practice so i just ended up as not happy with it. "Not recommented "is just that, a recommendation, just take it like that, you could get one and have your own and maybe different experience with it. But just saying this, if you want something specifially for passthrough and looking to buy, then simply why not get what the guy which does vfio have, like the 990FX which i think that is what aw has (?) not sure lol, but that paired with an 8 core cpu could be quite a performant option. But that is just a thought, nothing more. Mentioning coz that would be my choice right now, and most likely next time i will do exactly that.

aw

Member

Registered: 2013-10-04

Posts: 921

Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Tried 440fx , but it complains that it can't find "pcie.0" for the GPU . What am I missing ?

And no disks are found in UEFI (No fs# , just BLK# which are the floppies I think) .

This

-device ioh3420,bus=pcie.0,addr=1c.0,multifunction=on,port=1,chassis=1,id=root.1 \
-device vfio-pci,host=05:00.0,bus=root.1,addr=00.0,multifunction=on \
-device vfio-pci,host=05:00.1,bus=root.1,addr=00.1 \
-device vfio-pci,host=00:1b.0,bus=pcie.0 \

Becomes this with 440fx:

-device vfio-pci,host=05:00.0,addr=06.0,multifunction=on \
-device vfio-pci,host=05:00.1,addr=06.1 \
-device vfio-pci,host=00:1b.0,addr=7.0 \

I picked arbitrary addresses, feel free to change them.

OK , it booted fine , but still no disks . Host is running on legacy BIOS , perhaps that's the reason ?

The host using legacy/UEFI BIOS is completely separate from the guest, I think it more likely has something to do with how you're specifying the disks.  I typically specify an installation and virtio ISO like this:

-drive file=/ISOs/msdn/en_windows_8.1_professional_n_vl_with_update_x64_dvd_4065208.iso,if=none,id=dvd0,media=cdrom \
-device ide-cd,drive=dvd0,bootindex=2 \
-drive file=/ISOs/virtio-win-0.1-74.iso,if=none,id=dvd1,media=cdrom \
-device ide-cd,drive=dvd1 \

The guest disk should really look more like this:

-drive file=/path/to/guest/disk.img,cache=none,if=none,id=drive0,aio=threads \
-device virtio-blk-pci,drive=drive0,ioeventfd=on,bootindex=1 \

---

http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Denso

Member

Registered: 2014-08-30

Posts: 179

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

This

-device ioh3420,bus=pcie.0,addr=1c.0,multifunction=on,port=1,chassis=1,id=root.1 \
-device vfio-pci,host=05:00.0,bus=root.1,addr=00.0,multifunction=on \
-device vfio-pci,host=05:00.1,bus=root.1,addr=00.1 \
-device vfio-pci,host=00:1b.0,bus=pcie.0 \

Becomes this with 440fx:

-device vfio-pci,host=05:00.0,addr=06.0,multifunction=on \
-device vfio-pci,host=05:00.1,addr=06.1 \
-device vfio-pci,host=00:1b.0,addr=7.0 \

I picked arbitrary addresses, feel free to change them.

OK , it booted fine , but still no disks . Host is running on legacy BIOS , perhaps that's the reason ?

The host using legacy/UEFI BIOS is completely separate from the guest, I think it more likely has something to do with how you're specifying the disks.  I typically specify an installation and virtio ISO like this:

-drive file=/ISOs/msdn/en_windows_8.1_professional_n_vl_with_update_x64_dvd_4065208.iso,if=none,id=dvd0,media=cdrom \
-device ide-cd,drive=dvd0,bootindex=2 \
-drive file=/ISOs/virtio-win-0.1-74.iso,if=none,id=dvd1,media=cdrom \
-device ide-cd,drive=dvd1 \

The guest disk should really look more like this:

-drive file=/path/to/guest/disk.img,cache=none,if=none,id=drive0,aio=threads \
-device virtio-blk-pci,drive=drive0,ioeventfd=on,bootindex=1 \

Finally OVMF detected my Win8.iso , however trying to execute /EFI/BOOT/BOOTX64.EFI hangs the guest for 3 seconds then returns to the shell prompt .

Googling told me to execute that file to start the Windows installer  : (

et38

Member

Registered: 2014-09-29

Posts: 3

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

But just saying this, if you want something specifially for passthrough and looking to buy, then simply why not get what the guy which does vfio have, like the 990FX which i think that is what aw has (?) not sure lol, but that paired with an 8 core cpu could be quite a performant option. But that is just a thought, nothing more. Mentioning coz that would be my choice right now, and most likely next time i will do exactly that.

I've been thinking (a lot) of an 8-core Amd system, but the power consumption difference (vs. Intel 4c+HT) is just annoying. I'm just trying to see if there are any preferredly (Gigabyte/Asus/Z97/H97) motherboards offering relatively painless vga passthrough experience. Guests would probably be Steam OS / other dev distros.