You are not logged in.

#1501 2014-04-07 01:23:31

Nex7
Member
Registered: 2014-04-06
Posts: 5

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Cubex wrote:

Hello Nex7, I am issuing the same problems, everything works until I install the nvidia driver on Win7

Hey Cubex, I haven't yet gotten this to work, BUT, I have good news for you, IF you have qemu-1.7.4 or less (you probably have 1.7.0 like I did, from 'pacman -S qemu'). You need 1.7.5+. I can confirm that with the latest 'qemu-git' (and 'seabios-git'), my VM was able to boot into Windows with graphics, but a bunch of other crap was screwed up. I am currently attributing this to TOO NEW a qemu, since the qemu-git package seems to be grabbing the absolute bleeding edge. After dinner I will try to get the package to pull down something post 1.7.5 but not bleeding edge.

Offline

#1502 2014-04-07 09:08:41

trimm
Member
Registered: 2014-03-06
Posts: 31

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

nbhs: Could you perhaps tell us when you update qemu-git and seabios-git aswell? Or are these the same packages we get from AUR?

Offline

#1503 2014-04-08 03:56:38

nbhs
Member
From: Montevideo, Uruguay
Registered: 2013-05-02
Posts: 402

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

trimm wrote:

nbhs: Could you perhaps tell us when you update qemu-git and seabios-git aswell? Or are these the same packages we get from AUR?

These are git versions, so unless something breaks or fails to build theres no reason to update the package, you can use the AUR versions if you want since there are no extra patches needed

Offline

#1504 2014-04-08 18:40:26

Val532
Member
Registered: 2013-11-13
Posts: 35

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Hi all,

I have a small probleme, on my VM i want to start OCCT and i got a BSOD "driver_irql_not_less_or_equal" i have not any other information on the BSOD screen.

I run on qemu 2.0.0-RC1 and kernel 3.14, i have two graphique card AMD HD 6870 (crossfire not enable) on the VM (just for testing).

On dmesg i have :

[ 8704.000634] kvm_get_msr_common: 82 callbacks suppressed
[ 8704.000643] kvm [6821]: vcpu0 unhandled rdmsr: 0xce
[ 8704.000686] kvm [6821]: vcpu0 unhandled rdmsr: 0xce
[ 8704.004952] kvm [6821]: vcpu0 unhandled rdmsr: 0x637
[ 8704.004976] kvm [6821]: vcpu0 unhandled rdmsr: 0xe8
[ 8704.004989] kvm [6821]: vcpu0 unhandled rdmsr: 0xe7
[ 8704.005013] kvm [6821]: vcpu0 unhandled rdmsr: 0x31
[ 8704.005030] kvm [6821]: vcpu0 unhandled rdmsr: 0x35
[ 8704.005050] kvm [6821]: vcpu0 unhandled rdmsr: 0x39
[ 8704.005095] kvm [6821]: vcpu0 unhandled rdmsr: 0xce
[ 8704.005108] kvm [6821]: vcpu0 unhandled rdmsr: 0x194

Any help ?
Thanks

Last edited by Val532 (2014-04-08 18:40:56)

Offline

#1505 2014-04-08 19:03:24

nbhs
Member
From: Montevideo, Uruguay
Registered: 2013-05-02
Posts: 402

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Val532 wrote:

Hi all,

I have a small probleme, on my VM i want to start OCCT and i got a BSOD "driver_irql_not_less_or_equal" i have not any other information on the BSOD screen.

I run on qemu 2.0.0-RC1 and kernel 3.14, i have two graphique card AMD HD 6870 (crossfire not enable) on the VM (just for testing).

On dmesg i have :

[ 8704.000634] kvm_get_msr_common: 82 callbacks suppressed
[ 8704.000643] kvm [6821]: vcpu0 unhandled rdmsr: 0xce
[ 8704.000686] kvm [6821]: vcpu0 unhandled rdmsr: 0xce
[ 8704.004952] kvm [6821]: vcpu0 unhandled rdmsr: 0x637
[ 8704.004976] kvm [6821]: vcpu0 unhandled rdmsr: 0xe8
[ 8704.004989] kvm [6821]: vcpu0 unhandled rdmsr: 0xe7
[ 8704.005013] kvm [6821]: vcpu0 unhandled rdmsr: 0x31
[ 8704.005030] kvm [6821]: vcpu0 unhandled rdmsr: 0x35
[ 8704.005050] kvm [6821]: vcpu0 unhandled rdmsr: 0x39
[ 8704.005095] kvm [6821]: vcpu0 unhandled rdmsr: 0xce
[ 8704.005108] kvm [6821]: vcpu0 unhandled rdmsr: 0x194

Any help ?
Thanks

try

echo "options kvm ignore_msrs=1" >> /etc/modprobe.d/kvm.conf

Offline

#1506 2014-04-08 20:45:39

beckend
Member
Registered: 2014-03-16
Posts: 5

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Hours of tests, 3.14 posted on OP for me was SLOW to a crawl, 3.13 old kernel posted before gets me almost native speed.

Offline

#1507 2014-04-09 10:30:49

novist
Member
Registered: 2014-03-14
Posts: 47

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

beckend wrote:

Hours of tests, 3.14 posted on OP for me was SLOW to a crawl, 3.13 old kernel posted before gets me almost native speed.

i am experiencing unexpected slowdown too. could you please specify your hardware, exact kernel version (if its 3.13.something) and patches?


For others if anyone cares i managed to get my R9 270X gpu running as secondary on libvirt i440fx (default) chipset. here is xml:

<!--
WARNING: THIS IS AN AUTO-GENERATED FILE. CHANGES TO IT ARE LIKELY TO BE
OVERWRITTEN AND LOST. Changes to this xml configuration should be made using:
  virsh edit win
or other application using the libvirt API.
-->

<domain type='kvm'>
  <name>win</name>
  <uuid>9d99f9e4-c66f-427e-89a2-0285358ce620</uuid>
  <memory unit='KiB'>4194304</memory>
  <currentMemory unit='KiB'>4194304</currentMemory>
  <vcpu placement='static'>8</vcpu>
  <os>
    <type arch='x86_64' machine='pc-i440fx-2.0'>hvm</type>
    <boot dev='hd'/>
  </os>
  <features>
    <acpi/>
    <apic/>
    <pae/>
  </features>
  <cpu mode='host-passthrough'>
    <topology sockets='1' cores='4' threads='2'/>
  </cpu>
  <clock offset='localtime'>
    <timer name='rtc' tickpolicy='catchup'/>
    <timer name='pit' tickpolicy='delay'/>
    <timer name='hpet' present='no'/>
  </clock>
  <on_poweroff>destroy</on_poweroff>
  <on_reboot>restart</on_reboot>
  <on_crash>restart</on_crash>
  <pm>
    <suspend-to-mem enabled='no'/>
    <suspend-to-disk enabled='no'/>
  </pm>
  <devices>
    <emulator>/usr/local/bin/qemu-system-x86_64</emulator>
    <disk type='block' device='disk'>
      <driver name='qemu' type='raw' cache='none' io='native'/>
      <source dev='/dev/mapper/lvm-vm'/>
      <target dev='vda' bus='virtio'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x05' function='0x0'/>
    </disk>
    <controller type='usb' index='0' model='ich9-ehci1'>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x04' function='0x7'/>
    </controller>
    <controller type='usb' index='0' model='ich9-uhci1'>
      <master startport='0'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x04' function='0x0' multifunction='on'/>
    </controller>
    <controller type='usb' index='0' model='ich9-uhci2'>
      <master startport='2'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x04' function='0x1'/>
    </controller>
    <controller type='usb' index='0' model='ich9-uhci3'>
      <master startport='4'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x04' function='0x2'/>
    </controller>
    <controller type='pci' index='0' model='pci-root'/>
    <interface type='network'>
      <mac address='50:64:11:7c:b6:85'/>
      <source network='default'/>
      <model type='virtio'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x0'/>
    </interface>
    <input type='mouse' bus='ps2'/>
    <input type='keyboard' bus='ps2'/>
    <graphics type='vnc' port='-1' autoport='yes' keymap='en-us'/>
    <video>
      <model type='vga' vram='9216' heads='1'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x0'/>
    </video>
    <hostdev mode='subsystem' type='pci' managed='yes'>
      <source>
        <address domain='0x0000' bus='0x06' slot='0x00' function='0x0'/>
      </source>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x07' function='0x0'/>
    </hostdev>
    <hostdev mode='subsystem' type='pci' managed='yes'>
      <source>
        <address domain='0x0000' bus='0x06' slot='0x00' function='0x1'/>
      </source>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x09' function='0x0'/>
    </hostdev>
    <hostdev mode='subsystem' type='pci' managed='yes'>
      <source>
        <address domain='0x0000' bus='0x00' slot='0x13' function='0x0'/>
      </source>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x0a' function='0x0'/>
    </hostdev>
    <hostdev mode='subsystem' type='pci' managed='yes'>
      <source>
        <address domain='0x0000' bus='0x00' slot='0x13' function='0x2'/>
      </source>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x0b' function='0x0'/>
    </hostdev>
    <memballoon model='virtio'>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x08' function='0x0'/>
    </memballoon>
  </devices>
</domain>

Offline

#1508 2014-04-09 11:45:21

beckend
Member
Registered: 2014-03-16
Posts: 5

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

3.14 In the OP when he posted it the first time, and I dont see that changed.
If u want to know without decompressing the posted PKGBUILD file, here it is:
        'change-default-console-loglevel.patch'
        '0001-Bluetooth-allocate-static-minor-for-vhci.patch'
        '0002-module-allow-multiple-calls-to-MODULE_DEVICE_TABLE-p.patch'
        '0003-module-remove-MODULE_GENERIC_TABLE.patch'
        '0004-Revert-syscalls.h-use-gcc-alias-instead-of-assembler.patch'
        'http://repo-ck.com/source/gcc_patch/enable_additional_cpu_optimizations_for_gcc.patch$
        'radeon_load_vbios_from_file.patch'
        'acs.patch'
        'i915_314.patch'
        'debug_registers.patch'

Kernel version is pkgver=3.14 pkgrel=2 all from PKGBUILD. I dont use the 'override_for_missing_acs_capabilities.patch' since it did not work for me, however I edited the the the PKGBUILD to include a newer patch to fix the IOMMU group isolation from:  http://linux-kernel.2935.n7.nabble.com/ … 96884.html
pasted the patch into 'acs.patch'. The same thing was done for 3.13.6 (posted earlier from OP).

Hardware is a Lenovo W530, Ivy Bridge 3740QM, 8GB Ram. What was passthoughed is an external GPU, 680GTX to the virtual machine as mostly instructed from OP except I replaced all to virtio drivers. Performance of 3.13 is quite astounding, I still can't belive it.

Offline

#1509 2014-04-09 16:25:53

Eb
Member
Registered: 2014-04-08
Posts: 11

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

I'm unable to get qemu to boot the guest os. I've followed the instructions and used the packages in the main post, and have looked around the thread for a solution but I'm still clueless as to what could be wrong. Below is everything I could think of that will be helpful to troubleshoot, sorry that it's a bit much. Let me know if any more information would be helpful.

Command Run:

sudo qemu-system-x86_64 \
-enable-kvm \
-M q35 \
-m 1024 \
-cpu host \
-smp 6,sockets=1,cores=6,threads=1 \
-bios /usr/share/qemu/bios.bin \
-vga none \
-device ioh3420,bus=pcie.0,addr=1c.0,multifunction=on,port=1,chassis=1,id=root.1 \
-device vfio-pci,host=03:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on \
-device vfio-pci,host=03:00.1,bus=root.1,addr=00.1

Error:

qemu-system-x86_64: -device vfio-pci,host=03:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on: vfio: error, group 1 is not viable, please ensure all devices within the iommu_group are bound to their vfio bus driver.
qemu-system-x86_64: -device vfio-pci,host=03:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on: vfio: failed to get group 1
qemu-system-x86_64: -device vfio-pci,host=03:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on: Device initialization failed.
qemu-system-x86_64: -device vfio-pci,host=03:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on: Device 'vfio-pci' could not be initialized

Setup:

Asrock z87 Extreme9
Intel Integrated Graphics (not in use, aside for post and bios)
Radeon R9 270x (guest) [03:00.0 and 03:00.1, pci-stubbed]
Radeon HD 6670 (host)  [04:00.0 and 04:00.1]



Drivers Tried:

radeon
fglrx (current)

Please note that both drivers have the same issue.



Boot Options:

intel_iommu=on pcie_acs_override=downstream pci-stub.ids=1002:6810,1002:aab0

IOMMU Group 1:

ls -l /sys/kernel/iommu_groups/1/devices
total 0
lrwxrwxrwx 1 root root 0 Apr  9 11:33 0000:00:01.0 -> ../../../../devices/pci0000:00/0000:00:01.0
lrwxrwxrwx 1 root root 0 Apr  9 11:33 0000:01:00.0 -> ../../../../devices/pci0000:00/0000:00:01.0/0000:01:00.0
lrwxrwxrwx 1 root root 0 Apr  9 11:33 0000:02:08.0 -> ../../../../devices/pci0000:00/0000:00:01.0/0000:01:00.0/0000:02:08.0
lrwxrwxrwx 1 root root 0 Apr  9 11:33 0000:02:10.0 -> ../../../../devices/pci0000:00/0000:00:01.0/0000:01:00.0/0000:02:10.0
lrwxrwxrwx 1 root root 0 Apr  9 11:33 0000:02:11.0 -> ../../../../devices/pci0000:00/0000:00:01.0/0000:01:00.0/0000:02:11.0
lrwxrwxrwx 1 root root 0 Apr  9 11:33 0000:03:00.0 -> ../../../../devices/pci0000:00/0000:00:01.0/0000:01:00.0/0000:02:08.0/0000:03:00.0
lrwxrwxrwx 1 root root 0 Apr  9 11:33 0000:03:00.1 -> ../../../../devices/pci0000:00/0000:00:01.0/0000:01:00.0/0000:02:08.0/0000:03:00.1
lrwxrwxrwx 1 root root 0 Apr  9 11:33 0000:04:00.0 -> ../../../../devices/pci0000:00/0000:00:01.0/0000:01:00.0/0000:02:10.0/0000:04:00.0
lrwxrwxrwx 1 root root 0 Apr  9 11:33 0000:04:00.1 -> ../../../../devices/pci0000:00/0000:00:01.0/0000:01:00.0/0000:02:10.0/0000:04:00.1

Details For each device:


00.01.0:

00:01.0 PCI bridge: Intel Corporation Xeon E3-1200 v3/4th Gen Core Processor PCI Express x16 Controller (rev 06) (prog-if 00 [Normal decode])
	Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx+
	Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
	Latency: 0, Cache Line Size: 64 bytes
	Bus: primary=00, secondary=01, subordinate=05, sec-latency=0
	I/O behind bridge: 0000a000-0000bfff
	Memory behind bridge: dc900000-dcbfffff
	Prefetchable memory behind bridge: 00000000a0000000-00000000bfffffff
	Secondary status: 66MHz- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- <SERR- <PERR-
	BridgeCtl: Parity- SERR- NoISA- VGA+ MAbort- >Reset- FastB2B-
		PriDiscTmr- SecDiscTmr- DiscTmrStat- DiscTmrSERREn-
	Capabilities: [88] Subsystem: ASRock Incorporation Device 0c01
	Capabilities: [80] Power Management version 3
		Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0+,D1-,D2-,D3hot+,D3cold+)
		Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=0 PME-
	Capabilities: [90] MSI: Enable+ Count=1/1 Maskable- 64bit-
		Address: fee00298  Data: 0000
	Capabilities: [a0] Express (v2) Root Port (Slot+), MSI 00
		DevCap:	MaxPayload 256 bytes, PhantFunc 0
			ExtTag- RBE+
		DevCtl:	Report errors: Correctable- Non-Fatal- Fatal- Unsupported-
			RlxdOrd- ExtTag- PhantFunc- AuxPwr- NoSnoop-
			MaxPayload 256 bytes, MaxReadReq 128 bytes
		DevSta:	CorrErr- UncorrErr- FatalErr+ UnsuppReq- AuxPwr- TransPend-
		LnkCap:	Port #2, Speed 8GT/s, Width x16, ASPM L0s L1, Exit Latency L0s <1us, L1 <8us
			ClockPM- Surprise- LLActRep- BwNot+
		LnkCtl:	ASPM Disabled; RCB 64 bytes Disabled- CommClk-
			ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
		LnkSta:	Speed 8GT/s, Width x16, TrErr- Train- SlotClk+ DLActive- BWMgmt+ ABWMgmt+
		SltCap:	AttnBtn- PwrCtrl- MRL- AttnInd- PwrInd- HotPlug- Surprise-
			Slot #1, PowerLimit 75.000W; Interlock- NoCompl+
		SltCtl:	Enable: AttnBtn- PwrFlt- MRL- PresDet- CmdCplt- HPIrq- LinkChg-
			Control: AttnInd Unknown, PwrInd Unknown, Power- Interlock-
		SltSta:	Status: AttnBtn- PowerFlt- MRL- CmdCplt- PresDet+ Interlock-
			Changed: MRL- PresDet- LinkState-
		RootCtl: ErrCorrectable- ErrNon-Fatal- ErrFatal- PMEIntEna- CRSVisible-
		RootCap: CRSVisible-
		RootSta: PME ReqID 0000, PMEStatus- PMEPending-
		DevCap2: Completion Timeout: Not Supported, TimeoutDis-, LTR+, OBFF Via WAKE# ARIFwd-
		DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis-, LTR+, OBFF Via WAKE# ARIFwd-
		LnkCtl2: Target Link Speed: 8GT/s, EnterCompliance- SpeedDis-
			 Transmit Margin: Normal Operating Range, EnterModifiedCompliance- ComplianceSOS-
			 Compliance De-emphasis: -6dB
		LnkSta2: Current De-emphasis Level: -6dB, EqualizationComplete+, EqualizationPhase1+
			 EqualizationPhase2+, EqualizationPhase3+, LinkEqualizationRequest-
	Capabilities: [100 v1] Virtual Channel
		Caps:	LPEVC=0 RefClk=100ns PATEntryBits=1
		Arb:	Fixed- WRR32- WRR64- WRR128-
		Ctrl:	ArbSelect=Fixed
		Status:	InProgress-
		VC0:	Caps:	PATOffset=00 MaxTimeSlots=1 RejSnoopTrans-
			Arb:	Fixed+ WRR32- WRR64- WRR128- TWRR128- WRR256-
			Ctrl:	Enable+ ID=0 ArbSelect=Fixed TC/VC=ff
			Status:	NegoPending- InProgress-
	Capabilities: [140 v1] Root Complex Link
		Desc:	PortNumber=02 ComponentID=01 EltType=Config
		Link0:	Desc:	TargetPort=00 TargetComponent=01 AssocRCRB- LinkType=MemMapped LinkValid+
			Addr:	00000000fed19000
	Capabilities: [d94 v1] #19
	Kernel driver in use: pcieport
	Kernel modules: shpchp

   
01:00.0:

01:00.0 PCI bridge: PLX Technology, Inc. Device 8747 (rev ba) (prog-if 00 [Normal decode])
	Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx+
	Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
	Latency: 0, Cache Line Size: 64 bytes
	Region 0: Memory at dcb00000 (32-bit, non-prefetchable) [size=256K]
	Bus: primary=01, secondary=02, subordinate=05, sec-latency=0
	I/O behind bridge: 0000a000-0000bfff
	Memory behind bridge: dc900000-dcafffff
	Prefetchable memory behind bridge: 00000000a0000000-00000000bfffffff
	Secondary status: 66MHz- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- <SERR- <PERR-
	BridgeCtl: Parity- SERR- NoISA- VGA+ MAbort- >Reset- FastB2B-
		PriDiscTmr- SecDiscTmr- DiscTmrStat- DiscTmrSERREn-
	Capabilities: [40] Power Management version 3
		Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0+,D1-,D2-,D3hot+,D3cold+)
		Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=0 PME-
	Capabilities: [48] MSI: Enable+ Count=1/8 Maskable+ 64bit+
		Address: 00000000fee002f8  Data: 0000
		Masking: 000000ff  Pending: 00000000
	Capabilities: [68] Express (v2) Upstream Port, MSI 00
		DevCap:	MaxPayload 2048 bytes, PhantFunc 0
			ExtTag- AttnBtn- AttnInd- PwrInd- RBE+ SlotPowerLimit 75.000W
		DevCtl:	Report errors: Correctable- Non-Fatal- Fatal- Unsupported-
			RlxdOrd- ExtTag- PhantFunc- AuxPwr- NoSnoop+
			MaxPayload 256 bytes, MaxReadReq 128 bytes
		DevSta:	CorrErr+ UncorrErr- FatalErr- UnsuppReq+ AuxPwr- TransPend-
		LnkCap:	Port #0, Speed 8GT/s, Width x16, ASPM L1, Exit Latency L0s <4us, L1 <4us
			ClockPM- Surprise- LLActRep- BwNot-
		LnkCtl:	ASPM Disabled; Disabled- CommClk-
			ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
		LnkSta:	Speed 8GT/s, Width x16, TrErr- Train- SlotClk- DLActive- BWMgmt- ABWMgmt-
		DevCap2: Completion Timeout: Not Supported, TimeoutDis-, LTR+, OBFF Via message
		DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis-, LTR+, OBFF Disabled
		LnkCtl2: Target Link Speed: 8GT/s, EnterCompliance- SpeedDis-
			 Transmit Margin: Normal Operating Range, EnterModifiedCompliance- ComplianceSOS-
			 Compliance De-emphasis: -6dB
		LnkSta2: Current De-emphasis Level: -6dB, EqualizationComplete+, EqualizationPhase1+
			 EqualizationPhase2+, EqualizationPhase3+, LinkEqualizationRequest-
	Capabilities: [a4] Subsystem: PLX Technology, Inc. Device 8747
	Capabilities: [100 v1] Device Serial Number ba-87-00-10-b5-df-0e-00
	Capabilities: [fb4 v1] Advanced Error Reporting
		UESta:	DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
		UEMsk:	DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
		UESvrt:	DLP+ SDES+ TLP- FCP+ CmpltTO- CmpltAbrt- UnxCmplt- RxOF+ MalfTLP+ ECRC- UnsupReq- ACSViol-
		CESta:	RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr+
		CEMsk:	RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr+
		AERCap:	First Error Pointer: 1f, GenCap+ CGenEn- ChkCap+ ChkEn-
	Capabilities: [138 v1] Power Budgeting <?>
	Capabilities: [10c v1] #19
	Capabilities: [148 v1] Virtual Channel
		Caps:	LPEVC=0 RefClk=100ns PATEntryBits=8
		Arb:	Fixed- WRR32- WRR64- WRR128-
		Ctrl:	ArbSelect=Fixed
		Status:	InProgress-
		VC0:	Caps:	PATOffset=03 MaxTimeSlots=1 RejSnoopTrans-
			Arb:	Fixed- WRR32- WRR64+ WRR128- TWRR128- WRR256-

			Ctrl:	Enable+ ID=0 ArbSelect=WRR64 TC/VC=ff
			Status:	NegoPending- InProgress-
			Port Arbitration Table <?>
	Capabilities: [e00 v1] #12
	Capabilities: [b00 v1] Latency Tolerance Reporting
		Max snoop latency: 71680ns
		Max no snoop latency: 71680ns
	Capabilities: [b70 v1] Vendor Specific Information: ID=0001 Rev=0 Len=010 <?>
	Kernel driver in use: pcieport
	Kernel modules: shpchp

02:08.0

02:08.0 PCI bridge: PLX Technology, Inc. Device 8747 (rev ba) (prog-if 00 [Normal decode])
	Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx+
	Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
	Latency: 0, Cache Line Size: 64 bytes
	Bus: primary=02, secondary=03, subordinate=03, sec-latency=0
	I/O behind bridge: 0000b000-0000bfff
	Memory behind bridge: dca00000-dcafffff
	Prefetchable memory behind bridge: 00000000b0000000-00000000bfffffff
	Secondary status: 66MHz- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- <SERR- <PERR-
	BridgeCtl: Parity- SERR- NoISA- VGA- MAbort- >Reset- FastB2B-
		PriDiscTmr- SecDiscTmr- DiscTmrStat- DiscTmrSERREn-
	Capabilities: [40] Power Management version 3
		Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0+,D1-,D2-,D3hot+,D3cold+)
		Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=0 PME-
	Capabilities: [48] MSI: Enable+ Count=1/8 Maskable+ 64bit+
		Address: 00000000fee00318  Data: 0000
		Masking: 000000ff  Pending: 00000000
	Capabilities: [68] Express (v2) Downstream Port (Slot+), MSI 00
		DevCap:	MaxPayload 2048 bytes, PhantFunc 0
			ExtTag- RBE+
		DevCtl:	Report errors: Correctable- Non-Fatal- Fatal- Unsupported-
			RlxdOrd- ExtTag- PhantFunc- AuxPwr- NoSnoop+
			MaxPayload 256 bytes, MaxReadReq 128 bytes
		DevSta:	CorrErr+ UncorrErr- FatalErr- UnsuppReq+ AuxPwr- TransPend-
		LnkCap:	Port #8, Speed 8GT/s, Width x16, ASPM L1, Exit Latency L0s <4us, L1 <4us
			ClockPM- Surprise+ LLActRep+ BwNot+
		LnkCtl:	ASPM Disabled; Disabled- CommClk-
			ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
		LnkSta:	Speed 8GT/s, Width x16, TrErr- Train- SlotClk- DLActive+ BWMgmt+ ABWMgmt+
		SltCap:	AttnBtn- PwrCtrl- MRL- AttnInd- PwrInd- HotPlug- Surprise-
			Slot #72, PowerLimit 25.000W; Interlock- NoCompl-
		SltCtl:	Enable: AttnBtn- PwrFlt- MRL- PresDet- CmdCplt- HPIrq- LinkChg-
			Control: AttnInd Unknown, PwrInd Unknown, Power- Interlock-
		SltSta:	Status: AttnBtn- PowerFlt- MRL- CmdCplt- PresDet+ Interlock-
			Changed: MRL- PresDet- LinkState-
		DevCap2: Completion Timeout: Not Supported, TimeoutDis-, LTR+, OBFF Via message ARIFwd+
		DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis-, LTR+, OBFF Disabled ARIFwd-
		LnkCtl2: Target Link Speed: 8GT/s, EnterCompliance- SpeedDis-, Selectable De-emphasis: -6dB
			 Transmit Margin: Normal Operating Range, EnterModifiedCompliance- ComplianceSOS-
			 Compliance De-emphasis: -6dB
		LnkSta2: Current De-emphasis Level: -6dB, EqualizationComplete+, EqualizationPhase1+
			 EqualizationPhase2+, EqualizationPhase3+, LinkEqualizationRequest-
	Capabilities: [a4] Subsystem: PLX Technology, Inc. Device 8747
	Capabilities: [100 v1] Device Serial Number ba-87-00-10-b5-df-0e-00
	Capabilities: [fb4 v1] Advanced Error Reporting
		UESta:	DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
		UEMsk:	DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
		UESvrt:	DLP+ SDES+ TLP- FCP+ CmpltTO- CmpltAbrt- UnxCmplt- RxOF+ MalfTLP+ ECRC- UnsupReq- ACSViol-
		CESta:	RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr+
		CEMsk:	RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr+
		AERCap:	First Error Pointer: 1f, GenCap+ CGenEn- ChkCap+ ChkEn-
	Capabilities: [138 v1] Power Budgeting <?>
	Capabilities: [10c v1] #19
	Capabilities: [148 v1] Virtual Channel
		Caps:	LPEVC=0 RefClk=100ns PATEntryBits=8
		Arb:	Fixed- WRR32- WRR64- WRR128-
		Ctrl:	ArbSelect=Fixed
		Status:	InProgress-
		VC0:	Caps:	PATOffset=03 MaxTimeSlots=1 RejSnoopTrans-
			Arb:	Fixed- WRR32- WRR64+ WRR128- TWRR128- WRR256-
			Ctrl:	Enable+ ID=0 ArbSelect=WRR64 TC/VC=ff
			Status:	NegoPending- InProgress-
			Port Arbitration Table <?>
	Capabilities: [e00 v1] #12
	Capabilities: [f24 v1] Access Control Services
		ACSCap:	SrcValid+ TransBlk+ ReqRedir+ CmpltRedir+ UpstreamFwd+ EgressCtrl+ DirectTrans+
		ACSCtl:	SrcValid+ TransBlk- ReqRedir+ CmpltRedir+ UpstreamFwd+ EgressCtrl- DirectTrans-
	Capabilities: [b70 v1] Vendor Specific Information: ID=0001 Rev=0 Len=010 <?>
	Kernel driver in use: pcieport
	Kernel modules: shpchp

02:10.0

02:10.0 PCI bridge: PLX Technology, Inc. Device 8747 (rev ba) (prog-if 00 [Normal decode])
	Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx+
	Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
	Latency: 0, Cache Line Size: 64 bytes
	Bus: primary=02, secondary=04, subordinate=04, sec-latency=0
	I/O behind bridge: 0000a000-0000afff
	Memory behind bridge: dc900000-dc9fffff
	Prefetchable memory behind bridge: 00000000a0000000-00000000afffffff
	Secondary status: 66MHz- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- <SERR- <PERR-
	BridgeCtl: Parity- SERR- NoISA- VGA+ MAbort- >Reset- FastB2B-
		PriDiscTmr- SecDiscTmr- DiscTmrStat- DiscTmrSERREn-
	Capabilities: [40] Power Management version 3
		Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0+,D1-,D2-,D3hot+,D3cold+)
		Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=0 PME-
	Capabilities: [48] MSI: Enable+ Count=1/8 Maskable+ 64bit+
		Address: 00000000fee00338  Data: 0000
		Masking: 000000ff  Pending: 00000000
	Capabilities: [68] Express (v2) Downstream Port (Slot+), MSI 00
		DevCap:	MaxPayload 2048 bytes, PhantFunc 0
			ExtTag- RBE+
		DevCtl:	Report errors: Correctable- Non-Fatal- Fatal- Unsupported-
			RlxdOrd- ExtTag- PhantFunc- AuxPwr- NoSnoop+
			MaxPayload 256 bytes, MaxReadReq 128 bytes
		DevSta:	CorrErr+ UncorrErr- FatalErr- UnsuppReq+ AuxPwr- TransPend-
		LnkCap:	Port #16, Speed 8GT/s, Width x8, ASPM L1, Exit Latency L0s <2us, L1 <4us
			ClockPM- Surprise+ LLActRep+ BwNot+
		LnkCtl:	ASPM Disabled; Disabled- CommClk-
			ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
		LnkSta:	Speed 5GT/s, Width x8, TrErr- Train- SlotClk- DLActive+ BWMgmt+ ABWMgmt+
		SltCap:	AttnBtn- PwrCtrl- MRL- AttnInd- PwrInd- HotPlug- Surprise-
			Slot #80, PowerLimit 25.000W; Interlock- NoCompl-
		SltCtl:	Enable: AttnBtn- PwrFlt- MRL- PresDet- CmdCplt- HPIrq- LinkChg-
			Control: AttnInd Unknown, PwrInd Unknown, Power- Interlock-
		SltSta:	Status: AttnBtn- PowerFlt- MRL- CmdCplt- PresDet+ Interlock-
			Changed: MRL- PresDet- LinkState-
		DevCap2: Completion Timeout: Not Supported, TimeoutDis-, LTR+, OBFF Via message ARIFwd+
		DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis-, LTR+, OBFF Disabled ARIFwd-
		LnkCtl2: Target Link Speed: 8GT/s, EnterCompliance- SpeedDis-, Selectable De-emphasis: -6dB
			 Transmit Margin: Normal Operating Range, EnterModifiedCompliance- ComplianceSOS-
			 Compliance De-emphasis: -6dB
		LnkSta2: Current De-emphasis Level: -3.5dB, EqualizationComplete-, EqualizationPhase1-
			 EqualizationPhase2-, EqualizationPhase3-, LinkEqualizationRequest-
	Capabilities: [a4] Subsystem: PLX Technology, Inc. Device 8747
	Capabilities: [100 v1] Device Serial Number ba-87-00-10-b5-df-0e-00
	Capabilities: [fb4 v1] Advanced Error Reporting
		UESta:	DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
		UEMsk:	DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
		UESvrt:	DLP+ SDES+ TLP- FCP+ CmpltTO- CmpltAbrt- UnxCmplt- RxOF+ MalfTLP+ ECRC- UnsupReq- ACSViol-
		CESta:	RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr+
		CEMsk:	RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr+
		AERCap:	First Error Pointer: 1f, GenCap+ CGenEn- ChkCap+ ChkEn-
	Capabilities: [138 v1] Power Budgeting <?>
	Capabilities: [10c v1] #19
	Capabilities: [148 v1] Virtual Channel
		Caps:	LPEVC=0 RefClk=100ns PATEntryBits=8
		Arb:	Fixed- WRR32- WRR64- WRR128-
		Ctrl:	ArbSelect=Fixed
		Status:	InProgress-
		VC0:	Caps:	PATOffset=03 MaxTimeSlots=1 RejSnoopTrans-
			Arb:	Fixed- WRR32- WRR64+ WRR128- TWRR128- WRR256-
			Ctrl:	Enable+ ID=0 ArbSelect=WRR64 TC/VC=ff
			Status:	NegoPending- InProgress-
			Port Arbitration Table <?>
	Capabilities: [e00 v1] #12
	Capabilities: [f24 v1] Access Control Services
		ACSCap:	SrcValid+ TransBlk+ ReqRedir+ CmpltRedir+ UpstreamFwd+ EgressCtrl+ DirectTrans+
		ACSCtl:	SrcValid+ TransBlk- ReqRedir+ CmpltRedir+ UpstreamFwd+ EgressCtrl- DirectTrans-
	Capabilities: [b70 v1] Vendor Specific Information: ID=0001 Rev=0 Len=010 <?>
	Kernel driver in use: pcieport
	Kernel modules: shpchp

02:11.0

02:11.0 PCI bridge: PLX Technology, Inc. Device 8747 (rev ba) (prog-if 00 [Normal decode])
	Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx+
	Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
	Latency: 0, Cache Line Size: 64 bytes
	Bus: primary=02, secondary=05, subordinate=05, sec-latency=0
	I/O behind bridge: 0000f000-00000fff
	Memory behind bridge: fff00000-000fffff
	Prefetchable memory behind bridge: 00000000fff00000-00000000000fffff
	Secondary status: 66MHz- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- <SERR- <PERR-
	BridgeCtl: Parity- SERR- NoISA- VGA- MAbort- >Reset- FastB2B-
		PriDiscTmr- SecDiscTmr- DiscTmrStat- DiscTmrSERREn-
	Capabilities: [40] Power Management version 3
		Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0+,D1-,D2-,D3hot+,D3cold+)
		Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=0 PME-
	Capabilities: [48] MSI: Enable+ Count=1/8 Maskable+ 64bit+
		Address: 00000000fee00358  Data: 0000
		Masking: 000000ff  Pending: 00000000
	Capabilities: [68] Express (v2) Downstream Port (Slot+), MSI 00
		DevCap:	MaxPayload 2048 bytes, PhantFunc 0
			ExtTag- RBE+
		DevCtl:	Report errors: Correctable- Non-Fatal- Fatal- Unsupported-
			RlxdOrd+ ExtTag- PhantFunc- AuxPwr- NoSnoop+
			MaxPayload 256 bytes, MaxReadReq 128 bytes
		DevSta:	CorrErr+ UncorrErr- FatalErr- UnsuppReq+ AuxPwr- TransPend-
		LnkCap:	Port #17, Speed 8GT/s, Width x8, ASPM L1, Exit Latency L0s <4us, L1 <4us
			ClockPM- Surprise+ LLActRep+ BwNot+
		LnkCtl:	ASPM Disabled; Disabled- CommClk-
			ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
		LnkSta:	Speed 2.5GT/s, Width x0, TrErr- Train- SlotClk- DLActive- BWMgmt- ABWMgmt-
		SltCap:	AttnBtn- PwrCtrl- MRL- AttnInd- PwrInd- HotPlug- Surprise-
			Slot #81, PowerLimit 25.000W; Interlock- NoCompl-
		SltCtl:	Enable: AttnBtn- PwrFlt- MRL- PresDet- CmdCplt- HPIrq- LinkChg-
			Control: AttnInd Unknown, PwrInd Unknown, Power- Interlock-
		SltSta:	Status: AttnBtn- PowerFlt- MRL- CmdCplt- PresDet- Interlock-
			Changed: MRL- PresDet- LinkState-
		DevCap2: Completion Timeout: Not Supported, TimeoutDis-, LTR+, OBFF Via message ARIFwd+
		DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis-, LTR-, OBFF Disabled ARIFwd-
		LnkCtl2: Target Link Speed: 8GT/s, EnterCompliance- SpeedDis-, Selectable De-emphasis: -6dB
			 Transmit Margin: Normal Operating Range, EnterModifiedCompliance- ComplianceSOS-
			 Compliance De-emphasis: -6dB
		LnkSta2: Current De-emphasis Level: -3.5dB, EqualizationComplete-, EqualizationPhase1-
			 EqualizationPhase2-, EqualizationPhase3-, LinkEqualizationRequest-
	Capabilities: [a4] Subsystem: PLX Technology, Inc. Device 8747
	Capabilities: [100 v1] Device Serial Number ba-87-00-10-b5-df-0e-00
	Capabilities: [fb4 v1] Advanced Error Reporting
		UESta:	DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
		UEMsk:	DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
		UESvrt:	DLP+ SDES+ TLP- FCP+ CmpltTO- CmpltAbrt- UnxCmplt- RxOF+ MalfTLP+ ECRC- UnsupReq- ACSViol-
		CESta:	RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr+
		CEMsk:	RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr+
		AERCap:	First Error Pointer: 1f, GenCap+ CGenEn- ChkCap+ ChkEn-
	Capabilities: [138 v1] Power Budgeting <?>
	Capabilities: [10c v1] #19
	Capabilities: [148 v1] Virtual Channel
		Caps:	LPEVC=0 RefClk=100ns PATEntryBits=8
		Arb:	Fixed- WRR32- WRR64- WRR128-
		Ctrl:	ArbSelect=Fixed
		Status:	InProgress-
		VC0:	Caps:	PATOffset=03 MaxTimeSlots=1 RejSnoopTrans-
			Arb:	Fixed- WRR32- WRR64+ WRR128- TWRR128- WRR256-
			Ctrl:	Enable+ ID=0 ArbSelect=WRR64 TC/VC=ff
			Status:	NegoPending+ InProgress-
			Port Arbitration Table <?>
	Capabilities: [e00 v1] #12
	Capabilities: [f24 v1] Access Control Services
		ACSCap:	SrcValid+ TransBlk+ ReqRedir+ CmpltRedir+ UpstreamFwd+ EgressCtrl+ DirectTrans+
		ACSCtl:	SrcValid+ TransBlk- ReqRedir+ CmpltRedir+ UpstreamFwd+ EgressCtrl- DirectTrans-
	Capabilities: [b70 v1] Vendor Specific Information: ID=0001 Rev=0 Len=010 <?>
	Kernel driver in use: pcieport
	Kernel modules: shpchp

03:00.0

02:11.0 PCI bridge: PLX Technology, Inc. Device 8747 (rev ba) (prog-if 00 [Normal decode])
	Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx+
	Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
	Latency: 0, Cache Line Size: 64 bytes
	Bus: primary=02, secondary=05, subordinate=05, sec-latency=0
	I/O behind bridge: 0000f000-00000fff
	Memory behind bridge: fff00000-000fffff
	Prefetchable memory behind bridge: 00000000fff00000-00000000000fffff
	Secondary status: 66MHz- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- <SERR- <PERR-
	BridgeCtl: Parity- SERR- NoISA- VGA- MAbort- >Reset- FastB2B-
		PriDiscTmr- SecDiscTmr- DiscTmrStat- DiscTmrSERREn-
	Capabilities: [40] Power Management version 3
		Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0+,D1-,D2-,D3hot+,D3cold+)
		Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=0 PME-
	Capabilities: [48] MSI: Enable+ Count=1/8 Maskable+ 64bit+
		Address: 00000000fee00358  Data: 0000
		Masking: 000000ff  Pending: 00000000
	Capabilities: [68] Express (v2) Downstream Port (Slot+), MSI 00
		DevCap:	MaxPayload 2048 bytes, PhantFunc 0
			ExtTag- RBE+
		DevCtl:	Report errors: Correctable- Non-Fatal- Fatal- Unsupported-
			RlxdOrd+ ExtTag- PhantFunc- AuxPwr- NoSnoop+
			MaxPayload 256 bytes, MaxReadReq 128 bytes
		DevSta:	CorrErr+ UncorrErr- FatalErr- UnsuppReq+ AuxPwr- TransPend-
		LnkCap:	Port #17, Speed 8GT/s, Width x8, ASPM L1, Exit Latency L0s <4us, L1 <4us
			ClockPM- Surprise+ LLActRep+ BwNot+
		LnkCtl:	ASPM Disabled; Disabled- CommClk-
			ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
		LnkSta:	Speed 2.5GT/s, Width x0, TrErr- Train- SlotClk- DLActive- BWMgmt- ABWMgmt-
		SltCap:	AttnBtn- PwrCtrl- MRL- AttnInd- PwrInd- HotPlug- Surprise-
			Slot #81, PowerLimit 25.000W; Interlock- NoCompl-
		SltCtl:	Enable: AttnBtn- PwrFlt- MRL- PresDet- CmdCplt- HPIrq- LinkChg-
			Control: AttnInd Unknown, PwrInd Unknown, Power- Interlock-
		SltSta:	Status: AttnBtn- PowerFlt- MRL- CmdCplt- PresDet- Interlock-
			Changed: MRL- PresDet- LinkState-
		DevCap2: Completion Timeout: Not Supported, TimeoutDis-, LTR+, OBFF Via message ARIFwd+
		DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis-, LTR-, OBFF Disabled ARIFwd-
		LnkCtl2: Target Link Speed: 8GT/s, EnterCompliance- SpeedDis-, Selectable De-emphasis: -6dB
			 Transmit Margin: Normal Operating Range, EnterModifiedCompliance- ComplianceSOS-
			 Compliance De-emphasis: -6dB
		LnkSta2: Current De-emphasis Level: -3.5dB, EqualizationComplete-, EqualizationPhase1-
			 EqualizationPhase2-, EqualizationPhase3-, LinkEqualizationRequest-
	Capabilities: [a4] Subsystem: PLX Technology, Inc. Device 8747
	Capabilities: [100 v1] Device Serial Number ba-87-00-10-b5-df-0e-00
	Capabilities: [fb4 v1] Advanced Error Reporting
		UESta:	DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
		UEMsk:	DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
		UESvrt:	DLP+ SDES+ TLP- FCP+ CmpltTO- CmpltAbrt- UnxCmplt- RxOF+ MalfTLP+ ECRC- UnsupReq- ACSViol-
		CESta:	RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr+
		CEMsk:	RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr+
		AERCap:	First Error Pointer: 1f, GenCap+ CGenEn- ChkCap+ ChkEn-
	Capabilities: [138 v1] Power Budgeting <?>
	Capabilities: [10c v1] #19
	Capabilities: [148 v1] Virtual Channel
		Caps:	LPEVC=0 RefClk=100ns PATEntryBits=8
		Arb:	Fixed- WRR32- WRR64- WRR128-
		Ctrl:	ArbSelect=Fixed
		Status:	InProgress-
		VC0:	Caps:	PATOffset=03 MaxTimeSlots=1 RejSnoopTrans-
			Arb:	Fixed- WRR32- WRR64+ WRR128- TWRR128- WRR256-
			Ctrl:	Enable+ ID=0 ArbSelect=WRR64 TC/VC=ff
			Status:	NegoPending+ InProgress-
			Port Arbitration Table <?>
	Capabilities: [e00 v1] #12
	Capabilities: [f24 v1] Access Control Services
		ACSCap:	SrcValid+ TransBlk+ ReqRedir+ CmpltRedir+ UpstreamFwd+ EgressCtrl+ DirectTrans+
		ACSCtl:	SrcValid+ TransBlk- ReqRedir+ CmpltRedir+ UpstreamFwd+ EgressCtrl- DirectTrans-
	Capabilities: [b70 v1] Vendor Specific Information: ID=0001 Rev=0 Len=010 <?>
	Kernel driver in use: pcieport
	Kernel modules: shpchp

03.00.1

03:00.1 Audio device: Advanced Micro Devices, Inc. [AMD/ATI] Cape Verde/Pitcairn HDMI Audio [Radeon HD 7700/7800 Series]
	Subsystem: Gigabyte Technology Co., Ltd Device aab0
	Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
	Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
	Latency: 0, Cache Line Size: 64 bytes
	Interrupt: pin B routed to IRQ 10
	Region 0: Memory at dca60000 (64-bit, non-prefetchable) [size=16K]
	Capabilities: [48] Vendor Specific Information: Len=08 <?>
	Capabilities: [50] Power Management version 3
		Flags: PMEClk- DSI- D1+ D2+ AuxCurrent=0mA PME(D0-,D1-,D2-,D3hot-,D3cold-)
		Status: D0 NoSoftRst- PME-Enable- DSel=0 DScale=0 PME-
	Capabilities: [58] Express (v2) Legacy Endpoint, MSI 00
		DevCap:	MaxPayload 256 bytes, PhantFunc 0, Latency L0s <4us, L1 unlimited
			ExtTag+ AttnBtn- AttnInd- PwrInd- RBE+ FLReset-
		DevCtl:	Report errors: Correctable- Non-Fatal- Fatal- Unsupported-
			RlxdOrd- ExtTag+ PhantFunc- AuxPwr- NoSnoop+
			MaxPayload 256 bytes, MaxReadReq 512 bytes
		DevSta:	CorrErr- UncorrErr- FatalErr- UnsuppReq- AuxPwr- TransPend-
		LnkCap:	Port #8, Speed 8GT/s, Width x16, ASPM L0s L1, Exit Latency L0s <64ns, L1 <1us
			ClockPM- Surprise- LLActRep- BwNot-
		LnkCtl:	ASPM Disabled; RCB 64 bytes Disabled- CommClk-
			ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
		LnkSta:	Speed 8GT/s, Width x16, TrErr- Train- SlotClk+ DLActive- BWMgmt- ABWMgmt-
		DevCap2: Completion Timeout: Not Supported, TimeoutDis-, LTR-, OBFF Not Supported
		DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis-, LTR-, OBFF Disabled
		LnkSta2: Current De-emphasis Level: -6dB, EqualizationComplete-, EqualizationPhase1-
			 EqualizationPhase2-, EqualizationPhase3-, LinkEqualizationRequest-
	Capabilities: [a0] MSI: Enable- Count=1/1 Maskable- 64bit+
		Address: 0000000000000000  Data: 0000
	Capabilities: [100 v1] Vendor Specific Information: ID=0001 Rev=1 Len=010 <?>
	Capabilities: [150 v2] Advanced Error Reporting
		UESta:	DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
		UEMsk:	DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
		UESvrt:	DLP+ SDES+ TLP- FCP+ CmpltTO- CmpltAbrt- UnxCmplt- RxOF+ MalfTLP+ ECRC- UnsupReq- ACSViol-
		CESta:	RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr+
		CEMsk:	RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr+
		AERCap:	First Error Pointer: 00, GenCap+ CGenEn- ChkCap+ ChkEn-
	Kernel driver in use: vfio-pci
	Kernel modules: snd_hda_intel

04.00.0

04:00.0 VGA compatible controller: Advanced Micro Devices, Inc. [AMD/ATI] Turks XT [Radeon HD 6670/7670] (prog-if 00 [VGA controller])
	Subsystem: ASUSTeK Computer Inc. Device 0443
	Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx+
	Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
	Latency: 0, Cache Line Size: 64 bytes
	Interrupt: pin A routed to IRQ 66
	Region 0: Memory at a0000000 (64-bit, prefetchable) [size=256M]
	Region 2: Memory at dc920000 (64-bit, non-prefetchable) [size=128K]
	Region 4: I/O ports at a000 [size=256]
	Expansion ROM at dc900000 [disabled] [size=128K]
	Capabilities: [50] Power Management version 3
		Flags: PMEClk- DSI- D1+ D2+ AuxCurrent=0mA PME(D0-,D1-,D2-,D3hot-,D3cold-)
		Status: D0 NoSoftRst- PME-Enable- DSel=0 DScale=0 PME-
	Capabilities: [58] Express (v2) Legacy Endpoint, MSI 00
		DevCap:	MaxPayload 256 bytes, PhantFunc 0, Latency L0s <4us, L1 unlimited
			ExtTag+ AttnBtn- AttnInd- PwrInd- RBE+ FLReset-
		DevCtl:	Report errors: Correctable- Non-Fatal- Fatal- Unsupported-
			RlxdOrd- ExtTag+ PhantFunc- AuxPwr- NoSnoop+
			MaxPayload 256 bytes, MaxReadReq 512 bytes
		DevSta:	CorrErr- UncorrErr- FatalErr- UnsuppReq- AuxPwr- TransPend-
		LnkCap:	Port #16, Speed 5GT/s, Width x16, ASPM L0s L1, Exit Latency L0s <64ns, L1 <1us
			ClockPM- Surprise- LLActRep- BwNot-
		LnkCtl:	ASPM Disabled; RCB 64 bytes Disabled- CommClk-
			ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
		LnkSta:	Speed 5GT/s, Width x8, TrErr- Train- SlotClk+ DLActive- BWMgmt- ABWMgmt-
		DevCap2: Completion Timeout: Not Supported, TimeoutDis-, LTR-, OBFF Not Supported
		DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis-, LTR-, OBFF Disabled
		LnkCtl2: Target Link Speed: 5GT/s, EnterCompliance- SpeedDis-
			 Transmit Margin: Normal Operating Range, EnterModifiedCompliance- ComplianceSOS-
			 Compliance De-emphasis: -6dB
		LnkSta2: Current De-emphasis Level: -6dB, EqualizationComplete-, EqualizationPhase1-
			 EqualizationPhase2-, EqualizationPhase3-, LinkEqualizationRequest-
	Capabilities: [a0] MSI: Enable+ Count=1/1 Maskable- 64bit+
		Address: 00000000fee00678  Data: 0000
	Capabilities: [100 v1] Vendor Specific Information: ID=0001 Rev=1 Len=010 <?>
	Capabilities: [150 v1] Advanced Error Reporting
		UESta:	DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
		UEMsk:	DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
		UESvrt:	DLP+ SDES+ TLP- FCP+ CmpltTO- CmpltAbrt- UnxCmplt- RxOF+ MalfTLP+ ECRC- UnsupReq- ACSViol-
		CESta:	RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr+
		CEMsk:	RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr+
		AERCap:	First Error Pointer: 00, GenCap+ CGenEn- ChkCap+ ChkEn-
	Kernel driver in use: fglrx_pci
	Kernel modules: radeon, fglrx

04.00.1

04:00.1 Audio device: Advanced Micro Devices, Inc. [AMD/ATI] Turks/Whistler HDMI Audio [Radeon HD 6000 Series]
	Subsystem: ASUSTeK Computer Inc. Device aa90
	Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx+
	Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
	Latency: 0, Cache Line Size: 64 bytes
	Interrupt: pin B routed to IRQ 65
	Region 0: Memory at dc940000 (64-bit, non-prefetchable) [size=16K]
	Capabilities: [50] Power Management version 3
		Flags: PMEClk- DSI- D1+ D2+ AuxCurrent=0mA PME(D0-,D1-,D2-,D3hot-,D3cold-)
		Status: D0 NoSoftRst- PME-Enable- DSel=0 DScale=0 PME-
	Capabilities: [58] Express (v2) Legacy Endpoint, MSI 00
		DevCap:	MaxPayload 256 bytes, PhantFunc 0, Latency L0s <4us, L1 unlimited
			ExtTag+ AttnBtn- AttnInd- PwrInd- RBE+ FLReset-
		DevCtl:	Report errors: Correctable- Non-Fatal- Fatal- Unsupported-
			RlxdOrd- ExtTag+ PhantFunc- AuxPwr- NoSnoop+
			MaxPayload 256 bytes, MaxReadReq 512 bytes
		DevSta:	CorrErr- UncorrErr- FatalErr- UnsuppReq- AuxPwr- TransPend-
		LnkCap:	Port #16, Speed 5GT/s, Width x16, ASPM L0s L1, Exit Latency L0s <64ns, L1 <1us
			ClockPM- Surprise- LLActRep- BwNot-
		LnkCtl:	ASPM Disabled; RCB 64 bytes Disabled- CommClk-
			ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
		LnkSta:	Speed 5GT/s, Width x8, TrErr- Train- SlotClk+ DLActive- BWMgmt- ABWMgmt-
		DevCap2: Completion Timeout: Not Supported, TimeoutDis-, LTR-, OBFF Not Supported
		DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis-, LTR-, OBFF Disabled
		LnkSta2: Current De-emphasis Level: -6dB, EqualizationComplete-, EqualizationPhase1-
			 EqualizationPhase2-, EqualizationPhase3-, LinkEqualizationRequest-
	Capabilities: [a0] MSI: Enable+ Count=1/1 Maskable- 64bit+
		Address: 00000000fee00658  Data: 0000
	Capabilities: [100 v1] Vendor Specific Information: ID=0001 Rev=1 Len=010 <?>
	Capabilities: [150 v1] Advanced Error Reporting
		UESta:	DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
		UEMsk:	DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
		UESvrt:	DLP+ SDES+ TLP- FCP+ CmpltTO- CmpltAbrt- UnxCmplt- RxOF+ MalfTLP+ ECRC- UnsupReq- ACSViol-
		CESta:	RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr+
		CEMsk:	RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr+
		AERCap:	First Error Pointer: 00, GenCap+ CGenEn- ChkCap+ ChkEn-
	Kernel driver in use: snd_hda_intel
	Kernel modules: snd_hda_intel

I'm on week two of trying to get this set up, so any help at all is greatly appreciated!

Offline

#1510 2014-04-09 17:45:39

novist
Member
Registered: 2014-03-14
Posts: 47

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

@Eb: read the error. it says exactly what you have not done - you have not bound devices to vfio driver using vfio-bind script that you were supposed to make.

Offline

#1511 2014-04-09 17:49:58

Eb
Member
Registered: 2014-04-08
Posts: 11

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

novist wrote:

@Eb: read the error. it says exactly what you have not done - you have not bound devices to vfio driver using vfio-bind script that you were supposed to make.

I knew I forgot to mention something. I have the script created, and before attempting to run the KVM I run the script binding 03:00.0 and its subdevice 03:00.1

Offline

#1512 2014-04-09 18:02:01

aw
Member
Registered: 2013-10-04
Posts: 921
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Eb wrote:
novist wrote:

@Eb: read the error. it says exactly what you have not done - you have not bound devices to vfio driver using vfio-bind script that you were supposed to make.

I knew I forgot to mention something. I have the script created, and before attempting to run the KVM I run the script binding 03:00.0 and its subdevice 03:00.1

The error message says to make sure all of the devices in the group are bound to vfio.  The group, as you reported contains far more than just the secondary function of the device.  The IOMMU group on your system is large because the root port lacks PCIe ACS support.  If you wish to override that, look for the ACS override patch and options to use it (and caveats that enabling it implies).


http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Offline

#1513 2014-04-09 18:02:07

trimm
Member
Registered: 2014-03-06
Posts: 31

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Just bought a PCIe USB 3.0 card for my machine. I was going to use it solely for passthroughing but Im having difficulty getting it to work.

Error Im receiving:

(qemu) qemu-system-x86_64: -device vfio-pci,host=07:00.0,bus=pcie.0: vfio: error, group 9 is not viable, please ensure all devices within the iommu_group are bound to their vfio bus driver.
qemu-system-x86_64: -device vfio-pci,host=07:00.0,bus=pcie.0: vfio: failed to get group 9
qemu-system-x86_64: -device vfio-pci,host=07:00.0,bus=pcie.0: Device initialization failed.
qemu-system-x86_64: -device vfio-pci,host=07:00.0,bus=pcie.0: Device 'vfio-pci' could not be initialized
Done

lsgroup:

### Group 0 ###
    00:00.0 Host bridge: Intel Corporation 4th Gen Core Processor DRAM Controller (rev 06)
### Group 1 ###
    00:01.0 PCI bridge: Intel Corporation Xeon E3-1200 v3/4th Gen Core Processor PCI Express x16 Controller (rev 06)
    01:00.0 VGA compatible controller: NVIDIA Corporation GK104 [GeForce GTX 670] (rev a1)
    01:00.1 Audio device: NVIDIA Corporation GK104 HDMI Audio Controller (rev a1)
### Group 2 ###
    00:02.0 VGA compatible controller: Intel Corporation Xeon E3-1200 v3/4th Gen Core Processor Integrated Graphics Controller (rev 06)
### Group 3 ###
    00:03.0 Audio device: Intel Corporation Xeon E3-1200 v3/4th Gen Core Processor HD Audio Controller (rev 06)
### Group 4 ###
    00:14.0 USB controller: Intel Corporation 8 Series/C220 Series Chipset Family USB xHCI (rev 05)
### Group 5 ###
    00:16.0 Communication controller: Intel Corporation 8 Series/C220 Series Chipset Family MEI Controller #1 (rev 04)
### Group 6 ###
    00:19.0 Ethernet controller: Intel Corporation Ethernet Connection I217-V (rev 05)
### Group 7 ###
    00:1a.0 USB controller: Intel Corporation 8 Series/C220 Series Chipset Family USB EHCI #2 (rev 05)
### Group 8 ###
    00:1b.0 Audio device: Intel Corporation 8 Series/C220 Series Chipset High Definition Audio Controller (rev 05)
### Group 9 ###
    00:1c.0 PCI bridge: Intel Corporation 8 Series/C220 Series Chipset Family PCI Express Root Port #1 (rev d5)
    00:1c.1 PCI bridge: Intel Corporation 8 Series/C220 Series Chipset Family PCI Express Root Port #2 (rev d5)
    00:1c.2 PCI bridge: Intel Corporation 8 Series/C220 Series Chipset Family PCI Express Root Port #3 (rev d5)
    00:1c.3 PCI bridge: Intel Corporation 8 Series/C220 Series Chipset Family PCI Express Root Port #4 (rev d5)
    00:1c.4 PCI bridge: Intel Corporation 8 Series/C220 Series Chipset Family PCI Express Root Port #5 (rev d5)
    00:1c.5 PCI bridge: Intel Corporation 8 Series/C220 Series Chipset Family PCI Express Root Port #6 (rev d5)
    03:00.0 Network controller: Broadcom Corporation BCM4352 802.11ac Wireless Network Adapter (rev 03)
    04:00.0 SATA controller: ASMedia Technology Inc. ASM1062 Serial ATA Controller (rev 01)
    05:00.0 Ethernet controller: Intel Corporation I211 Gigabit Network Connection (rev 03)
    06:00.0 SATA controller: ASMedia Technology Inc. ASM1062 Serial ATA Controller (rev 01)
    07:00.0 USB controller: NEC Corporation uPD720200 USB 3.0 Host Controller (rev 04)
### Group 10 ###
    00:1d.0 USB controller: Intel Corporation 8 Series/C220 Series Chipset Family USB EHCI #1 (rev 05)
### Group 11 ###
    00:1f.0 ISA bridge: Intel Corporation Z87 Express LPC Controller (rev 05)
    00:1f.2 SATA controller: Intel Corporation 8 Series/C220 Series Chipset Family 6-port SATA Controller 1 [AHCI mode] (rev 05)
    00:1f.3 SMBus: Intel Corporation 8 Series/C220 Series Chipset Family SMBus Controller (rev 05)

Im running with acs override patch included with nbhs kernel(3.13.6-1-mainline) at the moment.
I can see there are just too many devices in group 9, many of them I do not want to pass through. (It seems Intel didnt follow some standards here)

I see beckend found a new acs override patch written by awilliams, but this would be the first time I wouldve manually copy a patch for myself.

I get the following while trying to compile:

patching file drivers/pci/quirks.c
patch: **** malformed patch at line 67:         u16 device;

Any help is appreciated, still got a lot to learn smile

Update: Managed to implement the patch myself. The pci devices are now divided much more than before and I got my USB PCie card passthrough working!

Last edited by trimm (2014-04-10 08:41:28)

Offline

#1514 2014-04-09 18:02:59

novist
Member
Registered: 2014-03-14
Posts: 47

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Still check if they are bound to vfio driver. For example my passed-through usb port 0000:00:13.0:

ls -la /sys/devices/pci0000:00/0000:00:13.0/driver/module/drivers

prints

pci:vfio-pci

Hope you can get it working. We have same GPU and im dying to find out if terrible performance im having is fault of GPU or what (hope its not discouraging!).

Offline

#1515 2014-04-09 18:05:20

Eb
Member
Registered: 2014-04-08
Posts: 11

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

aw wrote:
Eb wrote:
novist wrote:

@Eb: read the error. it says exactly what you have not done - you have not bound devices to vfio driver using vfio-bind script that you were supposed to make.

I knew I forgot to mention something. I have the script created, and before attempting to run the KVM I run the script binding 03:00.0 and its subdevice 03:00.1

The error message says to make sure all of the devices in the group are bound to vfio.  The group, as you reported contains far more than just the secondary function of the device.  The IOMMU group on your system is large because the root port lacks PCIe ACS support.  If you wish to override that, look for the ACS override patch and options to use it (and caveats that enabling it implies).

Could of sworn I already applied the patch, I'll look into it when i'm back at my system. Thanks.

Offline

#1516 2014-04-09 20:07:57

Cubex
Member
Registered: 2014-04-06
Posts: 24

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Yeah 770 works great under new qemu and using ubuntu kernel +  pci acs patch, what problem where you facing Nex7?

I got net working and now im looking at audio, seems that is somewhat laggy and cuts often... tried both nbhs way of adding soundcard and using -soundhw parametere but result is same, all tried with alsa and pulseaudio as driver and still the same hmm

EDIT: Also I changed a bit the vfio-bind script to ignore already binded pci devices, so I can add vfio-bind on the qemu script

#!/bin/bash

if [ ! $USER = "root" ]; then
    echo "Run as root!"
    exit
fi

modprobe vfio-pci

base="/tmp/vfio-bind"
mkdir $base -p
for dev in "$@"; do
    path="$base/$dev"
    if [ ! -f $path ]; then
        touch $path
        vendor=$(cat /sys/bus/pci/devices/$dev/vendor)
        device=$(cat /sys/bus/pci/devices/$dev/device)
        if [ -e /sys/bus/pci/devices/$dev/driver ]; then
                echo $dev > /sys/bus/pci/devices/$dev/driver/unbind
        fi
        echo $vendor $device > /sys/bus/pci/drivers/vfio-pci/new_id
    fi
done

Last edited by Cubex (2014-04-09 22:06:50)

Offline

#1517 2014-04-09 20:41:35

Eb
Member
Registered: 2014-04-08
Posts: 11

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Eb wrote:
aw wrote:
Eb wrote:

I knew I forgot to mention something. I have the script created, and before attempting to run the KVM I run the script binding 03:00.0 and its subdevice 03:00.1

The error message says to make sure all of the devices in the group are bound to vfio.  The group, as you reported contains far more than just the secondary function of the device.  The IOMMU group on your system is large because the root port lacks PCIe ACS support.  If you wish to override that, look for the ACS override patch and options to use it (and caveats that enabling it implies).

Could of sworn I already applied the patch, I'll look into it when i'm back at my system. Thanks.

I've re-installed the patched linux-mainline package listed on the main page, but I'm still having the same issue of all my graphics devices being in the same IOMMU group. I'm running a Z87 chipset, would that cause any issues? Is there a specific patch I should apply for this chipset? Thanks a ton for the help.

Edit:

I've also read that this can potentially disable dri, which is something I need. Am I looking at a lost cause here?

Last edited by Eb (2014-04-09 20:42:42)

Offline

#1518 2014-04-09 21:06:09

beckend
Member
Registered: 2014-03-16
Posts: 5

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Try the V2 of the IOMMU patch: http://linux-kernel.2935.n7.nabble.com/ … 96884.html
Do not apply the old acs patch.
Edit: only works on Intel.

Last edited by beckend (2014-04-09 21:08:06)

Offline

#1519 2014-04-09 21:37:12

Val532
Member
Registered: 2013-11-13
Posts: 35

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

nbhs wrote:
Val532 wrote:

Hi all,

I have a small probleme, on my VM i want to start OCCT and i got a BSOD "driver_irql_not_less_or_equal" i have not any other information on the BSOD screen.

I run on qemu 2.0.0-RC1 and kernel 3.14, i have two graphique card AMD HD 6870 (crossfire not enable) on the VM (just for testing).

On dmesg i have :

[ 8704.000634] kvm_get_msr_common: 82 callbacks suppressed
[ 8704.000643] kvm [6821]: vcpu0 unhandled rdmsr: 0xce
[ 8704.000686] kvm [6821]: vcpu0 unhandled rdmsr: 0xce
[ 8704.004952] kvm [6821]: vcpu0 unhandled rdmsr: 0x637
[ 8704.004976] kvm [6821]: vcpu0 unhandled rdmsr: 0xe8
[ 8704.004989] kvm [6821]: vcpu0 unhandled rdmsr: 0xe7
[ 8704.005013] kvm [6821]: vcpu0 unhandled rdmsr: 0x31
[ 8704.005030] kvm [6821]: vcpu0 unhandled rdmsr: 0x35
[ 8704.005050] kvm [6821]: vcpu0 unhandled rdmsr: 0x39
[ 8704.005095] kvm [6821]: vcpu0 unhandled rdmsr: 0xce
[ 8704.005108] kvm [6821]: vcpu0 unhandled rdmsr: 0x194

Any help ?
Thanks

try

echo "options kvm ignore_msrs=1" >> /etc/modprobe.d/kvm.conf

Nop, i test it but nothing change, and i remove one card, so now i have only one GPU on my VM.

[17229.256218] vfio-pci 0000:01:00.0: irq 56 for MSI/MSI-X
[17311.585365] kvm [11122]: vcpu0 ignored rdmsr: 0xce
[17311.585373] kvm [11122]: vcpu0 ignored rdmsr: 0xce
[17311.589590] kvm [11122]: vcpu0 ignored rdmsr: 0x637
[17311.593788] kvm [11122]: vcpu0 ignored rdmsr: 0x637
[17311.593804] kvm [11122]: vcpu0 ignored rdmsr: 0xe8
[17311.598002] kvm [11122]: vcpu0 ignored rdmsr: 0xe8
[17311.598013] kvm [11122]: vcpu0 ignored rdmsr: 0xe7
[17311.602225] kvm [11122]: vcpu0 ignored rdmsr: 0xe7
[17311.602252] kvm [11122]: vcpu0 ignored rdmsr: 0x31
[17311.602257] kvm [11122]: vcpu0 ignored rdmsr: 0x35

Offline

#1520 2014-04-09 21:58:29

apex8
Member
Registered: 2014-03-29
Posts: 60

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Just to get it right:
The IOMMU patch is meant to "rearrange" devices in the groups? At least this is what I understood from here
In my case the USB3 Controller and the sound card are in the same group as a pci bridge and several pci exporess root ports. Would be great if I could change that.

Offline

#1521 2014-04-09 22:12:53

nbhs
Member
From: Montevideo, Uruguay
Registered: 2013-05-02
Posts: 402

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Eb wrote:
Eb wrote:
aw wrote:

The error message says to make sure all of the devices in the group are bound to vfio.  The group, as you reported contains far more than just the secondary function of the device.  The IOMMU group on your system is large because the root port lacks PCIe ACS support.  If you wish to override that, look for the ACS override patch and options to use it (and caveats that enabling it implies).

Could of sworn I already applied the patch, I'll look into it when i'm back at my system. Thanks.

I've re-installed the patched linux-mainline package listed on the main page, but I'm still having the same issue of all my graphics devices being in the same IOMMU group. I'm running a Z87 chipset, would that cause any issues? Is there a specific patch I should apply for this chipset? Thanks a ton for the help.

Edit:

I've also read that this can potentially disable dri, which is something I need. Am I looking at a lost cause here?

You also need correct cmdline option in your bootloader otherwise the patch does nothing

Offline

#1522 2014-04-10 12:35:46

Eb
Member
Registered: 2014-04-08
Posts: 11

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

nbhs wrote:

You also need correct cmdline option in your bootloader otherwise the patch does nothing

I'm currently using:

intel_iommu=on pcie_acs_override=downstream pci-stub.ids=1002:6810,1002:aab0

Am I missing anything?

Also, does the linux-mainline package on the first page contain the second version of the patch?

beckend wrote:

Try the V2 of the IOMMU patch: http://linux-kernel.2935.n7.nabble.com/ … 96884.html
Do not apply the old acs patch.
Edit: only works on Intel.

I've tried to apply all three parts of the patch, however I'm getting "patch unexpectedly ends in middle of line" for 2/3 and 3/3. After doing this I realized that the patch may not work if there's an older version of the patch installed, or the same one is currently installed. Would I be correct or am I making some sort of silly mistake here?

Edit:

I've failed to mention that I'm fairly new with linux, sorry for any common questions that I may ask.

Last edited by Eb (2014-04-10 12:37:46)

Offline

#1523 2014-04-10 20:24:55

beckend
Member
Registered: 2014-03-16
Posts: 5

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

I believe they are conflicting patch (that was the case for me), skip the "override_for_missing_acs_capabilities.patch" if you plan to use V2 patch, start fresh undo all patches and go from there.

Offline

#1524 2014-04-10 20:27:59

gneville
Member
Registered: 2014-04-02
Posts: 27

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Still not having much luck with vfio vga passthrough on my AMD R9 290 sad

I've tried pretty much all the suggestions I've seen on here but no matter what I try the guest always crashes after installing video drivers.

I did swap from Win8 to Win7 and noticed that after installing the video drivers and rebooting when windows boots up it crashes with a BSOD saying

"PAGE_FAULT_IN_NONPAGED_AREA"

I've tried the following:
multiple linux kernels all with the patches rolled in.
multiple Qemu versions (1.7.0, 1.7.1, 2.0.0RC0,1 &2) and have applied the vga reset patches
Tried both machine types of q35 and i440fx, of which both successfully passthrough the vga card but crash after installing the drivers and rebooting windows
Tried just passing through the Video card leaving out the HDMI Audio.
Tried installing drivers without installing the Catalyst Control Centre as suggested by some other forums.
Tried both AMD drivers and Windows Provided drivers via Windows Updates.
Tried using RomFiles

I'm guessing the problem lies within Qemu somewhere.


gneville wrote:

On top of this I have just installed the OP linux-mainline (3.13.6-1-mainline),seabios (1.7.4.r1788.g8abc6a8-1) & qemu (2.0.r31714.g03d5142-1) versions (Just added reset patch and manually compiled (https://lists.gnu.org/archive/html/qemu … 0767.html))

and still no change sad



gneville wrote:

Hi All,

I'm hoping someone will be able to help me out please. I've been trying to get vga passthrough working with qemu and vfio for a few months on and off but having no luck. I'm getting similar problems to users redger & DanaGoyette.

My hardware: CPU: i7 4770s, MB: Jetway NF9J-Q87 GFX: AMD R9 290

I used to use Xen 4.3 and had vga passthrough working just passing through the intel hd4600 controller to hosts, but since I got the AMD R9 290 card Xen doesn't seem to passthrough the card whatsoever, so thought I would try qemu and vfio.

I've tried using different linux versions and qemu versions but not having much luck. I am able to successfully vfio-bind my gfx card and I can start QEMU and get video via the GFX Card HDMI cable, I'm also able to successfully install Windows 8.1. However after installation I'm getting problems with either windows installing the amd drivers or using the catalyst driver installer, the host just reboots.

My latest setup:

linux 3.13.7-1 + i915 patch
Seabios git 1.7.4-66 (31Mar14)
Qemu 2.0.0 RC0 + patched pcibus_reset & assertion (https://lists.nongnu.org/archive/html/q … 05901.html & https://lists.gnu.org/archive/html/qemu … 00767.html)

Before the pcibus_reset patch I kept getting the following messages when the host rebooted while trying to install the drivers:

"qemu-system-x86_64 shows: hw/pci/pci.c:250: pcibus_reset: Assertion `bus->irq_count[\i] == 0' failed."

dmesg shows:  2805.839608 vfio-pci 0000:01:00.0: irq 46 for MSI/MSI-X


After applying the pcibust_reset patch the host no longer reboots after installing the drivers however the screen just goes blank so I have to manually kill the host. I do see the following messages:

qemu-system-x86_64 shows: ehci warning: guest updated active QH

dmesg shows:  4681.189513 vfio-pci 0000:01:00.0: irq 46 for MSI/MSI-X

This is my QEMU config:

/usr/local/qemu-2-0-0-rc0-patched-pci/bin/qemu-system-x86_64 -name win8 -enable-kvm -M q35 -m 6144  \
-cpu Haswell,hv-time,hv_relaxed,hv_vapic,hv_spinlocks=0x1000 \
-boot order=dc \
-smp 2,sockets=1,cores=2,threads=1 \
-bios /root/vfio/bios.bin -vga none \
-usb -usbdevice tablet \
-spice port=5902,disable-ticketing \
-vnc 0.0.0.0:0 \
-device ioh3420,bus=pcie.0,addr=1c.0,multifunction=on,port=1,chassis=1,id=root.1 \
-device ahci,bus=pcie.0,id=ahci \
-net nic \
-net tap,ifname=tap0,script=no,downscript=no \
-device vfio-pci,host=01:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on \
-device vfio-pci,host=01:00.1,bus=root.1,addr=00.1 \
-device vfio-pci,host=00:14.0,bus=pcie.0 \
-drive file=/dev/mapper/vg1-winv1,media=disk,index=1,id=disk,cache=none,if=virtio \
-drive file=/dev/mapper/vg1-winv1data,media=disk,index=2,id=diskb,cache=none,if=virtio \
-drive file=/home/xen/iso/windows/en-gb_windows_8_1_enterprise_x64_dvd_2971910.iso,boot=on,media=cdrom,index=3,id=isocd  \
-device ide-cd,bus=ahci.1,drive=isocd \
-drive file=/root/virtio-win-0.1-74.iso,media=cdrom,index=4,id=virtio  \
-device ide-cd,bus=ahci.2,drive=virtio

If I enable "-vga cirrus" and VNC to the host I can install the AMD drivers successfully and after a reboot it shows in device manager the AMD card but with a yellow exclamation and code 43. If I then go back to "-vga none" windows no longer wants to boot and I need up in a boot loop (seabios>windows booting>seabios>windows booting).

Any ideas please?

Offline

#1525 2014-04-10 21:35:39

Eb
Member
Registered: 2014-04-08
Posts: 11

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

beckend wrote:

I believe they are conflicting patch (that was the case for me), skip the "override_for_missing_acs_capabilities.patch" if you plan to use V2 patch, start fresh undo all patches and go from there.

I've started fresh from the package on the main page, and have removed "override_for_missing_acs_capabilities.patch" from the pkgbuild and added in all three parts (in seperate files) of the ACS override patch. When I try to makepkg, I'm getting this:

==> Validating source files with md5sums...
    linux-3.14.tar.xz ... Passed
    config ... Passed
    config.x86_64 ... Passed
    linux.preset ... Passed
    change-default-console-loglevel.patch ... Passed
    0001-Bluetooth-allocate-static-minor-for-vhci.patch ... Passed
    0002-module-allow-multiple-calls-to-MODULE_DEVICE_TABLE-p.patch ... Passed
    0003-module-remove-MODULE_GENERIC_TABLE.patch ... Passed
    0004-Revert-syscalls.h-use-gcc-alias-instead-of-assembler.patch ... Passed
    enable_additional_cpu_optimizations_for_gcc.patch.gz ... Passed
    radeon_load_vbios_from_file.patch ... Passed
    aw_acs_1.patch ... Passed
    aw_acs_2.patch ... Passed
    aw_acs_3.patch ... Passed
    i915_314.patch ... Passed
==> Extracting sources...
  -> Extracting linux-3.14.tar.xz with bsdtar
  -> Extracting enable_additional_cpu_optimizations_for_gcc.patch.gz with gzip
==> Starting prepare()...
patching file kernel/printk/printk.c
Hunk #1 succeeded at 59 with fuzz 2 (offset 1 line).
patching file Documentation/devices.txt
patching file drivers/bluetooth/hci_vhci.c
patching file include/linux/miscdevice.h
patching file include/linux/module.h
patching file scripts/mod/file2alias.c
patching file include/linux/isapnp.h
patching file include/linux/module.h
patching file include/linux/compat.h
patching file include/linux/syscalls.h
patching file drivers/gpu/drm/radeon/radeon_bios.c
patching file drivers/pci/pci.c
Hunk #1 FAILED at 2180.
Hunk #2 FAILED at 2212.
2 out of 2 hunks FAILED -- saving rejects to file drivers/pci/pci.c.rej
patching file drivers/pci/quirks.c
Hunk #1 succeeded at 3461 with fuzz 2.
patching file include/linux/pci.h
Hunk #1 succeeded at 1530 with fuzz 1 (offset 20 lines).
Hunk #2 succeeded at 1543 with fuzz 2 (offset 20 lines).
==> ERROR: A failure occurred in prepare().
    Aborting...

I just know I'm missing something simple here, anyone see what it is? Do I need to edit the patches in some way?

Offline

Board footer

Powered by FluxBB