KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9 (Page 30) / Kernel & Hardware / Arch Linux Forums

You are not logged in.

Topics: Active | Unanswered

Pages: Previous 1 … 28 29 30 31 32 … 234 Next

Topic closed

#726 2013-11-13 21:45:29

nbhs: Member; From: Montevideo, Uruguay; Registered: 2013-05-02; Posts: 402

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Val532 wrote:

So i made some test with this methode but no luck. Virt Manager say to me a lot of difrente error.
Sometime AppArmor error, sometime connection reset by peer.
My Hardware is:
Corei5 4670S so HD4600 for the host
24Go ram (3*8Go)
Asrock Z87 Extrem 6
AMD HD7770 for the first VM
AMD HD5570 for the second VM
AMD HD5570 fot the third Vm
All of the graphic card work.
For the moment i've only Windows 7 on the first VM, the seconde and third come later.

Well try disabling apparmor, you could also use the qemu monitor for your vm's, launch qemu with -monitor stdio

Offline

#727 2013-11-13 22:21:17

cryptonymous: Member; Registered: 2013-11-05; Posts: 6

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

aw wrote:

aw wrote:
Testing a Linux guest w/ a GTX660, I can confirm that things go bad when using the NV_RegEnableMSI=1 option. This worked fine for me with a Quadro card. I see a single interrupt and the monitor goes out of sync and never comes back after OS bootup. I'll try to find some time to dig into it. I also note that the audio driver specifically checks for nvidia devices and blacklists MSI on snd-hda-intel, so at some point nvidia hasn't played nicely with Linux MSI support. Perhaps there's a reason why the Linux nvidia driver defaults to INTx as well.
Good news/bad news...
Good news: I figured out the problem and posted a patch that solves it on my GTX660: http://lists.nongnu.org/archive/html/qe … 01390.html
Bad news: This means that for every MSI interrupt, the Nvidia driver does a write to MMIO space at an offset that QEMU needs to trap in order to "virtualize" the device. That may negate any performance benefit of MSI vs INTx. As noted in the patch we might be able to use an ioeventfd to make handling more asynchronous, but the overhead is probably still more than INTx, which we can handle without exiting to QEMU userspace.

Thanks, "3950205 vfio-pci: Fix Nvidia MSI ACK through 0x88000 quirk" has fixed the interrupt issue on my GTX 650 in OS X guest.

Offline

#728 2013-11-13 22:37:37

Val532: Member; Registered: 2013-11-13; Posts: 35

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

nbhs wrote:

Well try disabling apparmor, you could also use the qemu monitor for your vm's, launch qemu with -monitor stdio

So i've disabled apparmor but it does not slove my problem.

I've tried a lot of differente .xml but some time it does not show in virt manager, sometime it's say Connection reset by peer (when i put pc-q35-1.7 instead of 440).

I'm completly lost ^^.

ps: my .xml

<domain type='kvm' xmlns:qemu='http://libvirt.org/schemas/domain/qemu/1.0'>
<name>windows7test</name>
<uuid>79afd66a-f479-8c9c-c66f-2ac53f1c110c</uuid>
<memory unit='KiB'>8388608</memory>
<currentMemory unit='KiB'>8388608</currentMemory>
<vcpu placement='static'>4</vcpu>
<os>
<type arch='x86_64' machine='pc-q35-1.7'>hvm</type>
<boot dev='hd'/>
<smbios mode='host'/>
</os>
<features>
<acpi/>
<apic/>
<pae/>
</features>
<cpu mode='custom' match='exact'>
<model fallback='allow'>Haswell</model>
<vendor>Intel</vendor>
<feature policy='require' name='tm2'/>
<feature policy='require' name='est'/>
<feature policy='require' name='vmx'/>
<feature policy='require' name='osxsave'/>
<feature policy='require' name='smx'/>
<feature policy='require' name='ss'/>
<feature policy='require' name='ds'/>
<feature policy='require' name='vme'/>
<feature policy='require' name='dtes64'/>
<feature policy='require' name='abm'/>
<feature policy='require' name='ht'/>
<feature policy='require' name='acpi'/>
<feature policy='require' name='pbe'/>
<feature policy='require' name='tm'/>
<feature policy='require' name='pdcm'/>
<feature policy='require' name='pdpe1gb'/>
<feature policy='require' name='ds_cpl'/>
<feature policy='require' name='rdrand'/>
<feature policy='require' name='f16c'/>
<feature policy='require' name='xtpr'/>
<feature policy='require' name='monitor'/>
</cpu>
<clock offset='localtime'/>
<on_poweroff>destroy</on_poweroff>
<on_reboot>restart</on_reboot>
<on_crash>restart</on_crash>
<devices>
<emulator>/usr/bin/qemu-system-x86_64</emulator>
<disk type='file' device='disk'>
<driver name='qemu' type='qcow2' cache='writethrough' io='native'/>
<source file='/home/val/windows2'/>
<target dev='sda' bus='sata'/>
<address type='drive' controller='0' bus='0' target='0' unit='0'/>
</disk>
<controller type='usb' index='0'>
<address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x2'/>
</controller>
<controller type='pci' index='0' model='pci-root'/>
<controller type='sata' index='0'>
<address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x0'/>
</controller>
<interface type='bridge'>
<mac address='52:54:00:4c:d9:7a'/>
<source bridge='br0'/>
<model type='e1000'/>
<address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x0'/>
</interface>
<memballoon model='virtio'>
<address type='pci' domain='0x0000' bus='0x00' slot='0x04' function='0x0'/>
</memballoon>
</devices>
<qemu:commandline>
<qemu:arg value='-device'/>
<qemu:arg value='ioh3420,bus=pcie.0,addr=1c.0,multifunction=on,port=1,chassis=1,id=root.1'/>
<qemu:arg value='-device'/>
<qemu:arg value='vfio-pci,host=03:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on'/>
<qemu:arg value='-device'/>
<qemu:arg value='vfio-pci,host=03:00.1,bus=root.1,addr=00.1'/>
<qemu:arg value='-bios'/>
<qemu:arg value='/usr/share/qemu/bios.bin'/>
</qemu:commandline>
</domain>

Last edited by Val532 (2013-11-13 23:03:54)

Offline

#730 2013-11-13 23:29:16

nbhs: Member; From: Montevideo, Uruguay; Registered: 2013-05-02; Posts: 402

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Norcoen wrote:

...
Assign PCI devices to pci-stub
...

You have to assign your devices to vfio-pci not pci-stub

Offline

#731 2013-11-13 23:33:14

aw: Member; Registered: 2013-10-04; Posts: 921; Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

nbhs wrote:

Norcoen wrote:
...
Assign PCI devices to pci-stub
...
You have to assign your devices to vfio-pci not pci-stub

And do not bind bridges to anything, leave them attached to pcieport. Maybe start with something simple, like one device instead of practically everything in the box.

http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses? Try https://www.redhat.com/mailman/listinfo/vfio-users

Offline

#732 2013-11-13 23:36:17

mafferri: Member; Registered: 2013-11-11; Posts: 7

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

aw wrote:

mafferri wrote:

quoto this because I wish I could do the reverse, for when I shutdown the vm
because if I restart the vm I have very bad performance
I have to reboot the host because everything works well

You can do the reverse. Here are the scripts I use:

vfio-group <group #>:

#!/bin/sh

if [ ! -e /sys/kernel/iommu_groups/$1 ]; then
	echo "IOMMU group $1 not found"
	exit 1
fi

if [ ! -e /sys/bus/pci/drivers/vfio-pci ]; then
	sudo modprobe vfio-pci
fi

for i in $(ls /sys/kernel/iommu_groups/$1/devices/); do
	if [ -e /sys/kernel/iommu_groups/$1/devices/$i/driver ]; then
		if [ "$(basename $(readlink -f \
	        	/sys/kernel/iommu_groups/$1/devices/$i/driver))" != \
			"pcieport" ]; then
			echo $i | sudo tee \
				/sys/kernel/iommu_groups/$1/devices/$i/driver/unbind
		fi
	fi
done

for i in $(ls /sys/kernel/iommu_groups/$1/devices/); do
	if [ ! -e /sys/kernel/iommu_groups/$1/devices/$i/driver ]; then
		VEN=$(cat /sys/kernel/iommu_groups/$1/devices/$i/vendor)
		DEV=$(cat /sys/kernel/iommu_groups/$1/devices/$i/device)
		echo $VEN $DEV | sudo tee \
			/sys/bus/pci/drivers/vfio-pci/new_id
	fi
done

vfio-ungroup <group #>

#!/bin/sh

if [ ! -e /sys/kernel/iommu_groups/$1 ]; then
	echo "IOMMU group $1 not found"
	exit 1
fi

for i in $(ls /sys/kernel/iommu_groups/$1/devices/); do
	VEN=$(cat /sys/kernel/iommu_groups/$1/devices/$i/vendor)
	DEV=$(cat /sys/kernel/iommu_groups/$1/devices/$i/device)
	echo $VEN $DEV | sudo tee \
		/sys/bus/pci/drivers/vfio-pci/remove_id
	echo $i | sudo tee \
		/sys/kernel/iommu_groups/$1/devices/$i/driver/unbind
done

for i in $(ls /sys/kernel/iommu_groups/$1/devices/); do
	echo $i | sudo tee /sys/bus/pci/drivers_probe
done

lsgroup

#!/bin/sh

BASE="/sys/kernel/iommu_groups"

for i in $(find $BASE -maxdepth 1 -mindepth 1 -type d); do
	GROUP=$(basename $i)
	echo "### Group $GROUP ###"
	for j in $(find $i/devices -type l); do
		DEV=$(basename $j)
		echo -n "    "
		lspci -s $DEV
	done
done

I'm not sure how this is going to help your performance though, it's pretty random whether re-loading the host driver is going to resolve anything that the guest driver can't. For graphics devices, if you're using kernel+qemu that support bus reset, that should provide something nearly as good as a host reboot.

much more convenient to handle thanks;)

Offline

#733 2013-11-13 23:50:53

Norcoen: Member; From: Norway; Registered: 2013-11-02; Posts: 23

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Thank you for your insanely fast reply
I didn't thought about it when I used pci-stub because I always used it with virt-manager before trying vfio...

I'm getting this error now:

qemu-system-x86_64: -device vfio-pci,host=04:00.0: vfio: error, group 9 is not viable, please ensure all devices within the iommu_group are bound to their vfio bus driver.
qemu-system-x86_64: -device vfio-pci,host=04:00.0: vfio: failed to get group 9
qemu-system-x86_64: -device vfio-pci,host=04:00.0: Device initialization failed.
qemu-system-x86_64: -device vfio-pci,host=04:00.0: Device 'vfio-pci' could not be initialized

Using the lsgroup script you just posted I got this:

$ ./lsgroup.sh 
### Group 0 ###
    00:00.0 Host bridge: Intel Corporation Xeon E3-1200 v3 Processor DRAM Controller (rev 06)
### Group 1 ###
    00:01.0 PCI bridge: Intel Corporation Xeon E3-1200 v3/4th Gen Core Processor PCI Express x16 Controller (rev 06)
    01:00.0 VGA compatible controller: Advanced Micro Devices, Inc. [AMD/ATI] Pitcairn PRO [Radeon HD 7850]
    01:00.1 Audio device: Advanced Micro Devices, Inc. [AMD/ATI] Cape Verde/Pitcairn HDMI Audio [Radeon HD 7700/7800 Series]
### Group 2 ###
    00:02.0 VGA compatible controller: Intel Corporation Xeon E3-1200 v3 Processor Integrated Graphics Controller (rev 06)
### Group 3 ###
    00:03.0 Audio device: Intel Corporation Xeon E3-1200 v3/4th Gen Core Processor HD Audio Controller (rev 06)
### Group 4 ###
    00:14.0 USB controller: Intel Corporation 8 Series/C220 Series Chipset Family USB xHCI (rev 05)
### Group 5 ###
    00:16.0 Communication controller: Intel Corporation 8 Series/C220 Series Chipset Family MEI Controller #1 (rev 04)
### Group 6 ###
    00:19.0 Ethernet controller: Intel Corporation Ethernet Connection I217-V (rev 05)
### Group 7 ###
    00:1a.0 USB controller: Intel Corporation 8 Series/C220 Series Chipset Family USB EHCI #2 (rev 05)
### Group 8 ###
    00:1b.0 Audio device: Intel Corporation 8 Series/C220 Series Chipset High Definition Audio Controller (rev 05)
### Group 9 ###
    00:1c.0 PCI bridge: Intel Corporation 8 Series/C220 Series Chipset Family PCI Express Root Port #1 (rev d5)
    00:1c.2 PCI bridge: Intel Corporation 8 Series/C220 Series Chipset Family PCI Express Root Port #3 (rev d5)
    00:1c.3 PCI bridge: Intel Corporation 8 Series/C220 Series Chipset Family PCI Express Root Port #4 (rev d5)
    00:1c.4 PCI bridge: Intel Corporation 8 Series/C220 Series Chipset Family PCI Express Root Port #5 (rev d5)
    00:1c.5 PCI bridge: Intel Corporation 8 Series/C220 Series Chipset Family PCI Express Root Port #6 (rev d5)
    03:00.0 SATA controller: ASMedia Technology Inc. ASM1062 Serial ATA Controller (rev 01)
    04:00.0 Ethernet controller: Intel Corporation I211 Gigabit Network Connection (rev 03)
    05:00.0 SATA controller: ASMedia Technology Inc. ASM1062 Serial ATA Controller (rev 01)
    06:00.0 PCI bridge: PLX Technology, Inc. Device 8603 (rev aa)
    07:01.0 PCI bridge: PLX Technology, Inc. Device 8603 (rev aa)
    07:02.0 PCI bridge: PLX Technology, Inc. Device 8603 (rev aa)
    08:00.0 SATA controller: ASMedia Technology Inc. ASM1062 Serial ATA Controller (rev 01)
    09:00.0 USB controller: Renesas Technology Corp. uPD720202 USB 3.0 Host Controller (rev 02)
### Group 10 ###
    00:1d.0 USB controller: Intel Corporation 8 Series/C220 Series Chipset Family USB EHCI #1 (rev 05)
### Group 11 ###
    00:1f.0 ISA bridge: Intel Corporation Z87 Express LPC Controller (rev 05)
    00:1f.2 SATA controller: Intel Corporation 8 Series/C220 Series Chipset Family 6-port SATA Controller 1 [AHCI mode] (rev 05)
    00:1f.3 SMBus: Intel Corporation 8 Series/C220 Series Chipset Family SMBus Controller (rev 05)

Why is there so much in group 9? From what I know, I can only passthrough complete groups.
But why would my PCIe Card be in the same group as a whole lot of onboard stuff?
Or is this the same problem as I had some pages back, because the PCIe Card has the same vendor:device ids as the onboard asmedia chip? (I think vfio-bind.sh automaticly gets them)

Offline

#734 2013-11-14 00:05:01

nbhs: Member; From: Montevideo, Uruguay; Registered: 2013-05-02; Posts: 402

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Norcoen wrote:

Thank you for your insanely fast reply
I didn't thought about it when I used pci-stub because I always used it with virt-manager before trying vfio...

I'm getting this error now:

qemu-system-x86_64: -device vfio-pci,host=04:00.0: vfio: error, group 9 is not viable, please ensure all devices within the iommu_group are bound to their vfio bus driver.
qemu-system-x86_64: -device vfio-pci,host=04:00.0: vfio: failed to get group 9
qemu-system-x86_64: -device vfio-pci,host=04:00.0: Device initialization failed.
qemu-system-x86_64: -device vfio-pci,host=04:00.0: Device 'vfio-pci' could not be initialized

Using the lsgroup script you just posted I got this:

$ ./lsgroup.sh 
### Group 0 ###
    00:00.0 Host bridge: Intel Corporation Xeon E3-1200 v3 Processor DRAM Controller (rev 06)
### Group 1 ###
    00:01.0 PCI bridge: Intel Corporation Xeon E3-1200 v3/4th Gen Core Processor PCI Express x16 Controller (rev 06)
    01:00.0 VGA compatible controller: Advanced Micro Devices, Inc. [AMD/ATI] Pitcairn PRO [Radeon HD 7850]
    01:00.1 Audio device: Advanced Micro Devices, Inc. [AMD/ATI] Cape Verde/Pitcairn HDMI Audio [Radeon HD 7700/7800 Series]
### Group 2 ###
    00:02.0 VGA compatible controller: Intel Corporation Xeon E3-1200 v3 Processor Integrated Graphics Controller (rev 06)
### Group 3 ###
    00:03.0 Audio device: Intel Corporation Xeon E3-1200 v3/4th Gen Core Processor HD Audio Controller (rev 06)
### Group 4 ###
    00:14.0 USB controller: Intel Corporation 8 Series/C220 Series Chipset Family USB xHCI (rev 05)
### Group 5 ###
    00:16.0 Communication controller: Intel Corporation 8 Series/C220 Series Chipset Family MEI Controller #1 (rev 04)
### Group 6 ###
    00:19.0 Ethernet controller: Intel Corporation Ethernet Connection I217-V (rev 05)
### Group 7 ###
    00:1a.0 USB controller: Intel Corporation 8 Series/C220 Series Chipset Family USB EHCI #2 (rev 05)
### Group 8 ###
    00:1b.0 Audio device: Intel Corporation 8 Series/C220 Series Chipset High Definition Audio Controller (rev 05)
### Group 9 ###
    00:1c.0 PCI bridge: Intel Corporation 8 Series/C220 Series Chipset Family PCI Express Root Port #1 (rev d5)
    00:1c.2 PCI bridge: Intel Corporation 8 Series/C220 Series Chipset Family PCI Express Root Port #3 (rev d5)
    00:1c.3 PCI bridge: Intel Corporation 8 Series/C220 Series Chipset Family PCI Express Root Port #4 (rev d5)
    00:1c.4 PCI bridge: Intel Corporation 8 Series/C220 Series Chipset Family PCI Express Root Port #5 (rev d5)
    00:1c.5 PCI bridge: Intel Corporation 8 Series/C220 Series Chipset Family PCI Express Root Port #6 (rev d5)
    03:00.0 SATA controller: ASMedia Technology Inc. ASM1062 Serial ATA Controller (rev 01)
    04:00.0 Ethernet controller: Intel Corporation I211 Gigabit Network Connection (rev 03)
    05:00.0 SATA controller: ASMedia Technology Inc. ASM1062 Serial ATA Controller (rev 01)
    06:00.0 PCI bridge: PLX Technology, Inc. Device 8603 (rev aa)
    07:01.0 PCI bridge: PLX Technology, Inc. Device 8603 (rev aa)
    07:02.0 PCI bridge: PLX Technology, Inc. Device 8603 (rev aa)
    08:00.0 SATA controller: ASMedia Technology Inc. ASM1062 Serial ATA Controller (rev 01)
    09:00.0 USB controller: Renesas Technology Corp. uPD720202 USB 3.0 Host Controller (rev 02)
### Group 10 ###
    00:1d.0 USB controller: Intel Corporation 8 Series/C220 Series Chipset Family USB EHCI #1 (rev 05)
### Group 11 ###
    00:1f.0 ISA bridge: Intel Corporation Z87 Express LPC Controller (rev 05)
    00:1f.2 SATA controller: Intel Corporation 8 Series/C220 Series Chipset Family 6-port SATA Controller 1 [AHCI mode] (rev 05)
    00:1f.3 SMBus: Intel Corporation 8 Series/C220 Series Chipset Family SMBus Controller (rev 05)

If you built the kernel i provided try this workarround: https://bbs.archlinux.org/viewtopic.php … 5#p1342995

Offline

#735 2013-11-14 00:40:36

Norcoen: Member; From: Norway; Registered: 2013-11-02; Posts: 23

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

No, I just built 3.12 from kernel.org with the config changes

    CONFIG_VFIO_IOMMU_TYPE1=y

    CONFIG_VFIO=y

    CONFIG_VFIO_PCI=y

    CONFIG_VFIO_PCI_VGA=y

I'll try building your kernel. Guess that will take some time, since I have to patch it manually. (And built my first kernel a week ago...^^)

Offline

#736 2013-11-14 00:48:55

nbhs: Member; From: Montevideo, Uruguay; Registered: 2013-05-02; Posts: 402

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Norcoen wrote:

I'll try building your kernel. Guess that will take some time, since I have to patch it manually. (And built my first kernel a week ago...^^)

If you're using arch, just unpack it, open a console then build it like this:

makepkg -s

then install it:

sudo pacman -U linux-mainline-*

Same for the qemu and seabios packages i linked.

Last edited by nbhs (2013-11-14 00:49:27)

Offline

#737 2013-11-14 02:16:44

mafferri: Member; Registered: 2013-11-11; Posts: 7

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Norcoen wrote:

Thank you for your insanely fast reply
I didn't thought about it when I used pci-stub because I always used it with virt-manager before trying vfio...

I'm getting this error now:

qemu-system-x86_64: -device vfio-pci,host=04:00.0: vfio: error, group 9 is not viable, please ensure all devices within the iommu_group are bound to their vfio bus driver.
qemu-system-x86_64: -device vfio-pci,host=04:00.0: vfio: failed to get group 9
qemu-system-x86_64: -device vfio-pci,host=04:00.0: Device initialization failed.
qemu-system-x86_64: -device vfio-pci,host=04:00.0: Device 'vfio-pci' could not be initialized

resolve this , add the group /dev/vfio/9 in the /etc/libvirt/qemu.conf edit: I thought you were the one who posted the libvirt domain
and "chmod 666 /dev/vfio/* " only this test

Last edited by mafferri (2013-11-14 02:18:24)

Offline

#738 2013-11-14 02:22:28

aw: Member; Registered: 2013-10-04; Posts: 921; Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

mafferri wrote:

Norcoen wrote:
Thank you for your insanely fast reply
I didn't thought about it when I used pci-stub because I always used it with virt-manager before trying vfio...
I'm getting this error now:
qemu-system-x86_64: -device vfio-pci,host=04:00.0: vfio: error, group 9 is not viable, please ensure all devices within the iommu_group are bound to their vfio bus driver.
qemu-system-x86_64: -device vfio-pci,host=04:00.0: vfio: failed to get group 9
qemu-system-x86_64: -device vfio-pci,host=04:00.0: Device initialization failed.
qemu-system-x86_64: -device vfio-pci,host=04:00.0: Device 'vfio-pci' could not be initialized
resolve this , add the group /dev/vfio/9 in the /etc/libvirt/qemu.conf edit: I thought you were the one who posted the libvirt domain
and "chmod 666 /dev/vfio/* " only this test

That's not the problem, the error message indicates how to fix it. Norcoen needs the ACS override patch and to boot with an option to prune the group down to something manageable.

http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses? Try https://www.redhat.com/mailman/listinfo/vfio-users

Offline

#739 2013-11-14 05:32:33

Val532: Member; Registered: 2013-11-13; Posts: 35

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Hi,

I work this night to make able to use virt-manager with GPU passtrough and it work now !!!

You need libvirt 1.1.4 and Virt-Manage 0.10.

Offline

#740 2013-11-14 14:16:06

ilya80: Member; Registered: 2013-11-12; Posts: 34

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Hi guys!

Need a bit of help here. My config:

Asrock Z68 Extreme 3 Gen 3, latest BIOS, Vt-d enabled in bios
Intel i5 3470, with HD 2500 as primary adapter, used in host system
NVidia GTX 680 as secondary adapter, nouveau blacklisted in host system - intended for passthrough
Running fresh Debian sid

I started on the whole thing, so far I`ve done the following:

1) Downloaded 3.12 source, applied "i915_fixes.patch" and "override_for_missing_acs_capabilities.patch", built with VFIO compiled into the kernel and rebooted. Tried both with and without "vfio_iommu_type1.allow_unsafe_interrupts=1" in kernel command line.
2) Cloned qemu git, applied "vfio_disallow_device_from_using_nosnoop_transactions.patch", built and installed via "make install"
3) Cloned seabios and built, have no use for it yet though.

My problem is that when I try to bind the GPU using "sudo vfio-bind 0000:01:00.0 0000:01:00.0", it fails with the following in /var/log/kern.log:

Nov 14 17:54:18 cave-lin kernel: [  145.427257] vfio-pci: probe of 0000:01:00.0 failed with error -22
Nov 14 17:54:18 cave-lin kernel: [  145.472245] vfio-pci: probe of 0000:01:00.0 failed with error -22
Nov 14 17:54:18 cave-lin kernel: [  145.472252] vfio-pci: probe of 0000:01:00.1 failed with error -22

I`m sure device is not yet claimed by any driver, "sudo lspci -k" gives:

01:00.0 VGA compatible controller: NVIDIA Corporation GK104 [GeForce GTX 680] (rev a1)
        Subsystem: NVIDIA Corporation Device 0969
01:00.1 Audio device: NVIDIA Corporation GK104 HDMI Audio Controller (rev a1)
        Subsystem: NVIDIA Corporation Device 0969

Any pointers on what I have missed / where I should look ?

Thanks!

Offline

#741 2013-11-14 14:20:31

aw: Member; Registered: 2013-10-04; Posts: 921; Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

ilya80 wrote:

Hi guys!
Need a bit of help here. My config:
Asrock Z68 Extreme 3 Gen 3, latest BIOS, Vt-d enabled in bios
Intel i5 3470, with HD 2500 as primary adapter, used in host system
NVidia GTX 680 as secondary adapter, nouveau blacklisted in host system - intended for passthrough
Running fresh Debian sid
I started on the whole thing, so far I`ve done the following:
1) Downloaded 3.12 source, applied "i915_fixes.patch" and "override_for_missing_acs_capabilities.patch", built with VFIO compiled into the kernel and rebooted. Tried both with and without "vfio_iommu_type1.allow_unsafe_interrupts=1" in kernel command line.
2) Cloned qemu git, applied "vfio_disallow_device_from_using_nosnoop_transactions.patch", built and installed via "make install"
3) Cloned seabios and built, have no use for it yet though.
My problem is that when I try to bind the GPU using "sudo vfio-bind 0000:01:00.0 0000:01:00.0", it fails with the following in /var/log/kern.log:
Nov 14 17:54:18 cave-lin kernel: [  145.427257] vfio-pci: probe of 0000:01:00.0 failed with error -22
Nov 14 17:54:18 cave-lin kernel: [  145.472245] vfio-pci: probe of 0000:01:00.0 failed with error -22
Nov 14 17:54:18 cave-lin kernel: [  145.472252] vfio-pci: probe of 0000:01:00.1 failed with error -22
I`m sure device is not yet claimed by any driver, "sudo lspci -k" gives:
01:00.0 VGA compatible controller: NVIDIA Corporation GK104 [GeForce GTX 680] (rev a1)
        Subsystem: NVIDIA Corporation Device 0969
01:00.1 Audio device: NVIDIA Corporation GK104 HDMI Audio Controller (rev a1)
        Subsystem: NVIDIA Corporation Device 0969
Any pointers on what I have missed / where I should look ?
Thanks!

Boot with intel_iommu=on

http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses? Try https://www.redhat.com/mailman/listinfo/vfio-users

Offline

#742 2013-11-14 14:36:42

ilya80: Member; Registered: 2013-11-12; Posts: 34

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

aw wrote:

Boot with intel_iommu=on

That did it, thanks! Both devices now successfully managed by vfio-pci. I`ll follow the guide in the first post. Maybe you could provide few clarifications: should I copy seabios binaries produced after building it somewhere into qemu? Or just try firing up qemu?

Offline

#743 2013-11-14 14:46:53

nbhs: Member; From: Montevideo, Uruguay; Registered: 2013-05-02; Posts: 402

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

ilya80 wrote:

aw wrote:
Boot with intel_iommu=on
That did it, thanks! Both devices now successfully managed by vfio-pci. I`ll follow the guide in the first post. Maybe you could provide few clarifications: should I copy seabios binaries produced after building it somewhere into qemu? Or just try firing up qemu?

Build it somewhere then fire qemu with qemu-system-x86_64 -bios /path/to/seabios/out/bios.bin ...

Offline

#744 2013-11-14 15:06:42

ilya80: Member; Registered: 2013-11-12; Posts: 34

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

nbhs wrote:

Build it somewhere then fire qemu with qemu-system-x86_64 -bios /path/to/seabios/out/bios.bin ...

Right, now I read it. Okay this requires me looking at the monitors, so I`ll have to come home first.

Thanks!

Offline

#745 2013-11-14 15:07:02

aw: Member; Registered: 2013-10-04; Posts: 921; Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

The bios provided by qemu should be fine.

http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses? Try https://www.redhat.com/mailman/listinfo/vfio-users

Offline

#746 2013-11-14 16:49:11

Norcoen: Member; From: Norway; Registered: 2013-11-02; Posts: 23

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

nbhs wrote:

If you're using arch, just unpack it, open a console then build it like this:
makepkg -s
then install it:
sudo pacman -U linux-mainline-*
Same for the qemu and seabios packages i linked.

Unfortunately I'm stuck with Fedora and/or CentOS if I don't want to use Windows here.
I already built the qemu and seabios packages you provided (~2 weeks ago, have there been any changes I need in the meantime?)

Now I just read the code in the PKGBUILD file and did the patching manually. No errors so far. But I'm kinda stuck on the .config file.
I copied the oldconfig from /boot/config/3.11.7 and compared it to your config.x86_64.

I think (hope) it's save, to leave configs on, that arent specified in your config, for example:
Your config: # CONFIG_KALLSYMS_ALL is not set
My oldconfig: CONFIG_KALLSYMS_ALL=y
Result: CONFIG_KALLSYMS_ALL=y

I think this also applies the other way around, when you set it, but it's not specified in my oldconfig yet:
Your config: CONFIG_TICK_CPU_ACCOUNTING=y
My oldconfig: # CONFIG_TICK_CPU_ACCOUNTING is not set
Result: CONFIG_TICK_CPU_ACCOUNTING=y

I hope I'm right to this point. Now what do I do with differences like this:

Your Config:                                             My oldconfig:
#								#
# RCU Subsystem							# RCU Subsystem
#								#
CONFIG_TREE_PREEMPT_RCU=y				      |	CONFIG_TREE_RCU=y
CONFIG_PREEMPT_RCU=y					      |	# CONFIG_PREEMPT_RCU is not set
CONFIG_RCU_STALL_COMMON=y					CONFIG_RCU_STALL_COMMON=y

Last edited by Norcoen (2013-11-14 16:54:44)

Offline

#747 2013-11-14 16:52:13

aw: Member; Registered: 2013-10-04; Posts: 921; Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

The fedora kernel config is fine to use as a starting point.

http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses? Try https://www.redhat.com/mailman/listinfo/vfio-users

Offline

#748 2013-11-14 16:58:25

Norcoen: Member; From: Norway; Registered: 2013-11-02; Posts: 23

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

But why isn't CONFIG_TREE_PREEMPT_RCU present in the fedora kernel config, while CONFIG_TREE_RCU isn't present in nbhs' config?
Aren't those related somehow? Or is it complety irrelevant?

Again, thank you for those insanely fast replies

Last edited by Norcoen (2013-11-14 17:04:30)

Offline

#749 2013-11-14 16:59:58

aw: Member; Registered: 2013-10-04; Posts: 921; Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Norcoen wrote:

But why isn't CONFIG_TREE_PREEMPT_RCU present in the fedora kernel config, while CONFIG_TREE_RCU is present isn't present in nbhs' config?
AreN't those related somehow? Or is it complety irrelevant?
Again, thank you for those insanely fast replies

Completely irrelevant to vga assignment

http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses? Try https://www.redhat.com/mailman/listinfo/vfio-users

Offline

#750 2013-11-14 18:41:29

Norcoen: Member; From: Norway; Registered: 2013-11-02; Posts: 23

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

It's working!
Booting kernel 3.12 with the provided patches and "pcie_acs_override=downstream" as a parameter, lsgroup now gives me this output:

$ ./lsgroup.sh 
### Group 0 ###
    00:00.0 Host bridge: Intel Corporation Xeon E3-1200 v3 Processor DRAM Controller (rev 06)
### Group 1 ###
    00:01.0 PCI bridge: Intel Corporation Xeon E3-1200 v3/4th Gen Core Processor PCI Express x16 Controller (rev 06)
### Group 2 ###
    00:02.0 VGA compatible controller: Intel Corporation Xeon E3-1200 v3 Processor Integrated Graphics Controller (rev 06)
### Group 3 ###
    00:03.0 Audio device: Intel Corporation Xeon E3-1200 v3/4th Gen Core Processor HD Audio Controller (rev 06)
### Group 4 ###
    00:14.0 USB controller: Intel Corporation 8 Series/C220 Series Chipset Family USB xHCI (rev 05)
### Group 5 ###
    00:16.0 Communication controller: Intel Corporation 8 Series/C220 Series Chipset Family MEI Controller #1 (rev 04)
### Group 6 ###
    00:19.0 Ethernet controller: Intel Corporation Ethernet Connection I217-V (rev 05)
### Group 7 ###
    00:1a.0 USB controller: Intel Corporation 8 Series/C220 Series Chipset Family USB EHCI #2 (rev 05)
### Group 8 ###
    00:1b.0 Audio device: Intel Corporation 8 Series/C220 Series Chipset High Definition Audio Controller (rev 05)
### Group 9 ###
    00:1c.0 PCI bridge: Intel Corporation 8 Series/C220 Series Chipset Family PCI Express Root Port #1 (rev d5)
### Group 10 ###
    00:1c.2 PCI bridge: Intel Corporation 8 Series/C220 Series Chipset Family PCI Express Root Port #3 (rev d5)
### Group 11 ###
    00:1c.3 PCI bridge: Intel Corporation 8 Series/C220 Series Chipset Family PCI Express Root Port #4 (rev d5)
### Group 12 ###
    00:1c.4 PCI bridge: Intel Corporation 8 Series/C220 Series Chipset Family PCI Express Root Port #5 (rev d5)
### Group 13 ###
    00:1c.5 PCI bridge: Intel Corporation 8 Series/C220 Series Chipset Family PCI Express Root Port #6 (rev d5)
### Group 14 ###
    00:1d.0 USB controller: Intel Corporation 8 Series/C220 Series Chipset Family USB EHCI #1 (rev 05)
### Group 15 ###
    00:1f.0 ISA bridge: Intel Corporation Z87 Express LPC Controller (rev 05)
    00:1f.2 SATA controller: Intel Corporation 8 Series/C220 Series Chipset Family 6-port SATA Controller 1 [AHCI mode] (rev 05)
    00:1f.3 SMBus: Intel Corporation 8 Series/C220 Series Chipset Family SMBus Controller (rev 05)
### Group 16 ###
    01:00.0 VGA compatible controller: Advanced Micro Devices, Inc. [AMD/ATI] Pitcairn PRO [Radeon HD 7850]
    01:00.1 Audio device: Advanced Micro Devices, Inc. [AMD/ATI] Cape Verde/Pitcairn HDMI Audio [Radeon HD 7700/7800 Series]
### Group 17 ###
    03:00.0 SATA controller: ASMedia Technology Inc. ASM1062 Serial ATA Controller (rev 01)
### Group 18 ###
    04:00.0 Ethernet controller: Intel Corporation I211 Gigabit Network Connection (rev 03)
### Group 19 ###
    05:00.0 SATA controller: ASMedia Technology Inc. ASM1062 Serial ATA Controller (rev 01)
### Group 20 ###
    06:00.0 PCI bridge: PLX Technology, Inc. Device 8603 (rev aa)
### Group 21 ###
    07:01.0 PCI bridge: PLX Technology, Inc. Device 8603 (rev aa)
### Group 22 ###
    07:02.0 PCI bridge: PLX Technology, Inc. Device 8603 (rev aa)
### Group 23 ###
    08:00.0 SATA controller: ASMedia Technology Inc. ASM1062 Serial ATA Controller (rev 01)
### Group 24 ###
    09:00.0 USB controller: Renesas Technology Corp. uPD720202 USB 3.0 Host Controller (rev 02)

I can boot the Windows7 image with this. It's working fine as far as I can tell. Thank you so much

I got an other little problem now, because Win7 does not recognize mouse and keyboard on the pcie-controller until I install the drivers for it, I need to passthrough my hosts usb devices.

$ lsusb
Bus 002 Device 002: ID 8087:8000 Intel Corp. 
Bus 002 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub
Bus 001 Device 002: ID 8087:8008 Intel Corp. 
Bus 001 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub
Bus 004 Device 002: ID 174c:3074 ASMedia Technology Inc. 
Bus 004 Device 001: ID 1d6b:0003 Linux Foundation 3.0 root hub
Bus 003 Device 013: ID 046d:c227 Logitech, Inc. G15 Refresh Keyboard
Bus 003 Device 011: ID 046d:c226 Logitech, Inc. G15 Refresh Keyboard
Bus 003 Device 010: ID 1532:0016 Razer USA, Ltd DeathAdder Mouse
Bus 003 Device 009: ID 10d5:1234 Uni Class Technology Co., Ltd 
Bus 003 Device 008: ID 0424:2524 Standard Microsystems Corp. USB MultiSwitch Hub
Bus 003 Device 002: ID 174c:2074 ASMedia Technology Inc. 
Bus 003 Device 007: ID 046d:c227 Logitech, Inc. G15 Refresh Keyboard
Bus 003 Device 006: ID 046d:c226 Logitech, Inc. G15 Refresh Keyboard
Bus 003 Device 005: ID 1532:0016 Razer USA, Ltd DeathAdder Mouse
Bus 003 Device 004: ID 10d5:1234 Uni Class Technology Co., Ltd 
Bus 003 Device 003: ID 0424:2524 Standard Microsystems Corp. USB MultiSwitch Hub
Bus 003 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub

Mouse and keyboard are 2x in this list, because both of the kvm-switch cables are connected to the host.
I want to passthrough the secondary ones, so the line in the startscript looks like this:

USB_DEVICES="host:046d:c226 host:046d:c227 host:1532:0016"

giving me this output+error:

$ ./startWin7.sh
cset: **> CPUSPEC "-4-3" has bad group "-4-3"
$QEMU $BIOS $CPU $MEMORY $QEMU_ARGS $DEV $USB_DEV $NET $SND $STORAGE $EXTRA_ARGS
QEMU 1.6.50 monitor - type 'help' for more information
(qemu) cset: **> shielding not active on system
taskset: Ungültige Option -- 4
Usage: taskset [options] [mask | cpu-list] [pid|cmd [args...]]

Options:
 -a, --all-tasks         operate on all the tasks (threads) for a given pid
 -p, --pid               operate on existing given pid
 -c, --cpu-list          display and specify cpus in list format
 -h, --help              display this help
 -V, --version           output version information

The default behavior is to run a new command:
    taskset 03 sshd -b 1024
You can retrieve the mask of an existing task:
    taskset -p 700
Or set it:
    taskset -p 03 700
List format uses a comma-separated list instead of a mask:
    taskset -pc 0,3,7-11 700
Ranges in list format can take a stride argument:
    e.g. 0-31:2 is equivalent to mask 0x55555555

For more information see taskset(1).
cset: **> shielding not active on system
taskset: Ungültige Option -- 3
Usage: taskset [options] [mask | cpu-list] [pid|cmd [args...]]

Options:
 -a, --all-tasks         operate on all the tasks (threads) for a given pid
 -p, --pid               operate on existing given pid
 -c, --cpu-list          display and specify cpus in list format
 -h, --help              display this help
 -V, --version           output version information

The default behavior is to run a new command:
    taskset 03 sshd -b 1024
You can retrieve the mask of an existing task:
    taskset -p 700
Or set it:
    taskset -p 03 700
List format uses a comma-separated list instead of a mask:
    taskset -pc 0,3,7-11 700
Ranges in list format can take a stride argument:
    e.g. 0-31:2 is equivalent to mask 0x55555555

For more information see taskset(1).
cset: **> shielding not active on system
taskset: Ungültige Option -- 2
Usage: taskset [options] [mask | cpu-list] [pid|cmd [args...]]

Options:
 -a, --all-tasks         operate on all the tasks (threads) for a given pid
 -p, --pid               operate on existing given pid
 -c, --cpu-list          display and specify cpus in list format
 -h, --help              display this help
 -V, --version           output version information

The default behavior is to run a new command:
    taskset 03 sshd -b 1024
You can retrieve the mask of an existing task:
    taskset -p 700
Or set it:
    taskset -p 03 700
List format uses a comma-separated list instead of a mask:
    taskset -pc 0,3,7-11 700
Ranges in list format can take a stride argument:
    e.g. 0-31:2 is equivalent to mask 0x55555555

For more information see taskset(1).
cset: **> shielding not active on system
taskset: Ungültige Option -- 1
Usage: taskset [options] [mask | cpu-list] [pid|cmd [args...]]

Options:
 -a, --all-tasks         operate on all the tasks (threads) for a given pid
 -p, --pid               operate on existing given pid
 -c, --cpu-list          display and specify cpus in list format
 -h, --help              display this help
 -V, --version           output version information

The default behavior is to run a new command:
    taskset 03 sshd -b 1024
You can retrieve the mask of an existing task:
    taskset -p 700
Or set it:
    taskset -p 03 700
List format uses a comma-separated list instead of a mask:
    taskset -pc 0,3,7-11 700
Ranges in list format can take a stride argument:
    e.g. 0-31:2 is equivalent to mask 0x55555555

For more information see taskset(1).
cset: **> shielding not active on system
pid 6715's current affinity list: 0-7
pid 6715's new affinity list: 0
qemu: could not add USB device 'host:046d:c226'
cset: **> shielding not active on system
taskset: failed to get pid 6755's affinity: Kein passender Prozess gefunden
cset: **> shielding not active on system
taskset: failed to get pid 6804's affinity: Kein passender Prozess gefunden
cset: **> shielding not active on system
taskset: failed to get pid 6853's affinity: Kein passender Prozess gefunden
cset: --> deactivating/reseting shielding
cset: **> shielding not active on system
All Done!!

Do I Need to unbind them somehow? Like the PCI devices?
Before I just selected them in virt-manager and they've been passed through without manually unbinding them.
Or is my syntax incorrect?

//edit:
ah, now I see that vendor:device IDs are the same again, like the problem I had with the sata-controller.
Is it possible to be more specific, on which device I want to pass?
virt-manager defines it in this xml structure:

<hostdev mode='subsystem' type='usb' managed='yes'>
      <source>
        <vendor id='0x1532'/>
        <product id='0x0016'/>
        <address bus='3' device='15'/>
      </source>
    </hostdev>

//edit2:
I found this: http://www.linux-kvm.org/page/USB_Host_ … d_to_Guest
I tried changing the startscript from

USB_DEVICES="host:046d:c226 host:046d:c227 host:1532:0016"

to:

USB_DEVICES="usb-host,hostbus=3,hostaddr=10 usb-host,hostbus=3,hostaddr=11 usb-host,hostbus=3,hostaddr=13"

but that did not work either.

Last edited by Norcoen (2013-11-14 19:02:23)

Offline

Pages: Previous 1 … 28 29 30 31 32 … 234 Next

Topic closed

Board footer

Atom topic feed

Arch Linux

#726 2013-11-13 21:45:29

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

#727 2013-11-13 22:21:17

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

#728 2013-11-13 22:37:37

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

#729 2013-11-13 23:25:13

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

#730 2013-11-13 23:29:16

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

#731 2013-11-13 23:33:14

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

#732 2013-11-13 23:36:17

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

#733 2013-11-13 23:50:53

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

#734 2013-11-14 00:05:01

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

#735 2013-11-14 00:40:36

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

#736 2013-11-14 00:48:55

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

#737 2013-11-14 02:16:44

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

#738 2013-11-14 02:22:28

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

#739 2013-11-14 05:32:33

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

#740 2013-11-14 14:16:06

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

#741 2013-11-14 14:20:31

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

#742 2013-11-14 14:36:42

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

#743 2013-11-14 14:46:53

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

#744 2013-11-14 15:06:42

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

#745 2013-11-14 15:07:02

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

#746 2013-11-14 16:49:11

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

#747 2013-11-14 16:52:13

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

#748 2013-11-14 16:58:25

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

#749 2013-11-14 16:59:58

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

#750 2013-11-14 18:41:29

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Board footer