Arch Linux

aw

Member

Registered: 2013-10-04

Posts: 921

Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

I also just tried assigning USB3 controllers and they just work for me.  I tried both an onboard Etron EJ168 and a plugin Renesas uPD720200.  Windows 7 guest, drivers installed, seems to work with an old USB thumb drive.  I picked up the Renesas card at Best Buy - http://www.rocketfishproducts.com/produ … 2USB3.html http://www.bestbuy.com/site/2-port-usb- … /1092716.p YMMV

---

http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

aw

Member

Registered: 2013-10-04

Posts: 921

Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

#!/bin/bash
QEMU_PA_SAMPLES=128 QEMU_AUDIO_DRV=pa qemu-system-x86_64 -enable-kvm -cpu qemu64 -M q35 -m 8192 \
-smp 2,sockets=1,cores=2,threads=1 \
-bios /usr/share/qemu/bios.bin \
-vga none \
-monitor stdio \
-nographic \
-device ahci,bus=pcie.0,id=ahci \
-drive file=/dev/bcache/by-uuid/d79f9dd0-c884-42f7-b5a9-79db832ea9d0,id=disk,format=raw \
-device ide-hd,bus=ahci.0,drive=disk \
-device ioh3420,bus=pcie.0,addr=1c.0,multifunction=on,port=1,chassis=1,id=root.1 \
-device vfio-pci,host=01:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on,rombar=0,romfile=/home/myuser/Tahiti.rom \
-device vfio-pci,host=01:00.1,bus=pcie.0,multifunction=on \
-device ich9-intel-hda,bus=pcie.0,addr=1b.0,id=sound0 \
-device hda-duplex,id=sound0-codec0,bus=sound0.0,cad=0 \
-usb -usbdevice host:04fc:05d8 \
-boot menu=on

rombar=0, romfile=<something> are conflicting.  The first tells it not to expose a ROM, the second tells it what to expose for the ROM.  I suppose maybe rombar=0 is getting ignored if you're getting output.

I can see the BIOS output and Windows starting animation on the monitor connected to the Radeon, but as soon as Catalyst tries to initialize the card, I get a BSOD with atikmpag.sys 116 error.
Additionally, after Seabios initializes the 7870, the color palette in Gnome running on the 7660D is totally messed up (I can to post a photo of the screen, if that could be any help). Qemu monitor is not showing any errors, but I get

I've seen color palette corruption on AMD as well, even without AMD primary display IIRC.

the follwing entries in journal:

[ 2204.807168] vfio-pci 0000:01:00.0: enabling device (0000 -> 0003)
[ 2204.831486] vfio_ecap_init: 0000:01:00.0 hiding ecap 0x19@0x270
[ 2204.831500] vfio_ecap_init: 0000:01:00.0 hiding ecap 0x1b@0x2d0
[ 2204.873898] vfio-pci 0000:01:00.1: enabling device (0000 -> 0002)
[ 2246.082840] kvm: zapping shadow pages for mmio generation wraparound
[ 2248.441789] AMD-Vi: Event logged [IO_PAGE_FAULT device=01:00.1 domain=0x0002 address=0x00000000857fc000 flags=0x0010]
[ 2248.441798] AMD-Vi: Event logged [IO_PAGE_FAULT device=01:00.1 domain=0x0002 address=0x00000000857fc800 flags=0x0010]
[ 2248.441801] AMD-Vi: Event logged [IO_PAGE_FAULT device=01:00.1 domain=0x0002 address=0x00000000857fc800 flags=0x0030]
[ 2248.441803] AMD-Vi: Event logged [IO_PAGE_FAULT device=01:00.1 domain=0x0002 address=0x00000000857fc800 flags=0x0010]
[ 2248.441805] AMD-Vi: Event logged [IO_PAGE_FAULT device=01:00.1 domain=0x0002 address=0x00000000857fc810 flags=0x0030]
[ 2248.441807] AMD-Vi: Event logged [IO_PAGE_FAULT device=01:00.1 domain=0x0002 address=0x00000000857fc800 flags=0x0010]
[ 2248.441809] AMD-Vi: Event logged [IO_PAGE_FAULT device=01:00.1 domain=0x0002 address=0x00000000857fc810 flags=0x0030]

The errors are from your audio card.  What about not assigning it at all?

---

http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

peaquino

Member

Registered: 2013-12-09

Posts: 5

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

The following script doesn't change much: the pagefaults are occuring on 01:00.0 device (main GPU) now (I still get a BSOD on Windows startup).

#!/bin/bash
QEMU_PA_SAMPLES=128 QEMU_AUDIO_DRV=pa qemu-system-x86_64 -enable-kvm -cpu qemu64 -M q35 -m 8192 \
-smp 2,sockets=1,cores=2,threads=1 \
-bios /usr/share/qemu/bios.bin \
-vga none \
-monitor stdio \
-nographic \
-device ahci,bus=pcie.0,id=ahci \
-drive file=/dev/bcache/by-uuid/d79f9dd0-c884-42f7-b5a9-79db832ea9d0,id=disk,format=raw \
-device ide-hd,bus=ahci.0,drive=disk \
-device ioh3420,bus=pcie.0,addr=1c.0,multifunction=on,port=1,chassis=1,id=root.1 \
-device vfio-pci,host=01:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on,romfile=/home/myuser/Tahiti.rom \
-device ich9-intel-hda,bus=pcie.0,addr=1b.0,id=sound0 \
-device hda-duplex,id=sound0-codec0,bus=sound0.0,cad=0 \
-usb -usbdevice host:04fc:05d8 \
-boot menu=on

rombar=0, romfile=<something> are conflicting.  The first tells it not to expose a ROM, the second tells it what to expose for the ROM.  I suppose maybe rombar=0 is getting ignored if you're getting output.

Thanks, I corrected that in the script.

andy123

Member

Registered: 2011-11-04

Posts: 169

Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

peaquino, your errors seem to be occuring on 01:00.1 not 01:00.0, which is your audio card.

[ 2248.441807] AMD-Vi: Event logged [IO_PAGE_FAULT device=01:00.1 domain=0x0002 address=0x00000000857fc800 flags=0x0010]
01:00.1 Audio device: Advanced Micro Devices, Inc. [AMD/ATI] Tahiti XT HDMI Audio [Radeon HD 7970 Series]

is there are reason why you aren't assigning it? (like so)

-device vfio-pci,host=01:00.0,bus=root.1,addr=0.0,multifunction=on,x-vga=on,romfile=/home/myuser/Tahiti.rom \
-device vfio-pci,host=01:00.1,bus=root.1,addr=0.1 \

---

i'm sorry for my poor english wirting skills…

aw

Member

Registered: 2013-10-04

Posts: 921

Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

peaquino, your errors seem to be occuring on 01:00.1 not 01:00.0, which is your audio card.

[ 2248.441807] AMD-Vi: Event logged [IO_PAGE_FAULT device=01:00.1 domain=0x0002 address=0x00000000857fc800 flags=0x0010]
01:00.1 Audio device: Advanced Micro Devices, Inc. [AMD/ATI] Tahiti XT HDMI Audio [Radeon HD 7970 Series]

is there are reason why you aren't assigning it? (like so)

-device vfio-pci,host=01:00.0,bus=root.1,addr=0.0,multifunction=on,x-vga=on,romfile=/home/myuser/Tahiti.rom \
-device vfio-pci,host=01:00.1,bus=root.1,addr=0.1 \

Numerous folks have reported BSODs which were solved by moving the audio device to the root complex rather than exposing it as a sub-function.  It may work, but exposing it on pcie.0 has had more success it seems.

---

http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

peaquino

Member

Registered: 2013-12-09

Posts: 5

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

peaquino, your errors seem to be occuring on 01:00.1 not 01:00.0, which is your audio card.

[ 2248.441807] AMD-Vi: Event logged [IO_PAGE_FAULT device=01:00.1 domain=0x0002 address=0x00000000857fc800 flags=0x0010]
01:00.1 Audio device: Advanced Micro Devices, Inc. [AMD/ATI] Tahiti XT HDMI Audio [Radeon HD 7970 Series]

is there are reason why you aren't assigning it? (like so)

-device vfio-pci,host=01:00.0,bus=root.1,addr=0.0,multifunction=on,x-vga=on,romfile=/home/myuser/Tahiti.rom \
-device vfio-pci,host=01:00.1,bus=root.1,addr=0.1 \

Numerous folks have reported BSODs which were solved by moving the audio device to the root complex rather than exposing it as a sub-function.  It may work, but exposing it on pcie.0 has had more success it seems.

That's what I read in th this thread -- I tried both options (exposing the soundcard as subdevice and as separate unit) -- none worked. Not passing the soundcard is not making any visible differences: instad of

AMD-Vi: Event logged [IO_PAGE_FAULT device=01:00.1 domain=0x0002 ...

I get

AMD-Vi: Event logged [IO_PAGE_FAULT device=01:00.0 domain=0x0002 ...

Sometimes the host reboots even before it has a chance to write theses errors to journal.

Moments ago I tried with pci-assign method and Windows Blue Preview: it doesn't BSOD, but the passed 7870 is not turning on and has a yellow exlamantion mark in device manager (Code 10 - if I remember correctly).

Still, thanks a lot! I *really* appreciate your help.

nbhs

Member

From: Montevideo, Uruguay

Registered: 2013-05-02

Posts: 402

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

I cant get pass the step from unbinding step. Everytime i execute the script or do

echo "0000:02:00.0" > /sys/bus/pci/devices/0000:02:00.0/driver/unbind

my system freezes and i have to switch it of holding the power button.

I have a lenovo y500 with two geforce gt650m cards (SLI).
Im using the nouveau driver. Can anyone help?

You'll probably have to use pci-stub or blacklist the nouveau driver

alphahere

Member

Registered: 2013-10-12

Posts: 7

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

I also just tried assigning USB3 controllers and they just work for me.  I tried both an onboard Etron EJ168 and a plugin Renesas uPD720200.  Windows 7 guest, drivers installed, seems to work with an old USB thumb drive.  I picked up the Renesas card at Best Buy - http://www.rocketfishproducts.com/produ … 2USB3.html http://www.bestbuy.com/site/2-port-usb- … /1092716.p YMMV

Thanks also to teekay for reporting that it does work with nec (renasas) usb 3.0 cards. The card does work fine under linux and windows natively.

Anyway I have wasted most of yesterday trying to figure this out and it must be my setup. Will try to compile everything again and see if I can figure it out. I can get a little crazy trying to get stuff to work.

Even the intel controller usb 3.0 on the motherboard does not work with qemu. It must be my setup because the default xhci driver does not even work anymore. This is the default built in usb 3.0 driver in qemu and it does not work!

-device nec-usb-xhci,id=xhci

This comes up as unknown device under windows 7.  I know I had this built in xhci driver working once.

Any hints at what to look for would be greatly apperciated.

Fun times!

Thanks

keatsfonam

Member

Registered: 2013-03-28

Posts: 3

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

EDIT2:

All problems solved by avoiding letting vfio-pci take the device from pci-stub upon starting the vm with the script provided.  Thanks

What exactly does that mean?

BTW, as of this morning everything except i915 vga arbiter stuff is in upstream, kernel and qemu.

Not executing vfio-bind and just starting the vm while the pci-stub is loaded is what I meant.

ilya80

Member

Registered: 2013-11-12

Posts: 34

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Not executing vfio-bind and just starting the vm while the pci-stub is loaded is what I meant.

Now that sounds really strange.

ddjdx

Member

Registered: 2013-07-16

Posts: 3

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Hi All. Encountering an issue that I cant seem to get a hold of a solution for.

I have installed the 3 packages in the OP, Kernel with default configuration.

I'm able to use pci-stub on my secondary AMD Radeon 5450 adapter and the VIFO-BIND Script appears to work.

When I start up the test Qemu, the screen that is plugged into the passthrough adapter comes out of standby but is blank. And a journalctl -r reveals the following :

Dec 11 14:28:05 tux kernel: ------------[ cut here ]------------
Dec 11 14:28:05 tux kernel: ---[ end trace 59e460fd7afe9fe2 ]---
Dec 11 14:28:05 tux kernel:  [<ffffffff815053ad>] system_call_fastpath+0x1a/0x1f
Dec 11 14:28:05 tux kernel:  [<ffffffff811b60c1>] SyS_ioctl+0x81/0xa0
Dec 11 14:28:05 tux kernel:  [<ffffffff811b5e60>] do_vfs_ioctl+0x2e0/0x4c0
Dec 11 14:28:05 tux kernel:  [<ffffffffa1042ea7>] vfio_fops_unl_ioctl+0x77/0x340 [vfio]
Dec 11 14:28:05 tux kernel:  [<ffffffffa1055439>] vfio_iommu_type1_ioctl+0x219/0x290 [vfio_iommu_type1]
Dec 11 14:28:05 tux kernel:  [<ffffffffa1054bc9>] vfio_dma_do_map+0x209/0x860 [vfio_iommu_type1]
Dec 11 14:28:05 tux kernel:  [<ffffffff813dab66>] iommu_map+0xe6/0x220
Dec 11 14:28:05 tux kernel:  [<ffffffff813e5f95>] intel_iommu_map+0x75/0xa0
Dec 11 14:28:05 tux kernel:  [<ffffffff813e5ea7>] __domain_mapping+0x3a7/0x420
Dec 11 14:28:05 tux kernel:  [<ffffffff81061c8a>] warn_slowpath_null+0x1a/0x20
Dec 11 14:28:05 tux kernel:  [<ffffffff81061bad>] warn_slowpath_common+0x7d/0xa0
Dec 11 14:28:05 tux kernel:  [<ffffffff814f6f10>] dump_stack+0x4d/0x6f
Dec 11 14:28:05 tux kernel: Call Trace:
Dec 11 14:28:05 tux kernel:  ffff8800ac626000 0000000000000201 0000000000000200 ffff880bea403c90
Dec 11 14:28:05 tux kernel:  ffff880bea403c80 ffffffff81061bad 0000000000000200 0000000be2200083
Dec 11 14:28:05 tux kernel:  0000000000000009 ffff880bea403c48 ffffffff814f6f10 0000000000000000
Dec 11 14:28:05 tux kernel: Hardware name: Gigabyte Technology Co., Ltd. To be filled by O.E.M./X79-UP4, BIOS F1 07/13/2012
Dec 11 14:28:05 tux kernel: CPU: 5 PID: 1246 Comm: qemu-system-x86 Not tainted 3.13.0-1-mainline #1
Dec 11 14:28:05 tux kernel:  snd_usb_audio evdev snd_usbmidi_lib snd_rawmidi dvb_core snd_seq_device rc_core sb_edac edac_core i2c_i801 radeon lpc_ich ttm drm_kms_helper drm i2c_algo_bi
Dec 11 14:28:05 tux kernel: Modules linked in: vfio_pci vfio_iommu_type1 vfio tun ipt_MASQUERADE iptable_nat nf_nat_ipv4 nf_nat nf_conntrack_ipv4 nf_defrag_ipv4 xt_conntrack nf_conntrac
Dec 11 14:28:05 tux kernel: WARNING: CPU: 5 PID: 1246 at drivers/iommu/intel-iommu.c:1860 __domain_mapping+0x3a7/0x420()
Dec 11 14:28:05 tux kernel: ------------[ cut here ]------------
Dec 11 14:28:05 tux kernel: ERROR: DMA PTE for vPFN 0x0 already set (to bec52f003 not be2200083)
Dec 11 14:28:03 tux kernel: vfio-pci 0000:03:00.0: enabling device (0000 -> 0003)

The stack trace is repeated over 100 times and eventually stops with Qemu still running and the secondary screen blank.

I'm not sure what to look at next as the stack trace doesn't seem to tell me anything. Though this line caught my eye:

Dec 11 14:28:05 tux kernel: ERROR: DMA PTE for vPFN 0x0 already set (to bec52f003 not be2200083)

But not sure what it means?

EDIT-
Tinkered around some more. Blacklisting the Radeon driver allows SeaBios to appear on the passthrough card. I'm assuming its because the Radeon driver is connecting to the card first before PCI-stub loads. I'm sure that can be fixed by compiling pci-stub into the kernel.But I'm,still getting 100's of the stack traces listed above. So I'm hesitant to go furthur and install an OS.

I've tried the following:
Disabling off-chipset Marvell SATA controller (Its known to cause problems with IOMMU), No difference.
Disabled Interrupt Mapping in the BIOS and enabled the workaround in the OP. No Difference
Disbaled all other Virtulisation settings in the bios except for VT-D and VT and enabled the Interrupt Mapping work around. No Difference.

Last edited by ddjdx (2013-12-11 08:54:02)

Owen77

Member

Registered: 2013-12-12

Posts: 7

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Thanks for the excellent guide, I've followed it through but having no luck with kernel 3.12 + qemu 1.7.
Does anyone know possible causes for vfio reporting the device does not support x-vga?
This is a headless server and no display drivers are loaded, nouveau is blacklisted

qemu-system-x86_64 -enable-kvm -M q35 -m 1024 -cpu host -smp 6,sockets=1,cores=6,threads=1 -bios /usr/share/qemu/bios.bin -vga none -device ioh3420,bus=pcie.0,addr=1c.0,multifunction=on,port=1,chassis=1,id=root.1 -device vfio-pci,host=04:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on -device vfio-pci,host=04:00.1,bus=root.1,addr=00.1

qemu-system-x86_64: -device vfio-pci,host=04:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on: vfio: Device does not support requested feature x-vga
qemu-system-x86_64: -device vfio-pci,host=04:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on: vfio: failed to get device 0000:04:00.0
qemu-system-x86_64: -device vfio-pci,host=04:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on: Device initialization failed.
qemu-system-x86_64: -device vfio-pci,host=04:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on: Device 'vfio-pci' could not be initialized

lspci | grep NVIDIA
04:00.0 3D controller: NVIDIA Corporation GF100GL [Tesla M2070-Q] (rev a3)
04:00.1 Audio device: NVIDIA Corporation GF100 High Definition Audio Controller (rev a1)

aw

Member

Registered: 2013-10-04

Posts: 921

Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Thanks for the excellent guide, I've followed it through but having no luck with kernel 3.12 + qemu 1.7.
Does anyone know possible causes for vfio reporting the device does not support x-vga?

grep VFIO from your kernel config and it might be pretty obvious.

---

http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Owen77

Member

Registered: 2013-12-12

Posts: 7

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Thanks for the excellent guide, I've followed it through but having no luck with kernel 3.12 + qemu 1.7.
Does anyone know possible causes for vfio reporting the device does not support x-vga?

grep VFIO from your kernel config and it might be pretty obvious.

cat /boot/config-3.12.0-031200-generic | grep VFIO
CONFIG_VFIO_IOMMU_TYPE1=m
CONFIG_VFIO=m
CONFIG_VFIO_PCI=m
CONFIG_VFIO_PCI_VGA=y

lsmod | grep vfio
vfio_pci               37175  0
vfio_iommu_type1       17885  0
vfio                   21113  2 vfio_pci,vfio_iommu_type1

Looks ok to me, anything I'm missing?

A colleague has just suggested x-vga may not be necessary for a headless config, would I be able to still access the NVidia card and install drivers to use CUDA for example with standard pci pass-through?

ilya80

Member

Registered: 2013-11-12

Posts: 34

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

cat /boot/config-3.12.0-031200-generic | grep VFIO
CONFIG_VFIO_IOMMU_TYPE1=m
CONFIG_VFIO=m
CONFIG_VFIO_PCI=m
CONFIG_VFIO_PCI_VGA=y

Why not compile VFIO support into the kernel instead of building as module?

Owen77

Member

Registered: 2013-12-12

Posts: 7

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Why not compile VFIO support into the kernel instead of building as module?

That's how it came.
It's an Ubuntu machine (trying to get this working without replacing the OS if possible) and I took the lazy option of installing a pre-built 3.12 kernel package.

ilya80

Member

Registered: 2013-11-12

Posts: 34

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Why not compile VFIO support into the kernel instead of building as module?

That's how it came.
It's an Ubuntu machine (trying to get this working without replacing the OS if possible) and I took the lazy option of installing a pre-built 3.12 kernel package.

I believe you need both upstream kernel and upstream qemu, prebuilt package wont do it for you.

Owen77

Member

Registered: 2013-12-12

Posts: 7

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

I believe you need both upstream kernel and upstream qemu, prebuilt package wont do it for you.

Ok, well I'm using  qemu 1.7 built from source (originally using 1.5 from a package).
I'll build the kernel if necessary but according to this post https://bbs.archlinux.org/viewtopic.php … 9#p1304619 packages should work ok.

aw

Member

Registered: 2013-10-04

Posts: 921

Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Thanks for the excellent guide, I've followed it through but having no luck with kernel 3.12 + qemu 1.7.
Does anyone know possible causes for vfio reporting the device does not support x-vga?

grep VFIO from your kernel config and it might be pretty obvious.

cat /boot/config-3.12.0-031200-generic | grep VFIO
CONFIG_VFIO_IOMMU_TYPE1=m
CONFIG_VFIO=m
CONFIG_VFIO_PCI=m
CONFIG_VFIO_PCI_VGA=y

lsmod | grep vfio
vfio_pci               37175  0
vfio_iommu_type1       17885  0
vfio                   21113  2 vfio_pci,vfio_iommu_type1

Looks ok to me, anything I'm missing?

A colleague has just suggested x-vga may not be necessary for a headless config, would I be able to still access the NVidia card and install drivers to use CUDA for example with standard pci pass-through?

CONFIG_VFIO_PCI_VGA is what I was looking for since I think you would have gotten the same error message if it was disabled (there's no need to build VFIO statically into the kernel, that's not the problem)  The problem though is that your Telsa card doesn't support VGA, note that the PCI class code is "3D controller" rather than "VGA compatible controller".  VFIO only enables VGA access to devices that claim to support VGA per the PCI spec.  Your Tesla is handled as if you had asked to enable VGA access to an Ethernet card.

Nvidia does support Tesla, Quadro, and GRID in virtualized environments, but only the K-series of each.  The supported configuration is to assign the GPU as a secondary device to the emulated VGA.  There's no need to use the x-vga option in the case.  For Quadro and GRID cards the Nvidia device becomes the primary display device when the Nvidia driver stack loads.  I don't know if this would work with your card, but VGA doesn't seem to be an option.  Also note that 3.12/1.7 is not sufficient, there are changes in both kernel and qemu that went in after those releases.

---

http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Owen77

Member

Registered: 2013-12-12

Posts: 7

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

CONFIG_VFIO_PCI_VGA is what I was looking for since I think you would have gotten the same error message if it was disabled (there's no need to build VFIO statically into the kernel, that's not the problem)  The problem though is that your Telsa card doesn't support VGA, note that the PCI class code is "3D controller" rather than "VGA compatible controller".  VFIO only enables VGA access to devices that claim to support VGA per the PCI spec.  Your Tesla is handled as if you had asked to enable VGA access to an Ethernet card.

Nvidia does support Tesla, Quadro, and GRID in virtualized environments, but only the K-series of each.  The supported configuration is to assign the GPU as a secondary device to the emulated VGA.  There's no need to use the x-vga option in the case.  For Quadro and GRID cards the Nvidia device becomes the primary display device when the Nvidia driver stack loads.  I don't know if this would work with your card, but VGA doesn't seem to be an option.  Also note that 3.12/1.7 is not sufficient, there are changes in both kernel and qemu that went in after those releases.

Aha, well that explains it. The M2070-Q adds the Quadro features to the M2070 and they advertise it as able to do remote visualisation, so maybe the second technique could work with this card.

I won't clutter up this thread with any more about this as it's clearly a different setup and it looks like we might need newer hardware but if you have any references about the latter option to speed up my research you could share I'd greatly appreciate it, thanks for the help so far.
(For disclosure this is a publicly funded research project and all our work will be written up and posted online for community use as we get things running.)

Last edited by Owen77 (2013-12-12 23:57:55)

aw

Member

Registered: 2013-10-04

Posts: 921

Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

CONFIG_VFIO_PCI_VGA is what I was looking for since I think you would have gotten the same error message if it was disabled (there's no need to build VFIO statically into the kernel, that's not the problem)  The problem though is that your Telsa card doesn't support VGA, note that the PCI class code is "3D controller" rather than "VGA compatible controller".  VFIO only enables VGA access to devices that claim to support VGA per the PCI spec.  Your Tesla is handled as if you had asked to enable VGA access to an Ethernet card.

Nvidia does support Tesla, Quadro, and GRID in virtualized environments, but only the K-series of each.  The supported configuration is to assign the GPU as a secondary device to the emulated VGA.  There's no need to use the x-vga option in the case.  For Quadro and GRID cards the Nvidia device becomes the primary display device when the Nvidia driver stack loads.  I don't know if this would work with your card, but VGA doesn't seem to be an option.  Also note that 3.12/1.7 is not sufficient, there are changes in both kernel and qemu that went in after those releases.

Aha, well that explains it. The M2070-Q adds the Quadro features to the M2070 and they advertise it as able to do remote visualisation, so maybe the second technique could work with this card.

I won't clutter up this thread with any more about this as it's clearly a different setup and it looks like we might need newer hardware but if you have any references about the latter option to speed up my research you could share I'd greatly appreciate it, thanks for the help so far.
(For disclosure this is a publicly funded research project and all our work will be written up and posted online for community use as we get things running.)

There's certainly a chance that assigning the GPU as a secondary device with emulated VGA could work, try it and see.  Do it just like you would if it's not a graphics device.  If it behaves like Quadro/GRID the display will switch once the Nvidia driver stack loads, so be sure you can get to the desktop over the network via VNC or some other remote access solution first.

---

http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

TripleSpeeder

Member

Registered: 2011-05-02

Posts: 47

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

So what's the status of KVM and VGA passthrough - Are there still specific patches necessary or is this now working out-of-the-box? I just installed Win7 in a VM, passing through my HD7880 and it seemed to work fine. After initial windows install and reboot I installed latest Catalyst, rebooted and it just works I did not do any performance tests so far, just wondering if in the meantime all relevant changes are already in the standard packages or anything still needs to be done?

firewing1

Member

Registered: 2013-12-17

Posts: 9

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

VGA passthrough is working well for me with a Radeon R9 270X (i.e. a rebranded 7870 GHz Edition), however after installing the latest Catalyst driver version 13.11 beta 9.5 on Windows 7 SP1 I am having trouble booting Windows.

I see 'Starting Windows...' with the windows splash screen animation but then the screen goes black and after a bit the monitor loses signal. In Safe mode, the event log shows error 0x00000116 - MSDN says this: "The VIDEO_TDR_ ERROR bug check has a value of 0x00000116. This indicates that an attempt to reset the display driver and recover from a timeout failed."

I have tried so moving the HDMI audio device (ID 0000:01:00.1) to pcie.0 instead of root.1 as well as eliminating passthrough of the HDMI audio device entirely, neither of which helped. I also tried manually specifying the ROM (rombar=0 romfile=...) from a ROM bin I dumped using GPU-Z. I dumped it from within the VM, which I read may not work but it didn't throw any errors and GPU-Z properly detected the card so I think it dumped the VBIOS correctly. Has anyone encountered something similar, or has any ideas on what I could do?

Full disclosure: I am on Fedora 20 using the virt-preview repository so I have kernel 3.12.5 and qemu-kvm 1.7.0. I'm using the same config as the main howto - AHCI, virtualized audio. Here's the command I use:

/usr/bin/qemu-system-x86_64 -machine accel=kvm -name win7 -S -M q35 -m 2048 \
-realtime mlock=off -rtc base=localtime \
-smp 4,sockets=1,cores=4,threads=1 \
-bios /usr/share/qemu/bios.bin -vga none \
-device ioh3420,bus=pcie.0,addr=1c.0,multifunction=on,port=1,chassis=1,id=root.1 \
-device vfio-pci,host=01:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on \
-device vfio-pci,host=01:00.1,bus=root.1,addr=00.1 \
-device ahci,bus=pcie.0,id=ahci \
-drive file=/home/VMs/W7.img,if=none,id=disk,format=qcow2 -device ide-hd,bus=ahci.0,drive=disk \
-drive file=/dev/null,id=isocd -device ide-cd,bus=ahci.1,drive=isocd \
-boot menu=on

VFIO passthrough was normally prior to installing the Catalyst drivers, so my guess is that pass-through is working correctly and the bug is in the machine config or the Windows driver.

Last edited by firewing1 (2013-12-17 05:28:52)

Owen77

Member

Registered: 2013-12-12

Posts: 7

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

There's certainly a chance that assigning the GPU as a secondary device with emulated VGA could work, try it and see.  Do it just like you would if it's not a graphics device.  If it behaves like Quadro/GRID the display will switch once the Nvidia driver stack loads, so be sure you can get to the desktop over the network via VNC or some other remote access solution first.

Hmmm, one last question about my slightly tangential issues in case anyone has any suggestions:

• The Tesla card is being passed through ok, I can login to a guest with the vga drivers and run CUDA or OpenCL programs.

• If the NVIDIA drivers are installed (tried 319.76 and 331.20) with appropriate xorg.conf created, the VNC screen just freezes after it loads the X server, if I then ssh in to the VM there are no errors in Xorg.0.log but X apps can't find a display, no matter what I set $DISPLAY to.

• If I start in runlevel 3 and login then run startx, the VNC screen then goes black but I can ssh in and the X server is now running! I can set $DISPLAY to :0.0 and run glxinfo and glxgears at ludicrous speeds from the terminal but have no way of seeing anything.

aw

Member

Registered: 2013-10-04

Posts: 921

Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

There's certainly a chance that assigning the GPU as a secondary device with emulated VGA could work, try it and see.  Do it just like you would if it's not a graphics device.  If it behaves like Quadro/GRID the display will switch once the Nvidia driver stack loads, so be sure you can get to the desktop over the network via VNC or some other remote access solution first.

Hmmm, one last question about my slightly tangential issues in case anyone has any suggestions:

• The Tesla card is being passed through ok, I can login to a guest with the vga drivers and run CUDA or OpenCL programs.

• If the NVIDIA drivers are installed (tried 319.76 and 331.20) with appropriate xorg.conf created, the VNC screen just freezes after it loads the X server, if I then ssh in to the VM there are no errors in Xorg.0.log but X apps can't find a display, no matter what I set $DISPLAY to.

• If I start in runlevel 3 and login then run startx, the VNC screen then goes black but I can ssh in and the X server is now running! I can set $DISPLAY to :0.0 and run glxinfo and glxgears at ludicrous speeds from the terminal but have no way of seeing anything.

Congratulation, it works.  When the Nvidia driver is used, the emulated VGA is only used for text mode and the GPU is used for graphical mode.  They're mutually exclusive, so it's expected that the emulated VGA "freezes" when X starts.  In your first case, apps can't find any display device, probably because the graphical desktop is sitting at a login prompt.  In the second case, you've started an X session, therefore there is a display to connect to.  This is the way that GRID cards work and Quadro as well if you ignore that it has a physical display connector.  You need to setup remote access to the desktop first.  You can run a vnc server in the guest or use some other remote'ing solution.  If you were expecting X on the emulated VGA with rendering in the GPU, that doesn't happen.

---

http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users