Arch Linux

tecnologic

Member

Registered: 2014-02-02

Posts: 12

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Ok stupid question, how do i pull the patch? Clone Kernel repo and then?

sry i'am not used to such deep linux things, but wanna get in to it. i'am more at home on embedded microprocessors.

tecnologic

Member

Registered: 2014-02-02

Posts: 12

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

ok found a discription for git apply.

are there any problems with the 3.14rc1? if not i'll take that to try my luck on this one

aw

Member

Registered: 2013-10-04

Posts: 921

Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

ok found a discription for git apply.

are there any problems with the 3.14rc1? if not i'll take that to try my luck on this one

v3.13 is likely a more stable target

---

http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

tecnologic

Member

Registered: 2014-02-02

Posts: 12

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Ok i'll use that. but this will take some time i need to read about this patching. how to apply your links to a branch

Flyser

Member

Registered: 2013-12-19

Posts: 29

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Has anyone found a way to reliably rebind usb devices to the host after the VM is shut down?
I tried to

echo $bus-$port > /sys/bus/usb/drivers_probe

but it does not seem work. Any ideas?

tecnologic

Member

Registered: 2014-02-02

Posts: 12

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Maybe it helps to eject the devices when shutting down the VM? some pages before there was an .exe mentiont to eject GPU maybe it works for USB parts to.

Flyser

Member

Registered: 2013-12-19

Posts: 29

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

I can't test it right now, but I don't think one can eject a usb keyboard?

toma222

Member

From: France

Registered: 2007-05-17

Posts: 7

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Hello,

I used my Radeon 7850 in a Windows 7/8/8.1 guest with pci-assign for months (with the only issue of the host crash in case of guest reboot).

Now, I want to use it with a Linux guest (Ubuntu) with vfio-pci, to make a kind of virtual Steam Machine + XBMC.

I have tried :
- secondary pasthrough (with a cirrus emulated graphic card) with vfio-pci : the guest start, the Radeon is detected and I have the display on the HDMI output, Unity start, I can play video, but when I try to use 3D (for example with fglrxinfo or Steam), I have only segmentation fault.
- primary passtrough with x-vga=on : I have built the 3.13 kernel (with options and patches from first post) and qemu from git, but I have exactly the same behaviour.

I have also tried with an old Radeon 6850 in secondary passthrough and vfio-pci, and it work quite well (except  some strange performance in Serious Sam 3).

Do you have an idea why I can't get the Radeon 7850 working normally ?

My host use a Core i7 4770S and is on Debian.

Thanks.

Just to let you know, with the latest commits in qemu.git, I now have my Radeon 7850 working normally in my Linux guest.
It seems that the issue was that before I can't start the guest with CPU model other than qemu64. Now it works with host model and the 3D is Ok.
I have tested secondary passthrough with unpatched kernel 3.13 from Debian and primary passthrough with patched one, and it work great for both (except for the memory not releases when the guest shutdown without the fix_memleak patch).

So thank you Alex for your work and all others for all the information in this thread.

mbroemme

Member

From: Cologne

Registered: 2014-02-05

Posts: 40

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Hi,

I've read this thread now for 8 months or so and I'm the guy who created this crude fix for the NVIDIA binary driver. Just to follow-up. I've already informed NVIDIA in their DevTalk Forum about this issue and it looks that there is some progress since January this year. You can follow it here:

https://devtalk.nvidia.com/default/topi … ation-lock

Anybody who needs this fixed should reply to it so that NVIDIA sees importance for it. Another thing that somebody here might observed and found a solution already is multi GPU passthrough to a single VM. I've asked already on qemu-devel and maybe Alex is reading it soon. I got multi GPU passthrough only working with Linux VMs for now but not with Windows. You can follow it here:

http://lists.nongnu.org/archive/html/qe … 00753.html

Any help or investigation to get it working is much appreciated.

--Maik

andy123

Member

Registered: 2011-11-04

Posts: 169

Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Hi,

I've read this thread now for 8 months or so and I'm the guy who created this crude fix for the NVIDIA binary driver. Just to follow-up. I've already informed NVIDIA in their DevTalk Forum about this issue and it looks that there is some progress since January this year. You can follow it here:

https://devtalk.nvidia.com/default/topi … ation-lock

Anybody who needs this fixed should reply to it so that NVIDIA sees importance for it. Another thing that somebody here might observed and found a solution already is multi GPU passthrough to a single VM. I've asked already on qemu-devel and maybe Alex is reading it soon. I got multi GPU passthrough only working with Linux VMs for now but not with Windows. You can follow it here:

http://lists.nongnu.org/archive/html/qe … 00753.html

Any help or investigation to get it working is much appreciated.

--Maik

Hi, thanks for creating this patch. I'm ajs124 on the nvidia forum. Seems like you failed to respond to the Nvidia guy, too. They have a strange notification system over there, so it took me months to follow up on his request for the nvidia-bug-report.log.gz and steps to reproduce the issue.

More replies/traffic over at nvidia devtalk from the people using the patch seems like a good idea to increase visibilty and maybe have this fixed upstream at some point.

I think I was credited for this patch in the original post for some reason. Maybe nbhs should change my name there to yours ^^

Last edited by andy123 (2014-02-05 20:07:20)

---

i'm sorry for my poor english wirting skills…

teekay

Member

Registered: 2011-10-26

Posts: 271

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

@mbroemme: responded there. Thanks a hundret times for the patch. W/o it my setup wouldn't work.

@andy123: something completely different. I rember you also have a Gigabyte board with funny Marvell 88SE91xx controller(s). See https://bugzilla.kernel.org/show_bug.cgi?id=42679#c30 for a patch and check if it helps. Andrew needs some more AMD testers. With that patch my controllers are fully usable on the host itself, and passthrough also works fine (though I meanwhile use a ZFS ZVOL for the VM rootfs).

mbroemme

Member

From: Cologne

Registered: 2014-02-05

Posts: 40

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

@mbroemme: responded there. Thanks a hundret times for the patch. W/o it my setup wouldn't work.

Thanks, but honestly most thanks go to aw who points me to the right direction on what to do.

mbroemme

Member

From: Cologne

Registered: 2014-02-05

Posts: 40

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

I was also able to retrieve the following output by running 'cat /proc/kmsg' during the crash

<6>[  996.181525] vfio-pci 0000:02:00.0: enabling device (0000 -> 0003)
<6>[  996.207159] vfio_ecap_init: 0000:02:00.0 hiding ecap 0x19@0x270
<6>[  998.065072] vgaarb: device changed decodes: PCI:0000:01:00.0,olddecodes=io+mem,decodes=none:owns=io+mem
<6>[  998.065076] vgaarb: transferring owner from PCI:0000:01:00.0 to PCI:0000:02:00.0
<3>[  998.095398] ata3: illegal qc_active transition (00000001->97b282a8)
<3>[  998.095540] ata3.00: exception Emask 0x2 SAct 0x0 SErr 0xd0000000 action 0x6 frozen
<3>[  998.095567] ata3: SError: { }
<3>[  998.095580] ata3.00: failed command: FLUSH CACHE EXT
<3>[  998.095607] ata3.00: cmd ea/00:00:00:00:00/00:00:00:00:00/a0 tag 0
<3>[  998.095607]          res 40/00:00:00:00:00/00:00:00:00:00/a0 Emask 0x2 (HSM violation)
<3>[  998.095637] ata3.00: status: { DRDY }
<6>[  998.095653] ata3: hard resetting link
<4>[  998.103575] Clocksource tsc unstable (delta = -147227892866 ns)

Thanks again for your help!

I had the exact same issue and I observed the same if I changed the NVIDIA card with an AMD using radeon driver. The solution for me was very simple but took me a couple of days to figure out.

After changing the default VGA device to use in BIOS of my GA-990FX-UD3 to the device which will be assigned to the VM later it started to work. So currently I do the following: I boot system via VGA on AMD Radeon card, bind it during boot via udev (before loading nouveau or radeon module) to VFIO and if nouveau or radeon is loaded it enables my secondary device which will be used for host system. After that I'm able to passthrough the AMD card (used for booting host) to a VM.

nbhs

Member

From: Montevideo, Uruguay

Registered: 2013-05-02

Posts: 402

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

...
I think I was credited for this patch in the original post for some reason. Maybe nbhs should change my name there to yours ^^

Actually i was crediting you for the link, but i also linked the original conversation on qemu-devel, and mbroemme thank you a lot for this patch

Tom_B

Member

Registered: 2014-02-05

Posts: 1

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Thanks for this awesome topic, I've been following this thread for a while and and I'm planning a system build that allows VGA passthrough but I have some questions:

1) Can you pass through an nvidia 770 or 780, there seems to be contradictory information on this as most places say you need to mod it into a quadro yet the front of this topic suggests otherwise
2) Can you run gpus from the same manufacturer in both the host and the guest OS? e.g. 2 radeons or 2 geforce cards? As I understand it there is a limitation on the catalyst drivers which means you cant use the official driver in the host, which isn't ideal

nbhs

Member

From: Montevideo, Uruguay

Registered: 2013-05-02

Posts: 402

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Thanks for this awesome topic, I've been following this thread for a while and and I'm planning a system build that allows VGA passthrough but I have some questions:

1) Can you pass through an nvidia 770 or 780, there seems to be contradictory information on this as most places say you need to mod it into a quadro yet the front of this topic suggests otherwise
2) Can you run gpus from the same manufacturer in both the host and the guest OS? e.g. 2 radeons or 2 geforce cards? As I understand it there is a limitation on the catalyst drivers which means you cant use the official driver in the host, which isn't ideal

1) iamtakingiteasy got a geforce 780 GTX working on a vm, also if you look back on the thread there's been a lot of nvidia cards reported working
2) I'm using 2 radeon cards myself, different models though using the open-source drivers, i havent tried the catalyst drivers but some people have reported problems using them on the host

Flyser

Member

Registered: 2013-12-19

Posts: 29

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

1) Can you pass through an nvidia 770 or 780, there seems to be contradictory information on this as most places say you need to mod it into a quadro yet the front of this topic suggests otherwise

The modding info is (or was?) only true for Xen VGA Passthrough as far as I know.

2) Can you run gpus from the same manufacturer in both the host and the guest OS? e.g. 2 radeons or 2 geforce cards? As I understand it there is a limitation on the catalyst drivers which means you cant use the official driver in the host, which isn't ideal

A friend of mine runs two different nvidia cards and it seems to work. An advantage is that you would not need the i915 patch in that case.

tecnologic

Member

Registered: 2014-02-02

Posts: 12

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Hi all,

i made some progress, i saw my first bios screen on the passthrough Geforce. Thanks to aw for the i915 kernel patches. But when i kill qemu due to no boot device, and start it a second time the card does not respond and the host freezes. When i use vrit-manager the passthrough does not funktion, but it resets the card properly.

so I need to look back in the thread i think the reset problem was mentioned before.

andy123

Member

Registered: 2011-11-04

Posts: 169

Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

@mbroemme: responded there. Thanks a hundret times for the patch. W/o it my setup wouldn't work.

@andy123: something completely different. I rember you also have a Gigabyte board with funny Marvell 88SE91xx controller(s). See https://bugzilla.kernel.org/show_bug.cgi?id=42679#c30 for a patch and check if it helps. Andrew needs some more AMD testers. With that patch my controllers are fully usable on the host itself, and passthrough also works fine (though I meanwhile use a ZFS ZVOL for the VM rootfs).

Hey teekay! Thanks for the link, I have an ASRock board, but this patch works for it after adding the PCI-ID of my controller. Once I've done more testing I'll report back to the upstream report.

---

i'm sorry for my poor english wirting skills…

anickname

Member

Registered: 2014-01-07

Posts: 23

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Hi all,
thank you for your time spent to create a working configuration.

I'm following the thread for more than 6 month and I hope, like you, to have a stable passthrough.

On my hardware Intel DQ67SW, i5 2400 I have succeeded to passthrough a radeon 7790 recently.
I also have a LSI 1068E HBA which I want to assign it to a FreeNAS VM.
I tried to load the HBA ROM from file and the card still will not be initialized in the VM.

In ESXi is working ok, but only when the device reset method is D3D0.
Can this be done using KVM ?

In ESXi possible values for the reset method include flr, d3d0, link, bridge, or default.
http://www.vmware.com/files/pdf/techpap … h_host.pdf.

The default setting is described as follows. If a device supports function level reset (FLR), ESXi always uses FLR. If the device does not support FLR, ESXi next defaults to link reset and bus reset in that order. Link reset and bus reset might prevent some devices from being assigned to different virtual machines, or from being assigned between the VMkernel and virtual machines. In the absence of FLR, it is possible to use PCI Power Management capability (D3 to D0 transitions) to trigger a reset. Most of the Intel NICs and various other HBAs support this mode.

I'm thinking to change my motherboard with an Asrock Z77 Professional Fatal1ty.
Does anyone had any luck with this board ?

Thanks.

noctavian

Member

Registered: 2013-07-11

Posts: 17

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

1) Can you pass through an nvidia 770 or 780, there seems to be contradictory information on this as most places say you need to mod it into a quadro yet the front of this topic suggests otherwise

noobman reported that GTX 770 works with secondary passthrough
https://bbs.archlinux.org/viewtopic.php … 4#p1364674

I'm thinking to change my motherboard with an Asrock Z77 Professional Fatal1ty.
Does anyone had any luck with this board ?

cmdr managed to passthrough a Nvidia GTX 470 on this motherboard
https://bbs.archlinux.org/viewtopic.php … 2#p1348932

Also you might want to take a look at
https://docs.google.com/spreadsheet/ccc … _web#gid=0

ilya80

Member

Registered: 2013-11-12

Posts: 34

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

After some time I wanted to give it another try, but still no luck. I`m running Core i5 3470 on Z68 chipset, Debian x64 with compiled 3.12.0 kernel, built qemu & seabios myself with few patches from original post applied ( patches seem to work as intended ). Trying to run GTX680 hardmodded into GRID K2 as a secondary adapter to an emulated vga, still getting Error code 43.

Only new thing i see in kern. log is :

Feb  9 13:45:45 cave-lin kernel: [19833.025719] vfio-pci 0000:01:00.0: irq 57 for MSI/MSI-X

Starting VM like this:

$ sudo qemu-system-x86_64 -enable-kvm -M pc -m 4096 -cpu SandyBridge -smp 3,sockets=1,cores=3,threads=1 -bios /opt/kalujny/kernel/seabios/out/bios.bin -device vfio-pci,host=01:00.0,addr=07.0,multifunction=on,romfile=/opt/kalujny/thrash/PNYGTX6802GB_GRIDK2.rom -device vfio-pci,host=01:00.1,addr=07.1 -drive file=/opt/kalujny/thrash/windows.img,if=none,id=disk,format=raw -device ide-hd,bus=ide.0,unit=0,drive=disk -usbdevice tablet

My next steps would be building a fresh kernel / qemu and probably give up afterwards.

Last edited by ilya80 (2014-02-09 10:00:37)

Kaidax

Member

Registered: 2014-02-09

Posts: 6

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Is there any way to increase KVM performance? Right now I experience atrocious framerates in Borderlands 2 under KVM, it runs at most at 55 fps and drops badly to 25. Dropping resolution to 640x480 and disabling every graphics option doesn't affect FPS at all. But under Xen I easily get 200 fps with the same Windows image! Pinning vcpus and hugepages increases FPS by 1-2, but nowhere near playable. Benchmarks indicate no problems, Cinebench CPU score is 766, 3DMark 11 Physics 9300, WEI 7.8.

I launch mainly with this script on Arch qemu-git, linux 3.13 with memleak and arbiter patches, but I've also tried secondary passthrough with pci-assign and vfio both on Arch and on stock Ubuntu 13.04 libvirt to no avail.

#!/bin/bash
# i440fx
QEMU_AUDIO_DRV=alsa
sudo qemu-system-x86_64 -enable-kvm \
-m 3072 -mem-prealloc -mem-path /dev/hugepages \
-cpu host -smp 8,sockets=1,cores=4,threads=8 \
-bios /usr/share/qemu/bios.bin -vga none \
-device vfio-pci,host=01:00.0,bus=pci.0,x-vga=on \
-device vfio-pci,host=01:00.1,bus=pci.0 \
-drive file=/run/media/abc/7582fbde-8935-4dd7-a6e4-42440830672e/home/nya/w-1.img,id=hd,format=raw,index=0 \
-drive file=/dev/disk/by-label/windowsgames,id=hd2,format=raw,index=1 \
-device ich9-intel-hda,bus=pci.0,addr=1b.0,id=sound0 -device hda-duplex,id=sound0-codec0,bus=sound0.0,cad=0 \
-netdev bridge,br=virbr0,id=hostnet0 -device rtl8139,netdev=hostnet0,id=net0,mac=52:54:00:f7:26:8a,bus=pci.0 \
-usb -usbdevice host:093a:2510 \
"$@"

My specs are:
Intel Core i7-4771
Asrock Z87 Extreme4
8GB RAM
AMD Radeon HD 7870
guest OS - Windows 7 x64

Can anyone post their framerates for comparison? Borderlands 2 is free on steam for today.

Last edited by Kaidax (2014-02-09 13:12:36)

Flyser

Member

Registered: 2013-12-19

Posts: 29

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

-cpu host -smp 8,sockets=1,cores=4,threads=8

That might be your problem. You should use -cpu host -smp 8,sockets=1,cores=4,threads=2 for your CPU.
Maybe using i440fx is slower as well? Why don't you use -machine type=q35,accel=kvm?

-drive file=/run/media/abc/7582fbde-8935-4dd7-a6e4-42440830672e/home/nya/w-1.img,id=hd,format=raw,index=0 \
-drive file=/dev/disk/by-label/windowsgames,id=hd2,format=raw,index=1 \
-netdev bridge,br=virbr0,id=hostnet0 -device rtl8139,netdev=hostnet0,id=net0,mac=52:54:00:f7:26:8a,bus=pci.0 \

Any reason you don't use virtual device drivers? e.g. like so:

-netdev tap,id=net.0,vhost=on,ifname=tap0,script=no,downscript=no -device virtio-net-pci,netdev=net.0 \
-drive id=disk1,file=/dev/sdg,if=none,index=0,media=disk -device virtio-blk-pci,drive=disk1,bootindex=1

It should not boost fps that much, but it should generally be a bit snappier.

Can anyone post their framerates for comparison? Borderlands 2 is free on steam for today.

I didn't measure FPS in detail, but steam in-house streaming reported 20fps at 1080p and highest details on my system.

Kaidax

Member

Registered: 2014-02-09

Posts: 6

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

That might be your problem. You should use -cpu host -smp 8,sockets=1,cores=4,threads=2 for your CPU.

No difference. I've already tried passing arbitrary number of cores, 2, 4, 3, 6, 8. No real difference.

Maybe using i440fx is slower as well? Why don't you use -machine type=q35,accel=kvm?

q35 memory access is generally slower than i440fx. I've tried q35, no difference in FPS, except q35 doesn't even load textures properly, so they stay blurred. Graphics performance is the same between KVM and Xen, unigine heaven scores are perfectly identical, but the game still lags on KVM.

Any reason you don't use virtual device drivers? e.g. like so:

-netdev tap,id=net.0,vhost=on,ifname=tap0,script=no,downscript=no -device virtio-net-pci,netdev=net.0 \
-drive id=disk1,file=/dev/sdg,if=none,index=0,media=disk -device virtio-blk-pci,drive=disk1,bootindex=1

It should not boost fps that much, but it should generally be a bit snappier.

I've used virtio drivers on Ubuntu's virt-manager. No difference.

I didn't measure FPS in detail, but steam in-house streaming reported 20fps at 1080p and highest details on my system.

20 fps? Ouch, that's bad. gtx 660 is capable of much-much more.

Last edited by Kaidax (2014-02-09 15:34:01)