You are not logged in.

#1951 2014-06-04 01:30:24

rabcor
Banned
Registered: 2013-02-09
Posts: 500

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

slis wrote:

hello there

amazin and long topic smile

hope someone can help me...

i got some problems... yesterday i managed to get passthrough running but performance was low... (DOTA2 native 120fps, in kvm 40-50fps, drops to 20fps sometimes jumps to 90fps

my config
i72600
Z77X-UD3H
host igpu
guest 560ti
jessie 13.4.4 + i915patch, acs and debug register


i am also interested in getting SLI running since i have 2 same cards, and is it possible to pass IGPU? so i can have hw video (quicksync) encode since 5xx series lack that and i can't stream 60fps



P.S.
what are proper patches,grub commands and kernel needed for my system?
do i need pci-stub, and vfio-bind, both?
can i pass it with pci-assign?

i can only reboot vm, after shutdown i must reboot host? is that normal? 
Thank you

Strange no one replied sooner tongue

Your setup is nearly identical to mine, same processor, different motherboard, but I have 550-Ti running on host (670 on guest). I don't know about SLI, but in theory if you have two GTX 560s and pass them both through then in theory you should be able to make them work in SLI but you need to leave one card for your host... Also you are most definitely going to need this patch for SLI to have a chance to work (makes nvidia drivers think you're not running a virtual machine) there's a different version of it if you scroll down further but I think yuo should use this one as it's a more surefire way than the other. (You need this if you're using the latest nvidia drivers anyways)

40-50 FPS in dota 2 is bad.  Sounds like you need the debug register patch but you said you have it. Why would you passthrough the igpu? I never managed to get good recordings in windows with any software, but on linux ffmpeg recording and encoding simultaneously was perfect and ffmpeg works in windows too. Learn to use that and you should be good without the igpu (use x264 or x265 format, alternatively vp9) I don't know if iGPU can be passthroughed, but I think even if it is possible it is a bad idea. If you don't see it as "VGA" in lspci don't even bother trying. But iGPUs work very differently from dedicated ones as they rely on your processor (i.e. they need direct access to your physical processor, this not only creates a lot of vulnerabilities but also might not be possible depending on how the system is structured)

Just download the linux-mainline.tar.gz from the OP and apply all the damned patches. You will also need tha VGA_ARB patch for the nvidia proprietary drivers if you have an nvidia card on your host. and don't forget the qemu patch I linked you earlier. Unless you have a card from the same manufacturer as your guest on your host you do not need pci-stub.

aw wrote:
rabcor wrote:

Thanks, but still not working

chown rabcor /dev/vfio/13
chmod 660 /dev/vfio/vfio
chown rabcor /dev/vfio/vfio
ulimit -l 8409088

Note: I had to do chown user on vfio because it denied me permission after running chmod 660 on it.

Ran ulimit -l 8409088 for 8gigs ((8192 + 20) * 1024 as the pdf you linked suggested)

Still get

qemu-system-x86_64: -device vfio-pci,host=02:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on: vfio_dma_map(0x7fe886977d80, 0x0, 0x80000000, 0x7fe750000000) = -12 (Cannot allocate memory)
qemu-system-x86_64: -device vfio-pci,host=02:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on: vfio_dma_map(0x7fe886977d80, 0x100000000, 0x80000000, 0x7fe7d0000000) = -12 (Cannot allocate memory)
qemu-system-x86_64: -device vfio-pci,host=02:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on: vfio: memory listener initialization failed for container
qemu-system-x86_64: -device vfio-pci,host=02:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on: vfio: failed to setup container for group 13
qemu-system-x86_64: -device vfio-pci,host=02:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on: vfio: failed to get group 13
qemu-system-x86_64: -device vfio-pci,host=02:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on: Device initialization failed.
qemu-system-x86_64: -device vfio-pci,host=02:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on: Device 'vfio-pci' could not be initialized

As noted in the slides, an additional 20MB was sufficient for that example config, with an assigned NIC that has much, much less MMIO space.  If you want to make sure things are covered, add an extra 1G.  Also make sure you're setting ulimit in a way that transfers to the user process, 'sudo ulimit ...' is not sufficient.

Thanks! This worked, I think the problem may have been that I needed to use su, run the ulimit command then run su again to go back into my own terminal rather than just type exit. The problem now would be that I will want to use this with the -runas function in qemu, how can I combine them?

Edit: Oh this was fairly easy, just use the /etc/security/limits.conf and do

@kvm soft memlock 8650752
@kvm hard memlock 9437184

soft lock on (8192+256)*1024, hard lock(roof) on (8192+1024)*1024 smile now it's set upon boot and I can use the asroot function (I hope)

Last edited by rabcor (2014-06-12 01:12:38)

Offline

#1952 2014-06-04 04:47:04

aw
Member
Registered: 2013-10-04
Posts: 921
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

sonnet wrote:

Thanks guys for your advices!
Does anybody have an idea about the overhead (in partiular % of cpu resources, considering I have a quadcore haswell) caused by this method with vnc?

Lots


http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Offline

#1953 2014-06-04 05:37:49

AKSN74
Member
From: Taichung, Taiwan
Registered: 2014-02-18
Posts: 52

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

aw wrote:
aw wrote:
syndtr wrote:

I think I found ultimate fix for code 43 on windows and RmInitAdapter failed on linux with nvidia devices with newer driver.
What I did was masking hypervisor cpuid signature and voala!!!

Apparently the driver checking for hypervisor cpuid signature and bail if detecting presence of those signature unless the gpu is nvidia GRID.
After 2 frustrating weeks at last.
F*ck you nvidia!!!

Wow, I was suspicious, but didn't actually think nvidia would sink this low to try it.  I suppose there's still a chance that it was an oversight and not an attempt to be malicious.  Maybe we can make a runtime option for qemu to hide the hypervisor.

It's also sufficient to just change the signature, for instance:

--- a/target-i386/kvm.c
+++ b/target-i386/kvm.c
@@ -528,7 +528,7 @@ int kvm_arch_init_vcpu(CPUState *cs)
         has_msr_hv_hypercall = true;
     }
 
-    memcpy(signature, "KVMKVMKVM\0\0\0", 12);
+    memcpy(signature, "kvmkvmkvm\0\0\0", 12);
     c = &cpuid_data.entries[cpuid_i++];
     c->function = KVM_CPUID_SIGNATURE | kvm_base;
     c->eax = 0;

Wow! I can't believe that NVIDIA do this force limitation.

Maybe it can solve the problem that I can't use CUDA in Ubuntu guest OS.
(My guess is, because GRID is earlier product, so Linux driver may earlier get this limitation.)

But... I'm trying to edit kvm.c, I don't know which change is useful? syndtr's or Alex's?
And does any other codes need to fix?


EDIT: After this changes, FINALLY!!!!!!!
Now Linux Guest OS can RUN CUDA WITH GTX480!!!!!!!
My guess is correct! NVIDIA also do this limitation on Linux Driver!


EDIT2: With code changes, now some CUDA samples can run very well (Like deviceQuery)
But some samples can't run (Like matrixMul)

And... VM still gets panic when reboot Ubuntu.
So, still have some problems need to solve.


EDIT3: When runs sample that get error, dmesg got this message

[784.918638] nvidia 0000:01:00.0: irq 45 for MSI/MSI-X
[785.394236] NVRM: os_pci_init_handle: invalid context!
[785.394362] NVRM: GPU at 0000:01:00: GPU-3222d28c-c7c6-ad1a-67ce-f0e84bed1fe0
[785.394364] NVRM: Xid (0000:01:00): 13, 0001 00000000 000090c0 00002390 00000000 00000000

And CUDA application got this error message (Example for matrixMul):

Failed to create start event (error code unknown error)!

Looks like Linux driver not only have this problem.
Here is some guest OS information

OS: Ubuntu 12.04-4 LTS (Kernel 3.11.0-15-generic x86_64)
NVIDIA Driver: 331.67
CUDA SDK version: 5.5.22

Last edited by AKSN74 (2014-06-04 07:16:38)

Offline

#1954 2014-06-04 14:57:51

shawly
Member
Registered: 2014-05-24
Posts: 22

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Ok, vga-passthrough on my machine is almost fully working (with OP patches and 3.14.4), BUT I got the same problem as slis, my fps drop from time to time.
Rabcor already described how to fix this, but which patches are now really needed and which patches are old?

The OP pkg has many patches, but for example, I can't apply all the patches (e.g. the i915 patch) from the OP pkg on the actual mainline (3.15).

Like I said, it's working but my question is which patches do I (really) need for 3.14.5-stable and which do I need for 3.15.*-mainline?
I would prefer the stable.
An up to date list with links to the patches with the corresponding kernel versions would be great.

My hardware:

CPU: Xeon E3-1245v3
MB: MSI Z97 Gaming 7
iGPU used by Host
NVidia GeForce GTX 770 passed to Windows Guest
AMD Radeon HD5450 passed to another Linux Guest

Edit: Can I use the code 43 fix to get the latest nvidia driver to work?

Last edited by shawly (2014-06-04 15:03:25)

Offline

#1955 2014-06-04 18:08:46

sinny
Member
Registered: 2014-05-24
Posts: 75

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

shawly wrote:

Ok, vga-passthrough on my machine is almost fully working (with OP patches and 3.14.4), BUT I got the same problem as slis, my fps drop from time to time.
Rabcor already described how to fix this, but which patches are now really needed and which patches are old?

The OP pkg has many patches, but for example, I can't apply all the patches (e.g. the i915 patch) from the OP pkg on the actual mainline (3.15).

Like I said, it's working but my question is which patches do I (really) need for 3.14.5-stable and which do I need for 3.15.*-mainline?
I would prefer the stable.
An up to date list with links to the patches with the corresponding kernel versions would be great.

My hardware:

CPU: Xeon E3-1245v3
MB: MSI Z97 Gaming 7
iGPU used by Host
NVidia GeForce GTX 770 passed to Windows Guest
AMD Radeon HD5450 passed to another Linux Guest

Edit: Can I use the code 43 fix to get the latest nvidia driver to work?

have you tried disabling sound in application you experience fps drops in?

i was having varied fps rates in win 7 (league of legends) between reboots. and one time, when fps was hard to bear i tried disabling sounds - it helped. after playing around with audio parameters, i tried the following:

export QEMU_PA_SAMPLES=8192
export QEMU_AUDIO_DRV=pa

so far problems with performance/sound quality are pretty much gone for the last 3 days after i switched to those settings.

p.s. i have only first core (both hyper threads) enabled for my host (isolcpus kernel parameter), win 7 guest is given 3 "dedicated" hyper threads (one from each core unused-by-host)

Offline

#1956 2014-06-04 18:45:56

shawly
Member
Registered: 2014-05-24
Posts: 22

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

sinny wrote:

have you tried disabling sound in application you experience fps drops in?

i was having varied fps rates in win 7 (league of legends) between reboots. and one time, when fps was hard to bear i tried disabling sounds - it helped. after playing around with audio parameters, i tried the following:

export QEMU_PA_SAMPLES=8192
export QEMU_AUDIO_DRV=pa

so far problems with performance/sound quality are pretty much gone for the last 3 days after i switched to those settings.

p.s. i have only first core (both hyper threads) enabled for my host (isolcpus kernel parameter), win 7 guest is given 3 "dedicated" hyper threads (one from each core unused-by-host)

I don't even have a soundcard in my vm at the moment, so it's not the sound.
But like I said the fps drops are not the problem, I wanna know which patches I need and which I don't.

Offline

#1957 2014-06-04 18:52:11

aw
Member
Registered: 2013-10-04
Posts: 921
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

shawly wrote:

CPU: Xeon E3-1245v3
MB: MSI Z97 Gaming 7
iGPU used by Host
NVidia GeForce GTX 770 passed to Windows Guest
AMD Radeon HD5450 passed to another Linux Guest

For this hardware running the latest 3.15-rc, you need the i915 patch, possibly the ACS override patch and matching cmdline option (depending on where the cards are attached), and the qemu patch to hide the hypervisor if you want to run the nvidia 337.88 driver.


http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Offline

#1958 2014-06-04 19:05:15

shawly
Member
Registered: 2014-05-24
Posts: 22

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

aw wrote:

For this hardware running the latest 3.15-rc, you need the i915 patch, possibly the ACS override patch and matching cmdline option (depending on where the cards are attached), and the qemu patch to hide the hypervisor if you want to run the nvidia 337.88 driver.

Yeah well, like I said, the i915 in the OP package won't apply to 3.15-rc and since I'm pretty new to this, I have no idea where I get the latest working patches. Sorry if I'm to blind...

Offline

#1959 2014-06-04 19:08:51

aw
Member
Registered: 2013-10-04
Posts: 921
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

shawly wrote:
aw wrote:

For this hardware running the latest 3.15-rc, you need the i915 patch, possibly the ACS override patch and matching cmdline option (depending on where the cards are attached), and the qemu patch to hide the hypervisor if you want to run the nvidia 337.88 driver.

Yeah well, like I said, the i915 in the OP package won't apply to 3.15-rc and since I'm pretty new to this, I have no idea where I get the latest working patches. Sorry if I'm to blind...

https://lkml.org/lkml/2014/5/9/517

Be sure to boot with i915.enable_hd_vgaarb=1 with this version.


http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Offline

#1960 2014-06-04 19:16:14

shawly
Member
Registered: 2014-05-24
Posts: 22

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

aw wrote:
shawly wrote:

Yeah well, like I said, the i915 in the OP package won't apply to 3.15-rc and since I'm pretty new to this, I have no idea where I get the latest working patches. Sorry if I'm to blind...

https://lkml.org/lkml/2014/5/9/517

Be sure to boot with i915.enable_hd_vgaarb=1 with this version.

Many thanks <3 will try that tomorrow, does the acs_override patch in the OP pkg apply to 3.15?

Offline

#1961 2014-06-04 19:17:00

aw
Member
Registered: 2013-10-04
Posts: 921
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

shawly wrote:
aw wrote:
shawly wrote:

Yeah well, like I said, the i915 in the OP package won't apply to 3.15-rc and since I'm pretty new to this, I have no idea where I get the latest working patches. Sorry if I'm to blind...

https://lkml.org/lkml/2014/5/9/517

Be sure to boot with i915.enable_hd_vgaarb=1 with this version.

Many thanks <3 will try that tomorrow, does the acs_override patch in the OP pkg apply to 3.15?

Dunno, I don't use it


http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Offline

#1962 2014-06-04 22:33:02

slis
Member
Registered: 2014-06-02
Posts: 127

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Strange no one replied sooner tongue

haha thank you!

anyway i got it almost all working.... except SLI...
my specs and setup are in that google doc

Problem with performace was in CPU config...
i did= -cpu host 4 1 4 2  for i7 2600 = DOTA 2 (40-50fps instead 110-120 native)

after 2 days i managed (haha) to change it to
-cpu host 8 1 8 1 = DOTA 2 (90-110fps)
and its pretty much same with qemu cpu...

with 770gtx it reaches 220fps, but drops to 30fps for no reason, i guess its driver 335 or cache in dota smile

Any suggestions on cpu config maybe to get that 5-10% more? smile it seems like cpu is lowering speed with throttling or something like that , will hugepages help or another model? and what about hv-time (what is that for)?

and about SLI... i didnt try that qemu patch but wiill do...
now: with downstream option, one card won't start either with error code 10 or 12, however after changing countless configs, audio device was reported working on second gpu but gpu itself still had error 12... not enough resources
i did try to define 2nd port root pci bridge in config but with no luck...


also did try 2 VM-s with 560ti each, dota 2 about 80fps each at same time... impressive... and i think that performace hit was due pcie x8 when there are 2 cards on my mbo. cpu usage 40%.

and about encoding... i dont think that u can do 1080p 60fps with software encode... i mean i can... 1 dota2 1080p 60fps stream ... i7 cpu about 70% usage and it cant hold 60 it falls to 40fps... not smooth...
but on windows with igpu everything is smooth constant  60fps... and i bet twice lower cpu usage but i didnt check.... steam doesnt support nvidia encode right now... only intel quick sync....


PS.
after further dota testing it seems like when it reaches 100% gpu usage to take it back to 90% for a half a second then take it back to 100%, and fps varies from 80-110fps for no reason, (it may be happening on windows at faster rate so its not that noticable).... but overall its working great, not a single BUG/CRASH in vm when i got nvidia driver up.... restart and shutdown is working with dumped bios.

Last edited by slis (2014-06-05 08:00:08)

Offline

#1963 2014-06-05 03:47:49

alexis_evo
Member
Registered: 2013-07-04
Posts: 33

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Using linux-mainline from OP and qemu-git from AUR... getting segfaults when I try to start the VM.

modprobe vfio-pci
echo 0000:03:00.0 > /sys/bus/pci/devices/0000\:03\:00.0/driver/unbind 
echo 1002 6738 > /sys/bus/pci/drivers/vfio-pci/new_id 
echo 0000:03:00.1 > /sys/bus/pci/devices/0000\:03\:00.1/driver/unbind 
echo 1002 aa88 > /sys/bus/pci/drivers/vfio-pci/new_id 

echo 2660 > /proc/sys/vm/nr_hugepages

/usr/bin/qemu-system-x86_64 -bios /usr/share/qemu/bios.bin -cpu host,hv-time,hv_relaxed,hv_vapic,hv_spinlocks=0x1000 \
-smp 8,sockets=1,cores=8,threads=1 -m 4G -mem-prealloc -mem-path /dev/hugepages -M q35 -enable-kvm \
-device ioh3420,bus=pcie.0,addr=1c.0,multifunction=on,port=1,chassis=1,id=root.1 \
-device vfio-pci,host=03:00.0,x-vga=on,addr=00.0,multifunction=on,bus=root.1 \
-device vfio-pci,host=03:00.1,addr=00.1,bus=root.1 \
-usb -usbdevice host:04d9:0167 -usbdevice host:0461:4d0f -usbdevice host:045e:028e \
-net nic,model=virtio -net bridge,br=br0 -device ich9-intel-hda,bus=pcie.0,addr=1b.0,id=sound0 -device hda-duplex,id=sound0-codec0,bus=sound0.0,cad=0 \
-drive file=/dev/sdc,id=disk,format=raw -device virtio-blk-pci,drive=disk -drive file=/home/alexis/Downloads/virtio-win-0.1-74.iso,id=isocd -device ide-cd,bus=ide.1,drive=isocd \
-boot menu=on -vga none
2021 Segmentation fault      (core dumped)
[root@snuggles alexis]# dmesg
[...]
[  878.248063] device tap0 entered promiscuous mode
[  878.248420] br0: port 2(tap0) entered forwarding state
[  878.248446] br0: port 2(tap0) entered forwarding state
[  879.850979] vfio-pci 0000:03:00.1: enabling device (0400 -> 0402)
[  879.969089] qemu-system-x86[2021]: segfault at 28 ip 00007fe527d78a10 sp 00007fffc6564c88 error 4 in qemu-system-x86_64[7fe527b88000+42e000]
[  881.162161] br0: port 2(tap0) entered disabled state
[  881.162429] device tap0 left promiscuous mode
[  881.162460] br0: port 2(tap0) entered disabled state

Hadn't updated qemu or linux-mainline in a couple months since everything was working. When I installed qemu-git, pacman made me uninstall seabios-git and libacard-git.

[2014-06-04 21:32] [PACMAN] Running 'pacman -U qemu-git-2.1.r33116.ge00fcfe-1-x86_64.pkg.tar.xz'
[2014-06-04 21:32] [PACMAN] removed libcacard-git (2.0.r31260.g4c288ac-1)
[2014-06-04 21:32] [PACMAN] removed seabios-git (1.7.3.r1736.g5f2d17d-1)
[2014-06-04 21:32] [PACMAN] upgraded qemu-git (2.0.r31260.g4c288ac-1 -> 2.1.r33116.ge00fcfe-1)

I can't downgrade qemu as I no longer have libcacard/seabios packages...

Last edited by alexis_evo (2014-06-05 03:49:36)

Offline

#1964 2014-06-05 05:20:17

slis
Member
Registered: 2014-06-02
Posts: 127

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

edit: solved

Last edited by slis (2014-06-10 03:58:42)

Offline

#1965 2014-06-05 05:51:27

AKSN74
Member
From: Taichung, Taiwan
Registered: 2014-02-18
Posts: 52

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

After some tests in #1953, I found that CUDA can't run some application with GPU Memory using.
Seems like the memory is full and can't be clean up automatically.
And maybe memory size has been limited with unknown forces.

Only can running well is some information applications (Like deviceQuery in NVIDIA CUDA Samples).
And I think driver still a big problem in Ubuntu. Even hide the hypervisor information.

Can someone test CUDA in VM that OS is Linux? I want to know it is OS issue or driver.

Offline

#1966 2014-06-05 09:49:27

mbroemme
Member
From: Cologne
Registered: 2014-02-05
Posts: 40

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

slis wrote:

Any suggestions on cpu config maybe to get that 5-10% more? smile it seems like cpu is lowering speed with throttling or something like that , will hugepages help or another model? and what about hv-time (what is that for)?

To reduce throttling under load I do the following:

echo 100 > /sys/devices/system/cpu/cpufreq/ondemand/sampling_down_factor
echo 75 > /sys/devices/system/cpu/cpufreq/ondemand/up_threshold

This helped to stabilize framerates in almost every game and CPU is still throttled when idle. I've automated this on boot using tmpfiles.d from systemd and added in /etc/tmpfiles.d/99-custom.conf the following:

w /sys/devices/system/cpu/cpufreq/ondemand/sampling_down_factor - - - - 100
w /sys/devices/system/cpu/cpufreq/ondemand/up_threshold - - - - 75

Last edited by mbroemme (2014-06-05 17:56:37)

Offline

#1967 2014-06-05 15:06:53

slis
Member
Registered: 2014-06-02
Posts: 127

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

alexis_evo wrote:

Using linux-mainline from OP and qemu-git from AUR... getting segfaults when I try to start the VM.


same here, clean install jessie, linux-vifio kernel and qemu-git, with config that worked with arch/debian qemu and aw qemu

[ 2902.040594] qemu-system-x86[2588]: segfault at 28 ip 00007fc0b7321fe4 sp 00007fff707ed3f0 error 4 in qemu-system-x86_64[7fc0b7132000+48a000]

it works with debian default qemu-kvm ....

jessie + linux vfio + default qemu-kvm

Last edited by slis (2014-06-05 15:33:50)

Offline

#1968 2014-06-05 15:42:46

aldum
Member
Registered: 2012-03-14
Posts: 8

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

aw wrote:
aw wrote:
syndtr wrote:

I think I found ultimate fix for code 43 on windows and RmInitAdapter failed on linux with nvidia devices with newer driver.
What I did was masking hypervisor cpuid signature and voala!!!

Apparently the driver checking for hypervisor cpuid signature and bail if detecting presence of those signature unless the gpu is nvidia GRID.
After 2 frustrating weeks at last.
F*ck you nvidia!!!

Wow, I was suspicious, but didn't actually think nvidia would sink this low to try it.  I suppose there's still a chance that it was an oversight and not an attempt to be malicious.  Maybe we can make a runtime option for qemu to hide the hypervisor.

It's also sufficient to just change the signature, for instance:

-    memcpy(signature, "KVMKVMKVM\0\0\0", 12);
+    memcpy(signature, "kvmkvmkvm\0\0\0", 12);

You sir, are a genius. Thanks to both of you!

Offline

#1969 2014-06-05 18:49:25

aw
Member
Registered: 2013-10-04
Posts: 921
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

aldum wrote:
aw wrote:
aw wrote:

Wow, I was suspicious, but didn't actually think nvidia would sink this low to try it.  I suppose there's still a chance that it was an oversight and not an attempt to be malicious.  Maybe we can make a runtime option for qemu to hide the hypervisor.

It's also sufficient to just change the signature, for instance:

-    memcpy(signature, "KVMKVMKVM\0\0\0", 12);
+    memcpy(signature, "kvmkvmkvm\0\0\0", 12);

You sir, are a genius. Thanks to both of you!

Current qemu.git now has an option to hide the hypervisor.  You can add "kvm=off" to your list of -cpu option, ex:

-cpu host,hv-time,kvm=off

http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Offline

#1970 2014-06-05 19:51:50

aldum
Member
Registered: 2012-03-14
Posts: 8

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

aw wrote:

Current qemu.git now has an option to hide the hypervisor.  You can add "kvm=off" to your list of -cpu option, ex:

-cpu host,hv-time,kvm=off

I already compiled a custom QEMU with ABS, but that would have made it a lot easier :3

Offline

#1971 2014-06-06 00:58:07

alexis_evo
Member
Registered: 2013-07-04
Posts: 33

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

So I managed to get it booting again by downgrading qemu-git AUR package. I grabbed the last PKGBUILD from some mirror website that hosts old PKGBUILDs, and added a git reset --hard into it to downgrade to some commit in April. Here's the PKGBUILD:

# Maintainer:  Devin Cofer <ranguvar{AT]archlinux[DOT}us>
# Contributor: Tobias Powalowski <tpowa@archlinux.org>
# Maintainer: Frederic Bezies <fredbezies@gmail.com>

pkgname=qemu-git
_gitname=qemu
pkgver=2.1.r32209.g2d03b49
pkgrel=1
pkgdesc="A generic and open source processor emulator which achieves a good emulation speed by using dynamic translation. Git version."
arch=('i686' 'x86_64')
license=('GPL2' 'LGPL2')
url="http://wiki.qemu.org/Index.html"
depends=('pixman' 'libjpeg' 'libpng' 'sdl' 'alsa-lib' 'nss' 'glib2'
         'gnutls>=2.4.1' 'bluez-libs' 'vde2' 'util-linux' 'curl' 'libsasl'
         'libgl' 'libpulse' 'libcap-ng' 'libaio' 'libseccomp'
         'libiscsi' 'spice' 'seabios-git' 'usbredir')
makedepends=('git' 'texi2html' 'perl' 'python2' 'dtc' 'spice-protocol')
optdepends=('ovmf-bin: UEFI Firmware (OVMF) with Secure Boot Support - for Virtual Machines (QEMU)')
conflicts=('qemu' 'kvm' 'kvm-git' 'qemu-spice' 'libcacard')
provides=('qemu' 'qemu-kvm' 'qemu-spice' 'libcacard')
backup=('etc/qemu/target-x86_64.conf')
source=('git://git.qemu.org/qemu.git'
        '65-kvm.rules'
        'qemu.install')
sha256sums=('SKIP'
            '9c8a15c34461a9481a34ca9e0ab4ae3825eabe8fd863227f2445325413cd755c'
            '9970c3050e8dc6153c5955d018d114f9fcbc091843b85f9e7b247eb28f09ba10')
install='qemu.install'
options=(!strip)

pkgver() {
        cd "${srcdir}/$_gitname"
        git reset --hard 2d03b49c3f225994c4b0b46146437d8c887d6774
        echo "2.1.r$(git rev-list --count master).g$(git log -1 --format="%h")"
}

build()
{
        cd $_gitname

        git reset --hard 2d03b49c3f225994c4b0b46146437d8c887d6774
        # qemu vs. make 4 == bad
        export ARFLAGS="rv"
        # http://permalink.gmane.org/gmane.comp.emulators.qemu/238740
        # gtk gui breaks keymappings at the moment
        ./configure --prefix=/usr --sysconfdir=/etc --audio-drv-list='pa alsa sdl' \
              --python=/usr/bin/python2 --smbd=/usr/bin/smbd \
              --enable-docs --libexecdir=/usr/lib/qemu \
              --disable-gtk --enable-linux-aio --enable-seccomp \
              --enable-spice --localstatedir=/var --disable-werror \
              --enable-tpm
        make V=99
}

package() {

  cd $_gitname

  git reset --hard 2d03b49c3f225994c4b0b46146437d8c887d6774
  make DESTDIR="${pkgdir}" libexecdir="/usr/lib/qemu" install
  # provided by seabios package
  rm "${pkgdir}/usr/share/qemu/bios.bin"
  rm "${pkgdir}/usr/share/qemu/acpi-dsdt.aml"
  rm "${pkgdir}/usr/share/qemu/q35-acpi-dsdt.aml"
  # remove conflicting /var/run directory
  rm -r "${pkgdir}/var"
  install -D -m644 "${srcdir}/65-kvm.rules" \
                   "${pkgdir}/usr/lib/udev/rules.d/65-kvm.rules"
  # bridge_helper needs suid
  # https://bugs.archlinux.org/task/32565
  chmod u+s "${pkgdir}/usr/lib/qemu/qemu-bridge-helper"
  # add sample config
  echo "allow br0" > ${pkgdir}/etc/qemu/bridge.conf.sample
  # strip scripts directory
    find "${pkgdir}/usr/src/linux-${_kernver}/scripts"  -type f -perm -u+w 2>/dev/null | while read binary ; do
      case "$(file -bi "$binary")" in
        *application/x-executable*) # Binaries
        /usr/bin/strip $STRIP_BINARIES "$binary";;
      esac
    done
  # remove libcacard files
  # rm -rf ${pkgdir}/usr/include/cacard
  # rm -rf ${pkgdir}/usr/lib/libcacard*
  # rm -rf ${pkgdir}/usr/lib/pkgconfig/libcacard.pc
  # rm -rf ${pkgdir}/usr/bin/vscclient
}

This is still using a rather old qemu build though, so I would greatly appreciate any help getting the latest qemu-git to work.

Offline

#1972 2014-06-06 01:14:06

andy123
Member
Registered: 2011-11-04
Posts: 169
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

alexis_evo wrote:

So I managed to get it booting again by downgrading qemu-git AUR package.
[…]
This is still using a rather old qemu build though, so I would greatly appreciate any help getting the latest qemu-git to work.

Interesting, I actually maintain the qemu-git aur package now and the upgrade removed seabios-git and libcacard-git, because its using qemu's internal versions now.

Since it works for me, I'm going to try to blame upstream here, but this isn't really helpful, so maybe try to run it in strace or gdb or something and see if you can get some usage output/error messages.


i'm sorry for my poor english wirting skills…

Offline

#1973 2014-06-06 01:39:39

Rusky
Member
Registered: 2014-06-06
Posts: 1

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

thehighhat wrote:

So I had a working vga passthrough of nvidia gtx470 with a windows 8 guest.  I let the windows store upgrade to windows 8.1 update.  now, the guest will not start correct.  it always goes into automatic system repair and fails.  the refresh pc, reset pc options ask for windwos cd, but cannot detect it.  if i just let windows try to start, it shows nohing but a black screen.  then i have reboot the host becaues i cannot shutdown the guest cleanly.

any ideas why the 8.1 update breaks everything?

I have this exact same problem with a Radeon R9 270X, starting with a working Windows 8.1 install and just doing a normal update. Automatic system repair always fails, all the bootrec tricks fail, building a new BCD from scratch fails, refreshing the install fails. The host is Arch with the linux-mainline kernel from the first post, using Qemu 2.0.0 from the repositories.

Offline

#1974 2014-06-06 05:06:28

slis
Member
Registered: 2014-06-02
Posts: 127

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

here is strace form segmentation fault with qemu-git, it crashes with vga none option....
i can give more logs and info if u instruct me how to smile
btw this is debian ... linux-vfio 3.15 with acs patch only


root@TEST:~# strace sh q
execve("/bin/sh", ["sh", "q"], [/* 15 vars */]) = 0
brk(0)                                  = 0x7fac63b8c000
access("/etc/ld.so.nohwcap", F_OK)      = -1 ENOENT (No such file or directory)
mmap(NULL, 8192, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7fac61bc3000
access("/etc/ld.so.preload", R_OK)      = -1 ENOENT (No such file or directory)
open("/etc/ld.so.cache", O_RDONLY|O_CLOEXEC) = 3
fstat(3, {st_mode=S_IFREG|0644, st_size=35337, ...}) = 0
mmap(NULL, 35337, PROT_READ, MAP_PRIVATE, 3, 0) = 0x7fac61bba000
close(3)                                = 0
access("/etc/ld.so.nohwcap", F_OK)      = -1 ENOENT (No such file or directory)
open("/lib/x86_64-linux-gnu/libc.so.6", O_RDONLY|O_CLOEXEC) = 3
read(3, "\177ELF\2\1\1\3\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0P\34\2\0\0\0\0\0"..., 832) = 832
fstat(3, {st_mode=S_IFREG|0755, st_size=1734080, ...}) = 0
mmap(NULL, 3840576, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_DENYWRITE, 3, 0) = 0x7fac615fc000
mprotect(0x7fac6179c000, 2097152, PROT_NONE) = 0
mmap(0x7fac6199c000, 24576, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x1a0000) = 0x7fac6199c000
mmap(0x7fac619a2000, 14912, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_ANONYMOUS, -1, 0) = 0x7fac619a2000
close(3)                                = 0
mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7fac61bb9000
mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7fac61bb8000
mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7fac61bb7000
arch_prctl(ARCH_SET_FS, 0x7fac61bb8700) = 0
mprotect(0x7fac6199c000, 16384, PROT_READ) = 0
mprotect(0x7fac61de2000, 8192, PROT_READ) = 0
mprotect(0x7fac61bc5000, 4096, PROT_READ) = 0
munmap(0x7fac61bba000, 35337)           = 0
getpid()                                = 2982
rt_sigaction(SIGCHLD, {0x7fac61bd9620, ~[RTMIN RT_1], SA_RESTORER, 0x7fac61631420}, NULL, 8) = 0
geteuid()                               = 0
brk(0)                                  = 0x7fac63b8c000
brk(0x7fac63bad000)                     = 0x7fac63bad000
getppid()                               = 2981
stat("/root", {st_mode=S_IFDIR|0700, st_size=4096, ...}) = 0
stat(".", {st_mode=S_IFDIR|0700, st_size=4096, ...}) = 0
open("q", O_RDONLY)                     = 3
fcntl(3, F_DUPFD, 10)                   = 10
close(3)                                = 0
fcntl(10, F_SETFD, FD_CLOEXEC)          = 0
rt_sigaction(SIGINT, NULL, {SIG_DFL, [], 0}, 8) = 0
rt_sigaction(SIGINT, {0x7fac61bd9620, ~[RTMIN RT_1], SA_RESTORER, 0x7fac61631420}, NULL, 8) = 0
rt_sigaction(SIGQUIT, NULL, {SIG_DFL, [], 0}, 8) = 0
rt_sigaction(SIGQUIT, {SIG_DFL, ~[RTMIN RT_1], SA_RESTORER, 0x7fac61631420}, NULL, 8) = 0
rt_sigaction(SIGTERM, NULL, {SIG_DFL, [], 0}, 8) = 0
rt_sigaction(SIGTERM, {SIG_DFL, ~[RTMIN RT_1], SA_RESTORER, 0x7fac61631420}, NULL, 8) = 0
read(10, "#!/bin/bash\nqemu-system-x86_64 -"..., 8192) = 613
stat("/usr/local/sbin/qemu-system-x86_64", 0x7fff18e9d8d0) = -1 ENOENT (No such file or directory)
stat("/usr/local/bin/qemu-system-x86_64", {st_mode=S_IFREG|0755, st_size=5998584, ...}) = 0
clone(child_stack=0, flags=CLONE_CHILD_CLEARTID|CLONE_CHILD_SETTID|SIGCHLD, child_tidptr=0x7fac61bb89d0) = 2983
wait4(-1, [{WIFSIGNALED(s) && WTERMSIG(s) == SIGSEGV}], 0, NULL) = 2983
--- SIGCHLD (Child exited) @ 0 (0) ---
rt_sigreturn(0x11)                      = 2983
write(2, "Segmentation fault\n", 19Segmentation fault
)    = 19
read(10, "", 8192)                      = 0
exit_group(139)                         = ?
root@TEST:~#

config

#!/bin/bash
qemu-system-x86_64 -enable-kvm -M q35 -m 3072 -cpu Haswell,hv-time -smp 4,sockets=1,cores=4,threads=1 \
-boot menu=on \
-vga none \
-device ioh3420,bus=pcie.0,addr=1c.0,multifunction=on,port=1,chassis=1,id=root.1 \
-device vfio-pci,host=01:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on,romfile=/root/GF114.rom \
-device vfio-pci,host=01:00.1,bus=root.1,addr=00.1 \
-drive file=/root/win.img,id=disk,format=raw -device ide-hd,bus=ide.0,drive=disk \
-usb -usbdevice host:413c:2003 -usbdevice host:046d:c042 \
-netdev tap,id=guest0,vhost=on,script=no \
-net nic,model=e1000,netdev=guest0 \
-vnc :1

Last edited by slis (2014-06-06 06:29:29)

Offline

#1975 2014-06-06 06:31:39

rman
Member
From: United States
Registered: 2013-08-08
Posts: 14

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Just wanted to chime in with another success story. It took me an entire weekend plus the following Monday, but I got my VGA passthrough setup fully functional! big_smile

Specs:
CPU: Core i5-4690
MB: ASRock H97m Pro 4
GPU 1: HD Graphics 4600 (used by Arch host)
GPU 2: AMD Radeon R7 265 (passed through to Windows 7)

The only patch I needed was the i915 arbiter patch. At first I used the 3.14 kernel from the OP, but I had some suspend to RAM issues, so I applied the patch against 3.15rc8 and it built and ran fine.

I cannot passthrough the Radeon's HDMI audio controller, or I get the dreaded atikmpag.sys BSOD. The VM is otherwise flawless though - I can even reboot it with absolutely no problems.

I had some issues with Qemu's emulated audio - Intel HDA was basically unusable with pops and static. I tried ac97, which worked much better, but I still had stuttering, noise, and choppiness. At that point I gave up and bought a USB sound card, which of course is working perfectly.

Anyway, thanks to everyone who contributed to this thread! It's good to know that this is still working with the latest GPU's. smile

Last edited by rman (2014-06-06 06:34:28)


Laptop: Lenovo L440, Intel Core i3-4000M, HD Graphics 4600
Desktop: Intel Core i5-4690, HD Graphics 4600, AMD Radeon R7 265 (KVM VGA passthrough)

Offline

Board footer

Powered by FluxBB