Attempting VGA Passthrough, no video output

Velociraptor_sb · 2016-09-23 12:58:29

Byter wrote:

The only difference for me is I've got a 670 instead of a 680 (same card underneath though).

I have a GTX680 and i can confirm , i have setup everything properly following the arch wiki , every step where there is an error check on the wiki leads to negative errors , and my pci is properly isolated and bound with the vfio-pci drivers (from the arch wiki it felt like an EITHER/OR between the vfio and pci_Stub method so i went with vfio since i have the latest official kernel, havenot tried pci_stub method sofar).

when i reach the step where i need to start up my vm it will startup (win10) only in standard "qxl" mode , but once i remove "qxl" and every other non needed component and bind my pci nvidia modules, there is absolutely no output to the screen and absolutely no error messages from qemu or journalctl or dmesg , i can even feel the cpu fan spinning with a higher rpm . i even at first suspected that because my pci was in the upper pci x16 slot it wasnot isolated properly (so it was in the same iommu group as the cpu's pci root port ), so i attempted to remove it and plug it in my second pci x16 slot , it correctly assigned both nvidia modules to a unique iommu groups , their drivers where listed as vfio-pci, and "dmesg | grep -i vfio" shows the pcis to be properly bound and yields no errors whatsover.

To sum up ,and given the previous posts, i am starting to think the nvidia GTX 680 is to blame,i am however going to rollback all my changes to the kernel and retrace my steps this time with the pci_Stub method if that fails i think i am going to get an AMD gpu.
if anyone with the same GPU as i, has positive results please let me know

my rig : i7 4770 (no OC), GPU: GTX680 Gainward, and intel GMA for host , RAM : 32 gigs ram 1600 Mhz

deafeningsylence · 2016-09-23 13:22:24

Hi Arch community,

I fought with gpu passthrough for a few days unti I got it right (total linux newby).

First of all, you can use vfio-pci and abandon pci-stub, if you are using OVMF anyway.
I had the missing ouput as well until I used the linux-vfio kernel from the AUR repository (it includes OVMF, iOMMU fixes, i915 and acs override patches. With that it worked quite easily.

A few tips for better performance.

If you are using Windows 10, and I think you are, set cpu to core2duo and define the cores and threads by hand. In your script that would be cpu=custom ... sockets=1 cores=4 threads=1 (example for an i5, dont copy this, since I do not know the exact script, I am using libvirt and xml). Otherwhise it can crash or hang during Windows updates and driver install (this is a known bug).
Also I strongly recommend cpu pinning (assign 2 cores or more to the VM only, best core 2 and 3, counting from 0-3, since those usually do not have to do much in most applications) and defining hugepages (giving half of your ram or more, provided you have enough to the VM). This gives quite a performance boost, at least in my case (30 vs 60 fps in BF4), since it reduces input lag and whatsoever.

Hope this helps,

deafeningsylence

deafeningsylence · 2016-09-23 13:28:25

Regarding the graphicscard, if you have no UEFI capable or EFI capable graphicscard (this can be checked for your card and vBios on techpowerup) and are using an Intel GPU for the host (Linux), then you need the ACS override and i915 arbiter patch (both included in linux-vfio kernel).

So yes, the 680 could potentially be blamed. Check it to be sure. I have successfully done it with the above kernel on an 7870 which officially does not support EFI vBios.

Last edited by deafeningsylence (2016-09-23 13:28:52)

Velociraptor_sb · 2016-09-23 13:47:02

deafeningsylence wrote:

Regarding the graphicscard, if you have no UEFI capable or EFI capable graphicscard (this can be checked for your card and vBios on techpowerup) and are using an Intel GPU for the host (Linux), then you need the ACS override and i915 arbiter patch (both included in linux-vfio kernel).
So yes, the 680 could potentially be blamed. Check it to be sure. I have successfully done it with the above kernel on an 7870 which officially does not support EFI vBios.

Thanks for your reply and ill try it with that method and update if it works, i have never tried the linux-vfio kernel coz i was afraid i could brick something but i guess now its the last solution short of buying an AMD which i am gonna do anyways.

deafeningsylence · 2016-09-23 13:51:54

Velociraptor_sb wrote:

deafeningsylence wrote:
Regarding the graphicscard, if you have no UEFI capable or EFI capable graphicscard (this can be checked for your card and vBios on techpowerup) and are using an Intel GPU for the host (Linux), then you need the ACS override and i915 arbiter patch (both included in linux-vfio kernel).
So yes, the 680 could potentially be blamed. Check it to be sure. I have successfully done it with the above kernel on an 7870 which officially does not support EFI vBios.
Thanks for your reply and ill try it with that method and update if it works, i have never tried the linux-vfio kernel coz i was afraid i could brick something but i guess now its the last solution short of buying an AMD which i am gonna do anyways.

I did not break anything so far. And for me it was the same descision, pay 300€ for a RX480 or use the vfio kernel. So the latter was the more appropriate decision, since there is no game currently my HD7870 cannot stem at least at medium settings.

If you install the linux-vfio kernel do not forget to update your initram ('mkinitcpio -p linux-vfio' and so on), also the install takes quite some time.

Good luck! It is worth it.

Velociraptor_sb · 2016-09-24 13:16:56

tried installing linux-vfio , it kept compiling for an hour almost then the installation failed with following warning

===> WARNING: Cannot install created packages
===> Removing installed dependencies...

then it removes the dependencies and rolls back, the linux-vfio package uses an older kernel than the one i have installed and is out-of-date according to its AUR page , I am no expert but it could be the reason. I tried to manually enable ACS override and i915 arbiter patch by adding the following lines to my EFI boot loader entry

options        pcie_acs_override=downstream
options        i915.enable_hd_vgaarb=1

but that also had absolutely no effect, my gtx680 still wont output anything.

I did some research about it and gtx680 apparently has no uefi support as u suggested i check earlier, so i am guessing thats the highest probable cause for the problem, and actually makes sense ,coz OVMF is a uefi firmware for QEMU, so it makes little sense trying to set it up with a bios card like mine.

so i guess that leaves me with two options .

the first : as stated in the Archwiki , the use of the legacy method (pci-stub) , combined with seabios instead of ovmf , but i gotta do some research on that , and on a first glance i found this tutorial about it from the arch forums

https://bbs.archlinux.org/viewtopic.php?id=162768

the second and obvious yet cost-intensive option: is to buy a uefi graphics card and use my gtx for host graphics.

But for now i guess i am gonna try the first option, i might also learn something new or two along the way

thanks for the help .

UPDATE

so i tried using Seabios instead of ovmf and the result is the same , no output . i guess my combination of GPUs is not meant for passthroughs after all , i guess now ill rollback all the changes and wait for some money to buy me a decent uefi enabled gpu

Last edited by Velociraptor_sb (2016-09-24 14:35:55)

Velociraptor_sb · 2016-09-24 17:22:24

finally i have managed to get my intelGMA(host)/GTX680(guest) combination to work for passthrough and wanted to share my approach for anyone who had my problem

the linux-vfio kernel failed to compile in my case so i went with the linux-vfio-lts (please note that u need to first import the gpg keys provided in the AUR page using a non standard server or else it might fail to import gng keys and thus fail to even start compiling)

after installing linux-vfio-lts , and updating initramfs and efi entries and efi bootloader so it starts using this newly installed custom kernel

i followed the arch wiki PCI passthrough via OVMF until the point "Setting up the guest OS" thats where i used a different approach to get my card to work

namely i didnot use ovmf but rather seabios to get my card to passthrough

so through the console i tested to see if its working(and it did) using the following

qemu-system-x86_64 -bios /usr/share/qemu/bios.bin -enable-kvm -m 1024 -cpu host,kvm=off \
-smp 4,sockets=1,cores=4,threads=1 \
-device vfio-pci,host=01:00.0,x-vga=on -device vfio-pci,host=01:00.1 \
-vga none

in my case 01:00.0 and 01:00.1 were the device ids for my nvidia gpu and its audio hdmi
please note that the -bios /usr/share/qemu/bios.bin qemu flag is extremely important , in fact in my case the passthrough failed without it as qemu booted by default over ovmf in uefi mode which yielded no output.
if for any reason you dont have the /usr/share/qemu/bios.bin file , u can download it as instructed in Seabios. then cd into the downloaded folder and run "make"
the software will compile and the new bios is then in out/bios.bin inside that same folder where seabios was compiled . then run qemu with -bios flag and path to your bios.bin which should force qemu to use it for booting up . It worked for me

in which case a black qemu output showed on the main screen and the Monitor connected to the guest booted into bios mode with "bla bla bla no bootable device " prompt , then it was just a matter of installing windows into the vm and configuring stuff like passing through mouse and keyboard (i used an extra mouse for host control as the passed keyboard and mouse no longer are controllable by arch) and also some hardware tweaking and optimizing as explained in the arch wiki.

Hope this help whomever had my same problem.

P.S. : also i added the following options to my /boot/loader/entries/arch-linux-vfio-lts.conf for the acs override and i915 arbiter patch , i dunno if it made a difference but i was like "as long as it works dont fix it"

options        pcie_acs_override=downstream
options        i915.enable_hd_vgaarb=1

Last edited by Velociraptor_sb (2016-09-24 17:31:02)

deafeningsylence · 2016-10-19 12:33:51

Hey velociraptor,

I just checked this post again to see if anything happened in the meantime (forgot to subscribe to it apparently). Nice that you came to a solution. And yes I forgot to mention, you have to import these keys first to install the vfio-kernel, sorry for that.

I think ovmf should work, no idea though why it didn't for you . I think the linux-vfio kernel is the only option for older GPUs at the moment, at least when you are not capable of patching the kernel yourself (I failed, downloaded abs and compiled a new kernel with the patch, but it just gave me some errors within the i915 driver lines). I figure this is why it says updated-i915-arbiter-patch in the sources of the linux-vfio kernel. Since I did not know how to get this updated patch on its own and were only able to find old versions for kernel 4.2+, I just gave up and installed the vfio-kernel again (although minimally outdated atm).

I also followed the wikipage about the passthrough and it gets a bit undetailed during the part where you have to install Windows and passthrough the graphicscard. I used virt-manager to setup the VM.
Do not forget to set cpu to core2duo and then custome the topology to whatever you want, otherwhise it might crash during Windows Update under Windows 10.

So, I set up the VM, without the pci devices passed through yet, then installed Windows, ran updates, rebooted, passed card through, started VM aand crash.
The crash is caused by Windows Update trying to install the AMD drivers. My solution was, dettaching the card again, install the drivers manually (download it from amd) without the card attached. Then set Display Server to VGA (from QXL) in virt-manager and voila, the VM booted with drivers and everything else and I only had to disable the second display in Windows.

I strongly recommend to setup hugepages for the VM Ram as well as pin two or more cpu cores to the VM, this will increase performance quite a bit, also use the virtio drivers for the hard drive and network.
In case you have cracking sound througput, use the ac'97 audio adapter and install its drivers under "Windows without driver signature check" (Settings > Recovery > Advanced reboot > troubleshooting > startup repair > f7).

All to late now, but I thought I'll just give you my thoughts anyway.

Happy gaming .

Velociraptor_sb · 2016-10-20 20:11:12

Thx nevertheless for the update , its up and running tip top with awesome sound and everything , got slight issues with bottlenecking sometimes (just for a couple of seconds) but i guess thats just my card not having enough ram (and me being greedy and setting everything on ULTRA settings xD), but other than that i got one thing to say : GAMING ON LINUX ROOOCKS !!! even though technically i am not gaming on linux but still, being able to browse the net on one screen and gaming on another is awesome not to mention the ability to run archlinux as my primary and sole OS which is something i have always wanted to do. even playing multiplayer games is smooth with almost the same ping and no network lags (using tap)

One more thing , while playing i noticed a massive improvement in fps when i did the following on my intel i7 4770 before starting the vm (i think the following only applies for Intel Sandybridge or newer i dunno about AMD processors)

echo performance | tee /sys/devices/system/cpu/cpu*/cpufreq/scaling_governor >/dev/null
echo 100 > /sys/devices/system/cpu/intel_pstate/min_perf_pct

the first line sets the cpu governor on all cores on performance mode and the second sets the minumum performance percentage to 100 % (default is 20%) so that intel_pstate driver uses only the highest cpu frequency state and doesnot keep switching back and forth according to the load , this helped reduce massive stuttering and lagging for me as the cpu kept running on its highest frequency state constantly.

Those changes arenot permanent and are reset after boot .

U have yourself a nice weekend and happy gaming to you too ^^

dominicm · 2016-10-20 20:36:48

Good to hear people getting this working. I am running the VM on Skylake 6700k with RX480 myself. Out of curiosity anyone else experience any issues with certain tasks running in the background of the host impacting performance? In particular I see stuttering when unrar is unpacking in the background even though cpu usage is quite low. The syatem drive should not be an issue as I am using a passed through ssd for the VM. Wonder if there's some hardware limitation...? I was hoping to have gaming PC and NAS in one but it's not without it's issues so far

Last edited by dominicm (2016-10-20 20:37:12)

Velociraptor_sb · 2016-10-21 10:25:42

dominicm wrote:

Good to hear people getting this working. I am running the VM on Skylake 6700k with RX480 myself. Out of curiosity anyone else experience any issues with certain tasks running in the background of the host impacting performance? In particular I see stuttering when unrar is unpacking in the background even though cpu usage is quite low. The syatem drive should not be an issue as I am using a passed through ssd for the VM. Wonder if there's some hardware limitation...? I was hoping to have gaming PC and NAS in one but it's not without it's issues so far

Could be related to RAM access and/or availability , Are you using hugepages for the vm ? did you preallocate memory for it ? as rar usually uses quite a lot of ram depending on the compression options . try the -mem-prealloc flag if your using qemu . for my vm with 12288 MB of ram

sudo (...) qemu-system-x86_64 (...)  -m 12288 -mem-path /dev/hugepages -mem-prealloc -cpu host,kvm=off,hv_time,hv_relaxed,hv_vapic,hv_spinlocks=0x1fff,hv_vendor_id=Nvidia -smp 4,sockets=1,cores=4,threads=1 (...)

Also if you are using hugepages , the number of pages u assigned to the vm will be completely unavailable to to the host , in my case i am left with 20 gigs for the host out of 32 gigs of total ram so keep an eye out for your available ram on the host.And knowing rar , if u used a relatively large word size during compression unrar might need a large amount of memory and if that memory isnot available for the host the hanging and stuttering starts (or maybe even a total freeze depending on the situation)

hope this helps

Last edited by Velociraptor_sb (2016-10-21 10:30:13)

Velociraptor_sb · 2016-10-21 12:49:25

deafeningsylence wrote:

I think ovmf should work, no idea though why it didn't for you

ovmf doesnot work on cards that are not uefi enabled , mine as well as most older cards are only bios enabled therefore its impossible to use ovmf.

Arch Linux

#26 2016-09-23 12:58:29

Re: Attempting VGA Passthrough, no video output

#27 2016-09-23 13:22:24

Re: Attempting VGA Passthrough, no video output

#28 2016-09-23 13:28:25

Re: Attempting VGA Passthrough, no video output

#29 2016-09-23 13:47:02

Re: Attempting VGA Passthrough, no video output

#30 2016-09-23 13:51:54

Re: Attempting VGA Passthrough, no video output

#31 2016-09-24 13:16:56

Re: Attempting VGA Passthrough, no video output

#32 2016-09-24 17:22:24

Re: Attempting VGA Passthrough, no video output

#33 2016-10-19 12:33:51

Re: Attempting VGA Passthrough, no video output

#34 2016-10-20 20:11:12

Re: Attempting VGA Passthrough, no video output

#35 2016-10-20 20:36:48

Re: Attempting VGA Passthrough, no video output

#36 2016-10-21 10:25:42

Re: Attempting VGA Passthrough, no video output

#37 2016-10-21 12:49:25

Re: Attempting VGA Passthrough, no video output

Board footer