Arch Linux

aw

Member

Registered: 2013-10-04

Posts: 921

Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

When you're trying OVMF, please make sure you're using the pure-EFI binary and not one that mentions the CSM.  We explicitly do not want the CSM, which can run the VGA BIOS and brings back all the issues with VGA arbitration.

---

http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

deniv

Member

Registered: 2013-10-16

Posts: 27

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Suddenly, i can not boot OVMF's built-in shell or start Windows8.1 installation CD in UEFI mode. There is just blank screen and a cursor on serial0, and nothing on any VGA or video output. I need OVMF to avoid IO_PAGE_FAULTS referring to my first GPU. The windows ISO checked to be correct and it does boot on real hardware.

I think I'm hitting the same bug as you. I've tried some combinations of qemu/ovmf versions and qemu options but still got blank screen whenever OVMF touches the EFI part of the vbios (vga works as I mentioned before). So what's left? I compared EfiRom output:

Image 2 -- Offset 0x10000
  ROM header contents
    Signature              0xAA55
    PCIR offset            0x001C
    Signature               PCIR
    Vendor ID               0x1002
    Device ID               0x683F
    Length                  0x0018
    Revision                0x0000
    DeviceListOffset        0x00
    Class Code              0x030000
    Image size              0xEE00
    Code revision:          0x0000
    MaxRuntimeImageLength   0x00
    ConfigUtilityCodeHeaderOffset 0x4F47
    DMTFCLPEntryPointOffset 0x2050
    Indicator               0x80   (last image)
    Code type               0x03   (EFI image)
  EFI ROM header contents
    EFI Signature          0x0EF1
    Compression Type       0x0001 (compressed)
    Machine type           0x8664 (unknown)
    Subsystem              0x000B (EFI boot service driver)
    EFI image offset       0x005C (@0x1005C)

with some vbioses from TechPowerUp. The result: there are no striking differences. The only lines that change between different vbioses: Device ID (obviously), Image Size, EFI image offset, and Image offset (the first line). I'm no expert, but judging from this I belive my vbios isn't broken. I can only assume it's a bug in OVMF. I've noticed aw's GT635 has "Image 2 -- Offset 0xF400" while my rom's "Image 2 -- Offset 0x10000" Could that additional digit make a difference for OVMF? In any case, 0x10000 isn't rare. Many R290 use the same offset.

P.S. Yes, I'm using pure-efi.

aw

Member

Registered: 2013-10-04

Posts: 921

Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Another sad fact of EFI is that vendors ship broken ROMs.  The XFX R7790 is one such card.  I thought I was lucky finding an EFI ROM on a 7700-series card, but it doesn't work.  If I install it as the primary graphics in my system and configure the BIOS to prefer the EFI BIOS, the system won't boot.  It behaves the same way in an OVMF VM.  I've contacted XFX support but they seem unable to do anything other than repackage a ROM provided by AMD and indicate that AMD has not provided EFI support for this card.  They seem unable to comprehend that they've shipped a card that not only doesn't support EFI, but in fact prevents using it since executing the EFI ROM seems to be the cause of the hang.  It would be interesting to trace the execution in the emulator and see where it breaks, maybe it could be patched.

---

http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

gyrfalco

Member

Registered: 2014-09-30

Posts: 10

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

hello, guys.

My spec: AMD A10-6700 with integrated Radeon HD 8670D (host GPU) + Radeon HD 7870 (GV-R787OC-2GD - guest GPU).

I have problems with Catalyst driver (13.*, 14.4, 14.7, 14.8, 14.9) install: system (win 8.1) crashes during driver installation, with or without CCC + display driver installation, or just updading default win display driver in device manager. 

I tried both seabios and OVMF with same result.

qemu-2.1.50, kernel 3.17.1:

iommu=pt pci-stub.ids=1002:6818,1002:aab0

Machine starts as:

qemu-system-x86_64 -enable-kvm -m 4096 -cpu host,hv_time,kvm=off \
-smp 4,sockets=1,cores=4,threads=1 -vga none -boot menu=on -rtc base=localtime \
-device vfio-pci,host=01:00.0,multifunction=on \
-drive file=windows.img,id=disk,format=raw,if=none -device ide-hd,bus=ide.0,drive=disk \ 
-drive if=pflash,format=raw,readonly,file=/usr/share/qemu/efi-bios.bin

Is it possible (actually only drivers installation, because without driver VM runs smooth and fine, through second monitor) when host gpu is integrated Radeon and guest gpu is 7870?

Thanks, sorry for my English.

Last edited by gyrfalco (2014-10-26 11:28:59)

doubledr

Member

Registered: 2013-12-26

Posts: 13

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Did you also modify the VBIOS for the new device ID?  If the VBIOS doesn't match the card seabios won't run it.  However, with a K2 you can avoid all the craziness of VGA and pass the card to the guest as a secondary display.  Pre-boot and early guest boot will happen on the emulated VGA.  When the Nvidia driver is loaded the emulated VGA is disabled and the assigned device will take over.  Use one of the standard emulated VGA devices and drop x-vga=on from the assigned device in your guest config.

Hi, aw.

I tried it today. Passing a real(not modded) Nvidia Grid k2 results in code 12 in my windows 2008 vm... Same as a Quadro 400... Look's like "x-vga" is a must...

Use 440FX not Q35

l have some difficulty with libvirt. No matter what I did, if I start my vm with libvirt, I always get the "Cannot allocate memory" error. "ulimit -l unlimited" doesn't help at all... Is anyone have similar issue? Now I can only start my vm manually with a long qemu command

If libvirt doesn't know you're doing device assignment it won't set your limits correctly.  If you use 440FX instead of Q35 you don't need any qemu:commandline sections and should be allowing libvirt to manage everything.

Thx, aw. It works!

I found that my libvirt used cgroup to control memory limit, maybe that is the reason why ulimit command doesn't work.

Nov 24 20:19:42 cave-lin kernel: [  368.001538] vfio-pci 0000:01:00.0: enabling device (0000 -> 0003)
Nov 24 20:19:42 cave-lin kernel: [  368.001719] vfio_ecap_init: 0000:01:00.0 hiding ecap 0x19@0x900
Nov 24 20:20:20 cave-lin kernel: [  405.590323] kvm: zapping shadow pages for mmio generation wraparound

deniv

Member

Registered: 2013-10-16

Posts: 27

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Another sad fact of EFI is that vendors ship broken ROMs.

That's sad indeed. How can they screw such an easy-to-test feature? At least vgaarb patches still work.

It would be interesting to trace the execution in the emulator and see where it breaks, maybe it could be patched.

Wish I could help, but it's beyond me. Please inform us if you manage to get any HD77xx card work in EFI mode.

Ansa89

Member

Registered: 2014-08-30

Posts: 20

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

First of all: my hard disk exploded (it was a bit old), so ATM I'm unable to proceed with further tests. However I managed to do a last test before it broke.

One way to at least reduce that is to use hardware en- and decoding. To enable hardware encoding you need to install Geforce experience, click on the "shadow play" button in its GUI and flip the left-positioned virtual switch that they show in the new popup windows: Turn it on once, then off again. After that you will need to access the Inhome-Streaming options in Steam's settings (on the client), open up the advanced host settings, and enable hardware encoding (You might also want to set the same option on the VM's steam installation for good measure, but it shouldn't be necessary).

You might also try to set the client options to "fast" and set the bandwidth limit in the advanced client options to automatic (should it not already be set to it).

I will try those settings after buying a new hard disk.

I've found that even with hardware en- and decoding I could still detect a slight difference between host and guest while inside an FPS, because the camera didn't rotate completely smoothly on the guest. Just enough that it would get me killed in multiplayer. The only way I could rid of any noticeable latency when Inhome-streaming through an emulated VirtIO network card was to limit the bandwith manually to 3Mbit, at which 1080p looks like someone threw several buckets full of different-coloured paint at my screen from the inside (and 720p is unacceptable to me).

Really not a good deal.

I do not own one, but I'm going to try that out when I have enough time to make an informed decision about picking one.

Informed decision:
- go on an e-commerce website
- click "networking" section
- click "pci ethernet" section
- sort items by "ascending price"
- close you eyes an choose one card
I really don't know if there is one card better than others.

It should. Just to be sure: You are able to stream regardless of whether your VM gets an emulated card or not (so both "-vga none" and "-vga std" work)?

Last test done: yes, I can stream regardless of "-vga std" or "-vga none" is passed to qemu.

Duelist

Member

Registered: 2014-09-22

Posts: 358

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Please, mark if you manage to get that cards doing something more useful than powering down when passing the ROM via romfile=.

Well, I can confirm that this rom works in vga mode. I used it to load my standard Win7 VM.
EFI, on the other hand, is a pain. First I tried to install Win8.1 just with QXL, but OVMF can't load the installation DVD (no drivers for the UDF filesystem?). I copied the DVD to a fat32 partition and managed to get the loader running, but it still fails quite soon with "can't find the media" error (maybe the installation process looks for UDF and dies because there is fat32?). Anyway, just to test EFI vbios I tried connecting the GPU annnnnd... nothing. I get a black screen and 100% one core utilisation, can't see OVMF, can't get to Windows installer.

P.S. I noticed in gpuz that this new bios (ver 015.032) bumps the default core clock to 900MHz while the original bios (ver 015.014) sets default to 820MHz.

Try installing windows 8.1 via USB flash drive. There was an official way of doing this. That should solve all controller-related issues, however, i've managed to install windows8 from virtio-scsi-pci scsi-cd to virtio-blk-pci using the virtio.iso image connected through ide-cd..
And the clock difference - that's why i warned you - my cards have 800MHz and they're pretty hot, so getting them work any faster would be dangerous.

We explicitly do not want the CSM

That's an interesting remark.

If I install it as the primary graphics in my system and configure the BIOS to prefer the EFI BIOS, the system won't boot.

Well, i'm going to visit my friend with ASUS-AMD-Windows system somewhere near wednesday and hopefully, i'll provide some testing information from the real hardware combination.

They seem unable to comprehend that they've shipped a card that not only doesn't support EFI, but in fact prevents using it since executing the EFI ROM seems to be the cause of the hang.  It would be interesting to trace the execution in the emulator and see where it breaks, maybe it could be patched.

Well, after i get some testing information, if the VBIOS is really broken - that would be extremely fun. ASUS support seem to actually understand what the hell i am trying to ask them - last time i've asked if it's possible to combine three HD7750 with Richland or Trinity-based CPU(not APU) on one of their motherboards, and suddenly, i've got an answer.

And i've finally managed to boot windows7(not 8!) installation ISO from OVMF. Let's see what will happen.

Last edited by Duelist (2014-10-25 12:14:58)

---

The forum rules prohibit requesting support for distributions other than arch.
I gave up. It was too late.
What I was trying to do.
The reference about VFIO and KVM VGA passthrough.

gyrfalco

Member

Registered: 2014-09-30

Posts: 10

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

And i've finally managed to boot windows7(not 8!) installation ISO from OVMF.

can you explain how? thanks.

PureTryOut

Member

Registered: 2014-09-23

Posts: 64

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

So I decided to reinstall my whole PC, this time in UEFI mode instead of the old BIOS.
Got everything up and running and was following this tutorial again but ran into some problems (again...).

This time i'm stuck at the following (again...):

qemu-system-x86_64: -device vfio-pci,host=02:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on: vfio: error no iommu_group for device
qemu-system-x86_64: -device vfio-pci,host=02:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on: Device initialization failed.
qemu-system-x86_64: -device vfio-pci,host=02:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on: Device 'vfio-pci' could not be initialized

I'm 100% sure I binded the devices, by both using systemd at boot, and manually running vfio-bind 0000:02:00.0 0000:02:00.1.
I get no errors so I just assume it works, but it clearly doesn't...

aw

Member

Registered: 2013-10-04

Posts: 921

Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

So I decided to reinstall my whole PC, this time in UEFI mode instead of the old BIOS.
Got everything up and running and was following this tutorial again but ran into some problems (again...).

This time i'm stuck at the following (again...):

qemu-system-x86_64: -device vfio-pci,host=02:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on: vfio: error no iommu_group for device
qemu-system-x86_64: -device vfio-pci,host=02:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on: Device initialization failed.
qemu-system-x86_64: -device vfio-pci,host=02:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on: Device 'vfio-pci' could not be initialized

I'm 100% sure I binded the devices, by both using systemd at boot, and manually running vfio-bind 0000:02:00.0 0000:02:00.1.
I get no errors so I just assume it works, but it clearly doesn't...

Intel_iommu=on boot option

---

http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

PureTryOut

Member

Registered: 2014-09-23

Posts: 64

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

So I decided to reinstall my whole PC, this time in UEFI mode instead of the old BIOS.
Got everything up and running and was following this tutorial again but ran into some problems (again...).

This time i'm stuck at the following (again...):

qemu-system-x86_64: -device vfio-pci,host=02:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on: vfio: error no iommu_group for device
qemu-system-x86_64: -device vfio-pci,host=02:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on: Device initialization failed.
qemu-system-x86_64: -device vfio-pci,host=02:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on: Device 'vfio-pci' could not be initialized

I'm 100% sure I binded the devices, by both using systemd at boot, and manually running vfio-bind 0000:02:00.0 0000:02:00.1.
I get no errors so I just assume it works, but it clearly doesn't...

Intel_iommu=on boot option

Aah, knew I forgot something, thanks!

Now i'm back to the initial problem: qemu only giving me a console, unless I remove the -vga none option.
I've read that several others have this problem too, but I haven't found a solution yet.

Last edited by PureTryOut (2014-10-25 21:48:45)

dwe11er

Member

Registered: 2014-03-18

Posts: 73

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Please, mark if you manage to get that cards doing something more useful than powering down when passing the ROM via romfile=.

Well, I can confirm that this rom works in vga mode. I used it to load my standard Win7 VM.
EFI, on the other hand, is a pain. First I tried to install Win8.1 just with QXL, but OVMF can't load the installation DVD (no drivers for the UDF filesystem?). I copied the DVD to a fat32 partition and managed to get the loader running, but it still fails quite soon with "can't find the media" error (maybe the installation process looks for UDF and dies because there is fat32?). Anyway, just to test EFI vbios I tried connecting the GPU annnnnd... nothing. I get a black screen and 100% one core utilisation, can't see OVMF, can't get to Windows installer.

P.S. I noticed in gpuz that this new bios (ver 015.032) bumps the default core clock to 900MHz while the original bios (ver 015.014) sets default to 820MHz.

Make sure you're not using ovmf from repo, it's useless.

risho

Member

Registered: 2011-09-06

Posts: 44

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

which ovmf are you using?

dave558

Member

Registered: 2013-11-28

Posts: 4

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Hi

I've just compiled a kernel (3.10) with vfio support (CONFIG_VFIO_PCI_VGA=y).

I have installed qemu-kvm through yum but it appears that this RPM has not been compiled for VFIO. When trying assign the graphic card to guests there is no option to assign device to vfio-pci.

I have read also that qemu 2.0+ supports VFIO and I am thinking of compiling qemu from the QEMU website. Although Qemu can run as a stand-alone emulator, I am assumming it will use the kernel-module automatically if available. Is this the case?

Also will there be any difference in performance between qemu compiled from source and qemu-kvm? Or is there anything else you can recommend? Thank yoo.

NOTE: This is a CentOS system, that is why I have installed the default RPM install qemu-kvm rather than QEMU.

dwe11er

Member

Registered: 2014-03-18

Posts: 73

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

which ovmf are you using?

ovmf-svn from AUR.

kristopher004

Member

Registered: 2014-06-30

Posts: 33

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Updated from 3.16 to  3.17.1-1 and now the vm skips. When I move the mouse or drag a window the mouse or window will freeze for a sec every few seconds. This happen to anyone else when they went to kernel 3.17?

My problem ended up being that I recompiled 3.17 and didn't recompile my nvidia driver, if anyone else has a similar problem.

Duelist

Member

Registered: 2014-09-22

Posts: 358

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

And i've finally managed to boot windows7(not 8!) installation ISO from OVMF.

can you explain how? thanks.

Actually windows 7 supports GPT and UEFI natibely, you just have to force it.
http://technet.microsoft.com/en-gb/libr … 10%29.aspx
You can use the shell, or you can type exit and navigate yourself through menus.
You have to use pure-efi OVMF image.

Last edited by Duelist (2014-10-26 21:28:18)

---

The forum rules prohibit requesting support for distributions other than arch.
I gave up. It was too late.
What I was trying to do.
The reference about VFIO and KVM VGA passthrough.

winie

Member

Registered: 2014-03-01

Posts: 17

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Is there a way to fix the new nvidia drivers looking for hyper-v parameters? possibly in future releases of qemu or from inside windows?

gyrfalco

Member

Registered: 2014-09-30

Posts: 10

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Actually windows 7 supports GPT and UEFI natibely, you just have to force it.
...
You have to use pure-efi OVMF image.

Thanks, this ovmf-svn image is pure-efi? If not, can you share link of pure-efi image.

Also, can you show qemu start command, which you run win7 with ovmf, thanks.

Last edited by gyrfalco (2014-10-27 08:41:59)

dwe11er

Member

Registered: 2014-03-18

Posts: 73

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Thanks, this ovmf-svn image is pure-efi? If not, can you share link of pure-efi image.

Yes, it is.

EDIT: ovmf qemu args

-drive file=/usr/share/ovmf/x64/ovmf_code_x64.bin,if=pflash,format=raw,unit=0,readonly=on -drive file=/var/lib/libvirt/qemu/nvram/windows_VARS.fd,if=pflash,format=raw,unit=1

Last edited by dwe11er (2014-10-27 10:33:20)

Duelist

Member

Registered: 2014-09-22

Posts: 358

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Thanks, this ovmf-svn image is pure-efi? If not, can you share link of pure-efi image.

Yes, it is.

EDIT: ovmf qemu args

-drive file=/usr/share/ovmf/x64/ovmf_code_x64.bin,if=pflash,format=raw,unit=0,readonly=on -drive file=/var/lib/libvirt/qemu/nvram/windows_VARS.fd,if=pflash,format=raw,unit=1

Hmm, do you explicitly need that unit=int64 parameter set if using if=pflash? It seem to be working without it, but i can't test it properly by changing some setting in UEFI setup menu - there is no settings.

---

The forum rules prohibit requesting support for distributions other than arch.
I gave up. It was too late.
What I was trying to do.
The reference about VFIO and KVM VGA passthrough.

dwe11er

Member

Registered: 2014-03-18

Posts: 73

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Thanks, this ovmf-svn image is pure-efi? If not, can you share link of pure-efi image.

Yes, it is.

EDIT: ovmf qemu args

-drive file=/usr/share/ovmf/x64/ovmf_code_x64.bin,if=pflash,format=raw,unit=0,readonly=on -drive file=/var/lib/libvirt/qemu/nvram/windows_VARS.fd,if=pflash,format=raw,unit=1

Hmm, do you explicitly need that unit=int64 parameter set if using if=pflash? It seem to be working without it, but i can't test it properly by changing some setting in UEFI setup menu - there is no settings.

No idea, probably not. It might be used to order pflash drives (I'm using libvirt).

aw

Member

Registered: 2013-10-04

Posts: 921

Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Is there a way to fix the new nvidia drivers looking for hyper-v parameters? possibly in future releases of qemu or from inside windows?

Are you asking for a solution beyond simply not using hyper-v extensions?  You might want to revert back to an old nvidia driver and test whether it would be worthwhile.  I don't know of anyone looking a further solutions.  If you need hyper-v extensions, nvidia professional series cards or AMD cards might be a better option.

---

http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Casperian

Member

Registered: 2014-10-28

Posts: 6

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Hey guys, been watching this thread for some time.
So I recently have set up a pass through and passed a 5450 to a Windows 7 Guest. This was easy but I had a BSOD issue caused by device reset which I worked round by not passing HDMI audio through. (BSOD 116)
From here I then tried to pass a GTX 750 through. Everything went fine, Qemu starts and gives me the warning "Bar 3 mmap unsupported" and the card is not outputting anything. I boot Qemu a second time without -vga none to see if I can find a problem and find the device is passed through, it is and I get the latest stable nvidia drivers installed to see if that fixes anything. The card is currently throwing code 12 in windows.

From what I found over the last few hours, I will check PCI-stub binding to the device tomorrow. Also thinking about trying pci-assign to see if it works.

The code I am using is;

sudo qemu-system-x86_64 -enable-kvm -M q35 -m 12288 -cpu host \
-smp 4,sockets=1,cores=4,threads=1 \
-bios /usr/share/qemu/bios.bin -vga none \
-device ioh3420,bus=pcie.0,addr=1c.0,multifunction=on,port=1,chassis=1,id=root.1 \
-device vfio-pci,host=04:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on \
-device vfio-pci,host=04:00.1,bus=root.1,addr=00.1 \
-drive file=/home/server/window_nvidia750.img,id=disk,format=raw -device ide-hd,bus=ide.0,drive=disk \
-boot menu=on 

Any advice on the issue would be awesome
I can't post anything more now as the computer is not on hand

Cheers
Casper

Last edited by Casperian (2014-10-28 09:05:00)