Arch Linux

aw

Member

Registered: 2013-10-04

Posts: 921

Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

This means you are not running a new enough QEMU.  You need at least 2.1 for the kvm=off property.

Ok, so got from git qemu 2.1.50, but when I then try to run my vm I just get a 'QEMU 2.1.50 monitor' prompt?
Even added full path /usr/local/bin/qemu... to make sure was using.
And then I still get the same KVM internal error. Suberror: 1 emulation failure after a short time :\
May have to put the card in another machine to confirm all is working ok for a start (used card I just picked up). Or just stick with ATI cards..

Does this resolve the emulation failure: https://git.kernel.org/cgit/virt/kvm/kv … 383dd0b7c8

---

http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Ansa89

Member

Registered: 2014-08-30

Posts: 20

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Those errors are what we need to debug.  rombar=0 tells QEMU to not expose a PCI option ROM for the device.  Following that with romfile= makes QEMU put that rom in the genroms section, which is sort of like having it builtin to the BIOS.  The problem with that is that it is no longer associated with the device, so unless you know ROM has the ability to scan through all the devices to find the hardware, you've probably just broken the option ROM execution.

Remove the rombar option and report the error, if there is one.

Ok, don't know about that.

I removed "rombar" and "romfile", magically qemu doesn't report any error/warning.
The only messages I got are from dmesg:

vfio-pci 0000:01:00.0: enabling device (0000 -> 0003)
vfio_ecap_init: 0000:01:00.0 hiding ecap 0x19@0x900
kvm: zapping shadow pages for mmio generation wraparound

However the problem persist: black screen and can't connect to VNC server inside Windows 7.
The strangest thing is that I can arping the VM and nmap report its ip as "up", but both nmap and vncviewer can't find open ports.

Wimma77

Member

Registered: 2014-08-12

Posts: 15

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Does this resolve the emulation failure: https://git.kernel.org/cgit/virt/kvm/kv … 383dd0b7c8

Thanks for the response.
Hmm, not sure sorry. Prob over my head
But I'm trying to install Win7, so not 8.1 and no nvidia driver (yet).
Can't even get Win7 to install.
So much more painful that AMD/ATI!

andy123

Member

Registered: 2011-11-04

Posts: 169

Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Does this resolve the emulation failure: https://git.kernel.org/cgit/virt/kvm/kv … 383dd0b7c8

Thanks for the response.
Hmm, not sure sorry. Prob over my head
But I'm trying to install Win7, so not 8.1 and no nvidia driver (yet).
Can't even get Win7 to install.
So much more painful that AMD/ATI!

How did you install the qemu git version? Applying patches really isn't that complicated…

---

i'm sorry for my poor english wirting skills…

Wimma77

Member

Registered: 2014-08-12

Posts: 15

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

How did you install the qemu git version? Applying patches really isn't that complicated…

git clone, config, make, install.
Didn't spit out any errors I noticed. Took a while to configure though.

slis

Member

Registered: 2014-06-02

Posts: 127

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

theres no pain with nvidia... it just works. you got config issues.... @ansa89 your last log shows it's running normally...

sinny

Member

Registered: 2014-05-24

Posts: 75

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

theres no pain with nvidia... it just works. you got config issues.... @ansa89 your last log shows it's running normally...

well, it actually differs.
while hd 6450 worked almost without any problems, i totally failed getting my gt 430 to work as well - there were many "iterations"/attempts to get it to work and lots of reading to try to get what i am doing wrong. either my card is somehow broken by itself or there is something else i am missing - for now i stick with hd 6450 only, the goal being simultaneous passthrough of 2 gpus to 2 different vms.

p.s. totally failed trying to get gt 430 working via ovmf, but that's another thing to look into. hard to understand what's wrong when no video is shown anywhere and there is no logs aside from usual "enabling device ..."

slis

Member

Registered: 2014-06-02

Posts: 127

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

well yeah... I tried 6 gpus from nvidia and 3 mbos form intel, all working, even on quad sli mbo 4gpu-s 4vm-s... but i guess u have amd mbo or (edit) NOT ..   i don't know but it works for me.

Last edited by slis (2014-08-31 07:26:00)

Ansa89

Member

Registered: 2014-08-30

Posts: 20

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

@ansa89 your last log shows it's running normally...

Ok, but the problem persists, so somewhere there should be something that fails.
Any idea/test/patch/voodoo-magic?

sinny

Member

Registered: 2014-05-24

Posts: 75

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

well yeah... I tried 6 gpus from nvidia and 3 mbos form intel, all working, even on quad sli mbo 4gpu-s 4vm-s... but i guess u have amd mbo or (edit) NOT .. :)  i don't know but it works for me.

MB: asrock z87 pro4
CPU: intel i7 4770
OS: fedora 20 x86_64
kernel: distro-based kernel-3.15.8 with i915 and vga arbiter patches
host GPU: integrated (intel i7 4770)
guest GPU0: radeon hd 6450 (Gigabyte GV-R645-1GI)  <-- worked almost flawlessly for several months already
guest GPU1: nvidia gt 430 (Palit GT 430)  <-- NOT working (most i got from it was showing video on first run (counting since host reboot/startup), than host freeze on second (again, counting since host reboot/startup) gpu passthrough attempt)

seems to me that there is something fundamentally different between nvidia and amd gpu passthrough (from configuration/setup/procedure point of view), but i haven't yet caught up on what exactly it is

slis

Member

Registered: 2014-06-02

Posts: 127

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

@sinny well for REBOOT u need rom file from gpu (atleast that was case for mine 560ti, should be same for 4xx), without rombar option...

@ansa89 you could try to remove that bios-256k option and u don't need gpu romfile option... and if u edit that script when adding gpu, care for spaces / new lines... that could be reason for not booting

Last edited by slis (2014-08-31 11:55:38)

Cenm

Member

Registered: 2014-08-29

Posts: 6

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

I've managed to get everything working for the first boot of the VM but it fails for every subsequent boot with the error 'vfio-pci 0000:03:00.0: Invalid ROM contents'

If I specify romfile=/path/to/rom I don't get the error but nothing appears on the screen.

I have also noticed when I kill the VM whatever was on the screen remains there until I start the VM again, at which point the screen turns off.

Any advice would be great.

Any ideas on getting this fixed? GPU is a GTX 560 Ti. As far as I can tell everything else works fine, but only once.

slis

Member

Registered: 2014-06-02

Posts: 127

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

@cenm are u dumping bios from gpu? or downloading from net? you should dump it... are u using latest qemu? etc...

aw

Member

Registered: 2013-10-04

Posts: 921

Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

p.s. totally failed trying to get gt 430 working via ovmf, but that's another thing to look into. hard to understand what's wrong when no video is shown anywhere and there is no logs aside from usual "enabling device ..."

Just added a new blog post: Does my graphics card support EFI?

---

http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

aw

Member

Registered: 2013-10-04

Posts: 921

Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

I've managed to get everything working for the first boot of the VM but it fails for every subsequent boot with the error 'vfio-pci 0000:03:00.0: Invalid ROM contents'

If I specify romfile=/path/to/rom I don't get the error but nothing appears on the screen.

I have also noticed when I kill the VM whatever was on the screen remains there until I start the VM again, at which point the screen turns off.

Any advice would be great.

Any ideas on getting this fixed? GPU is a GTX 560 Ti. As far as I can tell everything else works fine, but only once.

You haven't told us what you're running or how you're running it.  Recent QEMU, including 2.1 (maybe even 2.0) changed a lot about loading the ROM to make it more reliable.  The 3.17 kernel adds the ability for VFIO to do bus reset when the device is closed, but both functions of the device need to be bound to vfio-pci for this to work.  It works well on Nvidia, but not so well on newer AMD cards which falsely claim to support reset via PM state transition.

---

http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Cenm

Member

Registered: 2014-08-29

Posts: 6

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

@cenm are u dumping bios from gpu? or downloading from net? you should dump it... are u using latest qemu? etc...

You haven't told us what you're running or how you're running it.  Recent QEMU, including 2.1 (maybe even 2.0) changed a lot about loading the ROM to make it more reliable.  The 3.17 kernel adds the ability for VFIO to do bus reset when the device is closed, but both functions of the device need to be bound to vfio-pci for this to work.  It works well on Nvidia, but not so well on newer AMD cards which falsely claim to support reset via PM state transition.

Dumped BIOS from GPU. Running QEMU 2.1.0-2, Kernel 3.16.1-1, vfio bound both the video and audio components of the GPU

Last edited by Cenm (2014-08-31 17:24:26)

aw

Member

Registered: 2013-10-04

Posts: 921

Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

@cenm are u dumping bios from gpu? or downloading from net? you should dump it... are u using latest qemu? etc...

You haven't told us what you're running or how you're running it.  Recent QEMU, including 2.1 (maybe even 2.0) changed a lot about loading the ROM to make it more reliable.  The 3.17 kernel adds the ability for VFIO to do bus reset when the device is closed, but both functions of the device need to be bound to vfio-pci for this to work.  It works well on Nvidia, but not so well on newer AMD cards which falsely claim to support reset via PM state transition.

Dumped BIOS from GPU. Running QEMU 2.1.0-2, Kernel 3.16.1-1, vfio bound

You're still not telling us how you're running it or what hardware you have other than a GTX650Ti.  If you expect your stock distro kernel to work, please see the OP or read-up on the blog below for the condensed version.  If you'd like to report a problem, please provide the base version and list any patches you've applied for QEMU and kernel, the kernel command line options you're booting with, the host and guest graphics devices, whether this is an AMD or Intel system, and the commands/configuration used to start the guest.  Please don't expect us to guess.

---

http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

slis

Member

Registered: 2014-06-02

Posts: 127

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

are there any benefits with using ovfm other than getting rid of i915 patches/vga?

aw

Member

Registered: 2013-10-04

Posts: 921

Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

are there any benefits with using ovfm other than getting rid of i915 patches/vga?

If that isn't enough reason, you may not have a full grasp of the problems with i915 and VGA arbitration

Everything is currently upstream to be able to create OVMF VMs with assigned GPUs and manage them with libvirt.  Eventually your distro will include this.  AFAICT, i915 will never have working VGA arbitration and even if a plan comes about to support it, it could take years to deprecate the existing functionality to be able to implement it.  Not to mention all the other holes with VGA, like the vgacon driver not participating so you may end up with a host login prompt on the guest VGA console.  VGA is a fixed address space shared across the system and our management of it leaves much to be desired.  It's best to stick with the resources specific to your device, which is how we operate under OVMF.

---

http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

slis

Member

Registered: 2014-06-02

Posts: 127

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Thank you for your  answer. Since i use headless host, i guess i don't care about i915 working or not   I thought about using 440fx instead q35 does it make any difference?

aw

Member

Registered: 2013-10-04

Posts: 921

Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Thank you for your  answer. Since i use headless host, i guess i don't care about i915 working or not   I thought about using 440fx instead q35 does it make any difference?

The theory with Q35 is that it more closely matches the host hardware topology and therefore is more likely to "just work".  In a small number of cases, this is true.  The most apparent case is that of a Linux guest using the radeon driver, which attempts to negotiate link speed with the downstream port above it.  In a 440FX model there is no such downstream port and the driver code in the guest effectively segfaults.  If we want to use 440FX here we need to fix the driver rather than change the guest machine to accommodate the driver.  (Note that even in a Q35 model there is no negotiation with the parent downstream port.  The parent device registers are only emulated and do not support negotiation) 

In the majority of cases, the guest drivers work well independent of the VM topology.  This seems to be the case with nouveau on Linux as well as the Nvidia and AMD drivers on Windows (nvidia/fglrx on Linux is untested).  Therefore, using Q35 is often an unnecessary complication.

OVMF doesn't yet support Q35, so 440FX is currently your only option, but I expect that will eventually change.

---

http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

slis

Member

Registered: 2014-06-02

Posts: 127

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Thanks again for answers. Now i know more than i wanted i'll try omfv tommorow got some isp issues right now....

Cenm

Member

Registered: 2014-08-29

Posts: 6

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

@cenm are u dumping bios from gpu? or downloading from net? you should dump it... are u using latest qemu? etc...

You haven't told us what you're running or how you're running it.  Recent QEMU, including 2.1 (maybe even 2.0) changed a lot about loading the ROM to make it more reliable.  The 3.17 kernel adds the ability for VFIO to do bus reset when the device is closed, but both functions of the device need to be bound to vfio-pci for this to work.  It works well on Nvidia, but not so well on newer AMD cards which falsely claim to support reset via PM state transition.

Dumped BIOS from GPU. Running QEMU 2.1.0-2, Kernel 3.16.1-1, vfio bound

You're still not telling us how you're running it or what hardware you have other than a GTX650Ti.  If you expect your stock distro kernel to work, please see the OP or read-up on the blog below for the condensed version.  If you'd like to report a problem, please provide the base version and list any patches you've applied for QEMU and kernel, the kernel command line options you're booting with, the host and guest graphics devices, whether this is an AMD or Intel system, and the commands/configuration used to start the guest.  Please don't expect us to guess.

...

Hardware:
  3930K C2 Stepping
  Asus Extreme Rampage IV
  Nvidia GTX680 (Host (Nouveau Driver))
  Nvidia GTX560Ti (Guest)
 
Stock kernel unpatched, but >3.16 is the only thing mentioned as 'required'
No QEMU Patches are suggested...

Kernel param's
  intel_iommu=on
  pci-stub.ids=10de:1200,10de:0e0c (GTX 560Ti audio and video components)

Basic command on the first page to start the VM + romfile option

qemu-system-x86_64 -enable-kvm -M q35 -m 1024 -cpu host \
  -smp 6,sockets=1,cores=6,threads=1 \
  -bios /usr/share/qemu/bios.bin -vga none \
  -device ioh3420,bus=pcie.0,addr=1c.0,multifunction=on,port=1,chassis=1,id=root.1 \
  -device vfio-pci,host=03:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on,romfile=/path/to/rom \
  -device vfio-pci,host=03:00.1,bus=root.1,addr=00.1

aw

Member

Registered: 2013-10-04

Posts: 921

Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Hardware:
  3930K C2 Stepping
  Asus Extreme Rampage IV
  Nvidia GTX680 (Host (Nouveau Driver))
  Nvidia GTX560Ti (Guest)
 
Stock kernel unpatched, but >3.16 is the only thing mentioned as 'required'
No QEMU Patches are suggested...

Kernel param's
  intel_iommu=on
  pci-stub.ids=10de:1200,10de:0e0c (GTX 560Ti audio and video components)

Basic command on the first page to start the VM + romfile option

qemu-system-x86_64 -enable-kvm -M q35 -m 1024 -cpu host \
  -smp 6,sockets=1,cores=6,threads=1 \
  -bios /usr/share/qemu/bios.bin -vga none \
  -device ioh3420,bus=pcie.0,addr=1c.0,multifunction=on,port=1,chassis=1,id=root.1 \
  -device vfio-pci,host=03:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on,romfile=/path/to/rom \
  -device vfio-pci,host=03:00.1,bus=root.1,addr=00.1

It's possible you need the "other" VGA arbiter patch listed in question 4 of the FAQ.  That bug in arbitration was found with radeon, but if nouveau behaves similarly, the arbiter might be getting stuck with VGA routing enabled to the wrong device.

---

http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Cenm

Member

Registered: 2014-08-29

Posts: 6

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

It's possible you need the "other" VGA arbiter patch listed in question 4 of the FAQ.  That bug in arbitration was found with radeon, but if nouveau behaves similarly, the arbiter might be getting stuck with VGA routing enabled to the wrong device.

Thanks. Will try that, but the same thing happens when the host is running the nvidia drivers.