KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

dave558 · 2014-10-14 12:53:22

Hi all

I'm trying to pass through a graphic card to my virtual guests. Steps I've performed so far are as follows:
* Compiled kernel version 3.10 with vfio support (CONFIG_VFIO_PCI_VGA=y)
* Disabled radeon kernel modules via the blacklist in /etc/modprobe.d.
* Unbinded the graphics card from the PCI bus and assigned it the VFIO for pass through.

I have installed qemu-kvm through yum but it appears that this RPM has not been compiled for VFIO. When trying assign the graphic card to guests there is no option to assign device to vfio-pci. The partial command below results in errors:
/usr/libexec/qemu-kvm -device vfio-pci

I have read also that qemu 2.0+ supports VFIO and I am thinking of compiling qemu from the QEMU website. Although Qemu can run as a stand-alone emulator, I am assumming it will use the kernel-module automatically if available. Is this the case?

Also will there be any difference in performance between qemu compiled from source and qemu-kvm? Or is there anything else you can recommend? Thank you.

evilsephiroth · 2014-10-14 17:03:55

Hi to all...

I've got a headless server(Proxmox) with kvm/qemu git with fully working passthrough(pciex) to various vm.

I can passthrough nvidia gpu to windows vm, dvb-s pci express card to ubuntu and a pciexpress ethernet card to another ubuntu host.

My machine runs on kernel 3.16.2 with some patches (intel arbiter) and latest qemu.

I'm trying to create an openelec VM to use as media center and thus I need intel gpu passthrough.

Basically I don't need a monitor attached because I'm administering with ssh/webinterface.

Grub Settings

GRUB_CMDLINE_LINUX_DEFAULT="quiet intel_iommu=on  i915.enable_hd_vgaarb=1 pci-stub.ids=10de:1380,10de:0fbc,4254:0952,7470:3468,8086:0152,8086:1e20"

lspci

00:00.0 Host bridge: Intel Corporation Xeon E3-1200 v2/3rd Gen Core processor DRAM Controller (rev 09)
00:01.0 PCI bridge: Intel Corporation Xeon E3-1200 v2/3rd Gen Core processor PCI Express Root Port (rev 09)
00:02.0 VGA compatible controller: Intel Corporation Xeon E3-1200 v2/3rd Gen Core processor Graphics Controller (rev 09)
00:14.0 USB controller: Intel Corporation 7 Series/C210 Series Chipset Family USB xHCI Host Controller (rev 04)
00:16.0 Communication controller: Intel Corporation 7 Series/C210 Series Chipset Family MEI Controller #1 (rev 04)
00:1a.0 USB controller: Intel Corporation 7 Series/C210 Series Chipset Family USB Enhanced Host Controller #2 (rev 04)
00:1b.0 Audio device: Intel Corporation 7 Series/C210 Series Chipset Family High Definition Audio Controller (rev 04)
00:1c.0 PCI bridge: Intel Corporation 7 Series/C210 Series Chipset Family PCI Express Root Port 1 (rev c4)
00:1c.1 PCI bridge: Intel Corporation 7 Series/C210 Series Chipset Family PCI Express Root Port 2 (rev c4)
00:1c.2 PCI bridge: Intel Corporation 7 Series/C210 Series Chipset Family PCI Express Root Port 3 (rev c4)
00:1c.4 PCI bridge: Intel Corporation 7 Series/C210 Series Chipset Family PCI Express Root Port 5 (rev c4)
00:1d.0 USB controller: Intel Corporation 7 Series/C210 Series Chipset Family USB Enhanced Host Controller #1 (rev 04)
00:1e.0 PCI bridge: Intel Corporation 82801 PCI Bridge (rev a4)
00:1f.0 ISA bridge: Intel Corporation B75 Express Chipset LPC Controller (rev 04)
00:1f.2 SATA controller: Intel Corporation 7 Series/C210 Series Chipset Family 6-port SATA Controller [AHCI mode] (rev 04)
00:1f.3 SMBus: Intel Corporation 7 Series/C210 Series Chipset Family SMBus Controller (rev 04)
01:00.0 VGA compatible controller: NVIDIA Corporation GM107 [GeForce GTX 750 Ti] (rev a2)
01:00.1 Audio device: NVIDIA Corporation Device 0fbc (rev a1)
02:00.0 Multimedia video controller: Conexant Systems, Inc. CX23885 PCI Video and Audio Decoder (rev 04)
03:00.0 SATA controller: ASMedia Technology Inc. ASM1062 Serial ATA Controller (rev 01)
04:00.0 Ethernet controller: Realtek Semiconductor Co., Ltd. RTL8111/8168/8411 PCI Express Gigabit Ethernet Controller (rev 06)
05:00.0 Ethernet controller: Realtek Semiconductor Co., Ltd. RTL8111/8168/8411 PCI Express Gigabit Ethernet Controller (rev 06)

lspci -n

00:00.0 0600: 8086:0150 (rev 09)
00:01.0 0604: 8086:0151 (rev 09)
00:02.0 0300: 8086:0152 (rev 09)
00:14.0 0c03: 8086:1e31 (rev 04)
00:16.0 0780: 8086:1e3a (rev 04)
00:1a.0 0c03: 8086:1e2d (rev 04)
00:1b.0 0403: 8086:1e20 (rev 04)
00:1c.0 0604: 8086:1e10 (rev c4)
00:1c.1 0604: 8086:1e12 (rev c4)
00:1c.2 0604: 8086:1e14 (rev c4)
00:1c.4 0604: 8086:1e18 (rev c4)
00:1d.0 0c03: 8086:1e26 (rev 04)
00:1e.0 0604: 8086:244e (rev a4)
00:1f.0 0601: 8086:1e49 (rev 04)
00:1f.2 0106: 8086:1e02 (rev 04)
00:1f.3 0c05: 8086:1e22 (rev 04)
01:00.0 0300: 10de:1380 (rev a2)
01:00.1 0403: 10de:0fbc (rev a1)
02:00.0 0400: 14f1:8852 (rev 04)
03:00.0 0106: 1b21:0612 (rev 01)
04:00.0 0200: 10ec:8168 (rev 06)
05:00.0 0200: 10ec:8168 (rev 06)

KVM command

/usr/bin/kvm -id 105 -chardev 'socket,id=qmp,path=/var/run/qemu-server/105.qmp,server,nowait' -mon 'chardev=qmp,mode=control' -vnc unix:/var/run/qemu-server/105.vnc,x509,password -pidfile /var/run/qemu-server/105.pid -daemonize -smbios 'type=1,uuid=2233690f-30ac-499f-ad7e-91a1e2f62792' -name Openelec -smp 'sockets=1,cores=1' -nodefaults -boot 'menu=on' -vga none -cpu 'kvm64,kvm=off,+lahf_lm,+x2apic,+sep' -k it -m 2048 -readconfig /usr/share/qemu-server/pve-q35.cfg -device 'usb-tablet,id=tablet,bus=ehci.0,port=1' -device 'vfio-pci,host=00:02.0,id=hostpci0,bus=ich9-pcie-port-1,addr=0x0,x-vga=on' -device 'virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x3' -iscsi 'initiator-name=iqn.1993-08.org.debian:01:d4d6185a22' -drive 'file=/var/lib/vz/template/iso/ubuntu-14.04.1-desktop-amd64.iso,if=none,id=drive-ide2,media=cdrom,aio=native' -device 'ide-cd,bus=ide.1,unit=0,drive=drive-ide2,id=ide2,bootindex=200' -drive 'file=/var/lib/vz/images/105/vm-105-disk-1.qcow2,if=none,id=drive-ide0,format=qcow2,aio=native,cache=none' -device 'ide-hd,bus=ide.0,unit=0,drive=drive-ide0,id=ide0,bootindex=100' -netdev 'type=tap,id=net0,ifname=tap105i0,script=/var/lib/qemu-server/pve-bridge,downscript=/var/lib/qemu-server/pve-bridgedown,vhost=on' -device 'virtio-net-pci,mac=EE:4E:D1:CD:4A:51,netdev=net0,bus=pci.0,addr=0x12,id=net0,bootindex=300' -machine 'type=q35'

dmesg

Device is ineligible for IOMMU domain attach due to platform RMRR requirement.

Modprobe.d/blacklist.conf

i915
nouveau 
snd-hda-intel-hdmi
snd-hda-codec-hdmi

Intel audio/video is correctly assigned to pci-stub and iommu group are correct but when I run I get

vm: -device vfio-pci,host=00:02.0,id=hostpci0,bus=ich9-pcie-port-1,addr=0x0,x-vga=on: vfio: failed to set iommu for container: Operation not permitted
kvm: -device vfio-pci,host=00:02.0,id=hostpci0,bus=ich9-pcie-port-1,addr=0x0,x-vga=on: vfio: failed to setup container for group 2
kvm: -device vfio-pci,host=00:02.0,id=hostpci0,bus=ich9-pcie-port-1,addr=0x0,x-vga=on: vfio: failed to get group 2
kvm: -device vfio-pci,host=00:02.0,id=hostpci0,bus=ich9-pcie-port-1,addr=0x0,x-vga=on: Device initialization failed.
kvm: -device vfio-pci,host=00:02.0,id=hostpci0,bus=ich9-pcie-port-1,addr=0x0,x-vga=on: Device 'vfio-pci' could not be initialized

I've seen this patch

https://lkml.org/lkml/2014/7/3/580 that disables devices using rmrr.

My doubt is if I'm correctly not loading intel drivers so device is free for passthrough...

Thanks in advance

kaeptnb · 2014-10-14 17:10:23

aw wrote:

Does this resolve the emulation failure: https://git.kernel.org/cgit/virt/kvm/kv … 383dd0b7c8

Hi aw,

i have a similar regression - but without passing a nvidia card to the guest (i pass a ati radeon hd to the guest) whichs works flawless with 3.16, but with 3.17 (with and without your patch) i get:

KVM internal error. Suberror: 1
emulation failure
EAX=00000000 EBX=00000000 ECX=00000000 EDX=000106a5
ESI=00000000 EDI=00000000 EBP=00000000 ESP=00000000
EIP=0000fff2 EFL=00000002 [-------] CPL=0 II=0 A20=1 SMM=0 HLT=0
ES =0000 00000000 0000ffff 00009300
CS =f000 ffff0000 0000ffff 00009b00
SS =0000 00000000 0000ffff 00009300
DS =0000 00000000 0000ffff 00009300
FS =0000 00000000 0000ffff 00009300
GS =0000 00000000 0000ffff 00009300
LDT=0000 00000000 0000ffff 00008200
TR =0000 00000000 0000ffff 00008b00
GDT=     00000000 0000ffff
IDT=     00000000 0000ffff
CR0=60000010 CR2=00000000 CR3=00000000 CR4=00000000
DR0=0000000000000000 DR1=0000000000000000 DR2=0000000000000000 DR3=0000000000000000
DR6=00000000ffff0ff0 DR7=0000000000000400
EFER=0000000000000000
Code=90 90 eb c3 90 90 90 90 90 90 00 00 00 00 56 54 46 00 90 90 <eb> ac 90 90 90 90 90 90 90 90 90 90 90 90 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00

The only change is kernel upgrade from 3.16 to 3.17. I already tried your patch - doesn't seem to help

Greetings

zsph · 2014-10-14 18:52:48

aw wrote:

DanaGoyette wrote:
I actually get something similar when I stop or start (I forgot which) a VM, if I have the AMD High-Definition Audio Controller bound to the VM. Blacklisting snd-hda-intel on the host prevents the panic, but also prevents the integrated Realtek sound card from working on the host.
EDIT: By "similar", I mean a very similar, if not the same, backtrace.
Try using pci-stub.ids instead of blacklisting the driver

Thank you both. For the whole week not a single crush! I'm so happy

Calrama · 2014-10-14 19:05:58

Denso wrote:

Hi everyone .
Can anyone who uses OVMF successfully post his entire qemu launch command ?
OVMF might be the answer to all my hangs issues on VMs' reboots , but it still doesn't detect any of my drives .
I don't use libvirt , just plain shell scripts .
Much appreciated .

qemu-system-x86_64 \
    -pflash /usr/share/qemu/bios_ovmf.bin \
    -enable-kvm -cpu host,hv_time,kvm=off \
    -smp cpus=4,cores=4,threads=1 \
    -m 8GiB \
    -name WinGamingOVMF,process=WinGamingOVMF \
    -device vfio-pci,host=01:00.0,multifunction=on,x-vga=on \
    -device vfio-pci,host=01:00.1 \
    -daemonize \
    -rtc base=localtime \
    -soundhw hda \
    -vga none \
    -device virtio-scsi-pci,id=scsi \
    -drive file=/dev/partitions/wingamingovmf,if=none,id=disk_primary,format=raw \
        -device ide-hd,bus=ide.0,drive=disk_primary \
    -drive file=/dev/partitions/games,if=none,id=disk_games,format=raw \
        -device scsi-hd,bus=scsi.0,drive=disk_games \
    -netdev bridge,br=br0,id=veth0 -device virtio-net-pci,netdev=veth0 -usb

I'm using this on Gentoo, not on Arch, but the above command should be the same. The relevant stuff is this:
Kernel: 3.16.3-gentoo (with all the patches in the OP of this thread, I will try 3.17 without any of these patches later, because iirc none of the patches should be necessary when using OVMF)
Architecture: x86_64
QEMU: GIT master on commit 2472b6c07bb50179019589af1c22f43935ab7f5c
EDK2 (OVMF): SVN trunk on revision 16195

I know that you only asked for the command, but I'd like to share some of my notes from my adventure about using OVMF (started last saturday because pretty much every other VM boot would make my host freeze and force me to power down my whole system the old-fashioned way, got it working sunday):

First, the graphics card you want to pass through absolutely must have either a hybrid, or uefi-only bios; if your graphics card has only
legacy bios (as my GTX 660 TI did), you need to flash it to hybrid / uefi-only, or OVMF won't do anything for you. If you do try
to use a legacy-bios graphics card with OVMF, your virtual machine will detect the passed-through graphics card (even by name in the device manager in case of Windows), but it will not be useable (any monitor you connect to the graphics card will not receive any signal from it / Windows will not detect any monitor connected to the graphics card).

Furthermore, if you want to use Windows (which I'm going to assume), I think you need at least Windows 8 (I tested it only with Windows 8.1 and Windows 7, but Windows 8.1 and 8 extremely similar in nature, so I'm going to assume that they behave the same in regards to this - please correct me on this if I am wrong), because when trying to boot the Windows 7 installation disk, after the usual "Windows is loading files..." the Windows 7 install freezes as soon as the Windows logo shows up.

Additionally, the newer Q35 architecture (option "-M q35") does not seem to work well with Windows 8+. While I could install Windows 8.1 perfectly fine while having q35 enabled, after the installation - at first bootup - the "Getting devices ready" freezes at 15% (on multiple tries, every time). Even if the system is setup completely without q35 (or having it only enabled until first reboot, then shutoff the machine and disable it), enabling q35 will then consistently freeze the VM at the Windows logo.

When trying to install Windows 8+, I would suggest including the relevant disc images like this:

	-drive file=/path/to/${WINDOWS_IMAGE},id=iso_install,if=none \
		-device scsi-cd,drive=iso_install \
	-cdrom /path/to/virtio-win-0.1-81.iso`

These two should ensure that OVMF recognizes the windows disc image as a UEFI boot option, while having the windows virtio drivers available at install time. I had some trouble with QEMU+OVMF not recognizing any non-virtio disc images as UEFI-bootage (they did not show up on the UEFI shell / at the boot manager). On that note: When trying to install, you will likely automatically enter the EFI shell initially instead of the VM booting from the only available EFI boot option (the windows disc image). The OVMF EFI shell should be a bunch of yellow text (listing all pci devices you could theoretically try to boot from) on black background and a prompt. Quit the prompt (Enter "exit") and you should be taken to the UEFI bios, where you should select "Boot manager", followed by the (emulated) scsi cd drive, that contains the windows disc image.

Also, I suggest compiling OVMF yourself (The "bios_ovmf.bin" in the above command is in fact a symlink to $HOME/src/edk2/Build/OvmfX64/DEBUG_GCC48/QEMU/bios.bin), I found the instructions fairly easy to follow:
First setup and do first build like this:
http://tianocore.sourceforge.net/wiki/U … ke_systems
Afterwards build your real OVMF bios like this:
http://tianocore.sourceforge.net/wiki/How_to_build_OVMF

Lastly, if you use a consumer NVIDIA graphics card, the latest driver (imho) you should use is 340.52, because the 344.11 driver includes a new check whether or not "hv_time" is active and will make your card go to 800x600 with a "Code 43". If you want to use the latest driver, you could also remove the "hv_time" from the qemu command, but according to the QEMU changelog (http://wiki.qemu.org/ChangeLog/2.0) this option can give you a significant performance boost for exactly the type of calculations game engines tend to make heavy use of. See https://forums.geforce.com/default/topi … 00#4314345 for more information on this issue.

Alright, I think that should cover everything you need to know when switching from the usual passthrough to OVMF.
I hope the above information helps you and/or anyone else trying OVMF out, I wrote this down as more of a memo on how I did it than a proper guide, so forgive any lack in quality (I intend to make a proper gist about this when I have the time).

Last edited by Calrama (2014-10-14 19:34:16)

PureTryOut · 2014-10-14 19:23:00

I've been trying to get this too work for a few hours, but something is somehow going wrong and I don't understand how.

The problem is that it seems I can't use pci-stub.
When I add it to /etc/mkinitcpio.conf in the MODULES section it gives me

==> ERROR: Hook 'pci-stub' cannot be found
==> WARNING: errors were encountered during the build. The image may not be complete.

The output of lspci -nn gives me

02:00.0 VGA compatible controller [0300]: NVIDIA Corporation Device [10de:13c0] (rev a1)
02:00.1 Audio device [0403]: NVIDIA Corporation Device [10de:0fbb] (rev a1)

So i've added the following to my bootloader (Grub)

pci-stub.ids=10de:13c0,10de:0fbb

But after rebooting and running "dmesg | grep pci-stub" I only get the following output

[    0.000000] Command line: BOOT_IMAGE=/boot/vmlinuz-linux-mainline root=UUID=a0e2f143-41fc-4a4f-905d-6a12ac5b0237 rw quiet pci-stub.ids=10de:13c0,10de:0fbb
[    0.000000] Kernel command line: BOOT_IMAGE=/boot/vmlinuz-linux-mainline root=UUID=a0e2f143-41fc-4a4f-905d-6a12ac5b0237 rw quiet pci-stub.ids=10de:13c0,10de:0fbb

So stub didn't actually claim my graphic card.

If I continue without stub it only gives me more errors later on.
I can't just disable my graphic card driver because both of my cards are NVIDIA.

Using the kernel provided in the first post (3.17 + acs override patch + i915 vga arbiter fixes) and a GTX 980 to passthrough combined with a i7 4790k with VT-d enabled.

nbhs · 2014-10-14 19:33:11

PureTryOut wrote:

I've been trying to get this too work for a few hours, but something is somehow going wrong and I don't understand how.
The problem is that it seems I can't use pci-stub.
When I add it to /etc/mkinitcpio.conf in the MODULES section it gives me
==> ERROR: Hook 'pci-stub' cannot be found
==> WARNING: errors were encountered during the build. The image may not be complete.

nbhs wrote:

NOTE: If pci-stub was built as a module, you'll need to modify /etc/mkinitcpio.conf and add pci-stub in the MODULES section, after that you need to update your initramfs like this.

PureTryOut · 2014-10-14 19:50:19

So no answer then? Like I said I added pci-stub to the MODULES section, but it just gave me the error that it could not find the pci-stub hook...

nbhs · 2014-10-14 19:54:49

PureTryOut wrote:

So no answer then? Like I said I added pci-stub to the MODULES section, but it just gave me the error that it could not find the pci-stub hook...

Check the hooks section then, perhaps you're missing a '"' or, delete /etc/mkinitpio.conf then reinstall it (pacman -S mkinitcpio) and try again

PureTryOut · 2014-10-14 20:19:02

Thanks for your help, that actually worked!
I removed the file, and then reinstalled mkinitcpio, added it to modules, and this time it somehow worked.
No clue what went wrong.

Anyways now when I run the test

# qemu-system-x86_64 -enable-kvm -M q35 -m 1024 -cpu host \
-smp 6,sockets=1,cores=6,threads=1 \
-bios /usr/share/qemu/bios.bin -vga none \
-device ioh3420,bus=pcie.0,addr=1c.0,multifunction=on,port=1,chassis=1,id=root.1 \
-device vfio-pci,host=02:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on \
-device vfio-pci,host=02:00.1,bus=root.1,addr=00.1

it gives me the following errors:

qemu-system-x86_64: -device vfio-pci,host=02:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on: vfio: error no iommu_group for device
qemu-system-x86_64: -device vfio-pci,host=02:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on: Device initialization failed.
qemu-system-x86_64: -device vfio-pci,host=02:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on: Device 'vfio-pci' could not be initialized

Any clue what's happening?

nbhs · 2014-10-14 20:26:57

PureTryOut wrote:

Thanks for your help, that actually worked!
I removed the file, and then reinstalled mkinitcpio, added it to modules, and this time it somehow worked.
No clue what went wrong.

Anyways now when I run the test

# qemu-system-x86_64 -enable-kvm -M q35 -m 1024 -cpu host \
-smp 6,sockets=1,cores=6,threads=1 \
-bios /usr/share/qemu/bios.bin -vga none \
-device ioh3420,bus=pcie.0,addr=1c.0,multifunction=on,port=1,chassis=1,id=root.1 \
-device vfio-pci,host=02:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on \
-device vfio-pci,host=02:00.1,bus=root.1,addr=00.1

it gives me the following errors:

qemu-system-x86_64: -device vfio-pci,host=02:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on: vfio: error no iommu_group for device
qemu-system-x86_64: -device vfio-pci,host=02:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on: Device initialization failed.
qemu-system-x86_64: -device vfio-pci,host=02:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on: Device 'vfio-pci' could not be initialized

Any clue what's happening?

Uhm yes, you need to bind your devices to vfio-pci, i suggest you re-read the guide, also if you are using an nvidia card on your host, you might need to patch it (though im using an nvidia card myself on the host, with unpatched drivers), also since your card is pretty new and it suports uefi, you might want to try OVMF

Last edited by nbhs (2014-10-14 20:29:02)

PureTryOut · 2014-10-14 20:39:09

Yeah I figured out I hadn't done that, my mistake, sorry.

Now I have bind my device to vfio-pci, but I get the following:

qemu-system-x86_64: -device vfio-pci,host=02:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on: vfio: error, group 1 is not viable, please ensure all devices within the iommu_group are bound to their vfio bus driver.
qemu-system-x86_64: -device vfio-pci,host=02:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on: vfio: failed to get group 1
qemu-system-x86_64: -device vfio-pci,host=02:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on: Device initialization failed.
qemu-system-x86_64: -device vfio-pci,host=02:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on: Device 'vfio-pci' could not be initialized

I guess my main card (the one I use in the host, a GTX 770) is also in this group, which is of course not bound to the vfio bus driver.
Is this because I need to patch my driver?

Also, my system is using BIOS, not UEFI, or is this UEFI different? I rather not reinstall my pc to EUFI when I don't have too.

EDIT: Installed nvidia-dkms, didn't help sadly, still the same error.

Last edited by PureTryOut (2014-10-14 20:48:05)

nbhs · 2014-10-14 21:11:36

PureTryOut wrote:

Yeah I figured out I hadn't done that, my mistake, sorry.
Now I have bind my device to vfio-pci, but I get the following:
qemu-system-x86_64: -device vfio-pci,host=02:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on: vfio: error, group 1 is not viable, please ensure all devices within the iommu_group are bound to their vfio bus driver.
qemu-system-x86_64: -device vfio-pci,host=02:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on: vfio: failed to get group 1
qemu-system-x86_64: -device vfio-pci,host=02:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on: Device initialization failed.
qemu-system-x86_64: -device vfio-pci,host=02:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on: Device 'vfio-pci' could not be initialized
I guess my main card (the one I use in the host, a GTX 770) is also in this group, which is of course not bound to the vfio bus driver.
Is this because I need to patch my driver?
Also, my system is using BIOS, not UEFI, or is this UEFI different? I rather not reinstall my pc to EUFI when I don't have too.
EDIT: Installed nvidia-dkms, didn't help sadly, still the same error.

That error means that you need to bind ALL the devices on that iommu group to vfio pci, also, you dont need to reinstall anything to use OVMF

ls /sys/bus/pci/devices/0000\:02\:00.0/iommu_group/devices/

Last edited by nbhs (2014-10-14 21:15:26)

PureTryOut · 2014-10-14 21:28:46

Well yes I understand that, and I would love to bind all of the devices in that group to vfio pci, but it's not possible.
The GPU I use as display in the host (a GTX 770) is also in that group:

ls /sys/bus/pci/devices/0000\:02\:00.0/iommu_group/devices/
0000:00:01.0  0000:01:00.0  0000:02:00.0
0000:00:01.1  0000:01:00.1  0000:02:00.1

lspci -nn
00:01.0 PCI bridge [0604]: Intel Corporation Xeon E3-1200 v3/4th Gen Core Processor PCI Express x16 Controller [8086:0c01] (rev 06)
00:01.1 PCI bridge [0604]: Intel Corporation Xeon E3-1200 v3/4th Gen Core Processor PCI Express x8 Controller [8086:0c05] (rev 06)
01:00.0 VGA compatible controller [0300]: NVIDIA Corporation GK104 [GeForce GTX 770] [10de:1184] (rev a1)
01:00.1 Audio device [0403]: NVIDIA Corporation GK104 HDMI Audio Controller [10de:0e0a] (rev a1)
02:00.0 VGA compatible controller [0300]: NVIDIA Corporation Device [10de:13c0] (rev a1)
02:00.1 Audio device [0403]: NVIDIA Corporation Device [10de:0fbb] (rev a1)

When I bind it my screen just turns black and I can't do anything anymore without restarting my pc.

Anyways I will look into OVMF.

nbhs · 2014-10-14 21:58:03

PureTryOut wrote:

Well yes I understand that, and I would love to bind all of the devices in that group to vfio pci, but it's not possible.
The GPU I use as display in the host (a GTX 770) is also in that group:

ls /sys/bus/pci/devices/0000\:02\:00.0/iommu_group/devices/
0000:00:01.0  0000:01:00.0  0000:02:00.0
0000:00:01.1  0000:01:00.1  0000:02:00.1

lspci -nn
00:01.0 PCI bridge [0604]: Intel Corporation Xeon E3-1200 v3/4th Gen Core Processor PCI Express x16 Controller [8086:0c01] (rev 06)
00:01.1 PCI bridge [0604]: Intel Corporation Xeon E3-1200 v3/4th Gen Core Processor PCI Express x8 Controller [8086:0c05] (rev 06)
01:00.0 VGA compatible controller [0300]: NVIDIA Corporation GK104 [GeForce GTX 770] [10de:1184] (rev a1)
01:00.1 Audio device [0403]: NVIDIA Corporation GK104 HDMI Audio Controller [10de:0e0a] (rev a1)
02:00.0 VGA compatible controller [0300]: NVIDIA Corporation Device [10de:13c0] (rev a1)
02:00.1 Audio device [0403]: NVIDIA Corporation Device [10de:0fbb] (rev a1)

When I bind it my screen just turns black and I can't do anything anymore without restarting my pc.

Anyways I will look into OVMF.

That's what the acs override patch is for

dwe11er · 2014-10-14 22:42:48

I'd like to point out one thing:

If you're about to flash card without EFI GOP with BIOS (from similar card) that support EFI - don't do this (unless you're sure). You can pass BIOS with qemu adding 'romfile=...' like this:

-device vfio-pci,host=01:00.0,id=hostdev1,bus=pci.0,multifunction=on,addr=0x7,rombar=0,romfile=/var/lib/libvirt/firmware/HD7870_EFI.rom

Ansa89 · 2014-10-15 11:36:44

aw wrote:

Oh dear, you've gone to all this trouble to compile OVMF but haven't read this post to determine why you'd want it and how to use it? Have I missed something in describing the motivation and requirements there? OVMF isn't a "test", it's an alternative to VGA. Since you're not getting any output from the monitor, it seems like you have an issue with VGA routing. Avoiding VGA by using OVMF is one method of dealing with this.

Ok, now it's a bit more clear.

aw wrote:

Regarding the parse error, how recently did you grab that copy of rom-parser? I fixed a similar issue in the code on the 4th. Otherwise please send me a copy of the ROM binary so I can fix the code for it. Since you already have the edk2 tree, there's also a parser there you can use, details in the comments here.

I used latest code from your master branch.
At what address should I send you the bios?

aw wrote:

For the -bios option; you shouldn't ever need a -bios option. The default bios that comes with QEMU is fine for seabios use and for OVMF you'd add an option like:
-drive if=pflash,format=raw,readonly,file=/path/to/OVMF.fd

Ok, added that line and I can boot in a sort of efi shell, but can't see any kind of "boot menu" or something similar.
Consider that I'm connecting through vnc so I could miss some initial boot screens.

aw wrote:

Your previous posts indicate that you're using Windows 7, so please pay particular attention to the guest operating support for UEFI. You'd need to upgrade your guest to Windows 8 for OVMF.

Searching on the web, I found some guides about how booting/installing Windows 7 over UEFI. Can I follow such a tutorial or OVMF/VM is different?

aw wrote:

If that's not any option, we'll need to go back to debugging your VGA routing issue and 'dmesg | grep -i vga' might shed some light on what's wrong. Please also pastebin the entire dmesg log from the host.

# dmesg | grep -i vga
Command line: BOOT_IMAGE=/boot/vmlinuz-3.16.5 root=UUID=a4837900-ca63-43d4-bc2c-cf3f37cff7c6 ro i915.enable_hd_vgaarb=1 intel_iommu=on pci-stub.ids=10de:0fc6,10de:0e1b quiet
Kernel command line: BOOT_IMAGE=/boot/vmlinuz-3.16.5 root=UUID=a4837900-ca63-43d4-bc2c-cf3f37cff7c6 ro i915.enable_hd_vgaarb=1 intel_iommu=on pci-stub.ids=10de:0fc6,10de:0e1b quiet
Console: colour VGA+ 80x25
vgaarb: setting as boot device: PCI:0000:00:02.0
vgaarb: device added: PCI:0000:00:02.0,decodes=io+mem,owns=io+mem,locks=none
vgaarb: device added: PCI:0000:01:00.0,decodes=io+mem,owns=none,locks=none
vgaarb: loaded
vgaarb: bridge control possible 0000:01:00.0
vgaarb: no bridge control possible 0000:00:02.0
[drm] Replacing VGA console driver
vgaarb: device changed decodes: PCI:0000:00:02.0,olddecodes=io+mem,decodes=io:owns=io+mem

PS: sorry for the delay, but I got some trouble.

Duelist · 2014-10-15 12:53:24

https://bbs.archlinux.org/viewtopic.php … 7#p1465127

Downgraded bios to the oldest version having IOMMU support - still getting page faults.
If i pass only one GPU without sound - windows 7 still fails to boot with BSOD 116.

I am using nvidia gpu as a host gpu, by the way.

Passing second(host:02:00.00) GPU doesn't show any video output, but there is still
AMD-Vi: Event logged [IO_PAGE_FAULT device=02:00.0 domain=0x0021 address=0x0000000122dde000 flags=0x0010]
errors.

UPD: something wild and unknown happened - windows now may boot, but it'll do it with horrible memory corruption artifacts looking like horizontal stripes, and then BSOD. Cryptic messages in dmesg are gone...

Last edited by Duelist (2014-10-15 16:33:13)

Denso · 2014-10-15 14:56:33

Calrama wrote:

Denso wrote:
Hi everyone .
Can anyone who uses OVMF successfully post his entire qemu launch command ?
OVMF might be the answer to all my hangs issues on VMs' reboots , but it still doesn't detect any of my drives .
I don't use libvirt , just plain shell scripts .
Much appreciated .
qemu-system-x86_64 \
    -pflash /usr/share/qemu/bios_ovmf.bin \
    -enable-kvm -cpu host,hv_time,kvm=off \
    -smp cpus=4,cores=4,threads=1 \
    -m 8GiB \
    -name WinGamingOVMF,process=WinGamingOVMF \
    -device vfio-pci,host=01:00.0,multifunction=on,x-vga=on \
    -device vfio-pci,host=01:00.1 \
    -daemonize \
    -rtc base=localtime \
    -soundhw hda \
    -vga none \
    -device virtio-scsi-pci,id=scsi \
    -drive file=/dev/partitions/wingamingovmf,if=none,id=disk_primary,format=raw \
        -device ide-hd,bus=ide.0,drive=disk_primary \
    -drive file=/dev/partitions/games,if=none,id=disk_games,format=raw \
        -device scsi-hd,bus=scsi.0,drive=disk_games \
    -netdev bridge,br=br0,id=veth0 -device virtio-net-pci,netdev=veth0 -usb
I'm using this on Gentoo, not on Arch, but the above command should be the same. The relevant stuff is this:
Kernel: 3.16.3-gentoo (with all the patches in the OP of this thread, I will try 3.17 without any of these patches later, because iirc none of the patches should be necessary when using OVMF)
Architecture: x86_64
QEMU: GIT master on commit 2472b6c07bb50179019589af1c22f43935ab7f5c
EDK2 (OVMF): SVN trunk on revision 16195

I know that you only asked for the command, but I'd like to share some of my notes from my adventure about using OVMF (started last saturday because pretty much every other VM boot would make my host freeze and force me to power down my whole system the old-fashioned way, got it working sunday):
First, the graphics card you want to pass through absolutely must have either a hybrid, or uefi-only bios; if your graphics card has only
legacy bios (as my GTX 660 TI did), you need to flash it to hybrid / uefi-only, or OVMF won't do anything for you. If you do try
to use a legacy-bios graphics card with OVMF, your virtual machine will detect the passed-through graphics card (even by name in the device manager in case of Windows), but it will not be useable (any monitor you connect to the graphics card will not receive any signal from it / Windows will not detect any monitor connected to the graphics card).
Furthermore, if you want to use Windows (which I'm going to assume), I think you need at least Windows 8 (I tested it only with Windows 8.1 and Windows 7, but Windows 8.1 and 8 extremely similar in nature, so I'm going to assume that they behave the same in regards to this - please correct me on this if I am wrong), because when trying to boot the Windows 7 installation disk, after the usual "Windows is loading files..." the Windows 7 install freezes as soon as the Windows logo shows up.
Additionally, the newer Q35 architecture (option "-M q35") does not seem to work well with Windows 8+. While I could install Windows 8.1 perfectly fine while having q35 enabled, after the installation - at first bootup - the "Getting devices ready" freezes at 15% (on multiple tries, every time). Even if the system is setup completely without q35 (or having it only enabled until first reboot, then shutoff the machine and disable it), enabling q35 will then consistently freeze the VM at the Windows logo.
When trying to install Windows 8+, I would suggest including the relevant disc images like this:
	-drive file=/path/to/${WINDOWS_IMAGE},id=iso_install,if=none \
		-device scsi-cd,drive=iso_install \
	-cdrom /path/to/virtio-win-0.1-81.iso`
These two should ensure that OVMF recognizes the windows disc image as a UEFI boot option, while having the windows virtio drivers available at install time. I had some trouble with QEMU+OVMF not recognizing any non-virtio disc images as UEFI-bootage (they did not show up on the UEFI shell / at the boot manager). On that note: When trying to install, you will likely automatically enter the EFI shell initially instead of the VM booting from the only available EFI boot option (the windows disc image). The OVMF EFI shell should be a bunch of yellow text (listing all pci devices you could theoretically try to boot from) on black background and a prompt. Quit the prompt (Enter "exit") and you should be taken to the UEFI bios, where you should select "Boot manager", followed by the (emulated) scsi cd drive, that contains the windows disc image.
Also, I suggest compiling OVMF yourself (The "bios_ovmf.bin" in the above command is in fact a symlink to $HOME/src/edk2/Build/OvmfX64/DEBUG_GCC48/QEMU/bios.bin), I found the instructions fairly easy to follow:
First setup and do first build like this:
http://tianocore.sourceforge.net/wiki/U … ke_systems
Afterwards build your real OVMF bios like this:
http://tianocore.sourceforge.net/wiki/How_to_build_OVMF
Lastly, if you use a consumer NVIDIA graphics card, the latest driver (imho) you should use is 340.52, because the 344.11 driver includes a new check whether or not "hv_time" is active and will make your card go to 800x600 with a "Code 43". If you want to use the latest driver, you could also remove the "hv_time" from the qemu command, but according to the QEMU changelog (http://wiki.qemu.org/ChangeLog/2.0) this option can give you a significant performance boost for exactly the type of calculations game engines tend to make heavy use of. See https://forums.geforce.com/default/topi … 00#4314345 for more information on this issue.
Alright, I think that should cover everything you need to know when switching from the usual passthrough to OVMF.
I hope the above information helps you and/or anyone else trying OVMF out, I wrote this down as more of a memo on how I did it than a proper guide, so forgive any lack in quality (I intend to make a proper gist about this when I have the time).

Wow .. Thank you very much for this detailed walkthrough , I'm going to try ASAP and return with results !

Thank you for taking your time typing this detailed information !

EDIT !!!! :

YES ! It worked ! finally it detected my Windows 8.1 ISO ! The magic word is : SCSI .

It didn't detect regular IDE , but it detected SCSI .

Thank you Calrama !

Last edited by Denso (2014-10-15 15:13:44)

PureTryOut · 2014-10-15 16:48:16

nbhs wrote:

That's what the acs override patch is for

Yes that is what I understand as well... I have installed your version of the kernel (3.17 + acs override patch + i915 vga arbiter fixes), but it still gives me the same error.

Calrama · 2014-10-15 16:51:23

Denso wrote:

Calrama wrote:
Denso wrote:
Hi everyone .
Can anyone who uses OVMF successfully post his entire qemu launch command ?
OVMF might be the answer to all my hangs issues on VMs' reboots , but it still doesn't detect any of my drives .
I don't use libvirt , just plain shell scripts .
Much appreciated .
qemu-system-x86_64 \
    -pflash /usr/share/qemu/bios_ovmf.bin \
    -enable-kvm -cpu host,hv_time,kvm=off \
    -smp cpus=4,cores=4,threads=1 \
    -m 8GiB \
    -name WinGamingOVMF,process=WinGamingOVMF \
    -device vfio-pci,host=01:00.0,multifunction=on,x-vga=on \
    -device vfio-pci,host=01:00.1 \
    -daemonize \
    -rtc base=localtime \
    -soundhw hda \
    -vga none \
    -device virtio-scsi-pci,id=scsi \
    -drive file=/dev/partitions/wingamingovmf,if=none,id=disk_primary,format=raw \
        -device ide-hd,bus=ide.0,drive=disk_primary \
    -drive file=/dev/partitions/games,if=none,id=disk_games,format=raw \
        -device scsi-hd,bus=scsi.0,drive=disk_games \
    -netdev bridge,br=br0,id=veth0 -device virtio-net-pci,netdev=veth0 -usb
[...]
Wow .. Thank you very much for this detailed walkthrough , I'm going to try ASAP and return with results !
Thank you for taking your time typing this detailed information !
EDIT !!!! :
YES ! It worked ! finally it detected my Windows 8.1 ISO ! The magic word is : SCSI .
It didn't detect regular IDE , but it detected SCSI .
Thank you Calrama !

That's good to hear and you're welcome.

Thanks for pointing out that SCSI is the key and not VIRTIO itself, I'll take that into consideration when writing the gist.

Have you experienced any host freezes / crashes with OVMF? I'm just curious, because for me it has - so far - been the all-cure regarding that.
Since I started using OVMF I haven't had a single crash or host freeze and even better, OVMF seems to be a lot faster than the normal bios.
My Gaming VM takes only 15 seconds from the time I issue the qemu command until it's fully-booted onto the Windows desktop.

Denso · 2014-10-15 17:51:18

Calrama wrote:

Denso wrote:
Calrama wrote:
qemu-system-x86_64 \
    -pflash /usr/share/qemu/bios_ovmf.bin \
    -enable-kvm -cpu host,hv_time,kvm=off \
    -smp cpus=4,cores=4,threads=1 \
    -m 8GiB \
    -name WinGamingOVMF,process=WinGamingOVMF \
    -device vfio-pci,host=01:00.0,multifunction=on,x-vga=on \
    -device vfio-pci,host=01:00.1 \
    -daemonize \
    -rtc base=localtime \
    -soundhw hda \
    -vga none \
    -device virtio-scsi-pci,id=scsi \
    -drive file=/dev/partitions/wingamingovmf,if=none,id=disk_primary,format=raw \
        -device ide-hd,bus=ide.0,drive=disk_primary \
    -drive file=/dev/partitions/games,if=none,id=disk_games,format=raw \
        -device scsi-hd,bus=scsi.0,drive=disk_games \
    -netdev bridge,br=br0,id=veth0 -device virtio-net-pci,netdev=veth0 -usb
[...]
Wow .. Thank you very much for this detailed walkthrough , I'm going to try ASAP and return with results !
Thank you for taking your time typing this detailed information !
EDIT !!!! :
YES ! It worked ! finally it detected my Windows 8.1 ISO ! The magic word is : SCSI .
It didn't detect regular IDE , but it detected SCSI .
Thank you Calrama !
That's good to hear and you're welcome.
Thanks for pointing out that SCSI is the key and not VIRTIO itself, I'll take that into consideration when writing the gist.
Have you experienced any host freezes / crashes with OVMF? I'm just curious, because for me it has - so far - been the all-cure regarding that.
Since I started using OVMF I haven't had a single crash or host freeze and even better, OVMF seems to be a lot faster than the normal bios.
My Gaming VM takes only 15 seconds from the time I issue the qemu command until it's fully-booted onto the Windows desktop.

Hi

I can confirm that NOW I can reboot VM without freezing the host ! I believe that is because OVMF is legacy-free (VGA-free) .

On a side note , OVMF initializes the machine faster than SeaBIOS does . (a couple seconds faster) .

I can't thank you enough (I'm typing this reply from my new OVMF VM !)

evilsephiroth · 2014-10-15 18:09:28

so no one passing through intel gpu?

Duelist · 2014-10-15 18:21:09

http://awilliam.github.io/presentations/KVM-Forum-2014

Fresh aw's presentation works like sort of a FAQ for everybody. Now i know that i definitely need OVMF.

dwe11er · 2014-10-15 18:43:41

evilsephiroth wrote:

so no one passing through intel gpu?

You need KvmGT (which isn't released for wide public yet) or XenGT.

Arch Linux

#2951 2014-10-14 12:53:22

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

#2952 2014-10-14 17:03:55

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

#2953 2014-10-14 17:10:23

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

#2954 2014-10-14 18:52:48

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

#2955 2014-10-14 19:05:58

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

#2956 2014-10-14 19:23:00

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

#2957 2014-10-14 19:33:11

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

#2958 2014-10-14 19:50:19

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

#2959 2014-10-14 19:54:49

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

#2960 2014-10-14 20:19:02

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

#2961 2014-10-14 20:26:57

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

#2962 2014-10-14 20:39:09

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

#2963 2014-10-14 21:11:36

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

#2964 2014-10-14 21:28:46

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

#2965 2014-10-14 21:58:03

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

#2966 2014-10-14 22:42:48

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

#2967 2014-10-15 11:36:44

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

#2968 2014-10-15 12:53:24

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

#2969 2014-10-15 14:56:33

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

#2970 2014-10-15 16:48:16

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

#2971 2014-10-15 16:51:23

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

#2972 2014-10-15 17:51:18

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

#2973 2014-10-15 18:09:28

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

#2974 2014-10-15 18:21:09

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

#2975 2014-10-15 18:43:41

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Board footer