Arch Linux

fseven

Member

Registered: 2015-04-15

Posts: 16

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

@fseven: you can achieve a similar effect when doing -device virtio-blk-pci with drive /dev/sda or even /dev/sda4(but that will bork the partition table, as the VM will think it's a whole disk). If you poke windows and it's EFI-loader the right way, you may even boot baremetal from that partitions.

I have /dev/sda1 as 500~GB junkyard FS mounted to linux, and windows knows about it(since it has full /dev/sda), but neither OS can mount other's FS. Linux prevents you, windows doesn't do ext4.
There are some caching and scheduling issues present, but they don't affect the performance much.

I haven't heard any success-stories about dedicating a single port to some VM.

Right now I'm trying out the software raid method from the wiki, which seems to have quite passable performance (better with just the -hda option than with  -device virtio-blk-pci).
The host can see and mount the partition as well (though not at once (well I haven't tried, but it wouldn't be smart anyways).).
There are a few tweaks (see here for example  http://wiki.mikejung.biz/KVM_/_Xen?Page … VM_Guests) that might further improve performance, I'm trying that next.

Can you give some more details about "poking windows and it´s EFI-loader the right way"? Furthermore I would expect Windows to complain about changed chipset/driver mismatch and crashing, or is this an issue which doesn't exist in newer Windows versions?

In my experience if you pass the complete disk, you can try booting but windows always bluescreened for me (no wonder I guess, it's quite a different environment.). I don't think it's possible to boot both directly from the disk and via Qemu.

Last edited by fseven (2015-04-17 14:29:57)

mostlyharmless

Member

Registered: 2014-01-16

Posts: 72

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Used to be with Windows XP "hardware profiles", that you could dual boot a VM and bare metal from the same installation...as if you had a docking station.  You had to have a patch to one of the .sys files if I recall correctly.  Haven't tried 7+

fseven

Member

Registered: 2015-04-15

Posts: 16

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

I've now tried a few games and the graphics performance is way below my expectation (CS:GO for example frequently drops below 20 fps at lowest settings).
The graphics driver is installed and the device (Radeon HD6850) passed through as primary.
Could someone point me in the right direction as to what I can do to increase graphics performance? I'm reading as much of this thread as I can but 200 pages is a lot.
Thanks!

Duelist

Member

Registered: 2014-09-22

Posts: 358

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

I've now tried a few games and the graphics performance is way below my expectation (CS:GO for example frequently drops below 20 fps at lowest settings).
The graphics driver is installed and the device (Radeon HD6850) passed through as primary.
Could someone point me in the right direction as to what I can do to increase graphics performance? I'm reading as much of this thread as I can but 200 pages is a lot.
Thanks!

Check the CPU. And disks. I've sudden FPS drops when host heavily loads the hard drive, or spikes when host is changing CPU's frequency.
Dedicating(pinning) CPUs will help.

@mostlyharmless
In win7+ you do this same way as XP - enable the "generic" hard drive controller driver, boot up in a different environment and install the needed drivers. The system won't delete drivers for unplugged hardware, so you should be able to boot both ways.

As for "poking EFI loader the right way" - usually that involves copying EFIVARS from bare metal to VM and modifying them to point to the same ESP(partition with EFI loader) in different environments.

Last edited by Duelist (2015-04-17 22:53:15)

---

The forum rules prohibit requesting support for distributions other than arch.
I gave up. It was too late.
What I was trying to do.
The reference about VFIO and KVM VGA passthrough.

fseven

Member

Registered: 2015-04-15

Posts: 16

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

I've now tried a few games and the graphics performance is way below my expectation (CS:GO for example frequently drops below 20 fps at lowest settings).
The graphics driver is installed and the device (Radeon HD6850) passed through as primary.
Could someone point me in the right direction as to what I can do to increase graphics performance? I'm reading as much of this thread as I can but 200 pages is a lot.
Thanks!

Check the CPU. And disks. I've sudden FPS drops when host heavily loads the hard drive, or spikes when host is changing CPU's frequency.
Dedicating(pinning) CPUs will help.

@mostlyharmless
In win7+ you do this same way as XP - enable the "generic" hard drive controller driver, boot up in a different environment and install the needed drivers. The system won't delete drivers for unplugged hardware, so you should be able to boot both ways.

As for "poking EFI loader the right way" - usually that involves copying EFIVARS from bare metal to VM and modifying them to point to the same ESP(partition with EFI loader) in different environments.

The link in the first post is broken, where can I read up on dedicating/pinning CPUs?

Last edited by fseven (2015-04-17 23:28:48)

Duelist

Member

Registered: 2014-09-22

Posts: 358

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

I've now tried a few games and the graphics performance is way below my expectation (CS:GO for example frequently drops below 20 fps at lowest settings).
The graphics driver is installed and the device (Radeon HD6850) passed through as primary.
Could someone point me in the right direction as to what I can do to increase graphics performance? I'm reading as much of this thread as I can but 200 pages is a lot.
Thanks!

Check the CPU. And disks. I've sudden FPS drops when host heavily loads the hard drive, or spikes when host is changing CPU's frequency.
Dedicating(pinning) CPUs will help.

@mostlyharmless
In win7+ you do this same way as XP - enable the "generic" hard drive controller driver, boot up in a different environment and install the needed drivers. The system won't delete drivers for unplugged hardware, so you should be able to boot both ways.

As for "poking EFI loader the right way" - usually that involves copying EFIVARS from bare metal to VM and modifying them to point to the same ESP(partition with EFI loader) in different environments.

The link in the first post is broken, where can I read up on dedicating/pinning CPUs?

Apart from ITT, you can read redhat's guide. It's very useful when you're pursuiting high performance.

---

The forum rules prohibit requesting support for distributions other than arch.
I gave up. It was too late.
What I was trying to do.
The reference about VFIO and KVM VGA passthrough.

fkoehler

Member

Registered: 2015-04-17

Posts: 7

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Hello,
I have a problem. I guess this question came up before but I did not find it in the depths of this lengthy thread. I have been googleing and experimenting for many hours today and yesterday.  Binding my AMD HD7870 works fine and everything else seems to be fine as well. If I do not use OVMF I can install Windows using vga std on my partition /dev/sda3. If I use vga none and OVMF I see the output on the screen that is connected to the dedicated GPU (which is absolutely correct) but the system does not boot and I end up on the interactive EFI shell (I do not know waht to do with it). Am I just stupid?

I am using ovmf-svn from the AUR and the following command to start qemu:

qemu-system-x86_64 \
    -name win81 \
    -enable-kvm \
    -drive if=pflash,format=raw,readonly,file=/usr/share/ovmf/x64/ovmf_code_x64.bin \
    -drive if=pflash,format=raw,file=/usr/share/ovmf/x64/ovmf_vars_x64.bin \
    -cpu host,kvm=off \
    -smp 4,sockets=1 \
    -m 8G \
    -device vfio-pci,host=01:00.0 \
    -device vfio-pci,host=01:00.1 \
    -vga none \
    -device virtio-scsi-pci,id=scsi \
    -drive file=/home/fabian/win81.iso,id=cd0,if=none \
    -device scsi-cd,drive=cd0 \
    -drive file=/home/fabian/virtio.iso,id=cd1,if=none \
    -device ide-cd,bus=ide.1,drive=cd1 \
    -drive file=/dev/sda3,id=disk0,format=raw,if=none \
    -device scsi-hd,drive=disk0 \
    -boot menu=on

win81.iso is my Windows 8.1 x64 iso file and virtio.iso is the redhat iso with the virtio Windows drivers on it. /dev/sda3 is the drive I want to install Windows on.

Do you have any ideas?

Duelist

Member

Registered: 2014-09-22

Posts: 358

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Hello,
I have a problem. I guess this question came up before but I did not find it in the depths of this lengthy thread. I have been googleing and experimenting for many hours today and yesterday.  Binding my AMD HD7870 works fine and everything else seems to be fine as well. If I do not use OVMF I can install Windows using vga std on my partition /dev/sda3. If I use vga none and OVMF I see the output on the screen that is connected to the dedicated GPU (which is absolutely correct) but the system does not boot and I end up on the interactive EFI shell (I do not know waht to do with it). Am I just stupid?

I am using ovmf-svn from the AUR and the following command to start qemu:

qemu-system-x86_64 \
    -name win81 \
    -enable-kvm \
    -drive if=pflash,format=raw,readonly,file=/usr/share/ovmf/x64/ovmf_code_x64.bin \
    -drive if=pflash,format=raw,file=/usr/share/ovmf/x64/ovmf_vars_x64.bin \
    -cpu host,kvm=off \
    -smp 4,sockets=1 \
    -m 8G \
    -device vfio-pci,host=01:00.0 \
    -device vfio-pci,host=01:00.1 \
    -vga none \
    -device virtio-scsi-pci,id=scsi \
    -drive file=/home/fabian/win81.iso,id=cd0,if=none \
    -device scsi-cd,drive=cd0 \
    -drive file=/home/fabian/virtio.iso,id=cd1,if=none \
    -device ide-cd,bus=ide.1,drive=cd1 \
    -drive file=/dev/sda3,id=disk0,format=raw,if=none \
    -device scsi-hd,drive=disk0 \
    -boot menu=on

win81.iso is my Windows 8.1 x64 iso file and virtio.iso is the redhat iso with the virtio Windows drivers on it. /dev/sda3 is the drive I want to install Windows on.

Do you have any ideas?

Type exit or quit(whatever works) in the shell and you'll get into the interactive menu, where you'll find your way around.
The problem is - OVMF and windows are very picky about what type of controller you use to connect .iso images to the VM.
OVMF whitepaper suggests this way:

  # The Fedora installer disk appears as an IDE CD-ROM in the guest. This is
  # the 2nd boot option.
  OPTS="$OPTS -drive id=cd0,if=none,format=raw,readonly"
  OPTS="$OPTS,file=Fedora-Live-Xfce-x86_64-20-1.iso"
  OPTS="$OPTS -device ide-cd,bus=ide.1,drive=cd0,bootindex=1"

And you should connect virtio windows drivers iso(virtio.iso) in some similar way for windows to be able to use virtio.

P.S.
-cpu kvm=off is needed only for nvidia drivers which detect KVM and shut themselves down. Since you're using radeon, you can drop that line and activate hyper-v enlightenments instead.

Last edited by Duelist (2015-04-18 10:22:39)

---

The forum rules prohibit requesting support for distributions other than arch.
I gave up. It was too late.
What I was trying to do.
The reference about VFIO and KVM VGA passthrough.

lersek

Member

Registered: 2015-03-15

Posts: 38

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Hi,
when I want to use a newer version of the ovmf-svn package in my VM, is it possible to keep the part where the settings are stored in from the old build?

Thanks
apex

The split files for OVMF have been implemented for exactly this reason. When you upgrade the OVMF package on the host, the firmware binary (OVMF_CODE.fd) and the varstore template (OVMF_VARS.fd) are both updated. The code part is expected to see a change (that's what motivates the release of an updated OVMF package in the first place). The OVMF_VARS.fd file (the varstore template) is not expected to change.

In any case, since your virtual machines use their private variable stores (which had been originally copied from the then-installed OVMF_VARS.fd file, and then modified by the individual virtual machines in different, individual ways, and are all located under /var/lib/libvirt/qemu/nvram/ in a libvirt setup), such an OVMF package upgrade affects the firmware binary for *all* virtual machines (that's the goal), and it affects the varstores of *no* virtual machines (that is also the goal). The "new" OVMF_VARS.fd file (if it changed during the update, but it really doesn't) would only affect virtual machines that you'd *create* after the pkg update.

So, the short version is, if you use the split files (which you should), and your package maintainer is not doing crazy things (like changing from "secure boot disabled build" to "secure boot enabled build" in a simple package upgrade), then you should be all set; just upgrade the OVMF package and continue using your VM's own private varstore with the new binary.

(Note that I have no clue if "ovmf-svn" ships the split files.)

There's obviously one issue that needs to be mentioned here: what if the firmware binary (OVMF_CODE.fd) is modified in such a way that it becomes incompatible with the earlier varstore template (and consequently, with all individual varstores that had been instantiated from that earlier varstore template)? In that case the new package will (well... should!) install the firmware binary under a different name (ie. not OVMF_CODE.fd, but something else).

This way your old virtual machines won't even boot (because qemu won't find the central firmware binary under the original name), which is certainly favorable to crashing. There are two options then: either move your virtual machine to the new firmware (by deleting its old varstore, and creating the new varstore from the new template -- libvirt should do this for you automatically, if you delete the old varstore and also remove the nvram element from the domain XML), *or* you can keep the OVMF_CODE.fd file from the earlier package around, in a different location, and re-point the domain XML for all affected VMs to the new location of the old firmware binary.

If you pick option #1, then the VM will clearly lose its non-volatile variables. That should be no problem though, as described here:

http://blog.uncooperative.org/blog/2014 … partition/

installed UEFI OSes have (... should have) a fallback utility on their ESPs that get loaded if everything else fails, and they should regenerate some sane Boot#### options.

lersek

Member

Registered: 2015-03-15

Posts: 38

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Hello,
I have a problem. I guess this question came up before but I did not find it in the depths of this lengthy thread. I have been googleing and experimenting for many hours today and yesterday.  Binding my AMD HD7870 works fine and everything else seems to be fine as well. If I do not use OVMF I can install Windows using vga std on my partition /dev/sda3. If I use vga none and OVMF I see the output on the screen that is connected to the dedicated GPU (which is absolutely correct) but the system does not boot and I end up on the interactive EFI shell (I do not know waht to do with it). Am I just stupid?

I am using ovmf-svn from the AUR and the following command to start qemu:

qemu-system-x86_64 \
    -name win81 \
    -enable-kvm \
    -drive if=pflash,format=raw,readonly,file=/usr/share/ovmf/x64/ovmf_code_x64.bin \
    -drive if=pflash,format=raw,file=/usr/share/ovmf/x64/ovmf_vars_x64.bin \
    -cpu host,kvm=off \
    -smp 4,sockets=1 \
    -m 8G \
    -device vfio-pci,host=01:00.0 \
    -device vfio-pci,host=01:00.1 \
    -vga none \
    -device virtio-scsi-pci,id=scsi \
    -drive file=/home/fabian/win81.iso,id=cd0,if=none \
    -device scsi-cd,drive=cd0 \
    -drive file=/home/fabian/virtio.iso,id=cd1,if=none \
    -device ide-cd,bus=ide.1,drive=cd1 \
    -drive file=/dev/sda3,id=disk0,format=raw,if=none \
    -device scsi-hd,drive=disk0 \
    -boot menu=on

win81.iso is my Windows 8.1 x64 iso file and virtio.iso is the redhat iso with the virtio Windows drivers on it. /dev/sda3 is the drive I want to install Windows on.

Do you have any ideas?

First, you're using the varstore *template* as the VM's private variable store. That's wrong. Please use libvirt, or else instantiate the VM's own varstore from the template manually. See the "cp" command in the wiki. https://wiki.archlinux.org/index.php/PC … U_commands

Second, I think your virtio-scsi setup may not be right.  For -device scsi-hd and -device scsi-cd, I'd add the bus=scsi.0 property.

Finally, add the bootindex=0 property to your scsi-hd device, and bootindex=1 to scsi-cd. (If your target disk is empty / unformatted, this is the most convenient setting.) Otherwise, reverse the order.

I'd have to see your OVMF debug log (-debugcon file:debug.log -global isa-debugcon.iobase=0x402) to know more.

wikavalier

Member

Registered: 2013-01-22

Posts: 27

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Hello,
I have a problem. I guess this question came up before but I did not find it in the depths of this lengthy thread. I have been googleing and experimenting for many hours today and yesterday.  Binding my AMD HD7870 works fine and everything else seems to be fine as well. If I do not use OVMF I can install Windows using vga std on my partition /dev/sda3. If I use vga none and OVMF I see the output on the screen that is connected to the dedicated GPU (which is absolutely correct) but the system does not boot and I end up on the interactive EFI shell (I do not know waht to do with it). Am I just stupid?

I am using ovmf-svn from the AUR and the following command to start qemu:

qemu-system-x86_64 \
    -name win81 \
    -enable-kvm \
    -drive if=pflash,format=raw,readonly,file=/usr/share/ovmf/x64/ovmf_code_x64.bin \
    -drive if=pflash,format=raw,file=/usr/share/ovmf/x64/ovmf_vars_x64.bin \
    -cpu host,kvm=off \
    -smp 4,sockets=1 \
    -m 8G \
    -device vfio-pci,host=01:00.0 \
    -device vfio-pci,host=01:00.1 \
    -vga none \
    -device virtio-scsi-pci,id=scsi \
    -drive file=/home/fabian/win81.iso,id=cd0,if=none \
    -device scsi-cd,drive=cd0 \
    -drive file=/home/fabian/virtio.iso,id=cd1,if=none \
    -device ide-cd,bus=ide.1,drive=cd1 \
    -drive file=/dev/sda3,id=disk0,format=raw,if=none \
    -device scsi-hd,drive=disk0 \
    -boot menu=on

win81.iso is my Windows 8.1 x64 iso file and virtio.iso is the redhat iso with the virtio Windows drivers on it. /dev/sda3 is the drive I want to install Windows on.

Do you have any ideas?

I also spent many hours figuring out what I did wrong!!! But the thing is that you just need to type "exit" and an interactive menu will appear. Then select something like boot from file, search for the windows dvd and browse the EFI folder.

I cant exactly tell since you only need to do it for the installation. After that, it seems that the uefi knows from where to boot.

One question: is there any way to use the nvidia/nouveau drivers with vfio, like lxc or cgroups? I would like to passthrough and use the nvidia card for linux, not at the same time, but without rebooting

EDIT
hmmmmm.... perhaps kernel 4.0?

Last edited by wikavalier (2015-04-19 00:10:35)

Duelist

Member

Registered: 2014-09-22

Posts: 358

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

One question: is there any way to use the nvidia/nouveau drivers with vfio, like lxc or cgroups? I would like to passthrough and use the nvidia card for linux, not at the same time, but without rebooting

EDIT
hmmmmm.... perhaps kernel 4.0?

You can write up a small script which will unbind the devices from vfio-pci after the VM shutdown and bind them back to whatever thre driver you need.

You could make a systemd unit which binds the devices and runs the VM, and also shuts down the VM and rebinds the devices back on unit shutdown.
I think that may be possible to implement this via libvirt, but i'm not sure.

---

The forum rules prohibit requesting support for distributions other than arch.
I gave up. It was too late.
What I was trying to do.
The reference about VFIO and KVM VGA passthrough.

Naruni

Member

Registered: 2010-08-05

Posts: 34

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

So I've got a pretty simple stable instance running, not everything is tested for full performance.

Here is the script:

#!/bin/bash

#kvm
opt='-enable-kvm '

#Memory
opt+='-m 8192 '

#cpu
opt+='-cpu host,kvm=off '
opt+='-smp 4,cores=2,threads=2,sockets=1 '

#vga
#opt+='-vga qxl '

#gpu bind
opt+='-device vfio-pci,host=01:00.0 '

#ovmf
opt+='-drive if=pflash,format=raw,readonly,file=/usr/share/edk2.git/ovmf-x64/OVMF_CODE-pure-efi.fd '
opt+='-drive if=pflash,format=raw,file=/vm/7_vars.fd '

#virtio scsi controller
opt+='-device virtio-scsi-pci,id=scsi '

#disk
opt+='-drive file=/dev/mapper/vmdisks-win7,id=disk,format=raw,if=none -device scsi-hd,drive=disk '

#virtio drivers
#opt+='-drive file=/vm/virtio-win-0.1-100.iso,id=virtiocd,if=none -device ide-cd,bus=ide.1,drive=virtiocd '

#install cdrom
#opt+='-cdrom /vm/X17-58997.iso '

qemu-system-x86_64 $opt

Naruni

Member

Registered: 2010-08-05

Posts: 34

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

You can write up a small script which will unbind the devices from vfio-pci after the VM shutdown and bind them back to whatever thre driver you need.

You could make a systemd unit which binds the devices and runs the VM, and also shuts down the VM and rebinds the devices back on unit shutdown.
I think that may be possible to implement this via libvirt, but i'm not sure.

I tried that, and here's something I ran into (sorry for the funky column alignment):

$echo "0000:01:00.1" > /sys/bus/pci/drivers/vfio-pci/unbind
$ls /sys/bus/pci/devices/0000\:01\:00.0/
boot_vga              config                    device           enable         iommu_group    local_cpus  numa_node  rescan     resource1     resource3_wc  subsystem         uevent
broken_parity_status  consistent_dma_mask_bits  dma_mask_bits    firmware_node  irq            modalias    power      resource   resource1_wc  resource5     subsystem_device  vendor
class                 d3cold_allowed            driver_override  iommu          local_cpulist  msi_bus     remove     resource0  resource3     rom           subsystem_vendor

*Assumption disclaimer* There is no more driver file in the device's dir because pci-stub claimed it. So in order for the host to use it, the device would need to be bound to a vfio group and utilized by the host somehow? I've some more testing to do since I would like the same thing wikavalier wants.

Duelist

Member

Registered: 2014-09-22

Posts: 358

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

You can write up a small script which will unbind the devices from vfio-pci after the VM shutdown and bind them back to whatever thre driver you need.

You could make a systemd unit which binds the devices and runs the VM, and also shuts down the VM and rebinds the devices back on unit shutdown.
I think that may be possible to implement this via libvirt, but i'm not sure.

I tried that, and here's something I ran into (sorry for the funky column alignment):

$echo "0000:01:00.1" > /sys/bus/pci/drivers/vfio-pci/unbind
$ls /sys/bus/pci/devices/0000\:01\:00.0/
boot_vga              config                    device           enable         iommu_group    local_cpus  numa_node  rescan     resource1     resource3_wc  subsystem         uevent
broken_parity_status  consistent_dma_mask_bits  dma_mask_bits    firmware_node  irq            modalias    power      resource   resource1_wc  resource5     subsystem_device  vendor
class                 d3cold_allowed            driver_override  iommu          local_cpulist  msi_bus     remove     resource0  resource3     rom           subsystem_vendor

*Assumption disclaimer* There is no more driver file in the device's dir because pci-stub claimed it. So in order for the host to use it, the device would need to be bound to a vfio group and utilized by the host somehow? I've some more testing to do since I would like the same thing wikavalier wants.

Whaat?
You do the unbind. Then you do lspci to be sure that you've unbound your device.
And then you bind it to whatever driver you want.
pci-stub is a device driver too.
It is needed only for safety reasons, to make sure device doesn't get grabbed by some crazy driver which is loaded before vfio-bind systemd unit is launched.
When the system is fully booted, you can unbind the device from all drivers and have no kernel driver loaded for it.

---

The forum rules prohibit requesting support for distributions other than arch.
I gave up. It was too late.
What I was trying to do.
The reference about VFIO and KVM VGA passthrough.

fkoehler

Member

Registered: 2015-04-17

Posts: 7

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Hello,
I have a problem. I guess this question came up before but I did not find it in the depths of this lengthy thread. I have been googleing and experimenting for many hours today and yesterday.  Binding my AMD HD7870 works fine and everything else seems to be fine as well. If I do not use OVMF I can install Windows using vga std on my partition /dev/sda3. If I use vga none and OVMF I see the output on the screen that is connected to the dedicated GPU (which is absolutely correct) but the system does not boot and I end up on the interactive EFI shell (I do not know waht to do with it). Am I just stupid?

I am using ovmf-svn from the AUR and the following command to start qemu:

qemu-system-x86_64 \
    -name win81 \
    -enable-kvm \
    -drive if=pflash,format=raw,readonly,file=/usr/share/ovmf/x64/ovmf_code_x64.bin \
    -drive if=pflash,format=raw,file=/usr/share/ovmf/x64/ovmf_vars_x64.bin \
    -cpu host,kvm=off \
    -smp 4,sockets=1 \
    -m 8G \
    -device vfio-pci,host=01:00.0 \
    -device vfio-pci,host=01:00.1 \
    -vga none \
    -device virtio-scsi-pci,id=scsi \
    -drive file=/home/fabian/win81.iso,id=cd0,if=none \
    -device scsi-cd,drive=cd0 \
    -drive file=/home/fabian/virtio.iso,id=cd1,if=none \
    -device ide-cd,bus=ide.1,drive=cd1 \
    -drive file=/dev/sda3,id=disk0,format=raw,if=none \
    -device scsi-hd,drive=disk0 \
    -boot menu=on

win81.iso is my Windows 8.1 x64 iso file and virtio.iso is the redhat iso with the virtio Windows drivers on it. /dev/sda3 is the drive I want to install Windows on.

Do you have any ideas?

First, you're using the varstore *template* as the VM's private variable store. That's wrong. Please use libvirt, or else instantiate the VM's own varstore from the template manually. See the "cp" command in the wiki. https://wiki.archlinux.org/index.php/PC … U_commands

Second, I think your virtio-scsi setup may not be right.  For -device scsi-hd and -device scsi-cd, I'd add the bus=scsi.0 property.

Finally, add the bootindex=0 property to your scsi-hd device, and bootindex=1 to scsi-cd. (If your target disk is empty / unformatted, this is the most convenient setting.) Otherwise, reverse the order.

I'd have to see your OVMF debug log (-debugcon file:debug.log -global isa-debugcon.iobase=0x402) to know more.

Still having the UEFI issue. I am using vga=std to get everything to boot first. I made changes to my qemu start according to Duelist's and lersek's replies. My startup script is below. If I use an ubuntu iso it is getting detected automatically and I a land on the iso's grub (cannot boot as the kernel crashes for some reason). Tried archlinux iso which does not get detected. Therefore I type exit to get to the interactive menu where I go to Boot Maintenance Manager where the Archlinux iso was selectable I chose the x64 efi (pressed enter) but landed back on the Boot Maintenance Manager screen. My windows does not show up at all. debug.log is created but empty.

I also have another question that will only be relevant if I finally get it to work: using the vga=none graphics I get the output on another screen. To use mouse or keyboard I can give the vm control over the usb devices. But then there will be no way to switch it back to the host or will it?

#!/bin/bash

QEMU=qemu-system-x86_64
QEMU_OPTS=""

function add() {
    for opt in $*; do
        QEMU_OPTS+=" ${opt}"
    done
}

# name
add "-name win81"

# debug options
add "-debugcon file:debug.log -global isa-debugcon.iobase=0x402"

# enable kvm virtualization
add "-enable-kvm"

# UEFI via ovmf
add "-drive if=pflash,format=raw,readonly,file=/usr/share/ovmf/x64/ovmf_code_x64.bin"
add "-drive if=pflash,format=raw,file=/tmp/ovmf_vars.bin"

# cpu
add "-cpu host,hv_relaxed,hv_spinlocks=0x1fff,hv_vapic,hv_time"
add "-smp 4,sockets=1"

# memory
add "-m 8G"

# pci
add "-device vfio-pci,host=01:00.0"
add "-device vfio-pci,host=01:00.1"

# graphics mode
#add "-vga none"

# scsi controller
add "-device virtio-scsi-pci,id=scsi"

# hard drives
add "-drive file=/dev/sda3,id=disk0,format=raw,if=none -device scsi-hd,drive=disk0,bus=scsi.0,bootindex=1"

# cd drives
add "-drive file=/home/fabian/win81.iso,id=cd0,if=none,format=raw,readonly -device ide-cd,bus=ide.1,drive=cd0,bootindex=0"
add "-drive file=/home/fabian/virtio.iso,id=cd1,if=none,format=raw,readonly -device scsi-cd,bus=scsi.0,drive=cd1"

# usb
#add "-usb"
# logitech receiver
#add "-usbdevice host:046d:c52b"

# sound hardware
# add "-soundhw hda"

# boot opitions
add "-boot menu=on"

cp /usr/share/ovmf/x64/ovmf_vars_x64.bin /tmp/ovmf_vars.bin
echo "${QEMU} ${QEMU_OPTS}"
exec ${QEMU} ${QEMU_OPTS}

Duelist

Member

Registered: 2014-09-22

Posts: 358

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

I also have another question that will only be relevant if I finally get it to work: using the vga=none graphics I get the output on another screen. To use mouse or keyboard I can give the vm control over the usb devices. But then there will be no way to switch it back to the host or will it?

You can use -device qxl to make qemu provide you a small SDL-based surface for mouse and keyboard capturing purposes.
It won't show anything, and the device won't work, but who cares since we got the input.
In libvirt i have Graphics->VNC, and it gives me the same small window for same purposes.
Also, there is synergy and hardware KVM switches, spice and various stuff to try.
You could've just make qemu's VNC server running and connect to it from the host.

---

The forum rules prohibit requesting support for distributions other than arch.
I gave up. It was too late.
What I was trying to do.
The reference about VFIO and KVM VGA passthrough.

erganzi

Member

Registered: 2014-07-09

Posts: 19

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Hi, all:

my environment:
kernel: 3.18.6 with i915 patches.
qemu: 2.1.3
arch: Intel Xeon series
type: legacy VGA passthrough

There is a problem when I passthrough AMD HD7850 card to VM. the catd can passthrough successfully.
but when I installed driver, the vm crashed with follow error:
pcieport 0000:00:03.0: AER: Multiple Uncorrected (Fatal) error received: id=0000
vfio-pci 0000:04:00.0: PCIe Bus Error: severity=Uncorrected (Fatal), type=Unaccessible, id=0400(Unregistered Agent ID)
vfio-pci 0000:04:00.1: PCIe Bus Error: severity=Uncorrected (Fatal), type=Unaccessible, id=0401(Unregistered Agent ID)

qemu-kvm-passthrough: vfio_err_notifier_handler(0000:04:00.1) Unrecoverable error detected.  Please collect any data possible and then kill the guest
qemu-kvm-passthrough: vfio_err_notifier_handler(0000:04:00.0) Unrecoverable error detected.  Please collect any data possible and then kill the guest
qemu-kvm-passthrough: vfio_err_notifier_handler(0000:04:00.1) Unrecoverable error detected.  Please collect any data possible and then kill the guest
qemu-kvm-passthrough: vfio_err_notifier_handler(0000:04:00.0) Unrecoverable error detected.  Please collect any data possible and then kill the guest

If I just passthrough the VGA card, without hdmi audio, the VM worked well with driver. I don't know why, somebody can explain to me, thanks.

acehege

Member

Registered: 2015-04-20

Posts: 2

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Hey everybody,

I have gotten KVM VGA-Passthrough to work using this awesome forum and all of your experiences, code snippets and comments.

BUT, everytime i reboot/shutdown a VM with my passed-through Nvidia Cards ( GT630 ), it gives me the dreaded "Error 43" code, and the only fix is to reboot the arch linux host.

--- Info ---

1. I am running the linux-vfio kernel from AUR

2. I have four identical Nvidia GT630 cards

3. I am using OVFM in the VMs

4. The VMs are running Windows 8.1

5. kvm=off is set.

6. I have tried with and without x-vga=on

7. I have tried with and without the following in grub: pcie_acs_override=downstream i915.enable_hd_vgaarb=1

8. I have tried with virt-manager and also commandline qemu, same symptoms.

My current kernel version is 3.19.3.

--- /Info ---

Any help would be highly appreciated!

If any of you need more information or snippets from the system, just ask, and I will deliver

fseven

Member

Registered: 2015-04-15

Posts: 16

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

You can use -device qxl to make qemu provide you a small SDL-based surface for mouse and keyboard capturing purposes.
It won't show anything, and the device won't work, but who cares since we got the input.
In libvirt i have Graphics->VNC, and it gives me the same small window for same purposes.
Also, there is synergy and hardware KVM switches, spice and various stuff to try.
You could've just make qemu's VNC server running and connect to it from the host.

I was wondering if it is technically possible to emulate a kvm switch? Kind of like attaching and detaching the usb device, but leaving the slot active (real kvm switches do something similar so you dont have to wait 10 seconds for windows to load the drivers every time).
Because the network based options arent't really ideal when latency is important and switching by attaching/detaching always takes a few seconds.

Last edited by fseven (2015-04-20 16:45:03)

nbhs

Member

From: Montevideo, Uruguay

Registered: 2013-05-02

Posts: 402

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Hey everybody,

I have gotten KVM VGA-Passthrough to work using this awesome forum and all of your experiences, code snippets and comments.

BUT, everytime i reboot/shutdown a VM with my passed-through Nvidia Cards ( GT630 ), it gives me the dreaded "Error 43" code, and the only fix is to reboot the arch linux host.

--- Info ---

1. I am running the linux-vfio kernel from AUR

2. I have four identical Nvidia GT630 cards

3. I am using OVFM in the VMs

4. The VMs are running Windows 8.1

5. kvm=off is set.

6. I have tried with and without x-vga=on

7. I have tried with and without the following in grub: pcie_acs_override=downstream i915.enable_hd_vgaarb=1

8. I have tried with virt-manager and also commandline qemu, same symptoms.

My current kernel version is 3.19.3.

--- /Info ---

Any help would be highly appreciated!

If any of you need more information or snippets from the system, just ask, and I will deliver

Have you tried loading your gpu rom with the romfile= parameter? check out this post https://bbs.archlinux.org/viewtopic.php … 7#p1511177

akawoz

Member

Registered: 2015-02-22

Posts: 6

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Great success here - my gear / versions list.

Any questions please let me know - been battling this for ages.

Hardware:
Asrock 990FX Extreme9
AMD FX-8370 8-core CPU
32GB DDR3
1x GeForce GT 730 (host)
1x GeForce GTX 960 (VM)
1x Renasas 4x USB 3 PCIe 1x (VM)
no overclocking or special BIOS changes except:
IO-MMU=ON
USB 3 legacy support = OFF

Host OS:
Arch Linux stable repos
Kernel 3.19.3-3-vfio from: https://aur.archlinux.org/packages/linux-vfio/
No patches

Guest OS:
Windows 8.1 x64
Latest NVIDIA Experience drivers

[root@cloud ~]# lspci
00:00.0 Host bridge: Advanced Micro Devices, Inc. [AMD/ATI] RD890 PCI to PCI bridge (external gfx0 port B) (rev 02)
00:00.2 IOMMU: Advanced Micro Devices, Inc. [AMD/ATI] RD990 I/O Memory Management Unit (IOMMU)
00:02.0 PCI bridge: Advanced Micro Devices, Inc. [AMD/ATI] RD890 PCI to PCI bridge (PCI express gpp port B)
00:04.0 PCI bridge: Advanced Micro Devices, Inc. [AMD/ATI] RD890 PCI to PCI bridge (PCI express gpp port D)
00:05.0 PCI bridge: Advanced Micro Devices, Inc. [AMD/ATI] RD890 PCI to PCI bridge (PCI express gpp port E)
00:06.0 PCI bridge: Advanced Micro Devices, Inc. [AMD/ATI] RD890 PCI to PCI bridge (PCI express gpp port F)
00:07.0 PCI bridge: Advanced Micro Devices, Inc. [AMD/ATI] RD890 PCI to PCI bridge (PCI express gpp port G)
00:0a.0 PCI bridge: Advanced Micro Devices, Inc. [AMD/ATI] RD890 PCI to PCI bridge (external gfx1 port A)
00:0b.0 PCI bridge: Advanced Micro Devices, Inc. [AMD/ATI] RD890 PCI to PCI bridge (NB-SB link)
00:0c.0 PCI bridge: Advanced Micro Devices, Inc. [AMD/ATI] RD890S PCI Express bridge for GPP2 port 1
00:0d.0 PCI bridge: Advanced Micro Devices, Inc. [AMD/ATI] RD890 PCI to PCI bridge (external gfx1 port B)
00:11.0 SATA controller: Advanced Micro Devices, Inc. [AMD/ATI] SB7x0/SB8x0/SB9x0 SATA Controller [AHCI mode] (rev 40)
00:12.0 USB controller: Advanced Micro Devices, Inc. [AMD/ATI] SB7x0/SB8x0/SB9x0 USB OHCI0 Controller
00:12.2 USB controller: Advanced Micro Devices, Inc. [AMD/ATI] SB7x0/SB8x0/SB9x0 USB EHCI Controller
00:13.0 USB controller: Advanced Micro Devices, Inc. [AMD/ATI] SB7x0/SB8x0/SB9x0 USB OHCI0 Controller
00:13.2 USB controller: Advanced Micro Devices, Inc. [AMD/ATI] SB7x0/SB8x0/SB9x0 USB EHCI Controller
00:14.0 SMBus: Advanced Micro Devices, Inc. [AMD/ATI] SBx00 SMBus Controller (rev 42)
00:14.1 IDE interface: Advanced Micro Devices, Inc. [AMD/ATI] SB7x0/SB8x0/SB9x0 IDE Controller (rev 40)
00:14.2 Audio device: Advanced Micro Devices, Inc. [AMD/ATI] SBx00 Azalia (Intel HDA) (rev 40)
00:14.3 ISA bridge: Advanced Micro Devices, Inc. [AMD/ATI] SB7x0/SB8x0/SB9x0 LPC host controller (rev 40)
00:14.4 PCI bridge: Advanced Micro Devices, Inc. [AMD/ATI] SBx00 PCI to PCI Bridge (rev 40)
00:14.5 USB controller: Advanced Micro Devices, Inc. [AMD/ATI] SB7x0/SB8x0/SB9x0 USB OHCI2 Controller
00:15.0 PCI bridge: Advanced Micro Devices, Inc. [AMD/ATI] SB700/SB800/SB900 PCI to PCI bridge (PCIE port 0)
00:15.2 PCI bridge: Advanced Micro Devices, Inc. [AMD/ATI] SB900 PCI to PCI bridge (PCIE port 2)
00:16.0 USB controller: Advanced Micro Devices, Inc. [AMD/ATI] SB7x0/SB8x0/SB9x0 USB OHCI0 Controller
00:16.2 USB controller: Advanced Micro Devices, Inc. [AMD/ATI] SB7x0/SB8x0/SB9x0 USB EHCI Controller
00:18.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Family 15h Processor Function 0
00:18.1 Host bridge: Advanced Micro Devices, Inc. [AMD] Family 15h Processor Function 1
00:18.2 Host bridge: Advanced Micro Devices, Inc. [AMD] Family 15h Processor Function 2
00:18.3 Host bridge: Advanced Micro Devices, Inc. [AMD] Family 15h Processor Function 3
00:18.4 Host bridge: Advanced Micro Devices, Inc. [AMD] Family 15h Processor Function 4
00:18.5 Host bridge: Advanced Micro Devices, Inc. [AMD] Family 15h Processor Function 5
01:00.0 VGA compatible controller: NVIDIA Corporation GK208 [GeForce GT 730] (rev a1)
01:00.1 Audio device: NVIDIA Corporation GK208 HDMI/DP Audio Controller (rev a1)
02:00.0 SATA controller: ASMedia Technology Inc. ASM1062 Serial ATA Controller (rev 01)
03:00.0 USB controller: Etron Technology, Inc. EJ188/EJ198 USB 3.0 Host Controller
04:00.0 SATA controller: ASMedia Technology Inc. ASM1062 Serial ATA Controller (rev 01)
05:00.0 USB controller: Etron Technology, Inc. EJ188/EJ198 USB 3.0 Host Controller
06:00.0 USB controller: Renesas Technology Corp. uPD720201 USB 3.0 Host Controller (rev 03)
07:00.0 VGA compatible controller: NVIDIA Corporation GM206 [GeForce GTX 960] (rev a1)
07:00.1 Audio device: NVIDIA Corporation Device 0fba (rev a1)
08:00.0 Ethernet controller: Intel Corporation 82575EB Gigabit Network Connection (rev 02)
08:00.1 Ethernet controller: Intel Corporation 82575EB Gigabit Network Connection (rev 02)
09:00.0 Serial Attached SCSI controller: LSI Logic / Symbios Logic SAS2008 PCI-Express Fusion-MPT SAS-2 [Falcon] (rev 03)
0b:00.0 FireWire (IEEE 1394): VIA Technologies, Inc. VT6315 Series Firewire Controller (rev 01)
0c:00.0 Ethernet controller: Intel Corporation 82583V Gigabit Network Connection

[root@cloud ~]# cat /etc/mkinitcpio.conf
# vim:set ft=sh
# MODULES
# The following modules are loaded before any boot hooks are
# run.  Advanced users may wish to specify all system modules
# in this array.  For instance:
#     MODULES="piix ide_disk reiserfs"
MODULES="nouveau pci-stub btrfs"

[root@cloud ~]# cat /etc/default/grub
GRUB_DEFAULT=0
GRUB_TIMEOUT=5
GRUB_DISTRIBUTOR="Arch"
GRUB_CMDLINE_LINUX_DEFAULT="amd_iommu=on iommu=noagp pci-stub.ids=10de:1401,10de:0fba,1912:0014"

[root@cloud qemu]# cat gpu-bios2.xml
<!--
WARNING: THIS IS AN AUTO-GENERATED FILE. CHANGES TO IT ARE LIKELY TO BE
OVERWRITTEN AND LOST. Changes to this xml configuration should be made using:
  virsh edit gpu-bios2
or other application using the libvirt API.
-->

<domain type='kvm' xmlns:qemu='http://libvirt.org/schemas/domain/qemu/1.0'>
  <name>gpu-bios2</name>
  <uuid>dd9ac642-5098-4c65-9c87-8d7d33ed32b6</uuid>
  <memory unit='KiB'>4194304</memory>
  <currentMemory unit='KiB'>4194304</currentMemory>
  <vcpu placement='static'>8</vcpu>
  <os>
    <type arch='x86_64' machine='pc-q35-2.2'>hvm</type>
  </os>
  <features>
    <acpi/>
    <apic/>
    <pae/>
    <hyperv>
      <relaxed state='off'/>
      <vapic state='off'/>
      <spinlocks state='off'/>
    </hyperv>
    <kvm>
      <hidden state='on'/>
    </kvm>
  </features>
  <clock offset='localtime'>
    <timer name='rtc' tickpolicy='catchup'/>
    <timer name='pit' tickpolicy='delay'/>
    <timer name='hpet' present='no'/>
    <timer name='hypervclock' present='no'/>
  </clock>
  <on_poweroff>destroy</on_poweroff>
  <on_reboot>restart</on_reboot>
  <on_crash>restart</on_crash>
  <pm>
    <suspend-to-mem enabled='no'/>
    <suspend-to-disk enabled='no'/>
  </pm>
  <devices>
    <emulator>/usr/sbin/qemu-kvm</emulator>
    <disk type='file' device='disk'>
      <driver name='qemu' type='qcow2'/>
      <source file='/data/images/gpu-bios.qcow2'/>
      <target dev='hda' bus='ide'/>
      <boot order='1'/>
      <address type='drive' controller='0' bus='0' target='0' unit='0'/>
    </disk>
    <controller type='sata' index='0'>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x1f' function='0x2'/>
    </controller>
    <controller type='pci' index='0' model='pcie-root'/>
    <controller type='pci' index='1' model='dmi-to-pci-bridge'>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x1e' function='0x0'/>
    </controller>
    <controller type='pci' index='2' model='pci-bridge'>
      <address type='pci' domain='0x0000' bus='0x01' slot='0x01' function='0x0'/>
    </controller>
    <controller type='ide' index='0'/>
    <controller type='virtio-serial' index='0'>
      <address type='pci' domain='0x0000' bus='0x02' slot='0x04' function='0x0'/>
    </controller>
    <controller type='usb' index='0'>
      <address type='pci' domain='0x0000' bus='0x02' slot='0x03' function='0x0'/>
    </controller>
    <interface type='direct'>
      <mac address='52:54:00:99:bf:7d'/>
      <source dev='enp12s0' mode='bridge'/>
      <model type='virtio'/>
      <address type='pci' domain='0x0000' bus='0x02' slot='0x01' function='0x0'/>
    </interface>
    <interface type='network'>
      <mac address='52:54:00:79:67:c4'/>
      <source network='isolated'/>
      <model type='virtio'/>
      <address type='pci' domain='0x0000' bus='0x02' slot='0x07' function='0x0'/>
    </interface>
    <serial type='pty'>
      <target port='0'/>
    </serial>
    <console type='pty'>
      <target type='serial' port='0'/>
    </console>
    <sound model='ich6'>
      <address type='pci' domain='0x0000' bus='0x02' slot='0x02' function='0x0'/>
    </sound>
    <memballoon model='virtio'>
      <address type='pci' domain='0x0000' bus='0x02' slot='0x05' function='0x0'/>
    </memballoon>
  </devices>
  <qemu:commandline>
    <qemu:arg value='-device'/>
    <qemu:arg value='ioh3420,bus=pcie.0,addr=1c.0,multifunction=on,port=1,chassis=1,id=root.1'/>
    <qemu:arg value='-device'/>
    <qemu:arg value='vfio-pci,host=07:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on'/>
    <qemu:arg value='-device'/>
    <qemu:arg value='vfio-pci,host=07:00.1,bus=pcie.0'/>
    <qemu:arg value='-device'/>
    <qemu:arg value='vfio-pci,host=06:00.0'/>
    <qemu:arg value='-cpu'/>
    <qemu:arg value='host,kvm=off'/>
  </qemu:commandline>
</domain>

Last edited by akawoz (2015-04-20 19:10:38)

aw

Member

Registered: 2013-10-04

Posts: 921

Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

@akawoz

You're so close to having a pretty clean setup, why ruin it with q35 and all those <qemu:arg> statements?  You're already telling libvirt to use kvm=off with the hidden option, no need to specify it again.  If you swap to 440fx you can just use normal <hostdev> options for the GPU, audio, and USB.  And with a win8.1 guest and card that new, I'm not sure why you wouldn't use OVMF rather than BIOS to drop x-vga=on.

---

http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

akawoz

Member

Registered: 2015-02-22

Posts: 6

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Yeah agreed - but I've been playing around with this trying to get it working for months (started with an X10SAT, Radeon 7850) so I'm just pleased to finally have it working.

Good point re duplication of kvm=off statements.

Will take a look at <hostdev>.

Tried with OVMF but Windows kept freezing at boot - so followed the path of least resistance. Now I've got it running and stable I'd try other settings and options. Cheers.

@akawoz

You're so close to having a pretty clean setup, why ruin it with q35 and all those <qemu:arg> statements?  You're already telling libvirt to use kvm=off with the hidden option, no need to specify it again.  If you swap to 440fx you can just use normal <hostdev> options for the GPU, audio, and USB.  And with a win8.1 guest and card that new, I'm not sure why you wouldn't use OVMF rather than BIOS to drop x-vga=on.

fkoehler

Member

Registered: 2015-04-17

Posts: 7

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

I am back. I still had no luck with ovmf-svn from the AUR. ubuntu would boot (only without the graphics card as the radeon driver would crash), none of the archlinux entries would boot and for the windows iso I do not even have an option to boot from file. debug.log remained empty all the time. So I tried ovmf from this repo as described in the wiki. Now I get a black screen but at least the debug.log is not empty. The error messages lead me to old edk2 issues that were fixed years ago. Maybe you understand them?

startup script

#!/bin/bash

QEMU=qemu-system-x86_64
QEMU_OPTS=""

function add() {
    for opt in $*; do
        QEMU_OPTS+=" ${opt}"
    done
}

# name
add "-name win81"

# debug options
add "-debugcon file:debug.log -global isa-debugcon.iobase=0x402"

# enable kvm virtualization
add "-enable-kvm"

# uefi
add "-drive if=pflash,format=raw,readonly,file=/usr/share/edk2.git/ovmf-x64/OVMF_CODE-pure-efi.fd"
add "-drive if=pflash,format=raw,file=/tmp/ovmf_vars.fd"

# cpu
add "-cpu host,hv_relaxed,hv_spinlocks=0x1fff,hv_vapic,hv_time"
add "-smp 4,sockets=1"

# memory
add "-m 16G"

# pci
add "-device vfio-pci,host=01:00.0"
add "-device vfio-pci,host=01:00.1"

# graphics mode
add "-vga none"

# scsi controller
add "-device virtio-scsi-pci,id=scsi"

# hard drives
add "-drive file=/dev/sda3,id=disk0,format=raw,if=none -device scsi-hd,drive=disk0,bus=scsi.0,bootindex=0"

# cdrom
add "-drive file=/home/fabian/win81.iso,id=cd0,if=none,format=raw,readonly -device ide-cd,bus=ide.1,drive=cd0,bootindex=1"
add "-drive file=/home/fabian/virtio.iso,id=cd1,if=none,format=raw,readonly -device scsi-cd,bus=scsi.0,drive=cd1"

add "-boot menu=on"

# start
cp /usr/share/edk2.git/ovmf-x64/OVMF-pure-efi.fd /tmp/ovmf_vars.fd
echo "${QEMU} ${QEMU_OPTS}"
exec ${QEMU} ${QEMU_OPTS}

ovmf errors from debug.log

SecCoreStartupWithStack(0xFFFCC000, 0x818000)
Register PPI Notify: DCD0BE23-9586-40F4-B643-06522CED4EDE
Install PPI: 8C8CE578-8A3D-4F1C-9935-896185C32DD3
Install PPI: 5473C07A-3DCB-4DCA-BD6F-1E9689E7349A
The 0th FV start address is 0x00000820000, size is 0x000E0000, handle is 0x820000
Register PPI Notify: 49EDB1C1-BF21-4761-BB12-EB0031AABB39
Register PPI Notify: EA7CA24B-DED5-4DAD-A389-BF827E8F9B38
Install PPI: B9E0ABFE-5979-4914-977F-6DEE78C278A6
Install PPI: DBE23AA9-A345-4B97-85B6-B226F1617389
Loading PEIM at 0x00000838EA0 EntryPoint=0x00000839100 PcdPeim.efi
Install PPI: 06E81C58-4AD7-44BC-8390-F10265F72480
Install PPI: 01F34D25-4DE2-23AD-3FF3-36353FF323F1
Install PPI: 4D8B155B-C059-4C8F-8926-06FD4331DB8A
Install PPI: A60C6B59-E459-425D-9C69-0BCC9CB27D81
Loading PEIM at 0x000008421A0 EntryPoint=0x00000842400 StatusCodePei.efi
Install PPI: 229832D3-7A30-4B36-B827-F40CB7D45436
Loading PEIM at 0x00000847EA0 EntryPoint=0x00000848100 PlatformPei.efi
Select Item: 0x0
FW CFG Signature: 0x554D4551
Select Item: 0x1
FW CFG Revision: 0x1
QemuFwCfg interface is supported.
Platform PEIM Loaded
CMOS:
00: 56 00 04 00 21 00 02 20 04 15 26 02 10 80 00 00
10: 40 00 00 00 07 80 02 FF FF 00 00 00 00 00 00 00
20: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
30: FF FF 20 00 00 BF 00 20 30 00 00 00 00 12 00 00
40: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
50: 00 00 00 00 00 00 00 00 00 00 00 00 40 03 00 03
60: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
70: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
Select Item: 0x19
Select Item: 0x25
S3 support was detected on QEMU
Install PPI: 7408D748-FC8C-4EE6-9288-C4BEC092A410
PeiInstallPeiMemory MemoryBegin 0xBC000000, MemoryLength 0x4000000
QemuInitializeRam called
Allocated Memory unaligned: Address = 0xBFFD0000, Pages = 0x30, Type = 6 
After aligning to 0x10000 bytes: Address = 0xBFFD0000, Pages = 0x20 
Updated aligned-mem HOB with BaseAddress = BFFD0000, Length = 20000, MemoryType = 6 
Created after-mem HOB with BaseAddress = BFFF0000, Length = 10000, MemoryType = 7 
Reserved variable store memory: 0xBFFD0000; size: 128kb
Platform PEI Firmware Volume Initialization
Install PPI: 49EDB1C1-BF21-4761-BB12-EB0031AABB39
Notify: PPI Guid: 49EDB1C1-BF21-4761-BB12-EB0031AABB39, Peim notify entry point: 8242CA
The 1th FV start address is 0x00000900000, size is 0x00800000, handle is 0x900000
Temp Stack : BaseAddress=0x814000 Length=0x4000
Temp Heap  : BaseAddress=0x810000 Length=0x1980
Total temporary memory:    32768 bytes.
  temporary memory stack ever used: 16384 bytes.
  temporary memory heap used:       6528 bytes.
Old Stack size 16384, New stack size 131072
Stack Hob: BaseAddress=0xBC000000 Length=0x20000
Heap Offset = 0xBB810000 Stack Offset = 0xBB808000
TemporaryRamMigration(0x810000, 0xBC01C000, 0x8000)
Loading PEIM at 0x000BFFB6000 EntryPoint=0x000BFFB6260 PeiCore.efi
Reinstall PPI: 8C8CE578-8A3D-4F1C-9935-896185C32DD3
Reinstall PPI: 5473C07A-3DCB-4DCA-BD6F-1E9689E7349A
Reinstall PPI: B9E0ABFE-5979-4914-977F-6DEE78C278A6
Install PPI: F894643D-C449-42D1-8EA8-85BDD8C65BDE
Loading PEIM at 0x000BFFAC000 EntryPoint=0x000BFFAC260 DxeIpl.efi
Install PPI: 0AE8CE5D-E448-4437-A8D7-EBF5F194F731
Install PPI: 1A36E4E7-FAB6-476A-8E75-695A0576FDD7
Loading PEIM at 0x000BFFA1000 EntryPoint=0x000BFFA1260 S3Resume2Pei.efi
Install PPI: 6D582DBC-DB85-4514-8FCC-5ADF6227B147
DXE IPL Entry
Loading PEIM at 0x000BFF64000 EntryPoint=0x000BFF64260 DxeCore.efi
Loading DXE CORE at 0x000BFF64000 EntryPoint=0x000BFF64260
Install PPI: 605EA650-C65C-42E1-BA80-91A52AB618C6
InstallProtocolInterface: 5B1B31A1-9562-11D2-8E3F-00A0C969723B BFF938C8
HOBLIST address in DXE = 0xBFD19018
Memory Allocation 0x0000000A 0x818000 - 0x81FFFF
Memory Allocation 0x0000000A 0x810000 - 0x817FFF
Memory Allocation 0x0000000A 0x807000 - 0x807FFF
Memory Allocation 0x0000000A 0x800000 - 0x805FFF
Memory Allocation 0x0000000A 0x806000 - 0x806FFF
Memory Allocation 0x00000006 0xBFFD0000 - 0xBFFEFFFF
Memory Allocation 0x00000007 0xBFFF0000 - 0xBFFFFFFF
Memory Allocation 0x0000000A 0x820000 - 0x8FFFFF
Memory Allocation 0x00000004 0x900000 - 0x10FFFFF
Memory Allocation 0x00000004 0xBFF44000 - 0xBFF63FFF
Memory Allocation 0x00000004 0xBFFCF000 - 0xBFFCFFFF
Memory Allocation 0x00000004 0xBFFB6000 - 0xBFFCEFFF
Memory Allocation 0x00000004 0xBFFAC000 - 0xBFFB5FFF
Memory Allocation 0x00000004 0xBFFA1000 - 0xBFFABFFF
Memory Allocation 0x00000004 0xBFF64000 - 0xBFFA0FFF
Memory Allocation 0x00000003 0xBFF64000 - 0xBFFA0FFF
Memory Allocation 0x00000004 0xBFF44000 - 0xBFF63FFF
Memory Allocation 0x00000004 0xBFF02000 - 0xBFF43FFF
Memory Allocation 0x00000004 0xBC000000 - 0xBC01FFFF
FV Hob            0x900000 - 0x10FFFFF
InstallProtocolInterface: D8117CFE-94A6-11D4-9A3A-0090273FC14D BFF93360
InstallProtocolInterface: 8F644FA9-E850-4DB1-9CE2-0B44698E8DA4 BFD15270
InstallProtocolInterface: 09576E91-6D3F-11D2-8E39-00A0C969723B BFD15658
InstallProtocolInterface: 220E73B6-6BDB-4413-8405-B974B108619A BFD14C70
InstallProtocolInterface: EE4E5898-3914-4259-9D6E-DC7BD79403CF BFF93588
Loading driver 9B680FCE-AD6B-4F3A-B60B-F59899003443
InstallProtocolInterface: 5B1B31A1-9562-11D2-8E3F-00A0C969723B BF732180
Loading driver at 0x000BFE8C000 EntryPoint=0x000BFE8C2AF DevicePathDxe.efi
InstallProtocolInterface: BC62157E-3E33-4FEC-9920-2D3B36D750DF BF732E58
InstallProtocolInterface: 0379BE4E-D706-437D-B037-EDB82FB772A4 BFE98200
InstallProtocolInterface: 8B843E20-8132-4852-90CC-551A4E4A7F1C BFE98240
InstallProtocolInterface: 05C99A21-C70F-4AD2-8A5F-35DF3343F51E BFE98250
Loading driver 80CF7257-87AB-47F9-A3FE-D50B76D89541
InstallProtocolInterface: 5B1B31A1-9562-11D2-8E3F-00A0C969723B BF7325C0
Loading driver at 0x000BFE7F000 EntryPoint=0x000BFE7F2AF PcdDxe.efi
InstallProtocolInterface: BC62157E-3E33-4FEC-9920-2D3B36D750DF BF724018
InstallProtocolInterface: 11B34006-D85B-4D0A-A290-D5A571310EF7 BFE87C60
InstallProtocolInterface: 13A3F0F6-264A-3EF0-F2E0-DEC512342F34 BFE87D60
InstallProtocolInterface: 5BE40F57-FA68-4610-BBBF-E9C5FCDAD365 BFE87DF0
InstallProtocolInterface: FD0F4478-0EFD-461D-BA2D-E58C45FD5F5E BFE87E10
Loading driver 733CBAC2-B23F-4B92-BC8E-FB01CE5907B7
InstallProtocolInterface: 5B1B31A1-9562-11D2-8E3F-00A0C969723B BF727040
Loading driver at 0x000BFEC3000 EntryPoint=0x000BFEC32AF FvbServicesRuntimeDxe.efi
InstallProtocolInterface: BC62157E-3E33-4FEC-9920-2D3B36D750DF BF724698
QEMU Flash: Attempting flash detection at FFE00000
QemuFlashDetected => FD behaves as FLASH
QemuFlashDetected => Yes
Variable FV header is not valid. It will be reinitialized.
Installing QEMU flash FVB
InstallProtocolInterface: 8F644FA9-E850-4DB1-9CE2-0B44698E8DA4 BFEF0DF0
InstallProtocolInterface: 09576E91-6D3F-11D2-8E39-00A0C969723B BF727D98
Loading driver FEDE0A1B-BCA2-4A9F-BB2B-D9FD7DEC2E9F
InstallProtocolInterface: 5B1B31A1-9562-11D2-8E3F-00A0C969723B BF727340
Loading driver at 0x000BFEB9000 EntryPoint=0x000BFEB92AF StatusCodeRuntimeDxe.efi
InstallProtocolInterface: BC62157E-3E33-4FEC-9920-2D3B36D750DF BF727598
InstallProtocolInterface: D2B2B828-0826-48A7-B3DF-983C006024F0 BFEBF600
Loading driver B601F8C4-43B7-4784-95B1-F4226CB40CEE
InstallProtocolInterface: 5B1B31A1-9562-11D2-8E3F-00A0C969723B BF72FBC0
Loading driver at 0x000BFEB1000 EntryPoint=0x000BFEB12AF RuntimeDxe.efi
InstallProtocolInterface: BC62157E-3E33-4FEC-9920-2D3B36D750DF BF72F298
InstallProtocolInterface: B7DFB4E1-052F-449F-87BE-9818FC91B733 BFEB56C0
Loading driver F80697E9-7FD6-4665-8646-88E33EF71DFC
InstallProtocolInterface: 5B1B31A1-9562-11D2-8E3F-00A0C969723B BF72E040
Loading driver at 0x000BFDF9000 EntryPoint=0x000BFDF92AF SecurityStubDxe.efi
InstallProtocolInterface: BC62157E-3E33-4FEC-9920-2D3B36D750DF BF72F698
InstallProtocolInterface: 94AB2F58-1438-4EF1-9152-18941A3A0E68 BFE57B70
InstallProtocolInterface: A46423E3-4617-49F1-B9FF-D1BFA9115839 BFE57B68
Loading driver 13AC6DD0-73D0-11D4-B06B-00AA00BD6DE7
InstallProtocolInterface: 5B1B31A1-9562-11D2-8E3F-00A0C969723B BF72E400
Loading driver at 0x000BFDED000 EntryPoint=0x000BFDED2AF EbcDxe.efi
InstallProtocolInterface: BC62157E-3E33-4FEC-9920-2D3B36D750DF BF72EA18
InstallProtocolInterface: 13AC6DD1-73D0-11D4-B06B-00AA00BD6DE7 BF72E798
InstallProtocolInterface: 2755590C-6F3C-42FA-9EA4-A3BA543CDA25 BF72DE58
InstallProtocolInterface: AAEACCFD-F27B-4C17-B610-75CA1F2DFB52 BF72DE18
Loading driver 79CA4208-BBA1-4A9A-8456-E1E66A81484E
InstallProtocolInterface: 5B1B31A1-9562-11D2-8E3F-00A0C969723B BF72D140
Loading driver at 0x000BFDE8000 EntryPoint=0x000BFDE82AF Legacy8259.efi
InstallProtocolInterface: BC62157E-3E33-4FEC-9920-2D3B36D750DF BF72D8D8
InstallProtocolInterface: 38321DBA-4FE0-4E17-8AEC-413055EAEDC1 BFDEB040
Loading driver A19B1FE7-C1BC-49F8-875F-54A5D542443F
InstallProtocolInterface: 5B1B31A1-9562-11D2-8E3F-00A0C969723B BF72D500
Loading driver at 0x000BFDE2000 EntryPoint=0x000BFDE22AF CpuIo2Dxe.efi
InstallProtocolInterface: BC62157E-3E33-4FEC-9920-2D3B36D750DF BF72CF98
InstallProtocolInterface: AD61F191-AE5F-4C0E-B9FA-E869D288C64F BFDE53A0
Loading driver 1A1E4886-9517-440E-9FDE-3BE44CEE2136
InstallProtocolInterface: 5B1B31A1-9562-11D2-8E3F-00A0C969723B BF72CB00
Loading driver at 0x000BFDCF000 EntryPoint=0x000BFDCF2AF CpuDxe.efi
InstallProtocolInterface: BC62157E-3E33-4FEC-9920-2D3B36D750DF BF72C998
InstallProtocolInterface: 26BACCB1-6F42-11D4-BCE7-0080C73C8881 BFDDBF20
  Flushing GCD
  Flushing GCD
  Flushing GCD
  Flushing GCD
  Flushing GCD
  Flushing GCD
  Flushing GCD
  Flushing GCD
  Flushing GCD
  Flushing GCD
  Flushing GCD
  Flushing GCD
  Flushing GCD
  Flushing GCD
  Flushing GCD
  Flushing GCD
  Flushing GCD
Loading driver 6F0198AA-1F1D-426D-AE3E-39AB633FCC28
InstallProtocolInterface: 5B1B31A1-9562-11D2-8E3F-00A0C969723B BF72BE00
Loading driver at 0x000BFEAB000 EntryPoint=0x000BFEAB2AF KbcReset.efi
InstallProtocolInterface: BC62157E-3E33-4FEC-9920-2D3B36D750DF BF728D18
InstallProtocolInterface: 27CFAC88-46CC-11D4-9A38-0090273FC14D 0
Loading driver C8339973-A563-4561-B858-D8476F9DEFC4
InstallProtocolInterface: 5B1B31A1-9562-11D2-8E3F-00A0C969723B BF7288C0
Loading driver at 0x000BFDC9000 EntryPoint=0x000BFDC92AF Metronome.efi
InstallProtocolInterface: BC62157E-3E33-4FEC-9920-2D3B36D750DF BF728A98
InstallProtocolInterface: 26BACCB2-6F42-11D4-BCE7-0080C73C8881 BFDCBF20
Loading driver 22DC2B60-FE40-42AC-B01F-3AB1FAD9AAD8
InstallProtocolInterface: 5B1B31A1-9562-11D2-8E3F-00A0C969723B BF72B9C0
Loading driver at 0x000BFEA3000 EntryPoint=0x000BFEA32AF EmuVariableFvbRuntimeDxe.efi
InstallProtocolInterface: BC62157E-3E33-4FEC-9920-2D3B36D750DF BF726F98
EMU Variable FVB Started
Disabling EMU Variable FVB since flash variables appear to be supported.
Error: Image at 000BFEA3000 start failed: Aborted
Loading driver 2226F30F-3D5B-402D-9936-A97184EB4516
InstallProtocolInterface: 5B1B31A1-9562-11D2-8E3F-00A0C969723B BF726040
Loading driver at 0x000BED8A000 EntryPoint=0x000BED8A2AF VariableAuthRuntimeDxe.efi
InstallProtocolInterface: BC62157E-3E33-4FEC-9920-2D3B36D750DF BF72BA98
Variable Store header is corrupted

ASSERT_EFI_ERROR (Status = Volume Corrupt)
ASSERT /var/lib/jenkins/jobs/edk2/workspace/rpmbuild/rpm/BUILD/edk2.git/SecurityPkg/VariableAuthenticated/RuntimeDxe/VariableDxe.c(449): !EFI_ERROR (Status)