Arch Linux

myweb

Member

Registered: 2013-07-13

Posts: 69

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

And how did the 7750 work?

Windows 8 performance test shows: Graphics performance is 7.3 for 3d/2d.
But I noticed mouse hanging (not smooth moving) when Windows asks Administrator privileges (all screen covered with gray and just one dialog, that asks about administrator privileges, is active)

Did you happen to be using a pre-rc5 kernel that already included these?

No

I need help one more time ... I get no success to boot already installed Windows 7 to real drive.
Partition table looks like:
sda1 EFI
sda2 Linux Swap
sda3 GPT system
sda4 Arch Linux (KVM qemu host)
sda5 Windows 7

I specified:
-drive file=/dev/sda,id=disk,format=raw -device ide-hd,bus=ahci.0,drive=disk

But I got black screen, not windows booting.
Could you please help to boot from real hard drive?

aw

Member

Registered: 2013-10-04

Posts: 921

Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

And how did the 7750 work?

Windows 8 performance test shows: Graphics performance is 7.3 for 3d/2d.
But I noticed mouse hanging (not smooth moving) when Windows asks Administrator privileges (all screen covered with gray and just one dialog, that asks about administrator privileges, is active)

Did you happen to be using a pre-rc5 kernel that already included these?

No

I need help one more time ... I get no success to boot already installed Windows 7 to real drive.
Partition table looks like:
sda1 EFI
sda2 Linux Swap
sda3 GPT system
sda4 Arch Linux (KVM qemu host)
sda5 Windows 7

I specified:
-drive file=/dev/sda,id=disk,format=raw -device ide-hd,bus=ahci.0,drive=disk

But I got black screen, not windows booting.
Could you please help to boot from real hard drive?

This is effectively the same as moving a disk with a windows install from one physical system to another.  Windows does not do well with this.  There are those that can hack it to work, but for best results, it's recommended to use a separate install.

Last edited by aw (2013-10-29 14:17:41)

---

http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

aw

Member

Registered: 2013-10-04

Posts: 921

Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

I gave a talk about VFIO-VGA support at KVM Forum last week.  Unfortunately due to technology malfunction there's no video, but the slides are linked here:

https://plus.google.com/107691536544094 … y8EHhJ6PWz

---

http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

aw

Member

Registered: 2013-10-04

Posts: 921

Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Is the pcie_acs_override useful? Haven't used it up till now.

It's useful if you have an IOMMU group that includes too many devices.  Intel didn't include ACS support for the root ports in many of their chipsets and processors, without it we can't guarantee isolation between devices.  We're working with Intel to get a whitelist of root ports to resolve this.  In the meantime, yes the pcie_acs_override patch can be used as a workaround.  Note that it only changes IOMMU group topology.  It doesn't specifically fix or enabling anything new wrt VGA.

---

http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

empie

Member

From: The Netherlands

Registered: 2013-06-15

Posts: 9

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

It's useful if you have an IOMMU group that includes too many devices.  Intel didn't include ACS support for the root ports in many of their chipsets and processors, without it we can't guarantee isolation between devices.  We're working with Intel to get a whitelist of root ports to resolve this.  In the meantime, yes the pcie_acs_override patch can be used as a workaround.  Note that it only changes IOMMU group topology.  It doesn't specifically fix or enabling anything new wrt VGA.

Ok, thanks. Will give it a go in the next couple of days.

myweb

Member

Registered: 2013-07-13

Posts: 69

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

I use the following script to bind devices:

#!/bin/bash

modprobe vfio-pci

for dev in "$@"; do
        vendor=$(cat /sys/bus/pci/devices/$dev/vendor)
        device=$(cat /sys/bus/pci/devices/$dev/device)
        if [ -e /sys/bus/pci/devices/$dev/driver ]; then
                echo $dev > /sys/bus/pci/devices/$dev/driver/unbind
        fi
        echo $vendor $device > /sys/bus/pci/drivers/vfio-pci/new_id
done

without "pcie_acs_override=downstream" "ls /sys/bus/pci/devices/0000\:02\:00.0/iommu_group/devices/" shows:

0000:00:1c.0
0000:00:1c.4
0000:00:1c.5
0000:00:1c.7
0000:02:00.0
0000:02:00.1
0000:03:00.0
0000:04:01.0
0000:05:00.0
0000:06:00.0

with:

0000:02:00.0
0000:02:00.1

myweb

Member

Registered: 2013-07-13

Posts: 69

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

@nbhs, @aw
I think it would be great to create some kind of Wiki page with summary/list of successful configuration (CPU, Motherboard, VideoCard) and short note of specific settings for particular configuration fo example.
Do we have that page?

Is it correct that in order to use real partition along with virtual drive I need to specify:

-drive file=/home/nbhs/windows.img,id=bootDisk,format=raw -device ide-hd,bus=ahci.0,drive=bootDisk
-drive file=/dev/sdb,id=dataDisk,format=raw -device ide-hd,bus=ahci.1,drive=dataDisk

Could you please clarify how to specify boot order for seabios: what drive to boot?

Last edited by myweb (2013-10-30 11:09:15)

blitzschlag

Member

Registered: 2013-10-30

Posts: 11

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

hey

just wanted to chime in to report success on VGA passthrough with inspiration found in this Thread.

my Hardware
Gigabyte GA970A-UD3 (BIOS F8 beta with fixed ACPI-IVRS Table)
AMD Phenom II x6 1090T
EVGA Geforce GTX 285
Sapphire Radeon HD 5670

My Software
Debian Jessie
Kernel 3.12rc6 (Jessie Stock 3.10-3 got no vfio-pci-vga enabled anyway)
qemu-kvm 1.6.0+dfsg-2 (debian unstable)
Windows7 x64 in VM with latest Nvidia Driver

Only downside is that i have the reboot issue. e.g. i can't reboot the Windows VM without rebooting the Host aswell.

Cheers for the Inspiration and Hints

aw

Member

Registered: 2013-10-04

Posts: 921

Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

hey

just wanted to chime in to report success on VGA passthrough with inspiration found in this Thread.

my Hardware
Gigabyte GA970A-UD3 (BIOS F8 beta with fixed ACPI-IVRS Table)
AMD Phenom II x6 1090T
EVGA Geforce GTX 285
Sapphire Radeon HD 5670

My Software
Debian Jessie
Kernel 3.12rc6 (Jessie Stock 3.10-3 got no vfio-pci-vga enabled anyway)
qemu-kvm 1.6.0+dfsg-2 (debian unstable)
Windows7 x64 in VM with latest Nvidia Driver

Only downside is that i have the reboot issue. e.g. i can't reboot the Windows VM without rebooting the Host aswell.

Cheers for the Inspiration and Hints

Reboot should be fixed with current qemu.git or wait for 1.7 when it comes out.  Just to be clear, you're assigning the GTX285 and the HD5670 is your host display, right?

---

http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

blitzschlag

Member

Registered: 2013-10-30

Posts: 11

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Reboot should be fixed with current qemu.git or wait for 1.7 when it comes out.  Just to be clear, you're assigning the GTX285 and the HD5670 is your host display, right?

ah, cool

Yes, the GTX 285 is passed through to the Windows VM

kaeptnb

Member

Registered: 2013-09-04

Posts: 30

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Hi,

thanks for this thread! Using this setup for a couple of years under XEN - but with KVM and VFIO its easier and - the most important part - more stable (with xen the Guest-Reboot was not always successful - with KVM it is!).

Anyways i converted most of the manual qemu options into libvirt-xml - main reason was that libvirt handles vCPU pinning and i couln't find a way to do it manually.
Another positive effect is Host Startup/Shutdown handling due to the libvirt init script - they stop/start your Guest on Host startup/shutdown.

<domain type='kvm' id='26' xmlns:qemu='http://libvirt.org/schemas/domain/qemu/1.0'>                                     
  <name>windows7</name>                                                                                                 
  <uuid>xxxxxxx-xxxxx-xxxxx-xxxx-xxxxxxxxx</uuid>                                                                     
  <memory unit='KiB'>8388608</memory>                                                                                   
  <currentMemory unit='KiB'>8388608</currentMemory>                                                                     
  <memoryBacking>                                                                                                       
    <nosharepages/>                                                                                                     
    <locked/>                                                                                                           
  </memoryBacking>                                                                                                      
  <vcpu placement='static'>4</vcpu>                                                                                     
  <cputune>                                                                                                             
    <vcpupin vcpu='0' cpuset='2'/>                                                                                      
    <vcpupin vcpu='1' cpuset='6'/>                                                                                      
    <vcpupin vcpu='2' cpuset='3'/>                                                                                      
    <vcpupin vcpu='3' cpuset='7'/>                                                                                      
    <emulatorpin cpuset='0-1,4-5'/>                                                                                     
  </cputune>                                                                                                            
  <resource>                                                                                                            
    <partition>/machine</partition>                                                                                     
  </resource>                                                                                                           
  <os>                                                                                                                  
    <type arch='x86_64' machine='pc-q35-1.5'>hvm</type>                                                                 
    <boot dev='hd'/>                                                                                                    
    <smbios mode='host'/>                                                                                               
  </os>                                                                                                                 
  <cpu mode='host-passthrough'>                                                                                         
    <topology sockets='1' cores='2' threads='2'/>                                                                       
  </cpu>                                                                                                                
  <clock offset='localtime'/>                                                                                           
  <on_poweroff>destroy</on_poweroff>                                                                                    
  <on_reboot>restart</on_reboot>                                                                                        
  <on_crash>destroy</on_crash>                                                                                          
  <devices>                                                                                                             
    <emulator>/usr/local/kvm/bin/qemu-system-x86_64</emulator>                                                          
    <disk type='block' device='disk'>                                                                                   
      <driver name='qemu' type='raw' cache='none' discard='unmap'/>                                                     
      <source dev='/dev/ssd/win7system'/>                                                                               
      <target dev='vda' bus='virtio'/>                                                                                  
      <alias name='virtio-disk0'/>                                                                                      
      <address type='pci' domain='0x0000' bus='0x02' slot='0x03' function='0x0'/>                                       
    </disk>                                                                                                             
    <disk type='block' device='disk'>                                                                                   
      <driver name='qemu' type='raw' cache='none' discard='unmap'/>                                                     
      <source dev='/dev/mapper/win7games_cached'/>                                                                      
      <target dev='vdb' bus='virtio'/>                                                                                  
      <alias name='virtio-disk1'/>                                                                                      
      <address type='pci' domain='0x0000' bus='0x02' slot='0x04' function='0x0'/>                                       
    </disk>                                                                                                             
    <controller type='sata' index='0'>                                                                                  
      <alias name='sata0'/>                                                                                             
      <address type='pci' domain='0x0000' bus='0x00' slot='0x1f' function='0x2'/>                                       
    </controller>                                                                                                       
    <controller type='pci' index='0' model='pcie-root'>                                                                 
      <alias name='pcie.0'/>                                                                                            
    </controller>                                                                                                       
    <controller type='pci' index='1' model='dmi-to-pci-bridge'>                                                         
      <alias name='pci.1'/>                                                                                             
      <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x0'/>                                       
    </controller>                                                                                                       
    <controller type='pci' index='2' model='pci-bridge'>                                                                
      <alias name='pci.2'/>                                                                                             
      <address type='pci' domain='0x0000' bus='0x01' slot='0x01' function='0x0'/>                                       
    </controller>                                                                                                       
    <controller type='usb' index='0' model='none'>                                                                      
      <alias name='usb0'/>                                                                                              
      <address type='pci' domain='0x0000' bus='0x02' slot='0x01' function='0x0'/>                                       
    </controller>
    <interface type='bridge'>
      <mac address='00:xx:xx:xx:xx:xx'/>
      <source bridge='br0'/>
      <target dev='vnet0'/>
      <model type='virtio'/>
      <alias name='net0'/>
      <rom bar='off'/>
      <address type='pci' domain='0x0000' bus='0x02' slot='0x02' function='0x0'/>
    </interface>
    <memballoon model='none'>
      <alias name='balloon0'/>
    </memballoon>
  </devices>
  <seclabel type='none'/>
  <qemu:commandline>
    <qemu:arg value='-device'/>
    <qemu:arg value='ioh3420,bus=pcie.0,addr=1c.0,multifunction=on,port=1,chassis=1,id=root.1'/>
    <qemu:arg value='-device'/>
    <qemu:arg value='vfio-pci,host=08:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on'/>
    <qemu:arg value='-device'/>
    <qemu:arg value='vfio-pci,host=08:00.1,bus=root.1,addr=00.1'/>
    <qemu:arg value='-device'/>
    <qemu:arg value='vfio-pci,host=00:1d.0,bus=pcie.0'/>
    <qemu:arg value='-device'/>
    <qemu:arg value='vfio-pci,host=00:1d.1,bus=pcie.0'/>
    <qemu:arg value='-device'/>
    <qemu:arg value='vfio-pci,host=00:1d.2,bus=pcie.0'/>
    <qemu:arg value='-device'/>
    <qemu:arg value='vfio-pci,host=00:1d.7,bus=pcie.0'/>
    <qemu:arg value='-bios'/>
    <qemu:arg value='/usr/src/kvm/seabios-1.7.2-patched/seabios/out/bios.bin'/>
    <qemu:env name='DISPLAY' value=':0'/>
    <qemu:env name='QEMU_PA_SAMPLES' value='128'/>
    <qemu:env name='QEMU_AUDIO_DRV' value='pa'/>
  </qemu:commandline>
</domain>

Adapt UUID and MAC adress - and of course your PCI-IDs...

Greetings
K.

Last edited by kaeptnb (2013-10-30 20:54:40)

aw

Member

Registered: 2013-10-04

Posts: 921

Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Great tip!  I've parsed it down to what I think are the important parts if anyone wants to try to adapt their own guest xml.  libvirt does have vfio support now, but it doesn't (and shouldn't) know about the x-vga=on option.  If you just want to add ",x-vga=on" to a libvirt managed guest you can follow the instructions here to make a wrapper around the emulator. (a different bios shouldn't be necessary on qemu.git)

                                                                                                       
  <os>                                                                                                                 
    <type arch='x86_64' machine='pc-q35-1.5'>hvm</type>                                                                 
    ...
  </os>                                                                                                                 
  ...
  <devices>                                                                                                             
    <emulator>/usr/local/kvm/bin/qemu-system-x86_64</emulator>                                                         
    ...
  </devices>
  ...
  <qemu:commandline>
    <qemu:arg value='-device'/>
    <qemu:arg value='ioh3420,bus=pcie.0,addr=1c.0,multifunction=on,port=1,chassis=1,id=root.1'/>
    <qemu:arg value='-device'/>
    <qemu:arg value='vfio-pci,host=08:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on'/>
    <qemu:arg value='-device'/>
    <qemu:arg value='vfio-pci,host=08:00.1,bus=root.1,addr=00.1'/>
    ...
    <qemu:arg value='-bios'/>
    <qemu:arg value='/usr/src/kvm/seabios-1.7.2-patched/seabios/out/bios.bin'/>
    ...
  </qemu:commandline>

---

http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

kaeptnb

Member

Registered: 2013-09-04

Posts: 30

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

iirc if you pass through qemu args, you need the xmlns so that libvirt can validate the xml:

xmlns:qemu='http://libvirt.org/schemas/domain/qemu/1.0'

(1st line)

And regarding the CPU Pinning/Isolation: the "isolcpu" Kernel Parameter gets very handy - in my case : "isolcpus=2,3,6,7"

BTW it REALLY makes a difference (i went through great lenghts to benchmark it) to keep in mind cache trashing - so try to pass through the corresponding CPUs when you have a HT CPU like me (i7-920).

IRQ Balance is another hint - i have bad experience with the irqbalance daemon, so i made myself a ugly startup-script which puts the most intense interrupts to non-guest CPUs:

for i in $(grep -i ahci     /proc/interrupts | awk -F: '{print $1}') ; do echo 2 >/proc/irq/$i/smp_affinity ; done
...

Greetings!
K.

empie

Member

From: The Netherlands

Registered: 2013-06-15

Posts: 9

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Is it correct that in order to use real partition along with virtual drive I need to specify:

-drive file=/home/nbhs/windows.img,id=bootDisk,format=raw -device ide-hd,bus=ahci.0,drive=bootDisk
-drive file=/dev/sdb,id=dataDisk,format=raw -device ide-hd,bus=ahci.1,drive=dataDisk

Could you please clarify how to specify boot order for seabios: what drive to boot?

I use this to first boot from a slipstreamed iso, then the snapshot of the vm-disk on a logical volume:

-drive file=/dev/vg01/win7gtx560ti-slip4.1-snap01,cache=none,id=disk,format=raw -device ide-hd,bus=ahci.0,drive=disk \
-drive file=/data/win7hp-sp1-x64-ss20131008-rt7-4.iso,id=isocd -device ide-cd,bus=ahci.1,drive=isocd \
-boot dc

And thanks for the info regarding the binding of devices.

Last edited by empie (2013-10-30 18:17:18)

nbhs

Member

From: Montevideo, Uruguay

Registered: 2013-05-02

Posts: 402

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

...
main reason was that libvirt handles vCPU pinning and i couln't find a way to do it manually
...

Have you tried using cset and taskset? https://code.google.com/p/cpuset/, thats what i use on my script (on the front page) to pin my vm vcpus (6) to my cpu cores (2-7)

Last edited by nbhs (2013-10-30 18:27:09)

kaeptnb

Member

Registered: 2013-09-04

Posts: 30

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Yes, that works (taskset or cgroups) - but doesnt ensure 1:1 pinning;
i.e. i can say qemu with its 4 vcpus should run on cpus 2,3,6,7 - but nothing ensures the qemu threads aren't hopping between these 4 real cpus.
libvirt ensures that - it stops qemu after initialization, pins each thread, and continues qemu...

sad qemu doesn't support this directly - but OTOH its not bad to have libvirt and with all its benefits running, so...

Greetings

nbhs

Member

From: Montevideo, Uruguay

Registered: 2013-05-02

Posts: 402

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Yes, that works (taskset or cgroups) - but doesnt ensure 1:1 pinning;
i.e. i can say qemu with its 4 vcpus should run on cpus 2,3,6,7 - but nothing ensures the qemu threads aren't hopping between these 4 real cpus.
libvirt ensures that - it stops qemu after initialization, pins each thread, and continues qemu...

sad qemu doesn't support this directly - but OTOH its not bad to have libvirt and with all its benefits running, so...

Greetings

I believe qemu creates a thread per vcpu, so you can pin each thread to a core using taskset

kaeptnb

Member

Registered: 2013-09-04

Posts: 30

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Yes, each vcpu is a thread.
but which thread is which vcpu (important regarding cache-trashing)?
and (thats maybe just my incompetence) - how can you work with threads and taskset (afaik it only nows -p - the PID) ?

aw

Member

Registered: 2013-10-04

Posts: 921

Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

'info cpus' from the monitor gives the TID for each vcpu

---

http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

kaeptnb

Member

Registered: 2013-09-04

Posts: 30

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

ah nice. and taskset can handle TID also with -p i guess?

but the shell magic to automise this is probably more hassle than to adapt to libvirt

anyways - its worth the hassle either way - fewer microlags - i never tried it (i just use it for starting game.exe) but i bet you could even use it as a DAW station

Greetings
K.

Edit: "it" - being the windows7 kvm guest

Last edited by kaeptnb (2013-10-30 20:41:51)

kaeptnb

Member

Registered: 2013-09-04

Posts: 30

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

regarding latency:
i noticed a huge latency penalty with "-usbdevice tablet " and "-vnc 0.0.0.0:10" - oddly?!

meassurable with DPC Latency Checker ( http://www.thesycon.de/deu/latency_check.shtml )

Chetyre

Member

Registered: 2013-03-27

Posts: 34

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Hi,

thanks for this thread! Using this setup for a couple of years under XEN - but with KVM and VFIO its easier and - the most important part - more stable (with xen the Guest-Reboot was not always successful - with KVM it is!).

Anyways i converted most of the manual qemu options into libvirt-xml - main reason was that libvirt handles vCPU pinning and i couln't find a way to do it manually.
Another positive effect is Host Startup/Shutdown handling due to the libvirt init script - they stop/start your Guest on Host startup/shutdown.

<domain type='kvm' id='26' xmlns:qemu='http://libvirt.org/schemas/domain/qemu/1.0'>                                     
  <name>windows7</name>                                                                                                 
  <uuid>xxxxxxx-xxxxx-xxxxx-xxxx-xxxxxxxxx</uuid>                                                                     
  <memory unit='KiB'>8388608</memory>                                                                                   
  <currentMemory unit='KiB'>8388608</currentMemory>                                                                     
  <memoryBacking>                                                                                                       
    <nosharepages/>                                                                                                     
    <locked/>                                                                                                           
  </memoryBacking>                                                                                                      
  <vcpu placement='static'>4</vcpu>                                                                                     
  <cputune>                                                                                                             
    <vcpupin vcpu='0' cpuset='2'/>                                                                                      
    <vcpupin vcpu='1' cpuset='6'/>                                                                                      
    <vcpupin vcpu='2' cpuset='3'/>                                                                                      
    <vcpupin vcpu='3' cpuset='7'/>                                                                                      
    <emulatorpin cpuset='0-1,4-5'/>                                                                                     
  </cputune>                                                                                                            
  <resource>                                                                                                            
    <partition>/machine</partition>                                                                                     
  </resource>                                                                                                           
  <os>                                                                                                                  
    <type arch='x86_64' machine='pc-q35-1.5'>hvm</type>                                                                 
    <boot dev='hd'/>                                                                                                    
    <smbios mode='host'/>                                                                                               
  </os>                                                                                                                 
  <cpu mode='host-passthrough'>                                                                                         
    <topology sockets='1' cores='2' threads='2'/>                                                                       
  </cpu>                                                                                                                
  <clock offset='localtime'/>                                                                                           
  <on_poweroff>destroy</on_poweroff>                                                                                    
  <on_reboot>restart</on_reboot>                                                                                        
  <on_crash>destroy</on_crash>                                                                                          
  <devices>                                                                                                             
    <emulator>/usr/local/kvm/bin/qemu-system-x86_64</emulator>                                                          
    <disk type='block' device='disk'>                                                                                   
      <driver name='qemu' type='raw' cache='none' discard='unmap'/>                                                     
      <source dev='/dev/ssd/win7system'/>                                                                               
      <target dev='vda' bus='virtio'/>                                                                                  
      <alias name='virtio-disk0'/>                                                                                      
      <address type='pci' domain='0x0000' bus='0x02' slot='0x03' function='0x0'/>                                       
    </disk>                                                                                                             
    <disk type='block' device='disk'>                                                                                   
      <driver name='qemu' type='raw' cache='none' discard='unmap'/>                                                     
      <source dev='/dev/mapper/win7games_cached'/>                                                                      
      <target dev='vdb' bus='virtio'/>                                                                                  
      <alias name='virtio-disk1'/>                                                                                      
      <address type='pci' domain='0x0000' bus='0x02' slot='0x04' function='0x0'/>                                       
    </disk>                                                                                                             
    <controller type='sata' index='0'>                                                                                  
      <alias name='sata0'/>                                                                                             
      <address type='pci' domain='0x0000' bus='0x00' slot='0x1f' function='0x2'/>                                       
    </controller>                                                                                                       
    <controller type='pci' index='0' model='pcie-root'>                                                                 
      <alias name='pcie.0'/>                                                                                            
    </controller>                                                                                                       
    <controller type='pci' index='1' model='dmi-to-pci-bridge'>                                                         
      <alias name='pci.1'/>                                                                                             
      <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x0'/>                                       
    </controller>                                                                                                       
    <controller type='pci' index='2' model='pci-bridge'>                                                                
      <alias name='pci.2'/>                                                                                             
      <address type='pci' domain='0x0000' bus='0x01' slot='0x01' function='0x0'/>                                       
    </controller>                                                                                                       
    <controller type='usb' index='0' model='none'>                                                                      
      <alias name='usb0'/>                                                                                              
      <address type='pci' domain='0x0000' bus='0x02' slot='0x01' function='0x0'/>                                       
    </controller>
    <interface type='bridge'>
      <mac address='00:xx:xx:xx:xx:xx'/>
      <source bridge='br0'/>
      <target dev='vnet0'/>
      <model type='virtio'/>
      <alias name='net0'/>
      <rom bar='off'/>
      <address type='pci' domain='0x0000' bus='0x02' slot='0x02' function='0x0'/>
    </interface>
    <memballoon model='none'>
      <alias name='balloon0'/>
    </memballoon>
  </devices>
  <seclabel type='none'/>
  <qemu:commandline>
    <qemu:arg value='-device'/>
    <qemu:arg value='ioh3420,bus=pcie.0,addr=1c.0,multifunction=on,port=1,chassis=1,id=root.1'/>
    <qemu:arg value='-device'/>
    <qemu:arg value='vfio-pci,host=08:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on'/>
    <qemu:arg value='-device'/>
    <qemu:arg value='vfio-pci,host=08:00.1,bus=root.1,addr=00.1'/>
    <qemu:arg value='-device'/>
    <qemu:arg value='vfio-pci,host=00:1d.0,bus=pcie.0'/>
    <qemu:arg value='-device'/>
    <qemu:arg value='vfio-pci,host=00:1d.1,bus=pcie.0'/>
    <qemu:arg value='-device'/>
    <qemu:arg value='vfio-pci,host=00:1d.2,bus=pcie.0'/>
    <qemu:arg value='-device'/>
    <qemu:arg value='vfio-pci,host=00:1d.7,bus=pcie.0'/>
    <qemu:arg value='-bios'/>
    <qemu:arg value='/usr/src/kvm/seabios-1.7.2-patched/seabios/out/bios.bin'/>
    <qemu:env name='DISPLAY' value=':0'/>
    <qemu:env name='QEMU_PA_SAMPLES' value='128'/>
    <qemu:env name='QEMU_AUDIO_DRV' value='pa'/>
  </qemu:commandline>
</domain>

Adapt UUID and MAC adress - and of course your PCI-IDs...

Greetings
K.

I tried creating my own xml based on yours, but virsh fails with:

error: Failed to start domain windows7
error: internal error: early end of file from monitor: possible problem:
qemu-system-x86_64: -device vfio-pci,host=02:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on: vfio: error opening /dev/vfio/15: Permission denied
qemu-system-x86_64: -device vfio-pci,host=02:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on: vfio: failed to get group 15
qemu-system-x86_64: -device vfio-pci,host=02:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on: Device initialization failed.
qemu-system-x86_64: -device vfio-pci,host=02:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on: Device 'vfio-pci' could not be initialized

What could be wrong?

aw

Member

Registered: 2013-10-04

Posts: 921

Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

I tried creating my own xml based on yours, but virsh fails with:

error: Failed to start domain windows7
error: internal error: early end of file from monitor: possible problem:
qemu-system-x86_64: -device vfio-pci,host=02:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on: vfio: error opening /dev/vfio/15: Permission denied
qemu-system-x86_64: -device vfio-pci,host=02:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on: vfio: failed to get group 15
qemu-system-x86_64: -device vfio-pci,host=02:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on: Device initialization failed.
qemu-system-x86_64: -device vfio-pci,host=02:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on: Device 'vfio-pci' could not be initialized

What could be wrong?

I'd guess you need to change a few things in /etc/libvirt/qemu.conf

user & group should probably be root
clear_emulator_capabilities should probably be 0

You'll still need to setup the devices with vfio prior to starting the guest.

---

http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Chetyre

Member

Registered: 2013-03-27

Posts: 34

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

I tried creating my own xml based on yours, but virsh fails with:

error: Failed to start domain windows7
error: internal error: early end of file from monitor: possible problem:
qemu-system-x86_64: -device vfio-pci,host=02:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on: vfio: error opening /dev/vfio/15: Permission denied
qemu-system-x86_64: -device vfio-pci,host=02:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on: vfio: failed to get group 15
qemu-system-x86_64: -device vfio-pci,host=02:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on: Device initialization failed.
qemu-system-x86_64: -device vfio-pci,host=02:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on: Device 'vfio-pci' could not be initialized

What could be wrong?

I'd guess you need to change a few things in /etc/libvirt/qemu.conf

user & group should probably be root
clear_emulator_capabilities should probably be 0

You'll still need to setup the devices with vfio prior to starting the guest.

Now it gives me a different error

error: Failed to start domain windows7
error: internal error: early end of file from monitor: possible problem:
qemu-system-x86_64: -device vfio-pci,host=02:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on: vfio: error opening /dev/vfio/15: Operation not permitted
qemu-system-x86_64: -device vfio-pci,host=02:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on: vfio: failed to get group 15
qemu-system-x86_64: -device vfio-pci,host=02:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on: Device initialization failed.
qemu-system-x86_64: -device vfio-pci,host=02:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on: Device 'vfio-pci' could not be initialized

The devices are bound to vfio already.

aw

Member

Registered: 2013-10-04

Posts: 921

Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Now it gives me a different error

error: Failed to start domain windows7
error: internal error: early end of file from monitor: possible problem:
qemu-system-x86_64: -device vfio-pci,host=02:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on: vfio: error opening /dev/vfio/15: Operation not permitted
qemu-system-x86_64: -device vfio-pci,host=02:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on: vfio: failed to get group 15
qemu-system-x86_64: -device vfio-pci,host=02:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on: Device initialization failed.
qemu-system-x86_64: -device vfio-pci,host=02:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on: Device 'vfio-pci' could not be initialized

The devices are bound to vfio already.

vfio operates on iommu groups so you may need to attach more than just the devices being assigned to vfio-pci.  Look in /sys/bus/pci/devices/0000:02:00.0/iommu_group/devices/ and verify all the devices in there other than root ports are attached to vfio-pci.

---

http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users