Arch Linux

4kGamer

Member

Registered: 2014-10-29

Posts: 88

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

ok, I see. So more than 4 isn't gonna work.

I will use 4 then:

 <vcpu placement='static'>4</vcpu>
  <cputune>
    <vcpupin vcpu='0' cpuset='0'/>
    <vcpupin vcpu='1' cpuset='1'/>
    <vcpupin vcpu='2' cpuset='2'/>
    <vcpupin vcpu='3' cpuset='3'/>
 </cputune>

that looks simple. Hopefully I understood correctly.

Last edited by 4kGamer (2015-01-08 15:52:20)

4kGamer

Member

Registered: 2014-10-29

Posts: 88

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

sorry, double post

Last edited by 4kGamer (2015-01-08 15:51:55)

aw

Member

Registered: 2013-10-04

Posts: 921

Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

And change

  <cpu mode='host-passthrough'>
    <topology sockets='1' cores='4' threads='2'/>
  </cpu>

to threads='1'

---

http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

4kGamer

Member

Registered: 2014-10-29

Posts: 88

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

ah yes, of course. Thank you Alex.

kaeptnb

Member

Registered: 2013-09-04

Posts: 30

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

... although I don't know how they came up with those cpuset values.  AFAIK, cpusets are the same as processor numbers, where the cores are generally enumerated before the threads.  Thus on a 4-core w/ threads processor, 0-3 are cores and 4-7 are threads, where 4 is the thread sibling of 0, 5 of 1, etc.

this whole core/socket/thread/sibling really gave me a headache...
i found out - as you described, that 0/4 is actually my first core, 1/5 2nd and so on;
Its new to me that e.g. 4 is only "25%" of the power - relative to 0 ... but anyways it didn't matter to me because my goal was to represent my real cpu as close as possible to the guest.

hence the  threads='2' and the exact pinning as i posted (2-6-3-7) - that way i was quite sure that windows also sees that 2 of the cores are "thread" cores - and hopefully does "the right thing" regarding its internal scheduling.

benchmarks backed this: guest with vcpus pinned this way:
root@pc:/usr/local/sbin> cat vcpu-2637.sh
virsh vcpupin windows7 --vcpu 0 2
virsh vcpupin windows7 --vcpu 1 6
virsh vcpupin windows7 --vcpu 2 3
virsh vcpupin windows7 --vcpu 3 7

was faster in "wprime" thread benchmarking than guest with vcpus pinned that way:

root@pc:/usr/local/sbin> cat vcpu-2367.sh
virsh vcpupin windows7 --vcpu 0 2
virsh vcpupin windows7 --vcpu 1 3
virsh vcpupin windows7 --vcpu 2 6
virsh vcpupin windows7 --vcpu 3 7

so i concluded, a) windows "knows" about thread-cores b) my pinning was ok

HTH
Greetings

Last edited by kaeptnb (2015-01-08 16:28:25)

aw

Member

Registered: 2013-10-04

Posts: 921

Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

... although I don't know how they came up with those cpuset values.  AFAIK, cpusets are the same as processor numbers, where the cores are generally enumerated before the threads.  Thus on a 4-core w/ threads processor, 0-3 are cores and 4-7 are threads, where 4 is the thread sibling of 0, 5 of 1, etc.

this whole core/socket/thread/sibling really gave me a headache...
i found out - as you described, that 0/4 is actually my first core, 1/5 2nd and so on;
Its new to me that e.g. 4 is only "25%" of the power - relative to 0 ... but anyways it didn't matter to me because my goal was to represent my real cpu as close as possible to the guest.

hence the  threads='2' and the excact pinning as i posted - that way i was quite sure that windows also sees that 2 of the cores are "thread" cores - and hopefully does "the right thing" regarding its internal scheduling.

Aha, I missed that you used threads=2, so that makes a bit more sense.  So you're actually trying to represent exactly half of your CPU to the guest.

benchmarks backed this: guest with vcpus pinned this way:
root@pc:/usr/local/sbin> cat vcpu-2637.sh
virsh vcpupin windows7 --vcpu 0 2
virsh vcpupin windows7 --vcpu 1 6
virsh vcpupin windows7 --vcpu 2 3
virsh vcpupin windows7 --vcpu 3 7

So let's look at your actual mapping:

cpuset: core,thread
2: 2,0
6: 2,1
3: 3,0
7: 3,1

ie. core/thread/core/thread

was faster in "wprime" thread benchmarking than guest with vcpus pinned that way:

root@pc:/usr/local/sbin> cat vcpu-2367.sh
virsh vcpupin windows7 --vcpu 0 2
virsh vcpupin windows7 --vcpu 1 3
virsh vcpupin windows7 --vcpu 2 6
virsh vcpupin windows7 --vcpu 3 7

so i concluded, a) windows "knows" about thread-cores b) my pinning was ok

Here we've got:

2: 2,0
3: 3,0
6: 2,1
7: 3,1

ie. core/core/thread/thread

I'd expect the first mode would be what the guest expects.  Were you using the same cpu topology for these tests, cores=2/threads=2?

My host system shows:

$ egrep "^processor|^core id" /proc/cpuinfo 
processor	: 0
core id		: 0
processor	: 1
core id		: 1
processor	: 2
core id		: 2
processor	: 3
core id		: 3
processor	: 4
core id		: 0
processor	: 5
core id		: 1
processor	: 6
core id		: 2
processor	: 7
core id		: 3

So bare metal is core/core/core/core/thread/thread/thread/thread, which I think is pretty standard.

---

http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

kaeptnb

Member

Registered: 2013-09-04

Posts: 30

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

yes, both vcpu-pinnings were tried with host-passthrough cpu;
and always with "cores=2, threads=2", yes.

as i said - my goal was to represent my real cpu as close as possible.

here my host-system cpuinfo:

root@pc:~> egrep "^processor|^core id" /proc/cpuinfo
processor       : 0
core id         : 0
processor       : 1
core id         : 1
processor       : 2
core id         : 2
processor       : 3
core id         : 3
processor       : 4
core id         : 0
processor       : 5
core id         : 1
processor       : 6
core id         : 2
processor       : 7
core id         : 3

all of this is only a "problem" with intel hyperthreading cpu's - afaik amd "only" ships "real" cores, no hyperthreading, right?

P.S.: now i get your core/thread vs. core/core argument ; i'll try to redo my benchmarks, but iirc (its some time ago...) 2637 was superior to 2367...
It's afaik not documented what exactly libvirt does with "threads=2" config stanza - if it picks the "threads" itself and if it "tells" the guests these are "just" threading-cores... i don't know, so i benchmarked it; but - let me do it again

Greetings!

Last edited by kaeptnb (2015-01-08 17:32:00)

aw

Member

Registered: 2013-10-04

Posts: 921

Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

all of this is only a "problem" with intel hyperthreading cpu's - afaik amd "only" ships "real" cores, no hyperthreading, right?

Depends how you define "problem".  I find non-competitive performance to be a "problem".  Threading is a common way to try to make the most effective use of the pipeline.  It requires the scheduler, and in this case the user, know that not all the processors are completely independent of each other.  If you don't agree to that, disable it in your BIOS.  Back in the P4 days threads were of questionable benefit for quite a few workloads, but I think they've come a long way since then.

P.S.: now i get your core/thread vs. core/core argument ; i'll try to redo my benchmarks, but iirc (its some time ago...) 2637 was superior to 2367...
It's afaik not documented what exactly libvirt does with "threads=2" config stanza - if it picks the "threads" itself and if it "tells" the guests these are "just" threading-cores... i don't know, so i benchmarked it; but - let me do it again

It mirrors the QEMU args:

-smp [cpus=]n[,maxcpus=cpus][,cores=cores][,threads=threads][,sockets=sockets]
                set the number of CPUs to 'n' [default=1]
                maxcpus= maximum number of total cpus, including
                offline CPUs for hotplug, etc
                cores= number of CPU cores on one socket
                threads= number of threads on one CPU core
                sockets= number of discrete sockets in the system

---

http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

kaeptnb

Member

Registered: 2013-09-04

Posts: 30

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Ok, a quick re-run of wprime with both (2367 and 2637) configs didn't reveal any visible difference ...

this either means, the cache-trashing effect is negliable - or libvirt/qemu does the "right" thing, and tells the guest that cpu #2 and #3 (with 2367) are threading-cores, and #0 and #1 are "real" cores ...

I think its the later

4kGamer

Member

Registered: 2014-10-29

Posts: 88

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

are you using wprime in Windows or do you run it from host?

kaeptnb

Member

Registered: 2013-09-04

Posts: 30

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Ok this is bothering me, i re-did my benchmarks, this time more often for each case and under the very same conditions (guest was settled a few minutes after boot, so not too much interfered with the cpu-benchmark).
i did 6 benchmarks with wprime, 6 times the "32m" test;
here the results:

2367 setup:

1: 18,943
2: 20,157
3: 19,93
4: 19,058
5: 19,003
6: 18,95

2637 setup:

1: 18,892
2: 18,861
3: 18,943
4: 18,9
5: 19,011
6: 18,984

so - i retract my conclusion from post #3784 and say its better to tell qemu/libvirt what to do, it seems it doesn't do the right thing itself.

Last edited by kaeptnb (2015-01-08 18:09:00)

kaeptnb

Member

Registered: 2013-09-04

Posts: 30

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

4k: yes within the guest; you want to know which vcpu-pinning is better - for the guest. The host doesn't care much about your kvm-cpu pinnings

Greetings

aw

Member

Registered: 2013-10-04

Posts: 921

Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Ok this is bothering me, i re-did my benchmarks, this time more often for each case and under the very same conditions (guest was settled a few minutes after boot, so not too much interfered with the cpu-benchmark).
i did 6 benchmarks with wprime, 6 times the "32m" test;
here the results:

2367 setup:

1: 18,943
2: 20,157
3: 19,93
4: 19,058
5: 19,003
6: 18,95

2637 setup:

1: 18,892
2: 18,861
3: 18,943
4: 18,9
5: 19,011
6: 18,984

so - i retract my conclusion from post #3784 and say its better to tell qemu/libvirt what to do, it seems it doesn't do the right thing itself.

I don't think libvirt makes any attempt to re-interpret pinning.  You tell it to pin a vCPU to one or more CPU and that's what it does.  It doesn't care if the pinning target is in the correct order compared to the topology you're describing to the guest.  So I think you've reconfirmed the expected result here, right?  If a topology with theads is described to the guest, the first half of the vCPUs enumerate the cores with one thread, the second half re-enumerate the cores with a different thread.

---

http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

kaeptnb

Member

Registered: 2013-09-04

Posts: 30

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

It mirrors the QEMU args:

yes, so far so good; let me rephrase my question then: its (at least to me) unknown, what qemu does with this config stanza.

best case would be (with threads=>1), qemu searches for thread-cpu's, picks one of these as thread-cpus for the guest and TELLS the guest that this is a threading-cpu;
OTOH - we're defining the pinning ourselfs, so the rationale is i guess that if you do that, you know what you're doing, and nothing happens automatically.

well; i'll stick with my 2637 setup here, but your milage may vary.

So I think you've reconfirmed the expected result here, right?  If a topology with theads is described to the guest, the first half of the vCPUs enumerate the cores with one thread, the second half re-enumerate the cores with a different thread.

yes this is what i expected;

Rule of thumb: if you have a hyperthreading CPU and want 1:1 vcpu-pinning: take care of your thread-cores 

Greetings!

Last edited by kaeptnb (2015-01-08 18:25:23)

kaeptnb

Member

Registered: 2013-09-04

Posts: 30

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

@4kgamer: also remember to define "    <emulatorpin cpuset='0-1,4-5'/>" - it takes a lot of cpu-load (which qemu itself generates to manage your guest) away from your guest-cpus

What i also have active, but never measured the concrete impact regarding benchmarks is
" isolcpus=2,3,6,7 nohz_full=2,3,6,7 " in my kernel-cmdline;

and "CPUAffinity=0 1 4 5" in my systemd's system.conf; this and "isocpus" helps to free up cpu's 2,3,6,7 for only kvm guests;

nohz_full _should_ enable the host-kernel to lower the "Local timer interrupts" rate on these cpu's which in turn lowers the cpu's work it's doing without the guest knowing...
( look at http://www.breakage.org/2013/11/15/nohz_fullgodmode/ regarding nohz)

To free the kvm-cpus up from even more work, at boot i run this script:

#!/bin/bash

# 33 = 0,1,4,5
# 11 = 0,4
# 22 = 1,5
# 01 = 0
# 02 = 1
# 10 = 4
# 20 = 5

# irq balancing doesnt work well for more than one CPUs, so just one:
irqmask1=02
#for some devices other interrupts:
irqmask2=20
procmask=33
#rcuomask=$procmask
rcuomask=22

#Interrupts:
echo $irqmask1 >/proc/irq/default_smp_affinity
for irq in $(ls -1 /proc/irq/ ) ; do
        [ -w /proc/irq/$irq/smp_affinity ] && echo $irqmask1 >/proc/irq/$irq/smp_affinity 2>/dev/null
done

# some interrupt-intensive hardware:
for i in $(grep -i ahci     /proc/interrupts | awk -F: '{print $1}') ; do echo $irqmask2 >/proc/irq/$i/smp_affinity ; done
for i in $(grep -i eth      /proc/interrupts | awk -F: '{print $1}') ; do echo $irqmask2 >/proc/irq/$i/smp_affinity ; done
for i in $(grep -i nouveau  /proc/interrupts | awk -F: '{print $1}') ; do echo $irqmask2 >/proc/irq/$i/smp_affinity ; done
for i in $(grep -i xhci     /proc/interrupts | awk -F: '{print $1}') ; do echo $irqmask2 >/proc/irq/$i/smp_affinity ; done

# for mdadm
echo "$procmask" >/sys/devices/virtual/workqueue/writeback/cpumask
echo "$procmask" >/sys/devices/virtual/workqueue/raid5wq/cpumask
echo "$procmask" >/sys/bus/workqueue/devices/writeback/cpumask
echo 0 > /sys/bus/workqueue/devices/writeback/numa

# adjusts rcuo kernel threads - see : https://www.kernel.org/doc/Documentation/timers/NO_HZ.txt
for pid in $(pgrep rcuo) ; do
        taskset -p $rcuomask $pid >/dev/null
done

Greetings!

Last edited by kaeptnb (2015-01-08 19:02:17)

aw

Member

Registered: 2013-10-04

Posts: 921

Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

It mirrors the QEMU args:

yes, so far so good; let me rephrase my question then: its (at least to me) unknown, what qemu does with this config stanza.

best case would be (with threads=>1), qemu searches for thread-cpu's, picks one of these as thread-cpus for the guest and TELLS the guest that this is a threading-cpu;
OTOH - we're defining the pinning ourselfs, so the rationale is i guess that if you do that, you know what you're doing, and nothing happens automatically.

First, let's clear something up, what we're calling a "core" here is really just the first thread on the core.  If the other thread has no work, the first thread should get roughly the full core.  If the second thread has work and the first thread doesn't, the second thread should get roughly the whole core.  It's only when both have work to do that we have an interaction.  It's that interaction at the hardware level that we're trying to describe to the guest.  As far as QEMU/KVM is concerned these are all just configuration bits for the guest that don't have any substantive effect on the vCPU.  A vCPU is just a thread and it's up to the host scheduler where to run that thread.  By pinning the vCPUs, we know we're going to have an interaction at the hardware level that the guest needs to be aware of, otherwise it will assume each vCPU is completely independent and make poor scheduling decisions.

---

http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

aw

Member

Registered: 2013-10-04

Posts: 921

Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

@4kgamer: also remember to define "    <emulatorpin cpuset='0-1,4-5'/>" - it takes a lot of cpu-load (which qemu itself generates to manage your guest) away from your guest-cpus

FWIW, you're the only one I've seen using this option.  It makes sense, 4k would want to use 4-7, but I'm not sure how beneficial it is.

What i also have active, but never measured the concrete impact regarding benchmarks is
" isolcpus=2,3,6,7 nohz_full=2,3,6,7 " in my kernel-cmdline;

This _should_ enable the host-kernel to lower the "Local timer interrupts" rate on these cpu's which in turn lowers the cpu's work it's doing without the guest knowing...
( look at http://www.breakage.org/2013/11/15/nohz_fullgodmode/ regarding nohz)

Please be careful with isolcpus.  This option excludes the listed CPUs from the general scheduler, so they won't be available to normal host processes without manual scheduling, regardless of whether a guest is currently running.  If you're running the guest full time and want to reduce the host impact on it, this is a good option to use.

EDIT: correction, 4-7

Last edited by aw (2015-01-08 19:13:50)

---

http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

kaeptnb

Member

Registered: 2013-09-04

Posts: 30

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

right, all these hints are coming from someone who's kvm-guest is running 24/7 - as soon as the host boots up;

OTOH i am able to use all CPUs within the host - i just created a cpuset:

cgcreate -t kaeptnb -g cpuset:/allcpus
cgset -r cpuset.cpus=0-7 allcpus

and when i want to run a task on the host which needs full cpu i just do e.g.

cgexec -g cpuset:/allcpus mpv 4kmovie.mkv

Last edited by kaeptnb (2015-01-08 19:00:10)

4kGamer

Member

Registered: 2014-10-29

Posts: 88

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

thank you guys, I will try <emulatorpin cpuset='0-1,4-5'/>" and compare. Really grateful for this thread. Everyone that's helping: THANK YOU. I wouldn't be able to run the VM without you.

kaeptnb

Member

Registered: 2013-09-04

Posts: 30

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

FWIW, you're the only one I've seen using this option.  It makes sense, 4k would want to use 4-7, but I'm not sure how beneficial it is.

I just took a look:

ps -Leo cmd,pid,lwp,cputime,psr  

shows for the qemu processes:

qemu-system-x86_64 -enable- 23109 23109 00:01:23   5
qemu-system-x86_64 -enable- 23109 23112 00:06:22   2
qemu-system-x86_64 -enable- 23109 23113 00:05:31   6
qemu-system-x86_64 -enable- 23109 23114 00:05:15   3
qemu-system-x86_64 -enable- 23109 23115 00:06:52   7
qemu-system-x86_64 -enable- 23109 23122 00:00:06   2
qemu-system-x86_64 -enable- 23109 23123 00:00:01   2
qemu-system-x86_64 -enable- 23109 23124 00:00:24   3
qemu-system-x86_64 -enable- 23109 23125 00:00:00   7
qemu-system-x86_64 -enable- 23109 23126 00:00:00   3

so - only one thread is outside the vcpu-range, the first one;

the first and the last 5 threads seem to be the emulator-threads,
i expected more - all but 4 vcpu-threads to be exact - to be on cpu 0,1,4 or 5.

So you may be right, it's not that much helpful - but OTOH its not bad, at least one thread is "offloaded" from the guest-cpus;

Last edited by kaeptnb (2015-01-08 19:35:39)

aw

Member

Registered: 2013-10-04

Posts: 921

Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

So you may be right, it's not that much helpful - but OTOH its not bad, at least one thread is "offloaded" from the guest-cpus;

Theoretically, if you're using isolcpu, 2,3,6,7 shouldn't be available for general scheduling for the emulator anyway.

---

http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

kaeptnb

Member

Registered: 2013-09-04

Posts: 30

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Well i'm surprised as well and can't explain why only one of the 6 emulator-threads are on non-guest cpus... strange

kaeptnb

Member

Registered: 2013-09-04

Posts: 30

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Ok, i set the emulatorpin to just one cpu (    <emulatorpin cpuset='1'/> ), restarted the guest, ran the "1024m" wprime test (just to see which threads are the vcpu threads and which are the emulator threads)

watch "ps -Leo cmd,pid,lwp,cputime,psr  | grep qemu" 

reveals, that the emulator-threads are NOT pinned - they change their cpu from time to time.
And i still see changing them to cpu's which should be "protected" by isol_cpu... strange

Last edited by kaeptnb (2015-01-08 19:53:44)

Bronek

Member

From: London

Registered: 2014-02-14

Posts: 119

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

After conversion to libvirt I have this:

<domain type='kvm'>
  <name>windows1</name>
  <uuid>de644f73-fd03-4e6a-81dd-eb3e8cdd5305</uuid>
  <memory unit='KiB'>16777216</memory>
  <currentMemory unit='KiB'>16777216</currentMemory>
  <memoryBacking>
    <hugepages>
      <page size='2048' unit='KiB'/>
    </hugepages>
  </memoryBacking>
  <vcpu placement='static'>8</vcpu>
  <cputune>
    <vcpupin vcpu='0' cpuset='8'/>
    <vcpupin vcpu='1' cpuset='9'/>
    <vcpupin vcpu='2' cpuset='10'/>
    <vcpupin vcpu='3' cpuset='11'/>
    <vcpupin vcpu='4' cpuset='20'/>
    <vcpupin vcpu='5' cpuset='21'/>
    <vcpupin vcpu='6' cpuset='22'/>
    <vcpupin vcpu='7' cpuset='23'/>
  </cputune>
  <os>
    <type arch='x86_64' machine='pc-i440fx-2.2'>hvm</type>
    <loader type='rom'>/usr/share/qemu/bios.bin</loader>
  </os>
  <features>
    <acpi/>
    <apic/>
    <pae/>
    <hyperv>
      <relaxed state='on'/>
      <vapic state='on'/>
      <spinlocks state='on' retries='4096'/>
    </hyperv>
  </features>
  <cpu mode='custom' match='exact'>
    <model fallback='allow'>Penryn</model>
    <topology sockets='1' cores='4' threads='2'/>
  </cpu>
  <clock offset='localtime'>
    <timer name='rtc' tickpolicy='catchup'/>
    <timer name='pit' tickpolicy='delay'/>
    <timer name='hpet' present='no'/>
    <timer name='hypervclock' present='yes'/>
  </clock>
  <on_poweroff>destroy</on_poweroff>
  <on_reboot>restart</on_reboot>
  <on_crash>destroy</on_crash>
  <devices>
    <emulator>/usr/bin/qemu-system-x86_64.xvga</emulator>
    <disk type='block' device='disk'>
      <driver name='qemu' type='raw'/>
      <source dev='/dev/zvol/zdata/windows1'/>
      <target dev='vda' bus='virtio'/>
      <boot order='1'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x0'/>
    </disk>
    <disk type='file' device='cdrom'>
      <driver name='qemu' type='raw'/>
      <source file='/home/bronek/virtio-win-0.1-81.iso'/>
      <target dev='hdc' bus='ide'/>
      <readonly/>
      <address type='drive' controller='0' bus='0' target='0' unit='0'/>
    </disk>
    <controller type='usb' index='0' model='ich9-ehci1'>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x04' function='0x7'/>
    </controller>
    <controller type='usb' index='0' model='ich9-uhci1'>
      <master startport='0'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x04' function='0x0' multifunction='on'/>
    </controller>
    <controller type='ide' index='0'>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x1'/>
    </controller>
    <controller type='pci' index='0' model='pci-root'/>
    <interface type='bridge'>
      <mac address='52:54:01:34:56:f7'/>
      <source bridge='br0'/>
      <model type='virtio'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x0'/>
    </interface>
    <hostdev mode='subsystem' type='usb' managed='yes'>
      <source>
        <vendor id='0x04d9'/>
        <product id='0x2011'/>
      </source>
    </hostdev>
    <hostdev mode='subsystem' type='pci' managed='yes'>
      <driver name='vfio'/>
      <source>
        <address domain='0x0000' bus='0x00' slot='0x1b' function='0x0'/>
      </source>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x09' function='0x0'/>
    </hostdev>
    <hostdev mode='subsystem' type='pci' managed='yes'>
      <driver name='vfio'/>
      <source>
        <address domain='0x0000' bus='0x00' slot='0x1d' function='0x0'/>
      </source>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x0a' function='0x0'/>
    </hostdev>
    <hostdev mode='subsystem' type='pci' managed='yes'>
      <driver name='vfio'/>
      <source>
        <address domain='0x0000' bus='0x00' slot='0x1a' function='0x0'/>
      </source>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x0b' function='0x0'/>
    </hostdev>
    <hostdev mode='subsystem' type='pci' managed='yes'>
      <driver name='vfio'/>
      <source>
        <address domain='0x0000' bus='0x81' slot='0x00' function='0x0'/>
      </source>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x0e' function='0x0'/>
    </hostdev>
    <hostdev mode='subsystem' type='pci' managed='yes'>
      <driver name='vfio'/>
      <source>
        <address domain='0x0000' bus='0x83' slot='0x00' function='0x0'/>
      </source>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x0f' function='0x0'/>
    </hostdev>
    <hostdev mode='subsystem' type='pci' managed='yes'>
      <driver name='vfio'/>
      <source>
        <address domain='0x0000' bus='0x82' slot='0x00' function='0x0'/>
      </source>
      <rom file='/home/bronek/Turks.vga.rom'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x08' function='0x0'/>
    </hostdev>
    <memballoon model='virtio'>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x07' function='0x0'/>
    </memballoon>
  </devices>
</domain>

Note "fancy" <emulator>/usr/bin/qemu-system-x86_64.xvga</emulator>, which is:

root@gdansk ~ # cat /usr/bin/qemu-system-x86_64.xvga
#!/bin/sh
exec /usr/bin/qemu-system-x86_64 `echo "$@" | sed 's/vga.rom/vga.rom,x-vga=on/g'`

It works briliantly,  thanks aw for your hard work on this feature! Best of all, since it's virsh and I've setup core isolation on my kernel, I now know that this virtual machine is guaranteed full access to selected PCI, CPU and RAM resources - and it starts automatically when I switch on my (metal) machine - it's really fast

Small nigglees: there is now long list of devices under "safely remove" icon including all my passed thru PCI devices and Virtio ones as well. I hope I can find a way to somehow convince Windows that these should not be listed. Also, I had to "downgrade" VCPU from host, SandyBridge and Nehalem to Penryn (another stable choice was qemu64)

Duelist

Member

Registered: 2014-09-22

Posts: 358

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

...long story short...
I've finally flashed my cards with UEFI-compatible "secret"(i can't get it anywhere anymore) firmware, but appears like windows7 doesn't really works right with it. I turn off CSM, boot up the windows, it shows the splash logo and then image freezes. Yay for windows7. So, OVMF/UEFI and windows 7 is possible but troublesome, so most folks shouldn't bother about it.

P.S. if anyone needs UEFI-combatible VBIOS file for ASUS HD7750(silent, 800mhz one, but i think it's possible to edit it for any card) - send me an e-mail, because i seem to be unable to get it officially from support anymore.

---

The forum rules prohibit requesting support for distributions other than arch.
I gave up. It was too late.
What I was trying to do.
The reference about VFIO and KVM VGA passthrough.