Arch Linux

thearcherblog

Member

Registered: 2014-10-30

Posts: 165

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

I am having a terrible time getting audio working in a Windows 8.1 VM with qemu...
PCI passthrough is working perfectly with Radeon GPU, motherboard's 2nd ethernet port, and motherboard's 2nd SATA controller being passed to the VM. The last piece I need is working audio!

To try to enable sound (with pulseaudio used on host), I am doing qemu.sh:

export QEMU_AUDIO_DRV=pa

exec qemu-system-x86_64 \
	-enable-kvm \
	...... \
	-soundhw hda

,

and the VM fails to launch, with dmesg showing:

[79307.989184] kvm [22292]: vcpu0 unhandled rdmsr: 0xce
[79307.993410] kvm [22292]: vcpu0 unhandled rdmsr: 0x637
[79307.993418] kvm [22292]: vcpu0 unhandled rdmsr: 0xe8
[79307.993422] kvm [22292]: vcpu0 unhandled rdmsr: 0xe7
[79307.993426] kvm [22292]: vcpu0 unhandled rdmsr: 0x31
[79307.993430] kvm [22292]: vcpu0 unhandled rdmsr: 0x35
[79307.993434] kvm [22292]: vcpu0 unhandled rdmsr: 0x39
[79307.993448] kvm [22292]: vcpu0 unhandled rdmsr: 0xce
[79307.993451] kvm [22292]: vcpu0 unhandled rdmsr: 0x194
[79307.993454] kvm [22292]: vcpu0 unhandled rdmsr: 0x19a
[79313.196146] kvm_get_msr_common: 170 callbacks suppressed
[79313.196149] kvm [22292]: vcpu0 unhandled rdmsr: 0x150
[79313.196156] kvm [22292]: vcpu0 unhandled rdmsr: 0x150
[79313.196158] kvm [22292]: vcpu0 unhandled rdmsr: 0x150
[79313.196161] kvm [22292]: vcpu0 unhandled rdmsr: 0x150
[79313.205095] kvm [22292]: vcpu3 unhandled rdmsr: 0x150
[79313.205102] kvm [22292]: vcpu3 unhandled rdmsr: 0x150
[79313.205105] kvm [22292]: vcpu3 unhandled rdmsr: 0x150
[79313.205107] kvm [22292]: vcpu3 unhandled rdmsr: 0x150
[79313.214116] kvm [22292]: vcpu1 unhandled rdmsr: 0x150
[79313.214123] kvm [22292]: vcpu1 unhandled rdmsr: 0x150
[79318.241747] kvm_get_msr_common: 70 callbacks suppressed
...
...
...
[79950.317052] vfio-pci 0000:02:00.1: Refused to change power state, currently in D3
[79950.317119] vfio_pci_disable: Failed to reset device 0000:02:00.1 (-22)
[79950.328050] vfio-pci 0000:02:00.1: Refused to change power state, currently in D3
[84325.855696] vfio-pci 0000:02:00.0: enabling device (0400 -> 0403)
[84325.876928] vfio_ecap_init: 0000:02:00.0 hiding ecap 0x19@0x270
[84325.876931] vfio_ecap_init: 0000:02:00.0 hiding ecap 0x1b@0x2d0
[84325.878188] vfio-pci 0000:02:00.1: enabling device (0400 -> 0402)
[84325.900537] vfio-pci 0000:04:00.0: enabling device (0400 -> 0403)
[84326.906061] vfio-pci 0000:07:00.0: enabling device (0400 -> 0403)
[84330.202221] kvm: zapping shadow pages for mmio generation wraparound
[84338.019479] intel_iommu_map: iommu width (39) is not sufficient for the mapped address (c1f400001000)

Any insight into what I am experience? Thanks!

In my case I'm using the audio from the graphic card instead of the emulated one. And the performance is really good not like the emulated that is quite choppy and delayed (at least in my case).

mutiny

Member

Registered: 2015-02-15

Posts: 13

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

The monitor I am using doesn't have any speakers.

Also, the first set of messages (kvm [22292]: vcpu2 unhandled rdmsr: 0x150 etc) may not necessarily be related to the second set of messages (iommu width (39) is not sufficient for the mapped address) and non working audio/VM not working with -soundhw hda option, since I only got the 2nd set of messages after trying again.

EDIT:

Apparently it is an issue with passing through the HDMI audio of the video card that is causing the issue. It seems to work fine if I don't pass the HDMI audio on 02:00.1 and only pass the GPU itself to VM on 02:00.0. I guess it doesn't matter since I won't be using HDMI audio through the video card anyways.

I guess it makes sense from what dmesg says

vfio_pci_disable: Failed to reset device 0000:02:00.1 (-22)
vfio-pci 0000:02:00.1: Refused to change power state, currently in D3

It seems to cause an issue in trying to start the guest a 2nd time after shutting down if I don't pass the HDMI along with the GPU, but maybe it's another problem going on...
How would one resolve this, just in case they did want to be able to have the option? Is it a matter of ONLY having the option of passed through audio device or emulated audio through qemu and never both?

Last edited by mutiny (2015-02-24 10:07:55)

hotfunction

Member

Registered: 2015-02-05

Posts: 10

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

If 440fx and -nographic don't help and you're looking at the right output from the GPU, I'd next be suspicious of the ROM file.  It looks like one from techpowerup.  Does it actually match your card?  Can you dump the ROM from your card instead?  Do you even need a dump of the ROM?

hi sir aw.. i tried downloading various ROMs from other sites too, but none of them seems to get me working, they still gave me black screen output on the sscreen... and it seemed to be hard to find nvidia's ROM corresponding on that... so is it okay if i change into another GPU card? will it help me to work?

That black window is the sdl graphics head for the VM which appears because you haven't used the -nographic option.  The GPU output should be on a monitor connected to the assigned GPU.  I'd also recommend, like I almost always do, using 440fx rather than q35 for the VM.  It looks like you're using a Linux guest, but afaik, the nvidia driver does not have an issue with PCIe topology in Linux (until you get to lots of GPUs and that may already be fixed too).

and also about this part... i actually still confused, after some reading and stuff, i redone the steps from nbhs's page one, after making the single script and others.. i still see the black qemu command line window when adding -vga none... and also without a rom setting, i also tried rombar=0 option and -nographic still didnt give me output.. can you give me further explanation on this.. ?? i am very confused about the black qemu command line, and how does people use -vga none without -nographics to go straight to the VMS without the black qemu command line?

Edit : i tried adding -curses instead of -nographics, and its still giving me blank output, which i can type "quit" and i can go back to the qemu, which is that i am guessing that is still the black qemu command line output again.. removing that -vga none boots -curses to the VM.. so is the black qemu command line from the -vga none output is showing because of my GPU rom incorrect? or am i missing something..?

on this case, i tried to extract the ROM of my own GTX560Ti GPU, and i used it to launch my VMs, as the same that using -nographic and -vga none gives no output, any help on this anyone?

also i tried to put the romfile= option in the q35 machine ( becuase 440fx didn't work for me ) and i put my extracted rom from my GPU to the path, it still give me a same result instead.. -vga none still give me qemu command line monitor, adding -vga none and -nographics shows no output, and also if i remove -vga none and qemu command in q35 script, i can boot inside the VM, but it looks like the GPU is not utilized properly.. it shows in lspci in the linux, but the GPU passthrough didn't work, as i test opening youtube and it lags, and also tried unigine heaven benchmark and it still wont work, glx errors.. and also when i boot into windows 8, and adding kvm=off after -cpu argument, it won't let me boot in, soon after i log in into my account, its having an error_exception which keep restarts...

also i rechecked every steps and tried every possible thing on the nhbs's tutorial and went over some user's configuration here on this thread..
after i do some dmesg checking i found this on my dmesg:

[    0.029557] dmar: DMAR:[DMA Read] Request device [03:00.0] fault addr 40001000
[    0.029557] DMAR:[fault reason 06] PTE Read access is not set

here's my lspci :

00:00.0 Host bridge: Intel Corporation Xeon E3-1200 v2/3rd Gen Core processor DRAM Controller (rev 09)
00:01.0 PCI bridge: Intel Corporation Xeon E3-1200 v2/3rd Gen Core processor PCI Express Root Port (rev 09)
00:14.0 USB controller: Intel Corporation 7 Series/C210 Series Chipset Family USB xHCI Host Controller (rev 04)
00:16.0 Communication controller: Intel Corporation 7 Series/C210 Series Chipset Family MEI Controller #1 (rev 04)
00:1a.0 USB controller: Intel Corporation 7 Series/C210 Series Chipset Family USB Enhanced Host Controller #2 (rev 04)
00:1b.0 Audio device: Intel Corporation 7 Series/C210 Series Chipset Family High Definition Audio Controller (rev 04)
00:1c.0 PCI bridge: Intel Corporation 7 Series/C210 Series Chipset Family PCI Express Root Port 1 (rev c4)
00:1c.4 PCI bridge: Intel Corporation 7 Series/C210 Series Chipset Family PCI Express Root Port 5 (rev c4)
00:1c.5 PCI bridge: Intel Corporation 7 Series/C210 Series Chipset Family PCI Express Root Port 6 (rev c4)
00:1d.0 USB controller: Intel Corporation 7 Series/C210 Series Chipset Family USB Enhanced Host Controller #1 (rev 04)
00:1e.0 PCI bridge: Intel Corporation 82801 PCI Bridge (rev a4)
00:1f.0 ISA bridge: Intel Corporation B75 Express Chipset LPC Controller (rev 04)
00:1f.2 IDE interface: Intel Corporation 7 Series/C210 Series Chipset Family 4-port SATA Controller [IDE mode] (rev 04)
00:1f.3 SMBus: Intel Corporation 7 Series/C210 Series Chipset Family SMBus Controller (rev 04)
00:1f.5 IDE interface: Intel Corporation 7 Series/C210 Series Chipset Family 2-port SATA Controller [IDE mode] (rev 04)
01:00.0 PCI bridge: PLX Technology, Inc. PEX 8647 48-Lane, 3-Port PCI Express Gen 2 (5.0 GT/s) Switch (rev ab)
02:04.0 PCI bridge: PLX Technology, Inc. PEX 8647 48-Lane, 3-Port PCI Express Gen 2 (5.0 GT/s) Switch (rev ab)
02:08.0 PCI bridge: PLX Technology, Inc. PEX 8647 48-Lane, 3-Port PCI Express Gen 2 (5.0 GT/s) Switch (rev ab)
03:00.0 VGA compatible controller: Advanced Micro Devices, Inc. [AMD/ATI] R700 [Radeon HD 4870 X2]
03:00.1 Audio device: Advanced Micro Devices, Inc. [AMD/ATI] RV770 HDMI Audio [Radeon HD 4850/4870]
04:00.0 Display controller: Advanced Micro Devices, Inc. [AMD/ATI] R700 [Radeon HD 4870 X2]
05:00.0 VGA compatible controller: NVIDIA Corporation GF114 [GeForce GTX 560 Ti] (rev a1)
05:00.1 Audio device: NVIDIA Corporation GF114 HDMI Audio Controller (rev a1)
06:00.0 Ethernet controller: Realtek Semiconductor Co., Ltd. RTL8111/8168/8411 PCI Express Gigabit Ethernet Controller (rev 06)
07:00.0 Ethernet controller: Intel Corporation 82574L Gigabit Network Connection

cat /proc/cmdline/

BOOT_IMAGE=/vmlinuz-3.18.5 root=/dev/mapper/SSLVFIO--vg-root ro quiet i915.enable_hd_vgaarb=1 pcie_acs_override=downstream intel_iommu=on

qemu 2.2.0
linux 3.18-5, i patched linux-mainline 3.18-5 package from this thread

lspci -vnn to graphics cards:

03:00.0 VGA compatible controller [0300]: Advanced Micro Devices, Inc. [AMD/ATI] R700 [Radeon HD 4870 X2] [1002:9441] (prog-if 00 [VGA controller])
        Subsystem: ASUSTeK Computer Inc. Device [1043:0284]
        Flags: bus master, fast devsel, latency 0, IRQ 37
        Memory at d0000000 (64-bit, prefetchable) [size=256M]
        Memory at f6220000 (64-bit, non-prefetchable) [size=64K]
        I/O ports at b000 [size=256]
        Expansion ROM at f6200000 [disabled] [size=128K]
        Capabilities: [50] Power Management version 3
        Capabilities: [58] Express Legacy Endpoint, MSI 00
        Capabilities: [a0] MSI: Enable+ Count=1/1 Maskable- 64bit+
        Capabilities: [100] Vendor Specific Information: ID=0001 Rev=1 Len=010 <?>
        Kernel driver in use: radeon

03:00.1 Audio device [0403]: Advanced Micro Devices, Inc. [AMD/ATI] RV770 HDMI Audio [Radeon HD 4850/4870] [1002:aa30]
        Subsystem: ASUSTeK Computer Inc. Device [1043:aa30]
        Flags: bus master, fast devsel, latency 0, IRQ 36
        Memory at f6230000 (64-bit, non-prefetchable) [size=16K]
        Capabilities: [50] Power Management version 3
        Capabilities: [58] Express Legacy Endpoint, MSI 00
        Capabilities: [a0] MSI: Enable+ Count=1/1 Maskable- 64bit+
        Capabilities: [100] Vendor Specific Information: ID=0001 Rev=1 Len=010 <?>
        Kernel driver in use: snd_hda_intel

04:00.0 Display controller [0380]: Advanced Micro Devices, Inc. [AMD/ATI] R700 [Radeon HD 4870 X2] [1002:9441]
        Subsystem: ASUSTeK Computer Inc. Device [1043:0284]
        Flags: bus master, fast devsel, latency 0, IRQ 38
        Memory at c0000000 (64-bit, prefetchable) [size=256M]
        Memory at f6120000 (64-bit, non-prefetchable) [size=64K]
        I/O ports at a000 [size=256]
        Expansion ROM at f6100000 [disabled] [size=128K]
        Capabilities: [50] Power Management version 3
        Capabilities: [58] Express Legacy Endpoint, MSI 00
        Capabilities: [a0] MSI: Enable+ Count=1/1 Maskable- 64bit+
        Capabilities: [100] Vendor Specific Information: ID=0001 Rev=1 Len=010 <?>
        Kernel driver in use: radeon

05:00.0 VGA compatible controller [0300]: NVIDIA Corporation GF114 [GeForce GTX 560 Ti] [10de:1200] (rev a1) (prog-if 00 [VGA controller])
        Subsystem: Micro-Star International Co., Ltd. [MSI] Device [1462:2601]
        Flags: fast devsel, IRQ 16
        Memory at f4000000 (32-bit, non-prefetchable) [disabled] [size=32M]
        Memory at e0000000 (64-bit, prefetchable) [disabled] [size=128M]
        Memory at e8000000 (64-bit, prefetchable) [disabled] [size=64M]
        I/O ports at e000 [size=128]
        Expansion ROM at f6000000 [disabled] [size=512K]
        Capabilities: [60] Power Management version 3
        Capabilities: [68] MSI: Enable- Count=1/1 Maskable- 64bit+
        Capabilities: [78] Express Endpoint, MSI 00
        Capabilities: [b4] Vendor Specific Information: Len=14 <?>
        Capabilities: [100] Virtual Channel
        Capabilities: [128] Power Budgeting <?>
        Capabilities: [600] Vendor Specific Information: ID=0001 Rev=1 Len=024 <?>
        Kernel driver in use: vfio-pci

05:00.1 Audio device [0403]: NVIDIA Corporation GF114 HDMI Audio Controller [10de:0e0c] (rev a1)
        Subsystem: Micro-Star International Co., Ltd. [MSI] Device [1462:2601]
        Flags: bus master, fast devsel, latency 0, IRQ 17
        Memory at f6080000 (32-bit, non-prefetchable) [size=16K]
        Capabilities: [60] Power Management version 3
        Capabilities: [68] MSI: Enable- Count=1/1 Maskable- 64bit+
        Capabilities: [78] Express Endpoint, MSI 00
        Kernel driver in use: vfio-pci

- i think all steps were good.. suppose to be working.. but im always having black qemu command line on the result after adding -vga none..
- adding -vga none and -nographic together didn't show any result, no display at all..
- extracted ROM from my own NVIDIA card, and putting it romfile= into my scripts.. still not working at all
- tried using q35 and 440fx.. resulting q35 boots the VM, but 440fx only gave me black screen without any output.. ( both of them were not using -vga none and -nographic, of course, since those 2 options wont let me go in )
- booting on windows 8 without installing NVIDIA driver gives me error 43 , after installing, error 43 still shows up. ( q35, since 440fx gives no work for me )
- for windows 8 : added -cpu host,kvm=off, it wont let me boot, had an system_exceptional error and it bootloops. ( q35, since 440fx gives no work for me )

here are my scripts :
q35

#!/bin/bash

configfile=/etc/vfio-pci1.cfg

vfiobind() {
    dev="$1"
        vendor=$(cat /sys/bus/pci/devices/$dev/vendor)
        device=$(cat /sys/bus/pci/devices/$dev/device)
        if [ -e /sys/bus/pci/devices/$dev/driver ]; then
                echo $dev > /sys/bus/pci/devices/$dev/driver/unbind
        fi
        echo $vendor $device > /sys/bus/pci/drivers/vfio-pci/new_id

}

modprobe vfio-pci

cat $configfile | while read line;do
    echo $line | grep ^# >/dev/null 2>&1 && continue
        vfiobind $line
done

sudo qemu-system-x86_64 -M q35 -vga std -hda /home/sslab719/VMimages/w8.img -enable-kvm -m 2048 -cpu host \
-smp 2,sockets=1,cores=2,threads=1 \
-device ioh3420,bus=pcie.0,addr=1c.0,multifunction=on,port=1,chassis=1,id=root.1 \
-drive file=/home/sslab719/VMimages/w8.img,id=disk,format=qcow2 \
-device vfio-pci,host=05:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on,romfile=/home/sslab719/vbios1.rom \
-device vfio-pci,host=05:00.1,bus=root.1,addr=00.1 \
-net user,hostfwd=tcp::10022-:22 -net nic
#-boot menu=on

exit 0

i440fx

#!/bin/bash

configfile=/etc/vfio-pci1.cfg

vfiobind() {
    dev="$1"
        vendor=$(cat /sys/bus/pci/devices/$dev/vendor)
        device=$(cat /sys/bus/pci/devices/$dev/device)
        if [ -e /sys/bus/pci/devices/$dev/driver ]; then
                echo $dev > /sys/bus/pci/devices/$dev/driver/unbind
        fi
        echo $vendor $device > /sys/bus/pci/drivers/vfio-pci/new_id

}

modprobe vfio-pci

cat $configfile | while read line;do
    echo $line | grep ^# >/dev/null 2>&1 && continue
        vfiobind $line
done

sudo qemu-system-x86_64 -hda /home/sslab719/VMimages/w8.img -enable-kvm -M pc -nographic -vga none -m 2048 -cpu host,kvm=off \
-smp 2,sockets=1,cores=2,threads=1 \
-device vfio-pci,host=05:00.0,x-vga=on,romfile=/home/sslab719/vbios1.rom \
-device vfio-pci,host=05:00.1 \
-drive file=/home/sslab719/VMimages/w8.img,format=qcow2,media=disk \
-boot menu=on

exit 0

also i blacklisted nvidia from the host ( since i wanted to passthrough nvidia to the VMs ) it still doesn't make any change, and i tried removing the blacklist to standard configuration, not blacklisting any fglrx, nvidia, noveau, or radeon, but none worked for me..

additional : i tried testing running on linux ( q35, still no -vga none, no -nographic ), ubuntu 12.04 desktop version, and i had no glxinfo available, also on lspci shown standard graphics card 1234:1111, but i see my lspci showing my graphics card GTX 560 Ti, but it seems that my lspci is nothing, which doesn't give any good result, on additional drivers tab on settings it can automatically searches and installs nvidia drivers, but after install and reboot, it shows "this driver is installed but not activated"

also, i tried installing steam, and trying to type glxinfo, both of them saying that my GLX has errors, which mean its not really passthrough'in my GPU card...

i've been stuck here for 2 months and i am hoping this to work ..

EDIT 2 : I tried installing gnome on my host ubuntu server, and run a benchmark, installed Unigine Heaven Benchmark, the benchmark works correctly but it is showing Unknown GPU ( 256 MB ) -> ran on the host..

EDIT 3 : i finally tried using -vga none and -nographics menu, now the black qemu monitor shows up, then when i plug in my monitor to my secondary GPU ( the passthrough GPU ), finally i am able to see my VM using -vga none.. but this vm installs windows 8 with any installed NVIDIA drivers yet,  and also, this only works on the first boot of the VM, lets say i kill the VM and i change the script, then i start my VM again, the output won't show up again in the passthrough'd monitor, the only way to make it show up again is by rebooting the host.. and after it reboots i can only boot the VM again once... if i boot the VM twice, starting from that time it won't show the output again unless i reboot one more time..

any help on this please !! i think im almost there..

Last edited by hotfunction (2015-02-25 07:13:51)

instantepiphany

Member

Registered: 2013-09-04

Posts: 32

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

I am not using arch currently, but this is the best resource I could find with regards to vga passthrough on QEMU.
I have gotten everything working so far, except actual passthrough. When I run my vm, I hear the graphics card that I am passing through spin up like it does on boot, but when I switch to the input from that card, there is no display. What could be causing this?

FAQ #3/4

Aw,

I am using 3.18.5, and it seems the "other" vga patch was included in 3.17 - so I don't think that is causing my problem. What then do you think is causing my no output issue?

In case it is relevant - this is the only output I get in dmesg when running qemu (2:00.0 is a usb hub)

[20165.012459] vfio-pci 0000:02:00.0: enabling device (0400 -> 0402)
[20165.116200] vfio-pci 0000:01:00.0: enabling device (0400 -> 0403)
[20165.144582] vfio-pci 0000:01:00.1: enabling device (0400 -> 0402)

Last edited by instantepiphany (2015-02-24 10:58:33)

JohnyPea

Member

Registered: 2014-07-17

Posts: 17

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Did anybody try kernel 4.0-rc1? There are some experimental features for kvm regarding timers, which I was going to test. The VMs start and go through Seabios or OVMF, but then halt with first core maxed out. It seems that it's probably trying to allocate memory and gets stuck. It doesnt fail, nor progress. I didnt find anything regarding this, yet.

Duelist

Member

Registered: 2014-09-22

Posts: 358

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Did anybody try kernel 4.0-rc1? There are some experimental features for kvm regarding timers, which I was going to test. The VMs start and go through Seabios or OVMF, but then halt with first core maxed out. It seems that it's probably trying to allocate memory and gets stuck. It doesnt fail, nor progress. I didnt find anything regarding this, yet.

-debugcon file:debug.log -global isa-debugcon.iobase=0x402

Try adding this to see more logs from OVMF or seabios.

---

The forum rules prohibit requesting support for distributions other than arch.
I gave up. It was too late.
What I was trying to do.
The reference about VFIO and KVM VGA passthrough.

JohnyPea

Member

Registered: 2014-07-17

Posts: 17

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Did anybody try kernel 4.0-rc1? There are some experimental features for kvm regarding timers, which I was going to test. The VMs start and go through Seabios or OVMF, but then halt with first core maxed out. It seems that it's probably trying to allocate memory and gets stuck. It doesnt fail, nor progress. I didnt find anything regarding this, yet.

-debugcon file:debug.log -global isa-debugcon.iobase=0x402

Try adding this to see more logs from OVMF or seabios.

Thank you, this trick is very usefull. Did a quick test remotely and the log is quite long, but I dont see anything. Last lines:

Support(): UNDI3.1 found on handle BF29E298
 BlockSize : 512
 LastBlock : 95FFF
PartitionValidMbr: Bad MBR partition size EndingLBA(DEB56E8B) > LastLBA(95FFF)
 BlockSize : 512
 LastBlock : 3FFFF
 BlockSize : 512
 LastBlock : C6F6FFF
PartitionValidMbr: Bad MBR partition size EndingLBA(DEB56E8B) > LastLBA(C6F6FFF)
Select Item: 0x19
Select Item: 0x2B
SetBootOrderFromQemu: setting BootOrder: success
VideoFill: Past screen (Y)
PixelBlueGreenRedReserved8BitPerColor
PixelBlueGreenRedReserved8BitPerColor
Memory  Previous  Current    Next
 Type    Pages     Pages     Pages
======  ========  ========  ========
  0A    00000004  00000023  0000002B
  09    00000008  00000006  00000008
  00    00000004  00000000  00000004
  06    00000024  000000FB  00000139
  05    00000030  000000C9  000000FB
  03    00000180  00000538  00000686
  04    00000F00  00001226  000016AF
Booting Windows Boot Manager
InstallProtocolInterface: 5B1B31A1-9562-11D2-8E3F-00A0C969723B BF309E40
Loading driver at 0x00010000000 EntryPoint=0x000100061C0 bootmgfw.efi
InstallProtocolInterface: BC62157E-3E33-4FEC-9920-2D3B36D750DF BEF79918

I have noticed on this machine, which uses hugepages, that it probably allocated all of the memory. Therefore my initial thought was incorrect. I see that the boot process starts, on linux guest hangs on "loading initramfs" and on windows only displays logo, before animation. Will do more tests later with the simplest qemu VMs without periperals.

Krobar

Member

Registered: 2014-04-12

Posts: 17

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

A quick note for those wondering about using the GT620 I also had trouble with this card:
1) Would only work on first boot of the VM (No Reboot and unable tp stop and start after initial start)
2) Kodi under Linux would occasionally stop and then fast-forward video (Strange as the sound over HDMI was fine).

Geforce 960 does not suffer either of these issues and works prefectly within the same VM and config (Although serious overkill for HTPC VM use)

Last edited by Krobar (2015-02-24 17:57:08)

JohnyPea

Member

Registered: 2014-07-17

Posts: 17

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Did anybody try kernel 4.0-rc1? There are some experimental features for kvm regarding timers, which I was going to test. The VMs start and go through Seabios or OVMF, but then halt with first core maxed out. It seems that it's probably trying to allocate memory and gets stuck. It doesnt fail, nor progress. I didnt find anything regarding this, yet.

-debugcon file:debug.log -global isa-debugcon.iobase=0x402

Try adding this to see more logs from OVMF or seabios.

Thank you, this trick is very usefull. Did a quick test remotely and the log is quite long, but I dont see anything. Last lines:
...

So it may be some change in cgroups system which libvirt doesnt hadle well.

2015-02-24 17:56:03.746+0000: 5342: error : cgm_get_tasks:255 : cgmanager: cgm_get_tasks for controller=cpu, cgroup_path=machine/Win81.libvirt-qemu failed: Invalid arguments received in reply
2015-02-24 17:56:03.746+0000: 5342: error : cgm_get_tasks:255 : cgmanager: cgm_get_tasks for controller=cpuacct, cgroup_path=machine/Win81.libvirt-qemu failed: Invalid arguments received in reply
2015-02-24 17:56:03.746+0000: 5342: error : cgm_get_tasks:255 : cgmanager: cgm_get_tasks for controller=cpuset, cgroup_path=machine/Win81.libvirt-qemu failed: Invalid arguments received in reply

Qemu and kvm modules work. Tested with basic command line invocation and livecd.

JohnyPea

Member

Registered: 2014-07-17

Posts: 17

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

So it may be some change in cgroups system which libvirt doesnt hadle well.

2015-02-24 17:56:03.746+0000: 5342: error : cgm_get_tasks:255 : cgmanager: cgm_get_tasks for controller=cpu, cgroup_path=machine/Win81.libvirt-qemu failed: Invalid arguments received in reply
2015-02-24 17:56:03.746+0000: 5342: error : cgm_get_tasks:255 : cgmanager: cgm_get_tasks for controller=cpuacct, cgroup_path=machine/Win81.libvirt-qemu failed: Invalid arguments received in reply
2015-02-24 17:56:03.746+0000: 5342: error : cgm_get_tasks:255 : cgmanager: cgm_get_tasks for controller=cpuset, cgroup_path=machine/Win81.libvirt-qemu failed: Invalid arguments received in reply

Qemu and kvm modules work. Tested with basic command line invocation and livecd.

Well, sorry for this noise. Thats not it either. Happens exactly the same in 3.19 where the machine works flawlessly. Just keep an eye out for "lapic_timer_advance_ns" and "halt_poll_ns" in the future. I will just shut up now. :-)

Frans-Willem

Member

Registered: 2015-02-25

Posts: 9

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

I've got everything booting up on my system with a q35 machine model and qemu, and would like to convert it to a libvirt domain for easier start/stop management. However, I can't seem to find a XML example for Q35 and a PCIe device attached. Has anyone got this working, and if so, could you please share your XML ?

JohnyPea

Member

Registered: 2014-07-17

Posts: 17

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

I've got everything booting up on my system with a q35 machine model and qemu, and would like to convert it to a libvirt domain for easier start/stop management. However, I can't seem to find a XML example for Q35 and a PCIe device attached. Has anyone got this working, and if so, could you please share your XML ?

This is old and maybe not very good config:

<domain type='kvm' xmlns:qemu='http://libvirt.org/schemas/domain/qemu/1.0'>
  <name>GTX660Ti</name>
  <uuid>00000000-0000-0000-0000-000000000001</uuid>
  <memory unit='KiB'>6291456</memory>
  <currentMemory unit='KiB'>6291456</currentMemory>
  <memoryBacking>
    <hugepages>
      <page size='1048576' unit='KiB'/>
    </hugepages>
  </memoryBacking>
  <vcpu placement='static' cpuset='0-7'>8</vcpu>
  <cputune>
    <vcpupin vcpu='0' cpuset='0'/>
    <vcpupin vcpu='1' cpuset='1'/>
    <vcpupin vcpu='2' cpuset='2'/>
    <vcpupin vcpu='3' cpuset='3'/>
    <vcpupin vcpu='4' cpuset='4'/>
    <vcpupin vcpu='5' cpuset='5'/>
    <vcpupin vcpu='6' cpuset='6'/>
    <vcpupin vcpu='7' cpuset='7'/>
  </cputune>
  <os>
    <type arch='x86_64' machine='pc-q35-2.1'>hvm</type>
    <bootmenu enable='no'/>
  </os>
  <features>
    <acpi/>
    <apic/>
    <pae/>
    <hap/>
    <hyperv>
      <relaxed state='on'/>
      <vapic state='on'/>
      <spinlocks state='on' retries='4096'/>
    </hyperv>
    <kvm>
      <hidden state='on'/>
    </kvm>
    <pvspinlock state='on'/>
  </features>
  <cpu mode='host-passthrough'>
    <topology sockets='1' cores='8' threads='1'/>
  </cpu>
  <clock offset='utc'>
    <timer name='hypervclock' present='yes'/>
    <timer name='rtc' tickpolicy='catchup'/>
    <timer name='pit' tickpolicy='delay'/>
    <timer name='hpet' present='yes'/>
  </clock>
  <on_poweroff>destroy</on_poweroff>
  <on_reboot>restart</on_reboot>
  <on_crash>destroy</on_crash>
  <devices>
    <emulator>/usr/bin/qemu-system-x86_64</emulator>
    <disk type='file' device='disk'>
      <driver name='qemu' type='qcow2' cache='unsafe' io='threads' discard='unmap'/>
      <source file='/mnt/home/dzon/VM/WinVfio.qcow2'/>
      <target dev='vda' bus='virtio'/>
      <boot order='1'/>
      <address type='pci' domain='0x0000' bus='0x02' slot='0x04' function='0x0'/>
    </disk>
    <disk type='file' device='disk' snapshot='no'>
      <driver name='qemu' type='raw' cache='none' io='native'/>
      <source file='/dev/sda2'/>
      <target dev='vdb' bus='virtio'/>
      <address type='pci' domain='0x0000' bus='0x02' slot='0x03' function='0x0'/>
    </disk>
    <controller type='pci' index='0' model='pcie-root'/>
    <controller type='pci' index='1' model='dmi-to-pci-bridge'>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x1e' function='0x0'/>
    </controller>
    <controller type='pci' index='2' model='pci-bridge'>
      <address type='pci' domain='0x0000' bus='0x01' slot='0x01' function='0x0'/>
    </controller>
    <controller type='pci' index='3' model='pci-bridge'>
      <address type='pci' domain='0x0000' bus='0x01' slot='0x02' function='0x0'/>
    </controller>
    <controller type='usb' index='0' model='ich9-ehci1'>
      <address type='pci' domain='0x0000' bus='0x02' slot='0x01' function='0x7'/>
    </controller>
    <controller type='usb' index='0' model='ich9-uhci1'>
      <master startport='0'/>
      <address type='pci' domain='0x0000' bus='0x02' slot='0x01' function='0x0' multifunction='on'/>
    </controller>
    <controller type='usb' index='0' model='ich9-uhci2'>
      <master startport='2'/>
      <address type='pci' domain='0x0000' bus='0x02' slot='0x01' function='0x1'/>
    </controller>
    <controller type='usb' index='0' model='ich9-uhci3'>
      <master startport='4'/>
      <address type='pci' domain='0x0000' bus='0x02' slot='0x01' function='0x2'/>
    </controller>
    <controller type='sata' index='0'>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x1f' function='0x2'/>
    </controller>
    <controller type='ide' index='0'/>
    <interface type='network'>
      <mac address='52:54:00:12:34:01'/>
      <source network='virt'/>
      <model type='virtio'/>
      <driver name='vhost' txmode='iothread' ioeventfd='on' event_idx='on' queues='4'/>
      <rom bar='off'/>
      <address type='pci' domain='0x0000' bus='0x02' slot='0x02' function='0x0'/>
    </interface>
    <hostdev mode='subsystem' type='usb' managed='yes'>
      <source>
        <vendor id='0x046d'/>
        <product id='0xc52b'/>
      </source>
    </hostdev>
    <memballoon model='virtio'>
      <address type='pci' domain='0x0000' bus='0x02' slot='0x05' function='0x0'/>
    </memballoon>
  </devices>
  <qemu:commandline>
    <qemu:arg value='-device'/>
    <qemu:arg value='ioh3420,bus=pcie.0,addr=1c.0,multifunction=on,port=1,chassis=1,id=root.1'/>
    <qemu:arg value='-device'/>
    <qemu:arg value='vfio-pci,host=06:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on'/>
    <qemu:arg value='-device'/>
    <qemu:arg value='vfio-pci,host=06:00.1,bus=root.1,addr=00.1'/>
    <qemu:arg value='-set'/>
    <qemu:arg value='device.virtio-disk0.x-data-plane=off'/>
    <qemu:arg value='-set'/>
    <qemu:arg value='device.virtio-disk1.x-data-plane=on'/>
    <qemu:arg value='-set'/>
    <qemu:arg value='drive.drive-virtio-disk0.detect-zeroes=unmap'/>
  </qemu:commandline>
</domain>

I have switched to pc-i440fx machine. Libvirt handles it very well and I have generally less problems with it.

There is also a virsh command, try looking into this:

virsh help domxml-from-native
virsh domxml-from-native qemu-argv <config_file>

Last edited by JohnyPea (2015-02-25 10:33:02)

Denso

Member

Registered: 2014-08-30

Posts: 179

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Did anybody try kernel 4.0-rc1?

4.0-RC1 here . No issue . It was a completely transparent upgrade .

Last edited by Denso (2015-02-25 10:32:16)

JohnyPea

Member

Registered: 2014-07-17

Posts: 17

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Did anybody try kernel 4.0-rc1?

4.0-RC1 here . No issue . It was a completely transparent upgrade .

There has to be some specific error in my kernel or libvirt configs. At least I know it should be working. Thanks.

trimm

Member

Registered: 2014-03-06

Posts: 31

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

People with nvidia cards passthrough. Are you using it with the 340.52 driver and hv-time or using it with the latest driver with no hv-time?
Im unsure if I get better performance with the former or the latter. Anyone with some experience on this?

Duelist

Member

Registered: 2014-09-22

Posts: 358

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

People with nvidia cards passthrough. Are you using it with the 340.52 driver and hv-time or using it with the latest driver with no hv-time?
Im unsure if I get better performance with the former or the latter. Anyone with some experience on this?

Depends on the card.
If it's an old card like 4XX series - chances are there will be performance drawback just from newer drivers alone. I've witnessed that on a very old card, 250GTS(9800GTX+) - HDMI audio just broke, hardware GPU scaling got broken and there was a performance hit in GPU benchmarks.
If it's 9XX series card - it won't start with drivers-older-than-the-card because.. it's obvious why.

I'd suggest you googling driver benchmarks of some sort. As measured deeply back in time(there was a post from AW with screenshots from borderlands, somewhere last year maybe), performance impact of disabled hyper-v enlightenment and stuff is very little. I'd say, IOMMU and memory management(hugepages and such) issues chop off your performance more.

For example, my AMD system has some weird issues, getting 100%(compared to bare metal setup) GPU benchmarks(furmark, unigine heaven 4.0) BUT lagging in old games like NFS:Prostreet or rFactor. And i'm not using hugepages or cpu pinning or any other cool stuff this thread is suggesting.

P.S. word-parasite of the day: stuff.

---

The forum rules prohibit requesting support for distributions other than arch.
I gave up. It was too late.
What I was trying to do.
The reference about VFIO and KVM VGA passthrough.

Len

Member

Registered: 2015-01-06

Posts: 23

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

People with nvidia cards passthrough. Are you using it with the 340.52 driver and hv-time or using it with the latest driver with no hv-time?
Im unsure if I get better performance with the former or the latter. Anyone with some experience on this?

I did some performance tests on my GTX970, latest drivers with kvm=off.
Check it here https://bbs.archlinux.org/viewtopic.php … 4#p1505174

Tyrewt

Member

Registered: 2014-09-13

Posts: 14

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Trying to do some USB passthough in my config with:

usb -usbdevice host:056a:00b9 -usbdevice host:2672:000d -usbdevice host:046d:c21d

The camera (host:2672:000d) is seen by both the Host (Linux) and Guest (Windows7) when plugged in. Also, Windows fails to load a "MTP Device Driver". I believe it has something to do with USB passthough, usb-mtp.  Can't find much information on a fix. Please advise?

instantepiphany

Member

Registered: 2013-09-04

Posts: 32

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

My problem with no graphics output was solved by passing my graphics card bios file to qemu.

Performance is almost what I get on my dualboot windows 8 setup - with the exception of guild wars 2. I get equal performance - except for when I turn the camera, which drops the FPS down horribly to about 2, until I stop turning the camera. My vcpus aren't maxed out when i turn the camera, and neither is my virt-ram. What could be causing this? I have tried enabling huge pages, it hasn't helped.

Bronek

Member

From: London

Registered: 2014-02-14

Posts: 123

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Trying to do some USB passthough in my config with:

usb -usbdevice host:056a:00b9 -usbdevice host:2672:000d -usbdevice host:046d:c21d

The camera (host:2672:000d) is seen by both the Host (Linux) and Guest (Windows7) when plugged in. Also, Windows fails to load a "MTP Device Driver". I believe it has something to do with USB passthough, usb-mtp.  Can't find much information on a fix. Please advise?

I found that the best way for USB passthrough is to pass whole USB controller. You can do it also for USB controllers integrated into the south-bridge since they are attached to PCIe root like normal extension cards. This is useful for all types of USB devices because it basically removes USB root complex from host OS and also removes latency of USB serialisation through qemu. Also perfect for attaching things like USB-DAC for great quality sound.

thearcherblog

Member

Registered: 2014-10-30

Posts: 165

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Trying to do some USB passthough in my config with:

usb -usbdevice host:056a:00b9 -usbdevice host:2672:000d -usbdevice host:046d:c21d

The camera (host:2672:000d) is seen by both the Host (Linux) and Guest (Windows7) when plugged in. Also, Windows fails to load a "MTP Device Driver". I believe it has something to do with USB passthough, usb-mtp.  Can't find much information on a fix. Please advise?

I found that the best way for USB passthrough is to pass whole USB controller. You can do it also for USB controllers integrated into the south-bridge since they are attached to PCIe root like normal extension cards. This is useful for all types of USB devices because it basically removes USB root complex from host OS and also removes latency of USB serialisation through qemu. Also perfect for attaching things like USB-DAC for great quality sound.

Mmmmm thanks a lot! I will follow your advice because sometimes when I passthrough my USB for keyboard and mouse... if I shutdown the virtual computer and turn it on again... it give me a segmentation fault. I hope can be solved with that.

Thanks a for that

Duelist

Member

Registered: 2014-09-22

Posts: 358

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

My problem with no graphics output was solved by passing my graphics card bios file to qemu.

Performance is almost what I get on my dualboot windows 8 setup - with the exception of guild wars 2. I get equal performance - except for when I turn the camera, which drops the FPS down horribly to about 2, until I stop turning the camera. My vcpus aren't maxed out when i turn the camera, and neither is my virt-ram. What could be causing this? I have tried enabling huge pages, it hasn't helped.

Check your disk subsystem, maybe GW2 tries to load tons of small files when you turn around..

---

The forum rules prohibit requesting support for distributions other than arch.
I gave up. It was too late.
What I was trying to do.
The reference about VFIO and KVM VGA passthrough.

instantepiphany

Member

Registered: 2013-09-04

Posts: 32

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

My problem with no graphics output was solved by passing my graphics card bios file to qemu.

Performance is almost what I get on my dualboot windows 8 setup - with the exception of guild wars 2. I get equal performance - except for when I turn the camera, which drops the FPS down horribly to about 2, until I stop turning the camera. My vcpus aren't maxed out when i turn the camera, and neither is my virt-ram. What could be causing this? I have tried enabling huge pages, it hasn't helped.

Check your disk subsystem, maybe GW2 tries to load tons of small files when you turn around..

What should I check? The loading small files possibility is what I thought may be the case, and ended up in figuring out I should be using virtio driver rather than IDE for the disks I am passing through. But the problem still persists. I launch qemu with:

qemu-system-x86_64 -enable-kvm -m 4096 -smp 6,sockets=1,cores=6,threads=1 \
-cpu host \
-machine type=pc,accel=kvm \
-mem-path /libhugetlbfs \
-boot menu=on \
-device virtio-scsi-pci,id=scsi \
-drive file=/home/instantepiphany/qemu/windows7/windows7.img,if=virtio \
-device vfio-pci,host=02:00.0 \
-vga none \
-device vfio-pci,host=01:00.0,multifunction=on,x-vga=on,rombar=0,romfile=/home/instantepiphany/qemu/6950.rom \
-device vfio-pci,host=01:00.1 \
-debugcon file:debug.log -global isa-debugcon.iobase=0x402 \
-usb -usbdevice host:046d:c22a -usbdevice host:05e3:0607 \
-drive file=/dev/sdc,cache=none,if=virtio \
-drive file=/dev/sda,cache=none,if=virtio \

bubbacub

Member

Registered: 2015-02-26

Posts: 3

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Hi all,
Just hoping for a bit of advice.
I've got passthrough of a r9 270x on a nehalem based quadcore xeon and 3420gp server board with the following startup script on ubuntu 14.04:

#!/bin/bash

configfile=/etc/vfio-pci1.cfg

vfiobind() {
    dev="$1"
        vendor=$(cat /sys/bus/pci/devices/$dev/vendor)
        device=$(cat /sys/bus/pci/devices/$dev/device)
        if [ -e /sys/bus/pci/devices/$dev/driver ]; then
                echo $dev > /sys/bus/pci/devices/$dev/driver/unbind
        fi
        echo $vendor $device > /sys/bus/pci/drivers/vfio-pci/new_id
   
}

modprobe vfio-pci

cat $configfile | while read line;do
    echo $line | grep ^# >/dev/null 2>&1 && continue
        vfiobind $line
done

sudo qemu-system-x86_64 -enable-kvm -M q35 -m 4096 -cpu host \
-smp 4,sockets=1,cores=4,threads=1 \
-bios /usr/share/qemu/bios.bin -vga none \
-device ioh3420,bus=pcie.0,addr=1c.0,multifunction=on,port=1,chassis=1,id=root.1 \
-device vfio-pci,host=01:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on \
-device vfio-pci,host=01:00.1,bus=root.1,addr=00.1 \
-drive file=/dev/sdb,id=disk,format=raw -device ide-hd,bus=ide.0,drive=disk \
-drive file=~/virtio-win-0.1-94.iso,id=isocd -device ide-cd,bus=ide.1,drive=isocd \
-boot menu=on \
-usb -usbdevice host:045e:0745 \
-vnc :0
exit 0

Works great as a script and I have a nice host server with a multi TB zfs arrays running two guests, one is a headless linux install running plex server, transmission and filebot for TV and movies and the second is windows 8 instance for virtualised gaming using steam in home streaming over my home network.

My issue that I've been banging my head against a wall is getting the above script to run via libvirt so that I could use all the awesome VM management tools it comes with.

I've tried using domxml-from-native to create an xml file that sucessfully defines through virsh but doesn't let me start a vm.

my xml file based on the above qemu arguments is:

<domain type='kvm' xmlns:qemu='http://libvirt.org/schemas/domain/qemu/1.0'>
<name>win8qemu</name>
  <uuid>1009128f-983a-45e5-9d88-9295fcf0a692</uuid>
  <memory unit='KiB'>4194304</memory>
  <currentMemory unit='KiB'>4194304</currentMemory>
  <vcpu placement='static'>1</vcpu>
  <os>
    <type arch='i686' machine='q35'>hvm</type>
  </os>
  <features>
    <acpi/>
  </features>
  <cpu mode='custom' match='exact'>
    <model fallback='allow'>host</model>
  </cpu>
  <clock offset='utc'/>
  <on_poweroff>destroy</on_poweroff>
  <on_reboot>restart</on_reboot>
  <on_crash>destroy</on_crash>
  <devices>
    <emulator>/usr/bin/qemu-system-x86_64</emulator>
    <controller type='sata' index='0'/>
    <controller type='pci' index='0' model='pcie-root'/>
    <controller type='pci' index='1' model='dmi-to-pci-bridge'/>
    <controller type='pci' index='2' model='pci-bridge'/>
    <input type='mouse' bus='ps2'/>
    <input type='keyboard' bus='ps2'/>
    <graphics type='sdl'/>
    <video>
      <model type='cirrus' vram='9216' heads='1'/>
    </video>
    <hostdev mode='subsystem' type='usb' managed='no'>
      <source>
        <vendor id='0x045e'/>
        <product id='0x0745'/>
      </source>
    </hostdev>
    <memballoon model='virtio'/>
  </devices>
  <qemu:commandline>
    <qemu:arg value='qemu-system-x86_64'/>
    <qemu:arg value='\
-smp'/>
    <qemu:arg value='4,sockets=1,cores=4,threads=1'/>
    <qemu:arg value='\
-bios'/>
    <qemu:arg value='/usr/share/qemu/bios.bin'/>
    <qemu:arg value='\
-device'/>
    <qemu:arg value='ioh3420,bus=pcie.0,addr=1c.0,multifunction=on,port=1,chassis=1,id=root.1'/>
    <qemu:arg value='\
-device'/>
    <qemu:arg value='vfio-pci,host=01:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on'/>
    <qemu:arg value='\
-device'/>
    <qemu:arg value='vfio-pci,host=01:00.1,bus=root.1,addr=00.1'/>
    <qemu:arg value='\
-drive'/>
    <qemu:arg value='file=/dev/sdb,id=disk,format=raw'/>
    <qemu:arg value='-device'/>
    <qemu:arg value='ide-hd,bus=ide.0,drive=disk'/>
    <qemu:arg value='\
-drive'/>
    <qemu:arg value='file=~/virtio-win-0.1-94.iso,id=isocd'/>
    <qemu:arg value='-device'/>
    <qemu:arg value='ide-cd,bus=ide.1,drive=isocd'/>
    <qemu:arg value='\
-boot'/>
    <qemu:arg value='menu=on'/>
    <qemu:arg value='\
-usb'/>
    <qemu:arg value='\
-vnc'/>
    <qemu:arg value=':0'/>
  </qemu:commandline>
</domain>

I think this VM doesnt work because the bits of my vm script that sort out the pci passthrough:

#!/bin/bash

configfile=/etc/vfio-pci1.cfg

vfiobind() {
    dev="$1"
        vendor=$(cat /sys/bus/pci/devices/$dev/vendor)
        device=$(cat /sys/bus/pci/devices/$dev/device)
        if [ -e /sys/bus/pci/devices/$dev/driver ]; then
                echo $dev > /sys/bus/pci/devices/$dev/driver/unbind
        fi
        echo $vendor $device > /sys/bus/pci/drivers/vfio-pci/new_id
   
}

modprobe vfio-pci

cat $configfile | while read line;do
    echo $line | grep ^# >/dev/null 2>&1 && continue
        vfiobind $line
done

arn't been executed by libvirt. Any ideas for how I might get the above snippet of code running as a service continuously?

p.s. If I've broken the local ethos of the forum in any way by posting here, please accept my apologies, correct me and (hopefully help me!)

JohnyPea

Member

Registered: 2014-07-17

Posts: 17

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

...
arn't been executed by libvirt. Any ideas for how I might get the above snippet of code running as a service continuously?

p.s. If I've broken the local ethos of the forum in any way by posting here, please accept my apologies, correct me and (hopefully help me!)

That domxml-from-native went horrible. I tried it to see why, because my previous attempts were better. I had to edit the result a lot in the end, but I think this is a good approximation to your original machine (I don't exactly know the logic that qemu uses for address assignment so those can differ and cause problems for your guest). Seeing you are using fedoras virtio image, I set the disk type to virtio:

<domain type='kvm' xmlns:qemu='http://libvirt.org/schemas/domain/qemu/1.0'>
  <name>win8qemu</name>
  <uuid>6e68b59c-0977-49d9-82b7-325662a8ee13</uuid>
  <memory unit='KiB'>4194304</memory>
  <currentMemory unit='KiB'>4194304</currentMemory>
  <vcpu placement='static'>4</vcpu>
  <os>
    <type arch='x86_64' machine='pc-q35'>hvm</type>
    <loader type='rom'>/usr/share/qemu/bios.bin</loader>
    <boot dev='hd'/>
    <bootmenu enable='yes'/>
  </os>
  <features>
    <acpi/>
  </features>
  <cpu mode='host-passthrough'>
    <topology sockets='1' cores='4' threads='1'/>
  </cpu>
  <clock offset='utc'/>
  <on_poweroff>destroy</on_poweroff>
  <on_reboot>restart</on_reboot>
  <on_crash>destroy</on_crash>
  <devices>
    <emulator>/usr/bin/qemu-system-x86_64</emulator>
    <disk type='block' device='disk'>
      <driver name='qemu' type='qcow2' cache='none'/>
      <source file='/dev/sdb'/>
      <target dev='vda' bus='virtio'/>
      <address type='pci' domain='0x0000' bus='0x02' slot='0x02' function='0x0'/>
    </disk>
    <disk type='file' device='cdrom'>
      <driver name='qemu' type='raw'/>
      <source file='/home/dzon/Images/virtio-win-0.1-94.iso'/>
      <target dev='hda' bus='ide'/>
      <readonly/>
      <address type='drive' controller='0' bus='0' target='0' unit='0'/>
    </disk>
    <controller type='usb' index='0'>
      <address type='pci' domain='0x0000' bus='0x02' slot='0x01' function='0x0'/>
    </controller>
    <controller type='sata' index='0'>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x1f' function='0x2'/>
    </controller>
    <controller type='pci' index='0' model='pcie-root'/>
    <controller type='pci' index='1' model='dmi-to-pci-bridge'>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x1e' function='0x0'/>
    </controller>
    <controller type='pci' index='2' model='pci-bridge'>
      <address type='pci' domain='0x0000' bus='0x01' slot='0x01' function='0x0'/>
    </controller>
    <controller type='ide' index='0'/>
    <hostdev mode='subsystem' type='usb' managed='no'>
      <source>
        <vendor id='0x045e'/>
        <product id='0x0745'/>
      </source>
    </hostdev>
    <memballoon model='none'/>
  </devices>
  <qemu:commandline>
    <qemu:arg value='-device'/>
    <qemu:arg value='ioh3420,bus=pcie.0,addr=1c.0,multifunction=on,port=1,chassis=1,id=root.1'/>
    <qemu:arg value='-device'/>
    <qemu:arg value='vfio-pci,host=01:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on'/>
    <qemu:arg value='-device'/>
    <qemu:arg value='vfio-pci,host=01:00.1,bus=root.1,addr=00.1'/>
  </qemu:commandline>
</domain>

Regarding the vfio binding, you have to do it only once (if you dont use the device for anything else than passthrough, which I suppouse you dont). So it can be run in startup script (the simplest way for ubuntu: /etc/rc.local), or if compiled in kernel, you can do the binding through kernel command line directly to vfio driver instead of pci-stub. Didnt test this last one, yet.

There is also the possibility to run it through /etc/libvirt/hooks/qemu, but there you need to make tests for what event is occuring (start/stop, which machine). I'm using it for setting host tasks affinity to non-guest cores and reservation/release of hugepages.

Also I would suggest to use pc-i440fx machine type to avoid "qemu:arg" mayhem as much as possibe. Then libvirt does a pretty good job with management. But you would probably need to reinstall the guest.

I am thinking of getting 290x, out of curiosity and also because the effort needed to circumvent nVidias "helpfullness". So please let us know how it works out.