Arch Linux

tritron4

Member

Registered: 2012-04-14

Posts: 153

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

I think you have few options. You could use intel gpu for host and bind both cards and use them with guest. You can also use xen i guess xen 4.5 package will be out in aur soon. xen does not care about acs as much as kvm does.

aw

Member

Registered: 2013-10-04

Posts: 921

Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

I've been struggling with VGA passthrough for a few days and finally got something that works but unfortunately not in an optimal way. My plan was to use a radeon juniper card in guest and a radeon cedar card on host. The problem is that the hardware manufacturer for my cpu decided in their wisdom to not support ACS on the pcie root port.
Intel Corporation Xeon E3-1200 v3/4th Gen Core Processor PCI Express x16 Controller [8086:0c01]
Both graphic card therefore show up in the same iommu group. What options do I have for working around that problem?

Option 1: Use the ACS override patch and hope for the best.
Not a good idea in my opinion. I'm pretty sure the root port do per-to-per without any way of disabling that. Using the override patch is asking for problems.

Options 2: Rearrange the card to break apart the iommu group.
Unfortunately all 16X lanes go to the cpu through that root port. The 1X ports go through the chipset but I don't have any 1X graphics card. I could buy one but the selection is VERY limited. The only one I found only supports 2 monitors. I could do surgery on the 1X mobo slot or my 16X cedar graphics card to force it in but that not something I want to do.
http://i.imgur.com/NxzaK6L.png block diagram of my mobo. (Let's see if I'm allowed to post a link in my first post to this forum)

Option 3: Buy a new mobo
please no...

Option 4: Use the intel IGP for host graphics and non-legacy guest boot for working around the problems in IGP
My juniper card does not support EFI boot. I've seen some people do Macgyver style mods involving copy paste in hexeditors with other card's vbios. That is something I would like to avoid if possible.

Option 5: Use the intel IGP with the vga arbiter patch
That is what I'm using right now for testing but it's not a longterm solution. It's too inconvenient to apply a patch at every kernel update.

Strangely DRI is not disabled for me even with the vga arbiter patch. I assume I screwed up somewhere but I don't know where. The old patch submitted to the LKML doesn't apply on new kernels anymore. There is another patch named i915_317.patch that do apply to 3.19.1 with some fuss. I boot with the i915.enable_hd_vgaarb=1 argument and verify that argument is on with dmesg but I still get DRI.

You seem to have a good grasp of the options.  Personally I'd opt for using IGD for the host and replacing your guest card with one that supports UEFI.  You seem to be averse to the VGA BIOS hacking route, but that's actually one that's likely to continue working once you figure it out, with no future i915 kernel patching.  Remember you don't need to flash the card with the new ROM, you can use an external file as the ROM for the VM.  If you're willing to settle for a x1 graphics card, you can find x1-to-x16 adapters that will allow you to plug any low-profile x16 card into the x1 slot.  Double check your Xorg.0.log for DRI, DRI3 may still be enabled, but DRI2 has arbiter issues... don't ask me what the difference is, I don't know.

---

http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

VOT Productions

Member

Registered: 2011-10-22

Posts: 47

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Hmm, I'm still stuck with the Starting Windows screen when loading the installer, is my QEMU command right? (the paths are correct)

env QEMU_PA_SAMPLES=128 env QEMU_AUDIO_DRV=pa sudo qemu-system-x86_64 \
-enable-kvm -m 4096 -cpu host,kvm=off -smp cores=4,threads=1 \
-pflash /usr/share/ovmf/x64/ovmf_x64.bin \
-vga none \
-soundhw hda \
-device vfio-pci,host=01:00.0,multifunction=on \
-device vfio-pci,host=01:00.1 \
-device virtio-scsi-pci,id=scsi \
-drive file=/mnt/1TB_LinuxMedia/windows.img,id=disk,format=raw,if=none -device scsi-hd,drive=disk \
-drive file=/mnt/1TB_LinuxMedia/windows.iso,id=isocd,if=none -device scsi-cd,drive=isocd \
-drive file=/mnt/1TB_LinuxMedia/virtio.iso,id=virtiocd,if=none -device ide-cd,bus=ide.1,drive=virtiocd \
-usb -usbdevice host:04d9:a055 -usbdevice host:04d9:a04a \
-boot menu=on

Last edited by VOT Productions (2015-03-17 20:49:47)

tritron4

Member

Registered: 2012-04-14

Posts: 153

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

I found out that my quests run better under libvirt. I had my system freeze when left unattended. No I can leave it unattended  for days without single lockup.
I think options help 
<hyperv>
      <relaxed state='on'/>
      <vapic state='on'/>
      <spinlocks state='on' retries='8191'/>
    </hyperv>
  </features>
  <cpu mode='custom' match='exact'>
    <model fallback='allow'>Opteron_G5</model>
  </cpu>
  <clock offset='localtime'>
    <timer name='rtc' tickpolicy='catchup'/>
    <timer name='pit' tickpolicy='delay'/>
    <timer name='hpet' present='no'/>
    <timer name='hypervclock' present='yes'/>
  </clock>

tritron4

Member

Registered: 2012-04-14

Posts: 153

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

How do I ssetup secure boot for windows 8.1 ?

Draizer

Member

Registered: 2015-03-17

Posts: 8

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Hello everyone.

I followed the first part of this (excellent) guide thoroughly up to the part where I have to test output in bios, unfortunately it yielded no results. Repo QEMU shows a console, qemu-git only shows a menubar and nothing else. The vga arbiter is activated (before I activated it, I had a blank QEMU and glitchy graphics on the host once I started QEMU, after I activated it ... QEMU was still blank but at least my host didn't have glitches anymore), as is ACS override (tried both with and without). I tried rebuilding the kernel multiple times, with different Linux versions (3.9, 3.7, neither worked, I'm currently back to the kernel made by the OP). Triple checked if VT-d was activated on the motherboard, it is. Bindings seems correct as well. I uninstalled the radeon drivers, but still kept them blacklisted and bound the devices to stub just to be sure. I tried both with the i440fx and Q35 chipset. I tried to use the manipulation suggested if nothing happened. I browsed this thread and others on the web through and through, and the only mentions of troubles that remotely looked like mine were because the vga arbiter wasn't activated, which isn't my case. I'm basically at my wits ends, which is why I'm finally posting here.

My hardware :
-Intel Core i7 3770
-ASrock Z77 Extreme6
-Host GPU : Intel IGD
-Guest GPU : ATI Radeon 7970

I'm not sure if it is related, but dmesg brings the following error messages:

[drm:ivybridge_set_fifo_underrun_reporting] *ERROR* uncleared fifo underrun on pipe B
[drm:ivb_err_int_handler] *ERROR* Pipe B FIFO underrun
[drm:cpt_set_fifo_underrun_reporting] *ERROR* uncleared pch fifo underrun on pch transcoder A
[drm:cpt_serr_int_handler] *ERROR* PCH transcoder A FIFO underrun
[drm:cpt_set_fifo_underrun_reporting] *ERROR* uncleared pch fifo underrun on pch transcoder B
[drm:cpt_serr_int_handler] *ERROR* PCH transcoder B FIFO underrun

Didn't find anything helpful on this, either.

Evonat

Member

Registered: 2013-07-24

Posts: 12

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

One year and a half after I abandoned my attempt I'm back at it again, and of course it still doesn't fully work!
Or actually, everything seems to work fine once, then if the VM reboots/closes and opens again, the screen stays black.
In other words it looks like that old VFIO VGA reset problem for which there used to be some patches, now gone because it was supposed to be solved(?).

I'm using a linux-3.18 kernel patched with override_for_missing_acs_capabilities.patch (pci-stub.ids=1002:94b3,1002:aa38 pcie_acs_override=downstream) and QEMU 2.2.1.
I get no errors with QEMU or dmesg. The thing simply works the first time and the first time only, then I have to restart the host to get it to work again.

ceri

Member

Registered: 2013-10-12

Posts: 57

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

I just found out there's now a wiki page for this, however it's incomplete and requires more contributors (if you have time):

https://wiki.archlinux.org/index.php/PC … h_via_OVMF

kainet

Member

Registered: 2014-12-22

Posts: 9

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Again experiencing sound stuttering when gpu is pretty loaded MSI GTX 760

Some times waiting for some times helps to get back normal sound but i usually disable and enable audio device in Windows guest. Anybody knows which part of software is responsible for that ?

tritron4

Member

Registered: 2012-04-14

Posts: 153

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

How I can get sound using libvirt. I can hear sound if I use spice client but there is no sound from guest to alsa.

I see option like this:
The VM definition xml contains the following
<qemu:commandline>
<qemu:env name='QEMU_ALSA_DAC_BUFFER_SIZE' value='512'/>
<qemu:env name='QEMU_ALSA_DAC_PERIOD_SIZE' value='170'/>
<qemu:env name='QEMU_AUDIO_DRV' value='alsa'/>
</qemu:commandline>

Last edited by tritron4 (2015-03-18 18:23:57)

Naruni

Member

Registered: 2010-08-05

Posts: 34

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Try my theory - fetching the drivers for GPU before the install. Some OEMs do that, so there's definitely a way to do this. But it seems like windowns 7 can not into UEFI at all.

Thinking about that, the first approach I would try is figuring out if a driver is making that call during the install step, then replacing that driver with something that didn't make that call. On one of my notes I remember CLASSPNP.SYS causing lots of hangs when trying safe modes and such... I'll dig in a bit more.

Nothing came up as far as patching win7's graphics system, maybe next I could try ripping win8's graphics system and force it into win7?

On a side note, I have a working win8 now. It hangs at shutdown, I have to issue 'quit' in the qemu terminal, but windows doesn't complain about getting a bad shutdown.  I've tried disabling the power service in windows but that didn't do anything. Anyone else have experience with that?

Also, it was recommended I post this here: https://bbs.archlinux.org/viewtopic.php?id=194060

Duelist

Member

Registered: 2014-09-22

Posts: 358

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

How do I ssetup secure boot for windows 8.1 ?

http://www.linux-kvm.org/downloads/lers … 770f8c.txt

(6) Additionally, the "edk2.git-ovmf-x64" package seeks to simplify the
    enablement of Secure Boot in a virtual machine (strictly for development
    and testing purposes)

    - Boot the virtual machine off the CD-ROM image called
      "/usr/share/edk2.git/ovmf-x64/UefiShell.iso"; before or after installing
      the main guest operating system.

    - When the UEFI shell appears, issue the following commands:

      EnrollDefaultKeys.efi
      reset -s

Last edited by Duelist (2015-03-18 19:31:57)

---

The forum rules prohibit requesting support for distributions other than arch.
I gave up. It was too late.
What I was trying to do.
The reference about VFIO and KVM VGA passthrough.

Duelist

Member

Registered: 2014-09-22

Posts: 358

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Try my theory - fetching the drivers for GPU before the install. Some OEMs do that, so there's definitely a way to do this. But it seems like windowns 7 can not into UEFI at all.

Thinking about that, the first approach I would try is figuring out if a driver is making that call during the install step, then replacing that driver with something that didn't make that call. On one of my notes I remember CLASSPNP.SYS causing lots of hangs when trying safe modes and such... I'll dig in a bit more.

Nothing came up as far as patching win7's graphics system, maybe next I could try ripping win8's graphics system and force it into win7?

Also, it was recommended I post this here: https://bbs.archlinux.org/viewtopic.php?id=194060

Seems like you've missed some recent news about windows7:
https://bbs.archlinux.org/viewtopic.php … 0#p1511290
lersek has joined us and said that there's a FIX for win7 VM.
There's a whitepaper published here:
http://www.linux-kvm.org/downloads/lers … 770f8c.txt

And he also edited the wiki page of yours.

BTW: originally i've meant some way of tearing and ripping of win8's video drivers and shove them up win7. Yes, that sounds awful. That's the idea.

Last edited by Duelist (2015-03-18 22:39:54)

---

The forum rules prohibit requesting support for distributions other than arch.
I gave up. It was too late.
What I was trying to do.
The reference about VFIO and KVM VGA passthrough.

Denso

Member

Registered: 2014-08-30

Posts: 179

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Any possible way to "inject" UEFI support inside non-UEFI ROMs ?

I have a GT630 , which has a non-UEFI ROM , yet I found some ROMs from other vendors that have UEFI ROMs (techpowerup) which don't work directly on my GPU .

It is possible to insert the UEFI partition for some Radeons using a hex editor . How about NVIDIA ?

Duelist

Member

Registered: 2014-09-22

Posts: 358

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

How about NVIDIA ?

Weird. I have gigabyte GT610 which supports UEFI perfectly..
Anyway, try searching for NVidia's GOP technical presentations and etc - i've found one for AMD dating back to 2010, i think, saying "there will be an universal GOP for all cards". That presentation just proved the idea of copying GOP from one's GPU to another.

Also, try writing a support ticket to your hardware's vendor - in my ASUS case it just cleared up some things. But actually, they have an official update tool(which requires windowns installed..).

Also, maybe you just haven't got enough luck, since some nvidia cards may be converted to quadro just by firmware change.

You can also dump all UEFI-compatible ROMs from DB and try ripping uncompressed EFI driver same way we did it for AMD, it may suddenly be universal for all vendors and containing only GOP without any important bits. Supplying it as romfile= option will be a safe way of testing it out. The resulting file from firmware should be .efi image, you glue it with your normal vbios same way as AMD, using EfiRom(a tool from EDK2 btw). I don't recall properly, but i think EfiRom utility could decompress that efi image from ROM..

P.S. it wasn't a hex editor, it's an Elegant Weapon for a More Civilized Age of UEFI.

Last edited by Duelist (2015-03-19 10:02:48)

---

The forum rules prohibit requesting support for distributions other than arch.
I gave up. It was too late.
What I was trying to do.
The reference about VFIO and KVM VGA passthrough.

tholin

Member

Registered: 2015-03-17

Posts: 7

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

You seem to have a good grasp of the options.  Personally I'd opt for using IGD for the host and replacing your guest card with one that supports UEFI.  You seem to be averse to the VGA BIOS hacking route, but that's actually one that's likely to continue working once you figure it out, with no future i915 kernel patching.  Remember you don't need to flash the card with the new ROM, you can use an external file as the ROM for the VM.  If you're willing to settle for a x1 graphics card, you can find x1-to-x16 adapters that will allow you to plug any low-profile x16 card into the x1 slot.  Double check your Xorg.0.log for DRI, DRI3 may still be enabled, but DRI2 has arbiter issues... don't ask me what the difference is, I don't know.

I've been using the IGD for a few days and realized the drivers are too buggy for daily use. Maybe it's related to the i915 patch but many other people have the same problems. To be fair the open radeon drivers also got a lot of problems but I'm used to them and know how to work around the problems. I ordered that 1X radeon card instead.

It looks like dri2 is enabled. Here are some parts of Xorg.log

[   188.603] (II) xfree86: Adding drm device (/dev/dri/card0)
[   188.604] (--) PCI:*(0:0:2:0) 8086:0412:1458:d000 rev 6, Mem @ 0xf7400000/4194304, 0xc0000000/268435456, I/O @ 0x0000f000/64
[   188.604] (--) PCI: (0:3:0:0) 1002:68f9:174b:e153 rev 0, Mem @ 0xe0000000/268435456, 0xf7920000/131072, I/O @ 0x0000a000/256, BIOS @ 0x????????/131072
[   188.604] (--) PCI: (0:4:0:0) 1002:68ba:1458:220c rev 0, Mem @ 0xd0000000/268435456, 0xf7820000/131072, I/O @ 0x00009000/256, BIOS @ 0x????????/131072
...
[   188.618] (II) LoadModule: "intel"
[   188.618] (II) Loading /usr/lib64/xorg/modules/drivers/intel_drv.so
[   188.620] (II) Module intel: vendor="X.Org Foundation"
[   188.620]    compiled for 1.16.4, module version = 2.99.917
[   188.620]    Module class: X.Org Video Driver
[   188.620]    ABI class: X.Org Video Driver, version 18.0
[   188.620] (II) intel: Driver for Intel(R) Integrated Graphics Chipsets:
        i810, i810-dc100, i810e, i815, i830M, 845G, 854, 852GM/855GM, 865G,
        915G, E7221 (i915), 915GM, 945G, 945GM, 945GME, Pineview GM,
        Pineview G, 965G, G35, 965Q, 946GZ, 965GM, 965GME/GLE, G33, Q35, Q33,
        GM45, 4 Series, G45/G43, Q45/Q43, G41, B43
[   188.620] (II) intel: Driver for Intel(R) HD Graphics: 2000-6000
[   188.620] (II) intel: Driver for Intel(R) Iris(TM) Graphics: 5100, 6100
[   188.620] (II) intel: Driver for Intel(R) Iris(TM) Pro Graphics: 5200, 6200, P6300
[   188.620] (++) using VT number 7
[   188.626] (II) intel(0): Using Kernel Mode Setting driver: i915, version 1.6.0 20141121
[   188.627] (--) intel(0): Integrated Graphics Chipset: Intel(R) HD Graphics 4600
[   188.627] (--) intel(0): CPU: x86-64, sse2, sse3, ssse3, sse4.1, sse4.2, avx, avx2
[   188.627] (II) intel(0): Creating default Display subsection in Screen section
        "Default Screen Section" for depth/fbbpp 24/32
[   188.628] (==) intel(0): Depth 24, (--) framebuffer bpp 32
[   188.628] (==) intel(0): RGB weight 888
[   188.628] (==) intel(0): Default visual is TrueColor
[   188.628] (**) intel(0): Option "TearFree" "true"
[   188.628] (II) intel(0): Output VGA1 has no monitor section
[   188.628] (II) intel(0): Enabled output VGA1
[   188.628] (II) intel(0): Output HDMI1 has no monitor section
[   188.628] (II) intel(0): Enabled output HDMI1
[   188.628] (II) intel(0): Output HDMI2 has no monitor section
[   188.628] (II) intel(0): Enabled output HDMI2
[   188.628] (II) intel(0): Output DP1 has no monitor section
[   188.628] (II) intel(0): Enabled output DP1
[   188.628] (II) intel(0): Output HDMI3 has no monitor section
[   188.628] (II) intel(0): Enabled output HDMI3
[   188.628] (--) intel(0): Using a maximum size of 256x256 for hardware cursors
[   188.628] (II) intel(0): Output VIRTUAL1 has no monitor section
[   188.628] (II) intel(0): Enabled output VIRTUAL1
[   188.628] (--) intel(0): Output HDMI2 using initial mode 1920x1200 on pipe 0
[   188.628] (--) intel(0): Output DP1 using initial mode 1920x1200 on pipe 1
[   188.628] (**) intel(0): TearFree enabled
[   188.628] (==) intel(0): DPI set to (96, 96)
[   188.628] (II) Loading sub module "dri2"
[   188.628] (II) LoadModule: "dri2"
[   188.628] (II) Module "dri2" already built-in
[   188.628] (II) Loading sub module "present"
[   188.628] (II) LoadModule: "present"
[   188.628] (II) Module "present" already built-in
[   188.628] (==) Depth 24 pixmap format is 32 bpp
[   188.630] (II) intel(0): SNA initialized with Haswell (gen7.5, gt2) backend
[   188.630] (==) intel(0): Backing store enabled
[   188.630] (==) intel(0): Silken mouse enabled
[   188.631] (II) intel(0): HW Cursor enabled
[   188.631] (II) intel(0): RandR 1.2 enabled, ignore the following RandR disabled message.
[   188.631] (==) intel(0): DPMS enabled
[   188.631] (==) intel(0): display hotplug detection enabled
[   188.631] (II) intel(0): [DRI2] Setup complete
[   188.631] (II) intel(0): [DRI2]   DRI driver: i965
[   188.631] (II) intel(0): [DRI2]   VDPAU driver: i965
[   188.631] (II) intel(0): direct rendering: DRI2 enabled
[   188.631] (II) intel(0): hardware support for Present enabled
[   188.631] (--) RandR disabled
[   188.631] (II) Found 3 VGA devices: arbiter wrapping enabled
[   188.651] (II) AIGLX: enabled GLX_MESA_copy_sub_buffer
[   188.651] (II) AIGLX: enabled GLX_ARB_create_context
[   188.651] (II) AIGLX: enabled GLX_ARB_create_context_profile
[   188.651] (II) AIGLX: enabled GLX_EXT_create_context_es2_profile
[   188.651] (II) AIGLX: enabled GLX_INTEL_swap_event
[   188.651] (II) AIGLX: enabled GLX_SGI_swap_control and GLX_MESA_swap_control
[   188.651] (II) AIGLX: enabled GLX_EXT_framebuffer_sRGB
[   188.651] (II) AIGLX: enabled GLX_ARB_fbconfig_float
[   188.651] (II) AIGLX: GLX_EXT_texture_from_pixmap backed by buffer objects
[   188.651] (II) AIGLX: enabled GLX_ARB_create_context_robustness
[   188.651] (II) AIGLX: Loaded and initialized i965
[   188.651] (II) GLX: Initialized DRI2 GL provider for screen 0
[   188.653] (II) intel(0): switch to mode 1920x1200@60.0 on HDMI2 using pipe 0, position (0, 0), rotation normal, reflection none
[   188.667] (II) intel(0): switch to mode 1920x1200@60.0 on DP1 using pipe 1, position (0, 0), rotation normal, reflection none
[   188.677] (II) intel(0): Setting screen physical size to 508 x 317

Like most people her I'm having difficulty with sound. The hda hardware doesn't work at all and the ac79 got some crackling noise.
I've tried to play around with the qemu options but it doesn't make any difference. Even the QEMU_AUDIO_DRV=wav backend crackle. There is no crackle when playing steam trailers or music with audio players. I'm guessing games use a smaller audio buffer and are more susceptible.

I've tried forwarding my onboard mobo audio but it was never even detected. Has anyone got that to work? Last resort would be to buy an usb soundcard and forward that but I don't want to buy hardware to work around bugs in qemu.

Denso

Member

Registered: 2014-08-30

Posts: 179

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

How about NVIDIA ?

Weird. I have gigabyte GT610 which supports UEFI perfectly..
Anyway, try searching for NVidia's GOP technical presentations and etc - i've found one for AMD dating back to 2010, i think, saying "there will be an universal GOP for all cards". That presentation just proved the idea of copying GOP from one's GPU to another.

Also, try writing a support ticket to your hardware's vendor - in my ASUS case it just cleared up some things. But actually, they have an official update tool(which requires windowns installed..).

Also, maybe you just haven't got enough luck, since some nvidia cards may be converted to quadro just by firmware change.

You can also dump all UEFI-compatible ROMs from DB and try ripping uncompressed EFI driver same way we did it for AMD, it may suddenly be universal for all vendors and containing only GOP without any important bits. Supplying it as romfile= option will be a safe way of testing it out. The resulting file from firmware should be .efi image, you glue it with your normal vbios same way as AMD, using EfiRom(a tool from EDK2 btw). I don't recall properly, but i think EfiRom utility could decompress that efi image from ROM..

P.S. it wasn't a hex editor, it's an Elegant Weapon for a More Civilized Age of UEFI.

You are right , I have a GT610 too which has a UEFI ROM . But this new GT630 doesn't .

The card is from a vendor called "Point of View" , I'll try sending them a UEFI ROM request .

I did try ripping the UEFI partition from some ROMs I found in TPU , but that didn't succeed . Even AW's rom-parser didn't pick up the UEFI bits

Duelist

Member

Registered: 2014-09-22

Posts: 358

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Even AW's rom-parser didn't pick up the UEFI bits

Well, EfiRom will tell you something very alike. If AW's parser doesn't recognise any EFI drivers in ROM - then there is no EFI support in this ROM. At all.
Only one detail applies: nvidia cards sometimes have "pre-post rom". Adjust your plans accordingly.

---

The forum rules prohibit requesting support for distributions other than arch.
I gave up. It was too late.
What I was trying to do.
The reference about VFIO and KVM VGA passthrough.

Duelist

Member

Registered: 2014-09-22

Posts: 358

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Duelist strikes a menacing pose.

Laszlo, you are awesome. OVMF is awesome. Everything is awesome.

<domain type='kvm' id='10'>
  <name>vm1</name>
  <uuid>f6958833-174f-44e6-817b-2a6c73c6f041</uuid>
  <memory unit='KiB'>4194304</memory>
  <currentMemory unit='KiB'>4194304</currentMemory>
  <vcpu placement='static'>4</vcpu>
  <resource>
    <partition>/machine</partition>
  </resource>
  <os>
    <type arch='x86_64' machine='pc-i440fx-2.1'>hvm</type>
    <loader readonly='yes' type='pflash'>/usr/share/edk2.git/ovmf-x64/OVMF_CODE-pure-efi.fd</loader>
    <nvram>/var/lib/libvirt/qemu/nvram/vm1_VARS.fd</nvram>
  </os>
  <features>
    <acpi/>
    <apic/>
    <pae/>
  </features>
  <cpu mode='host-model'>
    <model fallback='allow'/>
  </cpu>
  <clock offset='localtime'>
    <timer name='rtc' tickpolicy='catchup'/>
    <timer name='pit' tickpolicy='delay'/>
    <timer name='hpet' present='no'/>
  </clock>
  <on_poweroff>destroy</on_poweroff>
  <on_reboot>restart</on_reboot>
  <on_crash>restart</on_crash>
  <pm>
    <suspend-to-mem enabled='no'/>
    <suspend-to-disk enabled='no'/>
  </pm>
  <devices>
    <emulator>/usr/bin/qemu-kvm</emulator>
    <disk type='file' device='cdrom'>
      <driver name='qemu' type='raw'/>
      <source file='/mnt/hdd/qemu/windows7.iso'/>
      <backingStore/>
      <target dev='hdb' bus='ide'/>
      <readonly/>
      <boot order='2'/>
      <alias name='ide0-0-1'/>
      <address type='drive' controller='0' bus='0' target='0' unit='1'/>
    </disk>
    <disk type='file' device='cdrom'>
      <driver name='qemu' type='raw'/>
      <source file='/mnt/hdd/qemu/virtio.iso'/>
      <backingStore/>
      <target dev='hdc' bus='ide'/>
      <readonly/>
      <alias name='ide0-1-0'/>
      <address type='drive' controller='0' bus='1' target='0' unit='0'/>
    </disk>
    <disk type='file' device='cdrom'>
      <driver name='qemu' type='raw'/>
      <source file='/usr/share/edk2.git/ovmf-x64/UefiShell.iso'/>
      <backingStore/>
      <target dev='hdd' bus='ide'/>
      <readonly/>
      <alias name='ide0-1-1'/>
      <address type='drive' controller='0' bus='1' target='0' unit='1'/>
    </disk>
    <disk type='block' device='disk'>
      <driver name='qemu' type='raw' cache='none' io='native'/>
      <source dev='/dev/sda'/>
      <backingStore/>
      <target dev='vda' bus='virtio'/>
      <boot order='1'/>
      <alias name='virtio-disk0'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x0c' function='0x0'/>
    </disk>
    <controller type='usb' index='0' model='ich9-ehci1'>
      <alias name='usb0'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x05' function='0x7'/>
    </controller>
    <controller type='usb' index='0' model='ich9-uhci1'>
      <alias name='usb0'/>
      <master startport='0'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x05' function='0x0' multifunction='on'/>
    </controller>
    <controller type='usb' index='0' model='ich9-uhci2'>
      <alias name='usb0'/>
      <master startport='2'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x05' function='0x1'/>
    </controller>
    <controller type='usb' index='0' model='ich9-uhci3'>
      <alias name='usb0'/>
      <master startport='4'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x05' function='0x2'/>
    </controller>
    <controller type='pci' index='0' model='pci-root'>
      <alias name='pci.0'/>
    </controller>
    <controller type='ide' index='0'>
      <alias name='ide0'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x1'/>
    </controller>
    <controller type='virtio-serial' index='0'>
      <alias name='virtio-serial0'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x06' function='0x0'/>
    </controller>
    <interface type='network'>
      <mac address='52:54:00:f0:d5:39'/>
      <source network='default'/>
      <target dev='vnet0'/>
      <model type='rtl8139'/>
      <alias name='net0'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x0'/>
    </interface>
    <serial type='pty'>
      <source path='/dev/pts/2'/>
      <target port='0'/>
      <alias name='serial0'/>
    </serial>
    <console type='pty' tty='/dev/pts/2'>
      <source path='/dev/pts/2'/>
      <target type='serial' port='0'/>
      <alias name='serial0'/>
    </console>
    <channel type='spicevmc'>
      <target type='virtio' name='com.redhat.spice.0'/>
      <alias name='channel0'/>
      <address type='virtio-serial' controller='0' bus='0' port='1'/>
    </channel>
    <input type='mouse' bus='usb'>
      <alias name='input0'/>
    </input>
    <input type='mouse' bus='ps2'/>
    <input type='keyboard' bus='ps2'/>
    <graphics type='vnc' port='5900' autoport='yes' listen='127.0.0.1'>
      <listen type='address' address='127.0.0.1'/>
    </graphics>
    <sound model='ich6'>
      <alias name='sound0'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x04' function='0x0'/>
    </sound>
    <video>
      <model type='qxl' ram='65536' vram='65536' heads='1'/>
      <alias name='video0'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x0'/>
    </video>
    <hostdev mode='subsystem' type='pci' managed='yes'>
      <driver name='vfio'/>
      <source>
        <address domain='0x0000' bus='0x01' slot='0x00' function='0x0'/>
      </source>
      <alias name='hostdev0'/>
      <rom bar='on' file='/mnt/hdd/qemu/hybridmagic.rom'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x08' function='0x0'/>
    </hostdev>
    <hostdev mode='subsystem' type='pci' managed='yes'>
      <driver name='vfio'/>
      <source>
        <address domain='0x0000' bus='0x01' slot='0x00' function='0x1'/>
      </source>
      <alias name='hostdev1'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x09' function='0x0'/>
    </hostdev>
    <hostdev mode='subsystem' type='pci' managed='yes'>
      <driver name='vfio'/>
      <source>
        <address domain='0x0000' bus='0x02' slot='0x00' function='0x0'/>
      </source>
      <alias name='hostdev2'/>
      <rom bar='on' file='/mnt/hdd/qemu/hybridmagic.rom'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x0a' function='0x0'/>
    </hostdev>
    <hostdev mode='subsystem' type='pci' managed='yes'>
      <driver name='vfio'/>
      <source>
        <address domain='0x0000' bus='0x02' slot='0x00' function='0x1'/>
      </source>
      <alias name='hostdev3'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x0b' function='0x0'/>
    </hostdev>
    <redirdev bus='usb' type='spicevmc'>
      <alias name='redir0'/>
    </redirdev>
    <redirdev bus='usb' type='spicevmc'>
      <alias name='redir1'/>
    </redirdev>
    <memballoon model='virtio'>
      <alias name='balloon0'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x07' function='0x0'/>
    </memballoon>
  </devices>
  <seclabel type='dynamic' model='selinux' relabel='yes'>
    <label>system_u:system_r:svirt_t:s0:c624,c898</label>
    <imagelabel>system_u:object_r:svirt_image_t:s0:c624,c898</imagelabel>
  </seclabel>
</domain>

So, how did it go:
1. QXL graphics came online, i did all the installation tasks via QXL.
2. I tried to install AMD drivers for win8.1 on win7 and it didn't work, showing code 10 on both GPUs.
3. But then.. DL'ed win7 drivers and after a bootup logo(which is dublicated on the physical screen alike tianocore logo and OVMF menus) QXL screen froze...
4. In the beginning there was silence and darkness all across the world; Then came the Cursor and a loading screen...
5. Windows popped up a message about turning QXL off, and QXL became code-10'd, both GPUs code 0.
6. It's safe to get rid of QXL, but Spice won't work, libvirt won't provide a surface for input grabbing purposes.

Seems like finally libvirt+crossfire+windows7+vfio got along with each other. There's some issues left with the setup, namely disk performance(and safety too), cpu pinning and input, but otherwise it's fine. And there's a bunch of unneeded devices and disk attached.

UPD:
Benchmarks show negligible performance hit in furmark, otherwise system works as usual and very similar to bare metal setup. I wonder if i can actually boot this in bare metal, since i'm using the most crazy way of managing disks: whole /dev/sda to the VM.

UPD2:
There's some sound issues, as my sound just falls off VM. It works, VM reboot and there's no sound device anymore. But i'm more worried about strange shutdown behavior:
Windowns 7 starts shutting itself down, kills explorer and stuff, and stops. I have a wallpaper, a mouse cursor which i can move, but nothing else happens. And it also boots very slowly, i guess there's something to do with the disk access.
If i force it off, on next boot it stalls at boot, refusing to execute OVMF at least.
Only rebooting the host helps. There's nothing related to GPUs in dmesg, so i assume it's libvirt trickery that fails to work.
GPU reset problems? Huh? Why?.. It was working fine with that before when i used qemu without libvirt.

Last edited by Duelist (2015-03-20 17:13:44)

---

The forum rules prohibit requesting support for distributions other than arch.
I gave up. It was too late.
What I was trying to do.
The reference about VFIO and KVM VGA passthrough.

The_Moves

Member

Registered: 2015-01-06

Posts: 59

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

How did you get Win7 to even load with more than 1 CPU with EFI boot? What is QXL?

My install would just freeze when the initial Windows Logo would appear after loading all the setup files (Black screen and bar loading on bottom).

Edit: Went back one page and found the post by Lersek: https://bbs.archlinux.org/viewtopic.php … 5#p1511585

Last edited by The_Moves (2015-03-20 14:11:18)

Duelist

Member

Registered: 2014-09-22

Posts: 358

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

How did you get Win7 to even load with more than 1 CPU with EFI boot? What is QXL?

My install would just freeze when the initial Windows Logo would appear after loading all the setup files (Black screen and bar loading on bottom).

Edit: Went back one page and found the post by Lersek: https://bbs.archlinux.org/viewtopic.php … 5#p1511585

Actually...
http://www.linux-kvm.org/downloads/lers … 770f8c.txt
Read all posts by lersek - he(and a bunch of other guys) develops OVMF. I mean, when we had initial troubles with VFIO, we asked alex(aw) for help, because he wrote it. Now we had problems with OVMF, and here he comes and explains it all.

I think, btw, his very first post and OVMF's whitepaper should be pinned into op-post, just for overall awareness.

---

The forum rules prohibit requesting support for distributions other than arch.
I gave up. It was too late.
What I was trying to do.
The reference about VFIO and KVM VGA passthrough.

tritron4

Member

Registered: 2012-04-14

Posts: 153

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Did anyone got sound working well using qemu emulation.
I removed spice display and vga emulated driver and I only use assigned video cards and my sound is send to alsa but it does not sound that great.
I was thinking about hot plugin sound card with virsh is that possible I see there is  attach-device                  attach device from an XML file option

The_Moves

Member

Registered: 2015-01-06

Posts: 59

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

I think, btw, his very first post and OVMF's whitepaper should be pinned into op-post, just for overall awareness.

I agree! Thanks for posting this!

VOT Productions

Member

Registered: 2011-10-22

Posts: 47

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

To everyone having issues with "Starting Windows" screen and it not progressing further:
Do -vga qxl while installing Windows. Then, install your GPU driver. Then put it back to -vga none, worked for me

However, I have a problem. I am trying to get Synergy to work, and this is my full QEMU command.

#!/bin/sh
sudo brctl addbr br0
sudo ip addr add 172.20.0.1/16 dev br0
sudo ip link set br0 up
sudo vfio-bind 0000:01:00.0 0000:01:00.1
cp /usr/share/edk2.git/ovmf-x64/OVMF_VARS-pure-efi.fd /tmp/my_vars.fd
QEMU_ALSA_DAC_BUFFER_SIZE=512 QEMU_ALSA_DAC_PERIOD_SIZE=170 QEMU_AUDIO_DRV=alsa sudo qemu-system-x86_64 \
-enable-kvm -m 4096 -cpu host,kvm=off -smp cores=4,threads=1 \
-drive if=pflash,format=raw,readonly,file=/usr/share/edk2.git/ovmf-x64/OVMF_CODE-pure-efi.fd \
-drive if=pflash,format=raw,file=/tmp/my_vars.fd \
-vga none \
-soundhw hda \
-device vfio-pci,host=01:00.0,multifunction=on \
-device vfio-pci,host=01:00.1 \
-device virtio-scsi-pci,id=scsi \
-drive file=/mnt/1TB_LinuxMedia/virtio.iso,id=virtiocd,if=none -device ide-cd,bus=ide.1,drive=virtiocd \
-drive file=/mnt/1TB_LinuxMedia/windows.img,id=disk,format=raw,if=none -device scsi-hd,drive=disk \
-drive file=/mnt/1TB_LinuxMedia/windows2.img,id=disk2,format=raw,if=none -device scsi-hd,drive=disk2 \
-usb -usbdevice host:04d9:a055 -usbdevice host:04d9:a04a \
-netdev user,id=net0 -device e1000,netdev=net0,mac=52:54:00:12:34:56 \
-netdev bridge,br=br0,id=net1 -device e1000,netdev=net1,mac=52:54:00:12:34:57 \
-usbdevice tablet 
ip link set br0 down

It makes two virtual networks on Windows, one which can access the internet, and one internal. The trouble is, having that internal network on slows down browsing on Windows MASSIVELY, to the point that some websites don't load. If I disable that adapter in Windows, the websites load instantly like they should. If anyone knows how to connect Qemu to the network with a wlan0 interface (so I can't bridge) or knows how to fix the slowdown, please help me

Denso

Member

Registered: 2014-08-30

Posts: 179

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

It makes two virtual networks on Windows, one which can access the internet, and one internal. The trouble is, having that internal network on slows down browsing on Windows MASSIVELY, to the point that some websites don't load. If I disable that adapter in Windows, the websites load instantly like they should. If anyone knows how to connect Qemu to the network with a wlan0 interface (so I can't bridge) or knows how to fix the slowdown, please help me

I use the same scenario as you , my way for setting up the internal network is explained here

No issues at all .

Last edited by Denso (2015-03-20 23:37:40)