Arch Linux

aw

Member

Registered: 2013-10-04

Posts: 921

Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

@morat

Perhaps it's a PATH issue then since you've taken the liberty to not call the qemu binary with the full path.

EDIT: Remove the -x for libvirt usage

Last edited by aw (2015-07-13 15:54:37)

---

http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

morat

Member

Registered: 2015-07-07

Posts: 38

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

@aw
Maybe, but probably not. exec /sbin/qemu-system-x86_64 still fails.
EDIT:
Removing the -x did not change anything.

Last edited by morat (2015-07-13 15:56:29)

aw

Member

Registered: 2013-10-04

Posts: 921

Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

@morat

Maybe the wrapper script should also be in /sbin, mine lives in /usr/libexec because that's where my qemu-kvm binary lives

---

http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

morat

Member

Registered: 2015-07-07

Posts: 38

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

@aw
Nope. Looks like we're running out of options
EDIT:
Diff-ing the two outputs (qemu-kvm.vga & qemu-system-x86_64) shows that they are identical, so it seems irrational that virsh still throws an error.

Last edited by morat (2015-07-13 16:04:27)

aw

Member

Registered: 2013-10-04

Posts: 921

Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

@morat

Change /bin/sh to /bin/bash?

Use full paths for echo and sed?

What distro are you on?  Maybe apparmor or similar is preventing execution.

---

http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

morat

Member

Registered: 2015-07-07

Posts: 38

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Well, it works. Good.
Seems unusual that it should require full paths, but w/e, I'm happy. Onto the next problem, whatever it will be.

EDIT:
I haven't yet built and switched to the patched kernel (which btw, at 4.2-rc1 is missing most of the i915 files targeted by your patch), and that is something that can definitely wait till morning. Good hustle, thanks for the help.

Last edited by morat (2015-07-13 16:09:39)

aw

Member

Registered: 2013-10-04

Posts: 921

Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

@morat

Perhaps you could post your working script and reference the distro to help those that come after you.

I would suggest not using 4.2-rc1, there's a reason rc kernels are rc.  Use a released kernel.

---

http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

morat

Member

Registered: 2015-07-07

Posts: 38

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

qemu-system-x86_64.vga:

#! /bin/bash
exec /sbin/qemu-system-x86_64 \
`/bin/echo "\$@" | \
 /bin/sed 's|01:00.0|01:00.0,x-vga=on|g'`

Working on Arch 4.0.7-2-ARCH

@aw
I'll just rebuild my current kernel then, seems like the safest route.

Duelist

Member

Registered: 2014-09-22

Posts: 358

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Strangely, although I am getting output on the physical card it isn't listed in device manager. Despite this, the AMD Catalyst installer detects the graphics hardware and tries (unsuccessfully) to install it. It gets up to installing the actual driver, the screen flickers once, then the physical display loses signal, the VM goes offline and ramps up to a constant 100% CPU usage with no disk activity. Happens using both the modified copy of my existing VBIOS and the vendor-provided updated one.

My hardware:
Motherboard: Gigabyte 990FXA-UD3
CPU: FX-8350
Host GPU: Gigabyte GTX 750 Ti OC
Guest GPU: Gigabyte R9 290 (initial reference cooler release)

I'll write a guide some day.
What guest OS and what device manager(devmgmt.msc?) you've used? I find it really odd that you can't see the card in there.
Also, which ROM did you bisect and took .efi part from? I remember finding 3 or 4 variants of GOP driver, maybe you could try different ones.

UPD: Catalyst 15.7 installed successfully, without any problems. Crossfire works as before.

Last edited by Duelist (2015-07-13 17:32:41)

---

The forum rules prohibit requesting support for distributions other than arch.
I gave up. It was too late.
What I was trying to do.
The reference about VFIO and KVM VGA passthrough.

impulse_255

Member

Registered: 2015-06-29

Posts: 22

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Another status update:

First of all, I use Witcher 3 as my benchmark. It loads roughly four CPUs and goes between being CPU and GPU limited about 50/50 on my (bare) hardware in the open game areas. I figure it makes a good way to check for CPU performance issues. I've had stuttering and nasty sound dropouts and I thought it happened when the game was doing IO. The game now runs from a passed through SATA controller's drive and the stuttering is still the same. Seems I was too quick to place blame.

I moved my setup on top of libvirt and gained a ghastly amount of qemu command line arguments which I don't know about and don't dare touch in case I break it. Why does it do so much extra stuff when the XML is really bare-bones?
Anyway, I pinned my CPUs and confirmed that qemu threads were only appearing on the selected CPUs (hit "1" when running top). I ended up giving Windows three cores and their hyperthreads so 6 vCpus. That did not help much either. Then I tried booting my host with isolcpus=2-7 so host processes only run on the first core and its hyperthread. Seems to work beautifully! I only ran it for a very brief time but the constant stutters were all gone. GPU usage and FPS were also up a little, suggesting it's less CPU bottlenecked now, though that was a very unscientific quick observation.

I had trouble understanding what exactly was involved in doing cpu pinning since so many different things and use cases have been documented. I'll take what worked and spell it out in case someone else needs help with it. This is for a HT quad-core with 8 logical CPUs. Adjust the CPU division to your hardware and needs.

On an existing libvirt setup, put this in the xml file:

<domain>
  ...
  <vcpu placement='static'>6</vcpu>
  <iothreads>1</iothreads>
  <cputune>
    <vcpupin vcpu='0' cpuset='2'/>
    <vcpupin vcpu='1' cpuset='3'/>
    <vcpupin vcpu='2' cpuset='4'/>
    <vcpupin vcpu='3' cpuset='5'/>
    <vcpupin vcpu='4' cpuset='6'/>
    <vcpupin vcpu='5' cpuset='7'/>
    <emulatorpin cpuset='0'/>
    <iothreadpin iothread='1' cpuset='1'/>
  </cputune>
  ...
</domain>

What this does is give the guest 6 vCpus and place the guest threads on cores 2-7. (Could be 0-5 instead as well, shouldn't matter. I opted to keep the first two for the host). I'm not sure if the emulatorpin and iothreadpin settings are necessary but they basically move qemu's internal tasks and IO handling to the specified CPUs. Read more: https://libvirt.org/formatdomain.html
This alone won't do much good because the host will still schedule its threads on the same CPUs. Adding the isolcpus=2-7 makes CPUs 2 through 7 unavailable for regular userspace processes. Some kernel tasks and interrupts still run on them but AFAIK it should be far less intrusive. The isolated CPUs are only used explicitly such as with cpuset. Kind of sucks when you want to use the CPUs on your host. I haven't looked up if the isolation could be done dynamically but that sounds like an ideal way.

Edit: Another random observation: using OVMF solves the problem of USB passthrough devices not working until the guest os has booted. At least I think so. Windows 8 has different error recovery menus though so I'm not sure if it loads USB drivers there. Legacy BIOS doesn't play nice with USB apparently. Same thing happens on my hardware BIOS if I disable its legacy USB hack mode.

Last edited by impulse_255 (2015-07-13 21:42:56)

Nazfellun

Member

Registered: 2012-10-22

Posts: 19

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Strangely, although I am getting output on the physical card it isn't listed in device manager. Despite this, the AMD Catalyst installer detects the graphics hardware and tries (unsuccessfully) to install it. It gets up to installing the actual driver, the screen flickers once, then the physical display loses signal, the VM goes offline and ramps up to a constant 100% CPU usage with no disk activity. Happens using both the modified copy of my existing VBIOS and the vendor-provided updated one.

My hardware:
Motherboard: Gigabyte 990FXA-UD3
CPU: FX-8350
Host GPU: Gigabyte GTX 750 Ti OC
Guest GPU: Gigabyte R9 290 (initial reference cooler release)

I'll write a guide some day.
What guest OS and what device manager(devmgmt.msc?) you've used? I find it really odd that you can't see the card in there.
Also, which ROM did you bisect and took .efi part from? I remember finding 3 or 4 variants of GOP driver, maybe you could try different ones.

UPD: Catalyst 15.7 installed successfully, without any problems. Crossfire works as before.

After further messing about, the card is showing up in device manager using the 'Microsoft Basic Display Adapter' driver. It doesn't list any errors, but Catalyst still won't install (tried 14.12 which was what I ran under SeaBIOS, also tried 15.7). Guest OS is Windows 8.1.

I didn't end up bisecting a driver for the EFI section, as the linked UEFIRomExtract repo only appears to have a binary for Macs, and I have no idea how to compile it from source. As such, I just used the GOP provided in the first version of the AMD-UEFI-GOP-MAKER tool, which I think is from a HD6450.

fld

Member

Registered: 2015-02-06

Posts: 9

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

UPD: Catalyst 15.7 installed successfully, without any problems. Crossfire works as before.

Yeah I just got the Catalyst 15.7 working on my HD7770 as well. I just needed to boot Win7 without the GPU passthrough and then hot-plug the GPU via Qemu console (device_add vfio-pci....). After that, the installer works.

I also figured out a solution to the xHCI/EHCI USB port mapping problem that someone else noted a few pages ago. The problem was that on some Intel based systems, all the physical USB ports found on the motherboard would get mapped behind the single Intel xHCI controller. Even when the chipset contains EHCI controllers that show up in lspci. The solution that I finally came up with, is to directly access the PCI config space of the xHCI controller and change the Port Mapping Mask register. The register simply has a bit for each physical port, selecting if it should show up behind the EHCI controller or the xHCI controller. So for example, with my ASrock board I ended up using: setpci -s 00:14.0 0xd0.W=2037, where 00:14.0 is the address of the xHCI controller, 0xd0 is the address of the port mapping mask register and 0x2037 is the value that maps the physical USB2.0 ports behind EHCI controllers. Now I'm able to passthrough one of the Intel EHCI controllers to my VM and gain 4 x real USB2.0 ports.

Denso

Member

Registered: 2014-08-30

Posts: 179

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

I also figured out a solution to the xHCI/EHCI USB port mapping problem that someone else noted a few pages ago. The problem was that on some Intel based systems, all the physical USB ports found on the motherboard would get mapped behind the single Intel xHCI controller. Even when the chipset contains EHCI controllers that show up in lspci. The solution that I finally came up with, is to directly access the PCI config space of the xHCI controller and change the Port Mapping Mask register. The register simply has a bit for each physical port, selecting if it should show up behind the EHCI controller or the xHCI controller. So for example, with my ASrock board I ended up using: setpci -s 00:14.0 0xd0.W=2037, where 00:14.0 is the address of the xHCI controller, 0xd0 is the address of the port mapping mask register and 0x2037 is the value that maps the physical USB2.0 ports behind EHCI controllers. Now I'm able to passthrough one of the Intel EHCI controllers to my VM and gain 4 x real USB2.0 ports.

I am having this exact issue with my X99 platform . Is there a detailed guide or a tutorial that you can redirect me to ?

Duelist

Member

Registered: 2014-09-22

Posts: 358

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

I didn't end up bisecting a driver for the EFI section, as the linked UEFIRomExtract repo only appears to have a binary for Macs, and I have no idea how to compile it from source. As such, I just used the GOP provided in the first version of the AMD-UEFI-GOP-MAKER tool, which I think is from a HD6450.

The thing is - that tools are from EDK2. I've found EFIRom somewhere there some time ago, but i've lost it. Anyway, what's the device IDs of that basic adapter?
I think something is broken deeply.

Yeah I just got the Catalyst 15.7 working on my HD7770 as well. I just needed to boot Win7 without the GPU passthrough and then hot-plug the GPU via Qemu console (device_add vfio-pci....). After that, the installer works.

I dunno, i have two HD7750s attached, i've ran the installer, it finished without error and asked for a reboot. After a guest reboot, everything was perfect, except the crossfire turned itself off. With a flick of a switch, all was done.

Last edited by Duelist (2015-07-14 13:49:42)

---

The forum rules prohibit requesting support for distributions other than arch.
I gave up. It was too late.
What I was trying to do.
The reference about VFIO and KVM VGA passthrough.

fld

Member

Registered: 2015-02-06

Posts: 9

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

I also figured out a solution to the xHCI/EHCI USB port mapping problem that someone else noted a few pages ago. The problem was that on some Intel based systems, all the physical USB ports found on the motherboard would get mapped behind the single Intel xHCI controller. Even when the chipset contains EHCI controllers that show up in lspci. The solution that I finally came up with, is to directly access the PCI config space of the xHCI controller and change the Port Mapping Mask register. The register simply has a bit for each physical port, selecting if it should show up behind the EHCI controller or the xHCI controller. So for example, with my ASrock board I ended up using: setpci -s 00:14.0 0xd0.W=2037, where 00:14.0 is the address of the xHCI controller, 0xd0 is the address of the port mapping mask register and 0x2037 is the value that maps the physical USB2.0 ports behind EHCI controllers. Now I'm able to passthrough one of the Intel EHCI controllers to my VM and gain 4 x real USB2.0 ports.

I am having this exact issue with my X99 platform . Is there a detailed guide or a tutorial that you can redirect me to ?

I'm afraid there are no guides or tutorials for this. I just used the X87 PCH datasheet to figure out the correct register and then experimented with different values, until I had figured out the mapping between the bitmask and all the physical ports.

So you would need to start with the X99 PCH datasheet at around page 650 where it gets into xHCI port routing.

Yeah I just got the Catalyst 15.7 working on my HD7770 as well. I just needed to boot Win7 without the GPU passthrough and then hot-plug the GPU via Qemu console (device_add vfio-pci....). After that, the installer works.

I dunno, i have two HD7750s attached, i've ran the installer, it finished without error and asked for a reboot. After a guest reboot, everything was perfect, except the crossfire turned itself off. With a flick of a switch, all was done.

I'm running without x-vga, so maybe that's why.

Denso

Member

Registered: 2014-08-30

Posts: 179

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

I'm afraid there are no guides or tutorials for this. I just used the X87 PCH datasheet to figure out the correct register and then experimented with different values, until I had figured out the mapping between the bitmask and all the physical ports.

So you would need to start with the X99 PCH datasheet at around page 650 where it gets into xHCI port routing.

Thanks ! It seems that I can only seperate it into 2 controllers only . I'm better off using my PCI-E USB3 card then

sunmast

Member

Registered: 2014-12-16

Posts: 4

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Is it possible to run OSX with OVMF method?

Yes. I have a proof-picture, if you're curious.

Hi Duelist, would you please share your qemu command line to run OSX with OVMF? Are you using Chameleon or Clover?

Thanks a lot!

Nektarios

Member

Registered: 2013-02-19

Posts: 90

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Is there a way to completely disable a graphics card (nvidia or ati) when not in use? And when I mean completely, I mean close to 0 power usage, no fan usage etc.

I want this so I can freely shut down the vga card that I use for the VM passthrough when the VM is not running.

aw

Member

Registered: 2013-10-04

Posts: 921

Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Is there a way to completely disable a graphics card (nvidia or ati) when not in use? And when I mean completely, I mean close to 0 power usage, no fan usage etc.

I want this so I can freely shut down the vga card that I use for the VM passthrough when the VM is not running.

Sure, buy an expensive server that supports physical PCI hotplug.  Most systems do not provide power control for PCI slots, nor is there any way to manipulate the hardware to disable slot power.  If you have a hotplug capable expresscard slot (ie. laptop - I haven't found one for desktop yet) and you're willing to take a significant performance penalty (5GT/s x1 PCIe), then something like the EXP GDC external video card slot will work (often with some commandline trickery).  You might also be surprised how little power some of the newer cards draw when idle.  I haven't tried my GTX750 in the external slot yet, but some smaller cards show less than 5W when unassigned.

---

http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Duelist

Member

Registered: 2014-09-22

Posts: 358

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Is it possible to run OSX with OVMF method?

Yes. I have a proof-picture, if you're curious.

Hi Duelist, would you please share your qemu command line to run OSX with OVMF? Are you using Chameleon or Clover?

Thanks a lot!

I can't. I've seen the picture of that. But only the picture.

I'm running without x-vga, so maybe that's why.

ಠ_ಠ
I am too, since i needed to patch my VBIOS to get GOP... I use OVMF, OVMF doesn't require x-vga.
If you run OVMF too with a factory-flashed GPU ROM - could you send me the file or at least the .efi part?

---

The forum rules prohibit requesting support for distributions other than arch.
I gave up. It was too late.
What I was trying to do.
The reference about VFIO and KVM VGA passthrough.

Denso

Member

Registered: 2014-08-30

Posts: 179

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

@aw :

Does vfio-pci ids REQUIRE pci-stub to be loaded ?

I'm asking because I've been having a hard time getting vfio-pci ids to be loaded at boot , and now that I added pci-stub module before vfio-pci in the boot modules chain IT WORKS ! And every device get bound to VFIO directly and correctly .

Please note that pci-stub is not built statically in Arch .

aw

Member

Registered: 2013-10-04

Posts: 921

Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

@Denso

No, vfio-pci has no dependency on pci-stub.

---

http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Denso

Member

Registered: 2014-08-30

Posts: 179

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

@ Well , that is strange . "options vfio-pci ids=" didn't work on both of my NVIDIAs until I added pci-stub to the top of the boot chain . Anyway it is working like a charm now and I can finally get rid of the vfio-bind script .

Bronek

Member

From: London

Registered: 2014-02-14

Posts: 123

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

@ Well , that is strange . "options vfio-pci ids=" didn't work on both of my NVIDIAs until I added pci-stub to the top of the boot chain . Anyway it is working like a charm now and I can finally get rid of the vfio-bind script .

@Denso could it be that, when pci-stub is not loaded, noveau driver loaded/probed your card thus preventing vfio-pci from working correctly, and loading pci-stub prevents this from happening (as is its purpose)?

Nazfellun

Member

Registered: 2012-10-22

Posts: 19

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

I didn't end up bisecting a driver for the EFI section, as the linked UEFIRomExtract repo only appears to have a binary for Macs, and I have no idea how to compile it from source. As such, I just used the GOP provided in the first version of the AMD-UEFI-GOP-MAKER tool, which I think is from a HD6450.

The thing is - that tools are from EDK2. I've found EFIRom somewhere there some time ago, but i've lost it. Anyway, what's the device IDs of that basic adapter?
I think something is broken deeply.

The listed device & vendor ID match up to the ones reported by lspci -nn on the host, and the PCI bus/slot listed for the device matches the libvirt configuration.

Last edited by Nazfellun (2015-07-15 09:57:27)