Arch Linux

pascal257

Member

Registered: 2014-01-02

Posts: 5

KVM/QEMU VFIO Passthrough of primary GPU

Hey there,

I'm sucessfully running VFIO Passthrough with iGPU and 2 GPUs on my home systems for nearly 2 years now.

My next project involves my office PC with the following specs:

- ASRock X99 Extreme4
- Intel Xeon E5-1620 v3
- 48 GB DDR4 ECC Memory (for virtualizing FreeNas which needs loads of memory)
- 2x R7 250(E) (1024MB Sapphire Radeon R7 250 Ultimate Passiv PCIe 3.0 x16 1xDVI/1xHDMI/1xDisplayPort (Full Retail) and 1024MB Sapphire Radeon R7 250 Low Profile Aktiv PCIe 3.0 x16 1xDVI/1xmicroHDMI/1xminiDP (Retail))

lspci lists them as
01:00.0 VGA compatible controller: Advanced Micro Devices, Inc. [AMD/ATI] Cape Verde PRO [Radeon HD 7750/8740 / R7 250E]

As said in the subject, I'm trying to passthrough both GPUs leaving the host headless/without GPU.
I tried this before on my home PC by disabling the iGPU in the BIOS, taking both GPUs via VFIO and setting GRUB_TERMINAL_OUTPUT=console which seemed to prevent GRUB from taking the GPU. (The screen showed the POST screen of the UEFI/BIOS and then went black). In this setup passing through the primary GPU appeared to be working. The iGPU was not visible in lspci, but I'm not sure if the BIOS helped here.

Back to my office PC, I can't stop GRUB from displaying the menu, thus taking the GPU and preventing me from passing it through.

This is what dmesg gives me when starting the VM

[  718.650918] vgaarb: device changed decodes: PCI:0000:01:00.0,olddecodes=io+mem,decodes=io+mem:owns=none
[  718.661448] vgaarb: device changed decodes: PCI:0000:01:00.0,olddecodes=io+mem,decodes=io+mem:owns=none
[  718.827018] device tap0 entered promiscuous mode
[  718.827248] vm_br: port 1(tap0) entered forwarding state
[  718.827257] vm_br: port 1(tap0) entered forwarding state
[  720.753715] vfio-pci 0000:01:00.0: enabling device (0100 -> 0103)
[  720.753805] vfio_ecap_init: 0000:01:00.0 hiding ecap 0x19@0x270
[  723.527948] vgaarb: device changed decodes: PCI:0000:02:00.0,olddecodes=io+mem,decodes=io+mem:owns=io+mem
[  723.527952] vgaarb: device changed decodes: PCI:0000:01:00.0,olddecodes=io+mem,decodes=io+mem:owns=none
[  723.871907] kvm: zapping shadow pages for mmio generation wraparound
[  733.859092] vm_br: port 1(tap0) entered forwarding state

I've tried dozens of different GRUB settings, kernel-parameters and using UEFI systemd-boot. Nothing seemed to work.
I'll check if the cards are UEFI compatible and try that route, although first tests with OVMF did not work.

And as a bonus:

Installing a third grapics card enables me to pass through the GPU, but after installing AMD drivers the guest BSODs (on boot) with atikmpag.sys. I've already read that this might be connected to the sound interface and I'll post back the results of my tests shortly.

Also, does anyone know how PCI adresses are assigned? After installing the third GPU in the first PCIe slot it showed up as: 02:00.0 VGA compatible controller: NVIDIA Corporation GF108 [GeForce GT 630] (rev a1)

Every help is appreciated, thanks in advance.

BTW I'm running the latest Arch Linux with linux-lts kernel

Last edited by pascal257 (2015-11-05 12:10:57)

pascal257

Member

Registered: 2014-01-02

Posts: 5

Re: KVM/QEMU VFIO Passthrough of primary GPU

After starting from scratch it works like a charm now.

LTS Kernel without patches, 1.8 Seabios, latest qemu, no UEFI/OVMF
Following this article: https://wiki.archlinux.org/index.php/PC … h_via_OVMF

Setting GRUB_TERMINAL_OUTPUT=console help with glitches and I did not notice any problems after passing through the primary graphics.

The r7 250E / 7750 8740 requires that you don't pass through the HDMI audio device or else it will BSOD on windows boot after installing amd drivers.

Hope this information helps others with the same problem.

soulgriever

Member

From: TN, USA

Registered: 2015-12-19

Posts: 4

Website

Re: KVM/QEMU VFIO Passthrough of primary GPU

I am glad you got this working, I am trying to do something similar following the same guide as well as a few others but with nvidia card's I was wondering if you could take a look at my thread and tell me where I am going wrong. https://bbs.archlinux.org/viewtopic.php?id=206406 or where to look next

---

Id rather be a noob eager to learn, then a professional closed minded to new information

balogh

Member

Registered: 2016-02-05

Posts: 7

Re: KVM/QEMU VFIO Passthrough of primary GPU

pascal257 -

I am trying to set up a similar headless home server plus one Windows VM using the same motherboard with a i7 5820k and one GTX970 in the primary VGA slot.
Unfortunately there is no output from the VM to the graphics card (not even the seabios screen). There are no errors in dmesg. Everything is set up according to the vfio blog and many other sources. Virtualization seems to be properly enabled judging by the dmesg output. I also get similar vfio, vgaarb and kvm logs in dmesg when starting the VM.

Assuming it is possible to install Windows in the VM using the passed-through graphics card:

Do you think the motherboard's chipset settings could affect this?
Could you please list your UEFI's chipset settings?

Thank you!

soulgriever

Member

From: TN, USA

Registered: 2015-12-19

Posts: 4

Website

Re: KVM/QEMU VFIO Passthrough of primary GPU

Balogh while I have since migrated to a different os I have had success by changing seabios to ovmf also if your i7 has integrated graphics, try setting it to primary in your bios. It helps linux let go of the graphics card.

---

Id rather be a noob eager to learn, then a professional closed minded to new information

andreigrig

Member

Registered: 2016-02-04

Posts: 20

Re: KVM/QEMU VFIO Passthrough of primary GPU

Hi there,

How did you manage to passthrough the primary GPU exactly? I get no output from the VM and vfio reports specific primary gpu passthrough errors as long as vgaarbt sets it as the boot GPU. Is there any way you can get ttys to display on the secondary GPU? I think this is why the memory cannot be reserved. I only managed to redirect the output in xorg.

Thanks!

Last edited by andreigrig (2016-02-05 12:11:10)

balogh

Member

Registered: 2016-02-05

Posts: 7

Re: KVM/QEMU VFIO Passthrough of primary GPU

soulgriever -

5820k doesn't have a iGPU.

andreigrig -

In order to prevent the kernel output to the primary graphics card I have added to the grub kernel arguments:

fbcon=map:1 (source: https://bbs.archlinux.org/viewtopic.php?id=162768&p=129 then a bit of reading in the kernel docs).

Alas, setting GRUB_TERMINAL_OUTPUT=console didn't prevent grub from displaying the menu to the primary VGA.

I'll post my settings if I succeed with primary passthrough.

Last edited by balogh (2016-02-05 13:50:13)

balogh

Member

Registered: 2016-02-05

Posts: 7

Re: KVM/QEMU VFIO Passthrough of primary GPU

I'm giving up on primary passthrough for now.
My primary concern was performance of the GTX970 in a PCIe 3.0 slot at x8. However, it seems that the performance loss is 1-2 fps, which is acceptable, so I'm giving it a try.
A EAH5450 Silent will sit in the primary VGA slot and the GTX970 in the x8 slot.

alphaniner

Member

From: Ancapistan

Registered: 2010-07-12

Posts: 2,810

Re: KVM/QEMU VFIO Passthrough of primary GPU

@ balogh - The BIOS may allow selection of primary GPU, have you looked into that?

---

But whether the Constitution really be one thing, or another, this much is certain - that it has either authorized such a government as we have had, or has been powerless to prevent it. In either case, it is unfit to exist.
-Lysander Spooner

andreigrig

Member

Registered: 2016-02-04

Posts: 20

Re: KVM/QEMU VFIO Passthrough of primary GPU

Thanks for the tip! However, the fbcon option does nothing for me. I have the same issue as before.

When I boot up I get this:

[    0.668496] VFIO - User Level meta-driver version: 0.3
[    0.679906] vfio_pci: add [10de:0fc6[ffff:ffff]] class 0x000000/00000000
[    0.693275] vfio_pci: add [10de:0e1b[ffff:ffff]] class 0x000000/00000000

However when I start the VM I get this as well:

[   53.528271] vfio_ecap_init: 0000:01:00.0 hiding ecap 0x19@0x900
[   53.529934] vfio-pci 0000:01:00.0: BAR 3: can't reserve [mem 0xd0000000-0xd1ffffff 64bit pref]
[   53.541345] vfio-pci 0000:01:00.1: enabling device (0000 -> 0002)
[   55.026840] vfio-pci 0000:01:00.0: Invalid ROM contents
[   55.026965] vfio-pci 0000:01:00.0: Invalid ROM contents

Those are all from dmesg. Any ideas?

Thanks!

Last edited by andreigrig (2016-02-05 19:28:54)

balogh

Member

Registered: 2016-02-05

Posts: 7

Re: KVM/QEMU VFIO Passthrough of primary GPU

The Asrock X99 Extreme4 doesn't have such an option in its UEFI.

alphaniner

Member

From: Ancapistan

Registered: 2010-07-12

Posts: 2,810

Re: KVM/QEMU VFIO Passthrough of primary GPU

That's weak sauce. Consider bringing it up on Asrock forums or with Asrock TS. There may be a UEFI f/w version that makes it possible. If nothing else, their official excuse should be interesting.

---

But whether the Constitution really be one thing, or another, this much is certain - that it has either authorized such a government as we have had, or has been powerless to prevent it. In either case, it is unfit to exist.
-Lysander Spooner

balogh

Member

Registered: 2016-02-05

Posts: 7

Re: KVM/QEMU VFIO Passthrough of primary GPU

andreigrig -

Try adding the rom file for your graphics card. Rom files can be found on the internet. The link that I posted above has an example of how to include the rom file.

andreigrig

Member

Registered: 2016-02-04

Posts: 20

Re: KVM/QEMU VFIO Passthrough of primary GPU

@balogh,

many thanks for your help. The thing is that the rom file is there and it works perfectly if I switch and use the card as a secondary GPU, but if I do this I get to use my card at x4 speed.

balogh

Member

Registered: 2016-02-05

Posts: 7

Re: KVM/QEMU VFIO Passthrough of primary GPU

The GTX970 as the secondary graphics card has successfully passed through. It seems that Nvidia cards are almost impossible to pass through as the primary card. The primary-passthrough success stories that I have read about all used AMD cards in the primary slot.
I have used a different linux distribution with which I am more familiar.

balogh

Member

Registered: 2016-02-05

Posts: 7

Re: KVM/QEMU VFIO Passthrough of primary GPU

The results are impressive. The difference between "Bare Metal, PCIe@x16" and "qemu/KVM, PCIe@x8" is so small that I regret spending so much time struggling to pass through as primary.
Here is a comparison. On the left is bare metal, on the right is qemu/KVM.