Arch Linux

nbhs

Member

From: Montevideo, Uruguay

Registered: 2013-05-02

Posts: 402

KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

THIS GUIDE IS SLIGHTLY OUTDATED AND I'M NOT MAINTAINING IT ANYMORE
PLEASE CHECK OUT ALEX'S BLOG FOR AN UP TO DATE GUIDE

This is intented for people who wish to pass-through a GPU to a virtual machine using the KVM hypervisor, QEMU and vfio-pci

NOTE: AMD RADEON 5xxx, 6xxx, 7xxx, 2xx and NVIDIA GEFORCE 7, 8, 4xx, 5xx, 6xx, 7xx 9xx have been reported working with this, passing though an intel IGD is not supported YET

NOTE: THIS IS EXPERIMENTAL SO IT MIGHT NOT WORK ON YOUR SYSTEM

This is the result on my radeon 6950:

This is the result on my geforce 470 gtx:

Recommended reads

Alex Williamson's blog and FAQ for detailed technical information
OVMF whitepaper.
GPU passthrough database by noctavian

At this moment there are two ways to achieve this using KVM: legacy VGA via Seabios or non-VGA (UEFI) via OVMF

The main practical advantage of using OVMF is that on intel systems you no longer need the i915 VGA arbiter patch which also disables DRI on the host

Requirements

• AMD-VI/VT-D enabled and working (On intel systems both your MB and you CPU must support it, to find out if you CPU has VT-D support go here: http://ark.intel.com/, you also need to boot with intel_iommu=on to enable it)

• At least 2 GPU's, one primary boot device and the card you wish to pass-through

• Qemu=>2.0

• Aditional kernel patches might be required if you're using an Intel CPU: ACS override patch and i915 VGA arbiter patch, you can find a kernel package with these patches included on AUR: linux-vfio

Aditional requirements for OVMF:

• An UEFI compatible GPU (Most modern gpus support this, SEE: Does my graphics card ROM support EFI?

• An UEFI compatible GUEST (ex: Windows 7 and up)

• OVMF:  Gerd Hoffmann's rpms or OVMF-svn from AUR

If you're building your kernel manually this option is required:

CONFIG_VFIO_PCI_VGA=y

Preparing your System

On intel system the i915 VGA arbiter patch is required if:

1. You're using the IGD on your host

2. You're doing VGA assignment

NOTE: Enabling this will disable DRI!

If you wish to avoid this, then then you should try OVMF

To enable it add this kernel parameter to your bootloader:

i915.enable_hd_vgaarb=1

Next, we need to prepare the GPU for vfio:

1. Blacklist radeon or nouveau or nvidia or fglrx on /etc/modprobe.d/blacklist.conf

   Example, blacklisting the opensource radeon module:

   echo "blacklist radeon" >> /etc/modprobe.d/blacklist.conf 

2. Use pci-stub

   In my case since i have 2 radeon cards blacklisting the radeon module is not an option, so i use pci-stub

   NOTE: If pci-stub was built as a module, you'll need to modify /etc/mkinitcpio.conf and add pci-stub in the MODULES section, after that you need to update your initramfs like this

   mkinitcpio -p linux-mainline

   lspci

   07:00.0 VGA compatible controller: Advanced Micro Devices, Inc. [AMD/ATI] Cayman PRO [Radeon HD 6950] <-- radeon 6950
   07:00.1 Audio device: Advanced Micro Devices, Inc. [AMD/ATI] Cayman/Antilles HDMI Audio [Radeon HD 6900 Series] <-- radeon 6950 audio

   lspci -n

   07:00.0 0300: 1002:6719 <-- radeon 6950
   07:00.1 0403: 1002:aa80 <-- radeon 6950 audio

   Now add this kernel parameter to your bootloader:

   pci-stub.ids=1002:6719,1002:aa80

   dmesg | grep pci-stub

   [    2.096151] pci-stub: add 1002:6719 sub=FFFFFFFF:FFFFFFFF cls=00000000/00000000
   [    2.096160] pci-stub 0000:07:00.0: claimed by stub
   [    2.096165] pci-stub: add 1002:AA80 sub=FFFFFFFF:FFFFFFFF cls=00000000/00000000
   [    2.096174] pci-stub 0000:07:00.1: claimed by stub
   [    2.096178] pci-stub: add 1B21:1042 sub=FFFFFFFF:FFFFFFFF cls=00000000/00000000

Setting up vfio and kvm modules

*********************************

This is only required if you get this message:

vfio_iommu_type1_attach_group: No interrupt remapping support.  Use the module param "allow_unsafe_interrupts" to enable VFIO IOMMU support on this platform

If your board doesn't enable interrupt remapping, you need to add this to your bootloader:

vfio_iommu_type1.allow_unsafe_interrupts=1

Or if vfio-pci was built as a module ( default on arch )

echo "options vfio_iommu_type1 allow_unsafe_interrupts=1" > /etc/modprobe.d/vfio_iommu_type1.conf

*********************************

Some applications like Passmark Performance Test and SiSoftware Sandra crash the VM without this:

echo "options kvm ignore_msrs=1" >> /etc/modprobe.d/kvm.conf

Binding a device to vfio-pci

We'll use this script to make life easier:

#!/bin/bash

modprobe vfio-pci

for dev in "$@"; do
        vendor=$(cat /sys/bus/pci/devices/$dev/vendor)
        device=$(cat /sys/bus/pci/devices/$dev/device)
        if [ -e /sys/bus/pci/devices/$dev/driver ]; then
                echo $dev > /sys/bus/pci/devices/$dev/driver/unbind
        fi
        echo $vendor $device > /sys/bus/pci/drivers/vfio-pci/new_id
done

Save it as /usr/bin/vfio-bind

chmod 755 /usr/bin/vfio-bind

Bind the GPU and GPU audio:

vfio-bind 0000:07:00.0 0000:07:00.1

Systemd service:

[Unit]
Description=Binds devices to vfio-pci
After=syslog.target

[Service]
EnvironmentFile=-/etc/vfio-pci.cfg
Type=oneshot
RemainAfterExit=yes
ExecStart=-/usr/bin/vfio-bind $DEVICES

[Install]
WantedBy=multi-user.target

cat /etc/vfio-pci.cfg

DEVICES="0000:00:11.0 0000:04:00.0 0000:05:00.0 0000:06:00.0 0000:07:00.0 000:07:00.1"

Testing if its working out

Seabios:

qemu-system-x86_64 -enable-kvm -m 1024 -cpu host,kvm=off \
-smp 4,sockets=1,cores=4,threads=1 \
-device vfio-pci,host=07:00.0,x-vga=on -device vfio-pci,host=07:00.1 \
-vga none

OVMF:

qemu-system-x86_64 -enable-kvm -m 1024 -cpu host,kvm=off  \
-smp 4,sockets=1,cores=4,threads=1 \
-drive if=pflash,format=raw,readonly,file=/usr/share/ovmf/x64/ovmf_code_x64.bin \
-drive if=pflash,format=raw,file=/usr/share/ovmf/x64/ovmf_vars_x64.bin \
-device vfio-pci,host=07:00.0 -device vfio-pci,host=07:00.1 \
-vga none

Note:

• kvm=off will hide the kvm hypervisor signature, this is required for NVIDIA cards, since its driver will refuse to work on an hypervisor and result in Code 43 on windows (unless you're using a QUADRO)

• x-vga=on is required for vga assignment

• -vga none disables the default vga device on QEMU, it is also required for vga assignment

You should see a black qemu window on your main display, and seabios/ovmf ouput on your monitor from your passthru'd card saying it cant find anything to boot

If you're using an intel cpu and nothing happens try this:

modprobe -r kvm_intel
modprobe kvm_intel emulate_invalid_guest_state=0

NOTE: There might be some problems using nvidia/fglrx/nouveau drivers on the host gpu, see the ISSUES section below on how to solve this

At this point you're ready to create your vm, you could use libvirt + virt-manager to create your vm using a nice graphical interface, there are
also some interesting examples in this thread, otherwise if you prefer to use the command line continue reading

DISK:

Creating a disk image:

Create a 30Gb raw image:

dd if=/dev/zero of=windows.img bs=1M seek=30000 count=0

Using a VIRTIO controller:

We create our virtio controller for qemu:

 -device virtio-scsi-pci,id=scsi

Attaching our hdd:

-drive file=/home/nbhs/windows.img,id=disk,format=raw,if=none -device scsi-hd,drive=disk

Attaching a cdrom:

-drive file=/home/nbhs/windows.iso,id=isocd,if=none -device scsi-cd,drive=isocd

To install windows, you'll need to download the virtio drivers iso from here

Attaching the virtio iso:

-drive file=/home/nbhs/virtio.iso,id=virtiocd,if=none -device ide-cd,bus=ide.1,drive=virtiocd

Windows will complain it cant find a disk to install to, just click browse, find the folder for your os/arch, select "RED HAT VirtIO pass-through controller" then click next.

For more info check out the Archlinux wiki's QEMU section

Using a physical disk or partition:

-drive file=/dev/sdb,id=disk,format=raw,if=none -device scsi-hd,drive=disk

You might need to change the bus depending on the controller you use

If you wish to use a physical partition and be able to read its contents later on, you can follow these guides:

http://fds-team.de/cms/articles/2013-12 … yer-u.html
https://wiki.archlinux.org/index.php/QE … disk_image

Passing a sata disk controller to qemu:

If you dual boot like me you can pass an entire sata controller (in ahci mode) and all the drives attached, and seabios will boot from them.

lspci

00:11.0 SATA controller: Advanced Micro Devices [AMD] nee ATI SB7x0/SB8x0/SB9x0 SATA Controller [AHCI mode] (rev 40)

Make sure all volumes are unmounted, then bind the controller to vfio:

vfio-bind 0000:00:11.0

Now we pass the controller to the vm:

-device vfio-pci,host=00:11.0

USB:

Its also possible to passthrough a usb controller, in my case an ASMEDIA USB3 Controller:

lspci

04:00.0 USB controller: ASMedia Technology Inc. ASM1042 SuperSpeed USB Host Controller
05:00.0 USB controller: ASMedia Technology Inc. ASM1042 SuperSpeed USB Host Controller
06:00.0 USB controller: ASMedia Technology Inc. ASM1042 SuperSpeed USB Host Controller

Bind them to vfio:

vfio-bind 0000:04:00.0 0000:05:00.0 0000:06:00.0

Pass them to the vm:

-device vfio-pci,host=04:00.0 \
-device vfio-pci,host=05:00.0 \
-device vfio-pci,host=06:00.0

Or a specific device:

lsusb

...
Bus 004 Device 010: ID 045e:00e1 Microsoft Corp. Wireless Laser Mouse 6000 Reciever <-- mouse
Bus 004 Device 011: ID 045e:074b Microsoft Corp. <-- keyboard
...

We pass them to the vm:

-usb -usbdevice host:045e:00e1 -usbdevice host:045e:074b

NETWORK:

Please see https://wiki.archlinux.org/index.php/QEMU#Networking

AUDIO EMULATION:

To hear the sound from the vm on your host speakers you'll need to add this lines:

-soundhw hda

You might need to start qemu like this:

QEMU_PA_SAMPLES=128 QEMU_AUDIO_DRV=pa qemu-system-x86_64...

alsa:

QEMU_ALSA_DAC_BUFFER_SIZE=512 QEMU_ALSA_DAC_PERIOD_SIZE=170 QEMU_AUDIO_DRV=alsa qemu-system-x86_64...

Note: these are the settings i found that work great on my system, if you get crackling/skipping audio you might want to try different settings

To see the available drivers and audio options:

qemu-system-x86_64 -audio-help

LOADING YOUR CARD ROM FROM A DUMP/FILE:

modify this line:

-device vfio-pci,host=07:00.0,...... \

change it to this:

-device vfio-pci,host=07:00.0,......,romfile=/path/to/your/gpu/bios.bin \

ISSUES:

• Using the latest NVIDIA drivers on the guest will result in code 43, but there's a workarround

• Using the latest NVIDIA drivers on the guest using hv enlightenments will result in code 43, see: https://forums.geforce.com/default/topi … 8/#4314318

• Using AMD proprieraty drivers on the host, see: https://bbs.archlinux.org/viewtopic.php … 2#p1273412 the only solution atm is using the opensource radeon driver

PERFORMANCE IMPROVEMENTS:

• SEE: https://bbs.archlinux.org/viewtopic.php … 1#p1270311

• Pin vcpu to pcpu, see http://www.linux-kvm.com/content/tip-ru … cific-cpus

• Use cgroups, specifically cpuset, see https://access.redhat.com/site/document … puset.html

• Use Hugepages, see http://pic.dhe.ibm.com/infocenter/lnxin … ttunhp.htm

• Use Hyper-V enlightenments, see: http://blog.wikichoon.com/2014/07/enabl … h-kvm.html

Last edited by nbhs (2015-07-11 16:23:26)

crondog

Member

Registered: 2011-04-21

Posts: 130

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Nice work. I currently use xen to achieve a similar thing since the upstream qemu never worked

I hope to give this a try soon so i can get rid of xen since i only ever run 1 vm and its a bit of a waste

nbhs

Member

From: Montevideo, Uruguay

Registered: 2013-05-02

Posts: 402

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

EDITED:

The single most important thing i found to improve the vm performance on my amd board is to disable nested page tables:

echo "options kvm-amd npt=0" > /etc/modprobe.d/kvm-amd.conf

These are some other things i did to improve the performance:

• -cpu host

• Using hugetlbfs instead of transparent huge pages

• Set Preemption Model to Voluntary on kernel

• Set Timer frequency  1000HZ on kernel

Im using -cpu Opteron_G4 instead of -cpu host, this shouldnt affect performance though.

NOTE: i use 6 cores (2-7) for the vm, 2 (0-1) are reserved for the host using cpuset cgroups
NOTE: Most of these settings affect vcpu performance, gpu performance remains about the same.

Last edited by nbhs (2014-01-20 22:02:03)

jgott

Member

Registered: 2011-04-10

Posts: 21

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Thank you for the very detailed post, nbhs! I've been struggling to get Xen working in either a primary or secondary passthrough situation over the past week, and I was about to start trying with KVM. This guide will sure come in handy.

I have a fairly similar configuration to you:
ASUS M5A99X EVO (990X/SB950)
AMD FX-8350 (4.8 GHz)
AMD Radeon HD 5750 (for Linux host)
AMD Radeon HD 7870 (for Windows guest)

Like you, my ASUS motherboard has a broken IVRS table (with precisely the same wrong values as yours, apparently). In addition to overriding it in the kernel, I actually went to all the bother to send ASUS a detailed and helpful support request detailing the AMD IOMMU spec and the chipset documentation, telling them how they could fix their firmware. My mistake, as the support guy continually misunderstands me due to the language barrier (and because he probably just wants me to go away). So I'm almost certainly wasting my time trying to get the actual responsible party to fix their broken firmware.

I'll keep you posted on any issues that I run into as I give KVM a try.

nbhs

Member

From: Montevideo, Uruguay

Registered: 2013-05-02

Posts: 402

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Like you, my ASUS motherboard has a broken IVRS table (with precisely the same wrong values as yours, apparently). In addition to overriding it in the kernel, I actually went to all the bother to send ASUS a detailed and helpful support request detailing the AMD IOMMU spec and the chipset documentation, telling them how they could fix their firmware. My mistake, as the support guy continually misunderstands me due to the language barrier (and because he probably just wants me to go away). So I'm almost certainly wasting my time trying to get the actual responsible party to fix their broken firmware.

I'll keep you posted on any issues that I run into as I give KVM a try.

Yeah so did i, unfortunately asus doesnt support linux, and their tech support are bunch of monkeys behind a keyboard, they remind me of this http://www.youtube.com/watch?v=C2Ph8zwpNyI, ill never buy a product from them ever again.

Last edited by nbhs (2013-05-11 01:54:31)

jgott

Member

Registered: 2011-04-10

Posts: 21

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

I've run into two problems so far.

First problem: I had to stop using the proprietary ATI graphics driver on the host, as running the VM and X at the same time would hang the system. Unfortunately, the fglrx module thinks it's a great idea to grab all the cards in the system and remap their IRQs (as shown by dmesg and /proc/interrupts) despite them actually being assigned to pci-stub. Apparently it just scans the PCI bus looking for ATI cards and then configures all of them, no matter which device the module was actually loaded for. Brilliant.

I thought I might be able to get around this, as I was able to hack the PCI ID for the 7870 out of the fglrx kernel module by modifying /usr/share/ati/build_mod/fglrxko_pci_ids.h and rebuilding. After I did this, the fglrx kernel module ignored the secondary card like I wanted it to. But the fglrx X driver would then refuse to let X start, since it was doing its own PCI bus scan, in which it recognized the secondary card and realized that the kernel module hadn't configured it. And to my great disgruntlement, the fglrx X driver ignores useful options like "AutoAddGPU" "false", plus I can't just hack their PCI scan to ignore the card because they've gone and duplicated a bunch of external code (like the PCI bus scanning). I basically can't change its behavior because it's closed source.

I suppose an older version of fglrx might work (pre-7000 series, so it wouldn't recognize the card). Hacking the kernel to misreport the PCI ID would probably work too. For now, though, just using the open source radeon driver is an acceptable solution.

Second problem: despite the new patches to qemu and linux, the card is apparently not being reset between guest runs. If I run the VM twice, the host will hang, gradually getting slower for a few seconds until it freezes, almost like it's experiencing an interrupt storm or something; eventually even Magic SysRq doesn't work. Suspending the host appears to be enough to reset the card, so for now, I'll just suspend/wake between VM runs. I plan to try the eject-before-shutdown method too.

Everything else seems to be working though. Performance is impressive! Now I need to buy a USB switch for input devices; or maybe I'll just solder/glue one out of spare parts. Those things are ridiculously overpriced for what amounts to a 4-pole, double-throw switch with USB ports on the ends.

(I was going to attach some screenshots to this post, but Print Screen made the guest freeze. So maybe I'll post those later.)

nbhs

Member

From: Montevideo, Uruguay

Registered: 2013-05-02

Posts: 402

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Thanks for the report!, I had no idea about fglrx, i havent used them since i had a Radeon 9800xt , and opensource is good enough for my 5450. You can try ejecting the card on windows just like you would eject a pendrive, method No3 is about the automatization of this process before shutting down/restarting the guest. You could also fill a bug report on qemu-devel about the card not reseting properly, in my case i can hear the gpu fan speeding up after a reset. I also re-uploaded linux-mainline based on 3.9.2 and qemu-git packages with reset patches, they're working fine on my pc.

EDIT: Could you try to load your gpu bios by file and see if it still crash? i remember a couple of times my vm crashed after a reset ( before i accidentally physically destroyed the bios chip , now it wont work without loading a rom file ) and after i manually loaded the rom, didnt happen again, you can do this by appending ',rombar=0, romfile=/path/to/the/rom.bin'

ex:

-device vfio-pci,host=07:00.0,x-vga=on,addr=0.0,multifunction=on,bus=root.1,rombar=0,romfile=/home/nbhs/vgabios-6950.rom

Or use my script, and modify VBIOS="" to VBIOS="/path/to/rom"

You can find yours here: http://www.techpowerup.com/vgabios/

Last edited by nbhs (2013-05-16 01:43:08)

apoapo

Member

Registered: 2013-05-16

Posts: 15

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Tried to get it working for me too, but with no success.

My system:
Radeon 7950
i5 3470 (non-K) incl Intel GPU for Host
AsRock Z77 Extreme4
8GB Ram

First thing I was trying was the old way, with pci-assign. I was able to first boot with VNC, installing ATI drivers. After reboot the screen woke up at login promt and the Perfomance was ok/good (I wasn't trying too much. Windows Performance Index at 7.5, so it should have worked ). But at reboot of VM the host crashed....

So I was looking around the internet and found this thread. Tried many hours to get it working, but I don't get any picture to the screen. No Seabios Texts and nothing in Windows after installing ATI drivers with vnc (and removing VNC again) like it was working with pci-assign.

Have tried Qemu-git from AUR, Qemu-1.5-rc1, seabios from your link, stock kernel (3.9.2-1) and kernel 3.10-rc1.
Did not change anything .

The only thing I havn't tried till now is to dump the VGA bios and put it in. Will do this today, when I´m at home.

IOMMU is enabled. logfiles look good (for me ). The only thing is "DMAR: No ATSR found". You have a Idea about this?
Full logfiles I can add later. 

Bind everything to vfio-pci seems to work. Also there isn't any error message at KVM-Start
I also tried all with pci-stub entries at grub and without, same with the interrupt remapping entry.

Do you have any Idea?

Thanks
apo

Last edited by apoapo (2013-05-16 14:07:21)

nbhs

Member

From: Montevideo, Uruguay

Registered: 2013-05-02

Posts: 402

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Tried to get it working for me too, but with no success.

My system:
Radeon 7950
i5 3470 (non-K) incl Intel GPU for Host
AsRock Z77 Extreme4
8GB Ram

First thing I was trying was the old way, with pci-assign. I was able to first boot with VNC, installing ATI drivers. After reboot the screen woke up at login promt and the Perfomance was ok/good (I wasn't trying too much. Windows Performance Index at 7.5, so it should have worked ). But at reboot of VM the host crashed....

So I was looking around the internet and found this thread. Tried many hours to get it working, but I don't get any picture to the screen. No Seabios Texts and nothing in Windows after installing ATI drivers with vnc (and removing VNC again) like it was working with pci-assign.

Have tried Qemu-git from AUR, Qemu-1.5-rc1, seabios from your link, stock kernel (3.9.2-1) and kernel 3.10-rc1.
Did not change anything .

The only thing I havn't tried till now is to dump the VGA bios and put it in. Will do this today, when I´m at home.

IOMMU is enabled. logfiles look good (for me ). The only thing is "DMAR: No ATSR found". You have a Idea about this?
Full logfiles I can add later. 

Bind everything to vfio-pci seems to work. Also there isn't any error message at KVM-Start
I also tried all with pci-stub entries at grub and without, same with the interrupt remapping entry.

Do you have any Idea?

Thanks
apo

Stock kernel wont work, you need to rebuild it to include:

    --- VFIO Non-Privileged userspace driver framework               
            <*>   VFIO support for PCI devices                               
              [*]     VFIO PCI support for VGA devices    

Please use the linux-mainline, and qemu-git packages that i posted on "RADEON RELATED PROBLEMS", it has this enabled by default.

If nothing appears on screen try:

modprobe -r kvm_intel
modprobe kvm_intel emulate_invalid_guest_state=0

Also remember when launching qemu you must use q35 machine mode, vga none,  and "x-vga=on" on your gpu

ex:

qemu-system-x86_64 --enable-kvm -M q35  -vga none \
-device ioh3420,bus=pcie.0,addr=1c.0,multifunction=on,port=1,chassis=1,id=root.1 \
-device vfio-pci,host=07:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on \
-device vfio-pci,host=07:00.1,bus=root.1,addr=00.1

Last edited by nbhs (2013-05-16 19:46:34)

apoapo

Member

Registered: 2013-05-16

Posts: 15

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

yeah. I forgot to say this.
I changed the configuration of the kernel . I enabled the 4 vfio things in the kernels I used.
All things should be like you said. Also tried reloading kvm_intel module ...
I will take a look in the next hours and post some Outputs and try the linux-mainline ....

Thx for helping

EDIT: Some outputs

$ dmesg  | grep -e DMAR -e IOMMU
[    0.000000] ACPI: DMAR 00000000bd969280 000B8 (v01 INTEL      SNB  00000001 INTL 00000001)
[    0.000000] Intel-IOMMU: enabled
[    0.144709] dmar: IOMMU 0: reg_base_addr fed90000 ver 1:0 cap c0000020e60262 ecap f0101a
[    0.144714] dmar: IOMMU 1: reg_base_addr fed91000 ver 1:0 cap c9008020660262 ecap f0105a
[    0.144785] IOAPIC id 2 under DRHD base  0xfed91000 IOMMU 1
[    0.404802] DMAR: No ATSR found
[    0.404822] IOMMU 0 0xfed90000: using Queued invalidation
[    0.404823] IOMMU 1 0xfed91000: using Queued invalidation
[    0.404824] IOMMU: Setting RMRR:
[    0.404832] IOMMU: Setting identity map for device 0000:00:02.0 [0xbf800000 - 0xcf9fffff]
[    0.405981] IOMMU: Setting identity map for device 0000:00:1d.0 [0xbd568000 - 0xbd59afff]
[    0.405997] IOMMU: Setting identity map for device 0000:00:1a.0 [0xbd568000 - 0xbd59afff]
[    0.406011] IOMMU: Setting identity map for device 0000:00:14.0 [0xbd568000 - 0xbd59afff]
[    0.406019] IOMMU: Prepare 0-16MiB unity mapping for LPC
[    0.406025] IOMMU: Setting identity map for device 0000:00:1f.0 [0x0 - 0xffffff]

ls -l /sys/bus/pci/drivers/vfio-pci/
insgesamt 0
lrwxrwxrwx 1 root root    0 17. Mai 00:53 0000:00:01.0 -> ../../../../devices/pci0000:00/0000:00:01.0
lrwxrwxrwx 1 root root    0 17. Mai 00:53 0000:01:00.0 -> ../../../../devices/pci0000:00/0000:00:01.0/0000:01:00.0
lrwxrwxrwx 1 root root    0 17. Mai 00:53 0000:01:00.1 -> ../../../../devices/pci0000:00/0000:00:01.0/0000:01:00.1
--w------- 1 root root 4096 17. Mai 00:53 bind
lrwxrwxrwx 1 root root    0 17. Mai 00:53 module -> ../../../../module/vfio_pci
--w------- 1 root root 4096 17. Mai 00:53 new_id
--w------- 1 root root 4096 17. Mai 00:53 remove_id
--w------- 1 root root 4096 17. Mai 00:48 uevent
--w------- 1 root root 4096 17. Mai 00:53 unbind

lspci
00:01.0 PCI bridge: Intel Corporation Xeon E3-1200 v2/3rd Gen Core processor PCI Express Root Port (rev 09)
01:00.0 VGA compatible controller: Advanced Micro Devices [AMD] nee ATI Tahiti PRO [Radeon HD 7950]
01:00.1 Audio device: Advanced Micro Devices [AMD] nee ATI Tahiti XT HDMI Audio [Radeon HD 7970 Series]

Starting KVM: dmesg
[  511.383247] vfio_ecap_init: 0000:01:00.0 hiding ecap 0x19@0x270
[  511.383256] vfio_ecap_init: 0000:01:00.0 hiding ecap 0x1b@0x2d0

qemu-system-x86_64 --enable-kvm -M q35 -m 2096 -cpu host \
-smp 4,sockets=1,cores=4,threads=1 \
-bios /usr/share/qemu/bios.bin  -vga none \
-device ioh3420,bus=pcie.0,addr=1c.0,multifunction=on,port=1,chassis=1,id=root.1 \
-device vfio-pci,host=01:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on \
-device vfio-pci,host=01:00.1,bus=root.1,addr=00.1

Last edited by apoapo (2013-05-16 22:59:45)

apoapo

Member

Registered: 2013-05-16

Posts: 15

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

I finally got some progress.

After trying mainline kernel and with vbios, I went to bios/uefi and turned the Primary VGA card from onboard to pci-express ...
Now I see the seabios in passthroughed screen, but with blinking cursor in it, so i will try some things .

nbhs

Member

From: Montevideo, Uruguay

Registered: 2013-05-02

Posts: 402

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

I finally got some progress.

After trying mainline kernel and with vbios, I went to bios/uefi and turned the Primary VGA card from onboard to pci-express ...
Now I see the seabios in passthroughed screen, but with blinking cursor in it, so i will try some things .

NIce! have you tried running windows with it yet? does it work if you dont load the rom file by hand?

apoapo

Member

Registered: 2013-05-16

Posts: 15

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

I finally got some progress.

After trying mainline kernel and with vbios, I went to bios/uefi and turned the Primary VGA card from onboard to pci-express ...
Now I see the seabios in passthroughed screen, but with blinking cursor in it, so i will try some things .

NIce! have you tried running windows with it yet? does it work if you dont load the rom file by hand?

Hi,

I tried it again and I captured this error message if I use the PCI-e card es Primary Card:

qemu-system-x86_64: -device vfio-pci,host=01:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on: VFIO 0000:01:00.0 BAR 0 mmap unsupported. Performance may be slow

This was, when I got some seabios picture.

After some reboots I wanted to try it again, I was spammed with errors . Dont have them to show you now...  will look for it.

Now I'm With the onboard as primary Card and I found this in dmesg:

[    0.401391] DMAR: No ATSR found
[    0.401412] IOMMU 0 0xfed90000: using Queued invalidation
[    0.401413] IOMMU 1 0xfed91000: using Queued invalidation
[    0.401415] IOMMU: Setting RMRR:
[    0.401422] IOMMU: Setting identity map for device 0000:00:02.0 [0xbf800000 - 0xcf9fffff]
[    0.402590] IOMMU: Setting identity map for device 0000:00:1d.0 [0xbd568000 - 0xbd59afff]
[    0.402606] IOMMU: Setting identity map for device 0000:00:1a.0 [0xbd568000 - 0xbd59afff]
[    0.402619] IOMMU: Setting identity map for device 0000:00:14.0 [0xbd568000 - 0xbd59afff]
[    0.402628] IOMMU: Prepare 0-16MiB unity mapping for LPC
[    0.402635] IOMMU: Setting identity map for device 0000:00:1f.0 [0x0 - 0xffffff]
[    0.402717] PCI-DMA: Intel(R) Virtualization Technology for Directed I/O
[    0.402782] ------------[ cut here ]------------
[    0.402786] WARNING: at drivers/pci/search.c:46 pci_find_upstream_pcie_bridge+0x83/0x90()
[    0.402787] Hardware name: To Be Filled By O.E.M.
[    0.402788] Modules linked in:
[    0.402790] Pid: 1, comm: swapper/0 Not tainted 3.9.2-1-mainline #1
[    0.402791] Call Trace:
[    0.402796]  [<ffffffff81058090>] warn_slowpath_common+0x70/0xa0
[    0.402797]  [<ffffffff8105817a>] warn_slowpath_null+0x1a/0x20
[    0.402799]  [<ffffffff8129ea43>] pci_find_upstream_pcie_bridge+0x83/0x90
[    0.402802]  [<ffffffff813b3c7b>] intel_iommu_add_device+0x4b/0x1f0
[    0.402805]  [<ffffffff813ab630>] ? bus_set_iommu+0x50/0x50
[    0.402806]  [<ffffffff813ab65a>] add_iommu_group+0x2a/0x50
[    0.402809]  [<ffffffff8135bf63>] bus_for_each_dev+0x63/0xa0
[    0.402811]  [<ffffffff813ab622>] bus_set_iommu+0x42/0x50
[    0.402814]  [<ffffffff819130b6>] intel_iommu_init+0xa83/0xb7b
[    0.402816]  [<ffffffff818dd10b>] ? memblock_find_dma_reserve+0x124/0x124
[    0.402818]  [<ffffffff818dd11d>] pci_iommu_init+0x12/0x3c
[    0.402820]  [<ffffffff8100210a>] do_one_initcall+0x10a/0x160
[    0.402823]  [<ffffffff818d5037>] kernel_init_freeable+0x15b/0x1dc
[    0.402824]  [<ffffffff818d4881>] ? do_early_param+0x88/0x88
[    0.402827]  [<ffffffff814b55f0>] ? rest_init+0x90/0x90
[    0.402828]  [<ffffffff814b55fe>] kernel_init+0xe/0x190
[    0.402830]  [<ffffffff814dad6c>] ret_from_fork+0x7c/0xb0
[    0.402832]  [<ffffffff814b55f0>] ? rest_init+0x90/0x9

nbhs

Member

From: Montevideo, Uruguay

Registered: 2013-05-02

Posts: 402

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Pease take a look at http://qemu.11.n7.nabble.com/VFIO-VGA-t … 01483.html, @Knut Omang this guy had it working using 2 discrete cards but he had to disable the integrated intel gpu

Last edited by nbhs (2013-05-17 01:57:12)

nbhs

Member

From: Montevideo, Uruguay

Registered: 2013-05-02

Posts: 402

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

You should set you primary boot device as IGP instead of pcie, vfio-vga wont work on you primary vga device.

lrwxrwxrwx 1 root root    0 17. Mai 00:53 0000:00:01.0 -> ../../../../devices/pci0000:00/0000:00:01.0
lrwxrwxrwx 1 root root    0 17. Mai 00:53 0000:01:00.0 -> ../../../../devices/pci0000:00/0000:00:01.0/0000:01:00.0
lrwxrwxrwx 1 root root    0 17. Mai 00:53 0000:01:00.1 -> ../../../../devices/pci0000:00/0000:00:01.0/0000:01:00.1
--w------- 1 root root 4096 17. Mai 00:53 bind
lrwxrwxrwx 1 root root    0 17. Mai 00:53 module -> ../../../../module/vfio_pci
--w------- 1 root root 4096 17. Mai 00:53 new_id
--w------- 1 root root 4096 17. Mai 00:53 remove_id
--w------- 1 root root 4096 17. Mai 00:48 uevent
--w------- 1 root root 4096 17. Mai 00:53 unbind

Looking back at it, i dont think your pcie root port should be owned by vfio, here's mine:

lrwxrwxrwx 1 root root    0 may 16 23:31 0000:07:00.0 -> ../../../../devices/pci0000:00/0000:00:0b.0/0000:07:00.0
lrwxrwxrwx 1 root root    0 may 16 23:31 0000:07:00.1 -> ../../../../devices/pci0000:00/0000:00:0b.0/0000:07:00.1
--w------- 1 root root 4096 may 16 23:31 bind
lrwxrwxrwx 1 root root    0 may 16 23:31 module -> ../../../../module/vfio_pci
--w------- 1 root root 4096 may 16 16:58 new_id
--w------- 1 root root 4096 may 16 23:31 remove_id
--w------- 1 root root 4096 may 16 16:58 uevent
--w------- 1 root root 4096 may 16 23:31 unbind

Anyway you could try playing around with intel_iommu kernel options like intel_iommu=on,igfx_off,pass-through, if it doesnt work you should fill a bug report on qemu-devel

Last edited by nbhs (2013-05-17 02:43:50)

apoapo

Member

Registered: 2013-05-16

Posts: 15

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

The Root-Port is in the same iommu-group with the graphicscard. I was wondering myself too

nbhs

Member

From: Montevideo, Uruguay

Registered: 2013-05-02

Posts: 402

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Have you tried setting intel_iommu=on,igfx_off or   intel_iommu=on,igfx_on with igd as primary?

apoapo

Member

Registered: 2013-05-16

Posts: 15

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Have you tried setting intel_iommu=on,igfx_off or   intel_iommu=on,igfx_on with igd as primary?

No change till now.
Will go to sleep now... its 5:00 in the morning here

Thx for your help . Will follow the conversation you linked me + try again with a second graphic card.
You will hear from me

PS:
some logoutput I don't have all the time at starting kvm (with intel GPU as primary):

[  109.801109] vfio-pci 0000:01:00.0: enabling device (0000 -> 0003)
[  109.827474] vfio_ecap_init: 0000:01:00.0 hiding ecap 0x19@0x270
[  109.827481] vfio_ecap_init: 0000:01:00.0 hiding ecap 0x1b@0x2d0
[  110.087849] vfio-pci 0000:01:00.1: enabling device (0000 -> 0002)

Last edited by apoapo (2013-05-31 13:27:14)

jgott

Member

Registered: 2011-04-10

Posts: 21

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Could you try to load your gpu bios by file and see if it still crash? i remember a couple of times my vm crashed after a reset ( before i accidentally physically destroyed the bios chip , now it wont work without loading a rom file ) and after i manually loaded the rom, didnt happen again, you can do this by appending ',rombar=0, romfile=/path/to/the/rom.bin'

I finally had time to give this a try, and it doesn't seem to make a difference. The VM works fine the first time, but the second time, the host freezes up as the guest boots. I tried a ROM image obtained via sysfs and one from techpowerup.com (different version for the same card); no visible difference.

What I may try next is swapping the cards. This isn't the configuration I'll want to use in the long term, but it will be interesting to see if the 5750 behaves any differently from the 7870 when passed through.

Incidentally, secondary passthrough (-vga cirrus or -vga std) doesn't cause any additional problems, which is nice, although it doesn't fix anything either.

jgott

Member

Registered: 2011-04-10

Posts: 21

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

What I may try next is swapping the cards. This isn't the configuration I'll want to use in the long term, but it will be interesting to see if the 5750 behaves any differently from the 7870 when passed through.

So I just gave this a try: 7870 as primary, 5750 as passthru. Started out without the ROM override. First boot, the guest works great. Second boot, the VM won't start, but I get this line in dmesg:

vfio-pci 0000:02:00.0: Invalid ROM contents

This didn't happen previously. Overriding the ROM on the second boot seems to get past this problem, but then the host freezes up, just like before.

nbhs

Member

From: Montevideo, Uruguay

Registered: 2013-05-02

Posts: 402

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Have you tried the "eject" method yet?

I'm still wondering why my particular card doesnt have this issue, i used to get performance degradation without reset patches but my host never crashed.

Are you sure the kernel and qemu are patched?

Last edited by nbhs (2013-05-17 06:24:34)

jgott

Member

Registered: 2011-04-10

Posts: 21

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Have you tried the "eject" method yet?

I'm still wondering why my particular card doesnt have this issue, i used to get performance degradation without reset patches but my host never crashed.

That's on my list of things to do, but first I'm going to reply to the qemu-devel mailing list.

I want to figure out why the pci reset from software isn't happening (presumably it is not happening, since suspending the host makes everything work properly). I don't know if it's possible to divine from the DEBUG_VFIO output why the reset attempt failed, or if the hardware is just silently ignoring the request.

Are you sure the kernel and qemu are patched?

Yes.

Last edited by jgott (2013-05-17 06:31:22)

jgott

Member

Registered: 2011-04-10

Posts: 21

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Oops. It seems that I forgot to change the pci-stub vendor and device IDs when I swapped cards. X wasn't running, but the radeon module might have been loaded. So maybe the card-swapping results were invalid. I don't want to switch everything over again though, what a pain.

apoapo

Member

Registered: 2013-05-16

Posts: 15

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

I put in a second graphicscard and tested it again.
The good thing is. I had no problems to pass it to the VM... have seen the bios of the Card + seabios on the screen + Windows was booting (was booting to system repair, but I think that's because of other storage type)

The bad thing is, that both graphicscards are in the same iommu-group + 2 PCI-Controller 01.0 and 01.1. Think that is because my 2 PCIe Ports share share the 16 lanes, that are normally used by the first port.
So I think I will be abled to pass the card to the VM, but can't really use the primary card for Linux and I have to wait to get the Intel as Primary working ...

Any other thoughts about this?

nbhs

Member

From: Montevideo, Uruguay

Registered: 2013-05-02

Posts: 402

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

I put in a second graphicscard and tested it again.
The good thing is. I had no problems to pass it to the VM... have seen the bios of the Card + seabios on the screen + Windows was booting (was booting to system repair, but I think that's because of other storage type)

The bad thing is, that both graphicscards are in the same iommu-group + 2 PCI-Controller 01.0 and 01.1. Think that is because my 2 PCIe Ports share share the 16 lanes, that are normally used by the first port.
So I think I will be abled to pass the card to the VM, but can't really use the primary card for Linux and I have to wait to get the Intel as Primary working ...

Any other thoughts about this?

Unfortunately, no, your best bet at this point is fill a bug @ qemu-devel, if you're using my script to bind the devices, you could try doing it manually, only binding the gpu and the gpu_snd, but i dont think vfio will work if all the devices of the same group are not binded to it.