Arch Linux

slis

Member

Registered: 2014-06-02

Posts: 127

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

so... what does -smp intends to do ?

And, how I could block internet connection for one guest and let the other enjoy the internet ?

-smp is number of processor sockets

iptables drop rule?

smth like

iptables -A FORWARD -s client ip -d 0.0.0.0/0 -j DROP
iptables -A FORWARD -s client ip -d LANsubnet -j ACCEPT

maybe... depending on your network configuration....

Last edited by slis (2014-07-11 08:16:16)

MCMjolnir

Member

Registered: 2014-06-19

Posts: 9

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Has anyone been able to get the HDMI audio working?

slis

Member

Registered: 2014-06-02

Posts: 127

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Has anyone been able to get the HDMI audio working?

what do you mean? on nvidia it works i think... but i have some latency issues maybe cause of pcie3.0 and sandy bridge... still waiting for ivy to arrive...
on 560ti pcie2.0 card it works flawless...

Last edited by slis (2014-07-11 08:26:02)

penetal

Member

Registered: 2014-06-15

Posts: 10

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

I am trying to create an audio infterface on the VM, but I end up with the scratchy / crackling sound problem that some others have come across in this thread.

I tried to set the sample rate and driver for QEMU, but that resulted in the problem becomming 100 times worse so I commented it out.
Startup Script:

#!/bin/sh

vfio-bind 0000:01:00.0 0000:01:00.1

#export QEMU_PA_SAMPLES=128
#export QEMU_AUDIO_DRV=pa
qemu-system-x86_64 -enable-kvm -M q35 -m 10240 -cpu host \
        -smp 6,sockets=1,cores=3,threads=2 \
        -bios /usr/share/qemu/bios.bin \
        -device ioh3420,bus=pcie.0,addr=1c.0,multifunction=on,port=1,chassis=1,id=root.1 \
        -device vfio-pci,host=01:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on \
        -device vfio-pci,host=01:00.1,bus=root.1,addr=00.1 \
        -drive file=/home/USER/vm/GamingWindows.img,id=disk,format=raw -device ide-hd,bus=ide.0,drive=disk \
        -vga none \
        -device ich9-intel-hda,bus=pcie.0,addr=1b.0,id=sound0 \
        -device hda-duplex,id=sound0-codec0,bus=sound0.0,cad=0 \
        -usb -usbdevice host:046d:c52b \
        -monitor stdio

I also disabled nested tables in kvm_intel:

% systool -vm kvm_intel
Module = "kvm_intel"

  Attributes:
    coresize            = "135528"
    initsize            = "0"
    initstate           = "live"
    refcnt              = "0"
    taint               = ""
    uevent              = <store method only>

  Parameters:
    emulate_invalid_guest_state= "Y"
    enable_apicv        = "N"
    enable_shadow_vmcs  = "Y"
    ept                 = "Y"
    eptad               = "Y"
    fasteoi             = "Y"
    flexpriority        = "Y"
    nested              = "N"
    ple_gap             = "128"
    ple_window          = "4096"
    unrestricted_guest  = "Y"
    vmm_exclusive       = "Y"
    vpid                = "Y"

  Sections:
    .bss                = "0xffffffffa00a7140"
    .data               = "0xffffffffa00a6800"
    .data..read_mostly  = "0xffffffffa00a6e68"
    .data.unlikely      = "0xffffffffa00a6ea0"
    .exit.text          = "0xffffffffa00971d2"
    .fixup              = "0xffffffffa0095bdc"
    .gnu.linkonce.this_module= "0xffffffffa00a6ec0"
    .init.text          = "0xffffffffa00ad000"
    .note.gnu.build-id  = "0xffffffffa0097268"
    .parainstructions   = "0xffffffffa00a5800"
    .rodata             = "0xffffffffa00972c0"
    .rodata.str1.1      = "0xffffffffa00a5bf0"
    .rodata.str1.8      = "0xffffffffa00a5cc8"
    .smp_locks          = "0xffffffffa00a5b8c"
    .strtab             = "0xffffffffa00b0b30"
    .symtab             = "0xffffffffa00add10"
    .text               = "0xffffffffa008a000"
    .text.unlikely      = "0xffffffffa00959d0"
    __bug_table         = "0xffffffffa00a6089"
    __ex_table          = "0xffffffffa00a5160"
    __jump_table        = "0xffffffffa00a6cd0"
    __mcount_loc        = "0xffffffffa00a6248"
    __param             = "0xffffffffa00a60a8"
    __verbose           = "0xffffffffa00a6e78"

Has anyone been able to solve this problem when those two things didn't help?

Edit:
I'm becomming unsure if the nested option is refering to the page tables or something else.

Last edited by penetal (2014-07-11 17:28:44)

Midov

Member

Registered: 2013-10-22

Posts: 9

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Has anyone tried to get snapshots working?
It seems that when i use -M -q35 snapshoting results in an -22 error.
I do have qcow2 disk

rafalcieslak

Member

Registered: 2014-07-04

Posts: 4

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

So I am into the fourth week of experimenting with passing through a GTX770 (well, apparently a number of others were successful with it!).
The card appars in the device manager of a windows7, where it is recognized as a "Standard VGA Graphics Adapter". Seems like it got passed through well.
However, it's status is:

This device cannot start. (Code 10)

Any nvidia driver fails to install, claiming it "failed to find supported hardware".
This is the closest I got to  a working setup. Has anyone seen that code 10 error? Google searching has not provided me with anything useful about this problem.  What could I make out of it? What should I investigate?

Last edited by rafalcieslak (2014-07-12 20:01:25)

slis

Member

Registered: 2014-06-02

Posts: 127

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

So I am into the fourth week of experimenting with passing through a GTX770 (well, apparently a number of others were successful with it!).
The card appars in the device manager of a windows7, where it is recognized as a "Standard VGA Graphics Adapter". Seems like it got passed through well.
However, it's status is:

This device cannot start. (Code 10)

Any nvidia driver fails to install, claiming it "failed to find supported hardware".
This is the closest I got to  a working setup. Has anyone seen that code 10 error? Google searching has not provided me with anything useful about this problem.  What could I make out of it? What should I investigate?

post your configs and hardware specs?

error code 10 looks like its not passed correctly...

rafalcieslak

Member

Registered: 2014-07-04

Posts: 4

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

post your configs and hardware specs?

error code 10 looks like its not passed correctly...

Okay, there it goes.

The hardware:

Intel 4790K (with an integrated intel HD graphics 4600)
NVIDIA GTX 770 (Gigabyte)
ASRock Z97 Extreme6

I use stock kernel 3.15.3 configured with:

$ cat .config | grep -i vfi
CONFIG_VFIO_IOMMU_TYPE1=m
CONFIG_VFIO=m
CONFIG_VFIO_PCI=m
CONFIG_VFIO_PCI_VGA=y
CONFIG_KVM_VFIO=y

and patched with vga arbiter fixes (inclucing these for i915) and acs overrides.

Therefore, my /etc/modules contains:

vfio
vfio_iommu_type1
vfio-pci

The setup I use is meant to get the host system to use the the intel GPU, and I am passing the nvidia gtx 770 to the guest.
BIOS is set to use intel GPU as the primary graphics adapter (to ensure it won't initialize the gtx 770).

My kenel boot commandline:

linux	/vmlinuz-3.15.3 root=UUID=3b7c22f4-85dc-480c-90e7-58e743c1b5d8 ro nvidia.blacklist=yes nvidia-current.blacklist=yes nvidia_337.blacklist=yes nvidiafb.blacklist=yes i915.enable_hd_vgaarb=1 intel_iommu=on pci-stub.ids=10de:1184,10de:0e0a  quiet

(I might skip some of these blacklists, these are just to force the host to leave the gtx770 untoched).
I also use

options kvm allow_unsafe_assigned_interrupts=1

Host:

$ lspci | grep VGA
00:02.0 VGA compatible controller: Intel Corporation Xeon E3-1200 v3/4th Gen Core Processor Integrated Graphics Controller (rev 06)
01:00.0 VGA compatible controller: NVIDIA Corporation GK104 [GeForce GTX 770] (rev a1)
$ lspci | grep NVIDIA
01:00.0 VGA compatible controller: NVIDIA Corporation GK104 [GeForce GTX 770] (rev a1)
01:00.1 Audio device: NVIDIA Corporation GK104 HDMI Audio Controller (rev a1)
$ lspci -n | grep 01:00
01:00.0 0300: 10de:1184 (rev a1)
01:00.1 0403: 10de:0e0a (rev a1)

I am working with qemu pulled from git.
Before I run qemu, I do:

vfio-bind 0000:01:00.0 0000:01:00.1

I start qemu with:

qemu-system-x86_64 -enable-kvm -M q35 -m 4096 -cpu host \
 -smp cores=4 \
 -bios /usr/share/qemu/bios.bin \
 -device ioh3420,bus=pcie.0,addr=1c.0,multifunction=on,port=1,chassis=1,id=root.1 \
 -device vfio-pci,host=01:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on \
 -device vfio-pci,host=01:00.1,bus=root.1,addr=00.1 \
 -monitor stdio \
 -vga cirrus \
 -boot menu=on \
 -drive file=/dev/sdb3,format=raw,media=disk,id=disk1 \
 -drive file=/dev/mapper/main-win,format=raw,media=disk,id=disk2 \
 -device ide-hd,bus=ide.0,drive=disk1 \
 -device ide-hd,bus=ide.1,drive=disk2

It seems to work, so the host indeed never uses the nvidia GPU, there are no problems mentioned in dmesg, vfio binding is successful, and qemu never complains about a thing.
A linux distro installed as a guest apparently detects the GPU, running lspci I get, among other entries:

01:00.0 VGA compatible controller: NVIDIA Corporation GK104 [GeForce GTX 770] (rev a1)

which is probably cool.

I will repeat here (for reference and completeness) that a windows7 installed there will detect the passed through device as a "Standard VGA Graphics Adapter" in its Device Manager, but the status of this devices is

This device cannot start. (Code 10)

and the nvidia drivers fail to install, as it "cannot find supported hardware".

Other configurations I have tried that did not help in any combination
kvm=off does not change anything.
Using a romfile for my graphics card (as apparently many others do).
vga -none  is probably quite pointless before I get the drivers to install, there is never any output from the passed GPU.

I am looking both for ideas on what might be wron in my setup, as well as hints on how should I troubleshoot it to investigate the problem further.
Cheers!

Last edited by rafalcieslak (2014-07-13 10:56:20)

slis

Member

Registered: 2014-06-02

Posts: 127

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

-monitor stdio \
-vga cirrus \
-boot menu=on \

u cant pass it with video cirrus .... u must disable that with -vga none or smth like that... that is error code 10. i think,
u must have SEABIOS screen through 770 when it works with -vga none and i915 patches<<<

hints:
(aka 770 must be PRIMARY card not secondary with cirrus, it can be secondary if u mod it to quadro/grid)

geforce 6xx+ doesn't need rom file... 5xx or lower needs...

your geforce must work regradless of driver installed.... you must have seabios screen on gtx770

try lower driver than 340 aka 331 to be sure driver works with -vga none ofc....

here is my xml with all options for best perfomance i think... i am only misssing x-data-plane option for hdd i guess libvirt still doesn't support it...

this setup works for 2 weeks 0-24, win 8.1 x64, debian 3.15, z77, dual vm, dual 680gtx, used for streaming win games...

<domain type='kvm' xmlns:qemu='http://libvirt.org/schemas/domain/qemu/1.0'>                                         
  <name>GTX680</name>                                                                                               
  <uuid>00000000-0000-0000-0000-000000000001</uuid>                                                                 
  <memory unit='KiB'>6242304</memory>                                                                               
  <currentMemory unit='KiB'>6242304</currentMemory>                                                                 
  <vcpu placement='static' cpuset='0-7'>4</vcpu>                                                                     
  <os>                                                                                                               
    <type arch='x86_64' machine='pc-q35-2.1'>hvm</type>                                                             
    <boot dev='hd'/>                                                                                                 
    <bootmenu enable='no'/>                                                                                         
  </os>                                                                                                             
  <features>                                                                                                         
    <acpi/>                                                                                                         
    <apic/>                                                                                                         
    <hap/>                                                                                                           
    <hyperv>                                                                                                         
      <relaxed state='on'/>                                                                                         
      <vapic state='on'/>                                                                                           
      <spinlocks state='on' retries='4096'/>                                                                         
    </hyperv>                                                                                                       
    <pvspinlock state='on'/>                                                                                         
  </features>
  <cpu mode='custom' match='exact'>
    <model fallback='allow'>Haswell</model>
    <vendor>Intel</vendor>
    <topology sockets='1' cores='4' threads='1'/>
    <feature policy='force' name='rtm'/>
    <feature policy='force' name='invpcid'/>
    <feature policy='force' name='lahf_lm'/>
    <feature policy='force' name='erms'/>
    <feature policy='force' name='bmi2'/>
    <feature policy='force' name='smep'/>
    <feature policy='force' name='avx2'/>
    <feature policy='force' name='hle'/>
    <feature policy='force' name='bmi1'/>
    <feature policy='force' name='fsgsbase'/>
    <feature policy='force' name='movbe'/>
    <feature policy='force' name='fma'/>
  </cpu>
  <clock offset='utc'>
    <timer name='hypervclock' present='yes'/>
  </clock>
  <on_poweroff>destroy</on_poweroff>
  <on_reboot>restart</on_reboot>
  <on_crash>destroy</on_crash>
  <devices>
    <emulator>/usr/local/bin/qemu-system-x86_64</emulator>
    <disk type='file' device='disk'>
      <driver name='qemu' type='raw' cache='none' io='native'/>
      <source file='/root/win1.img'/>
      <target dev='vda' bus='virtio'/>
      <address type='pci' domain='0x0000' bus='0x02' slot='0x03' function='0x0'/>
    </disk>
    <disk type='file' device='disk'>
      <driver name='qemu' type='raw' cache='none' io='native'/>
      <source file='/root/storage/data1.img'/>
      <target dev='vdb' bus='virtio'/>
      <address type='pci' domain='0x0000' bus='0x02' slot='0x04' function='0x0'/>
    </disk>
    <controller type='pci' index='0' model='pcie-root'/>
    <controller type='pci' index='1' model='dmi-to-pci-bridge'>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x1e' function='0x0'/>
    </controller>
    <controller type='pci' index='2' model='pci-bridge'>
      <address type='pci' domain='0x0000' bus='0x01' slot='0x01' function='0x0'/>
    </controller>
    <controller type='pci' index='3' model='pci-bridge'>
      <address type='pci' domain='0x0000' bus='0x01' slot='0x02' function='0x0'/>
    </controller>
    <controller type='usb' index='0'>
      <address type='pci' domain='0x0000' bus='0x02' slot='0x01' function='0x0'/>
    </controller>
    <controller type='sata' index='0'>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x1f' function='0x2'/>
    </controller>
    <interface type='bridge'>
      <mac address='52:54:00:12:34:01'/>
      <source bridge='br0'/>
      <model type='virtio'/>
      <rom bar='off'/>
      <address type='pci' domain='0x0000' bus='0x02' slot='0x02' function='0x0'/>
    </interface>
    <hostdev mode='subsystem' type='usb' managed='yes'>
      <source>
        <vendor id='0x1532'/>
        <product id='0x010b'/>
      </source>
    </hostdev>
    <hostdev mode='subsystem' type='usb' managed='yes'>
      <source>
        <vendor id='0x1532'/>
        <product id='0x000a'/>
      </source>
    </hostdev>
    <memballoon model='virtio'>
      <address type='pci' domain='0x0000' bus='0x02' slot='0x05' function='0x0'/>
    </memballoon>
  </devices>
  <seclabel type='none'/>
  <qemu:commandline>
    <qemu:arg value='-device'/>
    <qemu:arg value='ioh3420,bus=pcie.0,addr=1c.0,multifunction=on,port=1,chassis=1,id=root.1'/>
    <qemu:arg value='-device'/>
    <qemu:arg value='vfio-pci,host=04:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on'/>
    <qemu:arg value='-device'/>
    <qemu:arg value='vfio-pci,host=04:00.1,bus=root.1,addr=00.1'/>
    <qemu:arg value='-device'/>
    <qemu:arg value='ich9-intel-hda,bus=pcie.0,addr=1b.0,id=sound0'/>
    <qemu:arg value='-device'/>
    <qemu:arg value='hda-duplex,id=sound0-codec0,bus=sound0.0,cad=0'/>
    <qemu:arg value='-set'/>
    <qemu:arg value='device.virtio-disk0.scsi=off'/>
    <qemu:arg value='-set'/>
    <qemu:arg value='device.virtio-disk0.config-wce=off'/>
    <qemu:env name='DISPLAY' value=':0'/>
  </qemu:commandline>
</domain>

Last edited by slis (2014-07-13 12:49:10)

buraktamturk

Member

Registered: 2014-05-28

Posts: 13

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

I have a problem with opengl driver. I am using intel i915 on host and ati radeon 5400 on guest, i applied "i915_315.patch" and enabled "i915.enable_hd_vgaarb=1" in kernel procline. DRI is working without vgaarb patch, not working with vaarb patch.

I am pretty sure i was able to play game (Don't Starve) on host, while guest is running with ati radeon 5400 card. But can't play now i don't know what problem is it.

From Xorg log:

(WW) intel(0): [DRI2] Direct rendering is not supported when VGA arb is necessary for the device

Is there any patch that i miss?

Thank you.

rafalcieslak

Member

Registered: 2014-07-04

Posts: 4

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

slis, huge thanks for your hints! With this help and some slight experiments, I have managed to get my setup working completely. The performance is pretty good, even without tuning.
One particular thing I would like to mention for others who might experience similar problems is that it would seem that I have missed

pcie_acs_override=downstream

in my kernel command line. I am not entirely sure why would be necessary in my setup, but it never works without it, so it is apparently crucial.

Now I am stuck with maintaining my windows installation, as it boots correctly for the first 10-20 times after installing, and then it refuses to cooperate and crashes on boot But that is another story... Passthrough is working well, thanks all for the aboundance of information in this topic!

slis

Member

Registered: 2014-06-02

Posts: 127

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

pcie_acs_override=downstream

you need that if u have multiple devices / gpu-s that use same group behind pcie bridge... so u separate them with it....

Last edited by slis (2014-07-14 11:38:07)

buraktamturk

Member

Registered: 2014-05-28

Posts: 13

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Since i can't use DRI with vga arb patch, I attached a nvidia gpu to the host, and installed non-free nvidia drivers from nvidia website. (340.24)

01:00.0 VGA compatible controller: Advanced Micro Devices, Inc. [AMD/ATI] Device 68f0
01:00.1 Audio device: Advanced Micro Devices, Inc. [AMD/ATI] Cedar HDMI Audio [Radeon HD 5400/6300 Series]
02:00.0 VGA compatible controller: NVIDIA Corporation GF114 [GeForce GTX 560] (rev a1)
02:00.1 Audio device: NVIDIA Corporation GF114 HDMI Audio Controller (rev a1)

But when i try to attach ATI to guest. I get

[  710.872104] vfio-pci 0000:01:00.0: Invalid ROM contents

I have dumped ati card rom, and appened romfile=/home/buraktamturk/rom.rom to qemu command line.

    <qemu:arg value='-device'/>
    <qemu:arg value='ioh3420,bus=pcie.0,addr=1c.0,multifunction=on,port=1,chassis=1,id=root.1'/>
    <qemu:arg value='-device'/>
    <qemu:arg value='vfio-pci,host=01:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on,romfile=/home/buraktamturk/rom.rom'/>
    <qemu:arg value='-device'/>
    <qemu:arg value='vfio-pci,host=01:00.1,bus=root.1,addr=00.1'/>

Now guest starts with no video. No invalid rom contents error either.

NOTE: with old configuration, gpu passthrought was working without romfile, and after appending romfile to old working configuration, no video shows up too, it's was just fine without romfile.

Any throughts?

Thank you.

andy123

Member

Registered: 2011-11-04

Posts: 169

Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Since i can't use DRI with vga arb patch, I attached a nvidia gpu to the host, and installed non-free nvidia drivers from nvidia website. (340.24)

01:00.0 VGA compatible controller: Advanced Micro Devices, Inc. [AMD/ATI] Device 68f0
01:00.1 Audio device: Advanced Micro Devices, Inc. [AMD/ATI] Cedar HDMI Audio [Radeon HD 5400/6300 Series]
02:00.0 VGA compatible controller: NVIDIA Corporation GF114 [GeForce GTX 560] (rev a1)
02:00.1 Audio device: NVIDIA Corporation GF114 HDMI Audio Controller (rev a1)

But when i try to attach ATI to guest. I get

[  710.872104] vfio-pci 0000:01:00.0: Invalid ROM contents

I have dumped ati card rom, and appened romfile=/home/buraktamturk/rom.rom to qemu command line.

    <qemu:arg value='-device'/>
    <qemu:arg value='ioh3420,bus=pcie.0,addr=1c.0,multifunction=on,port=1,chassis=1,id=root.1'/>
    <qemu:arg value='-device'/>
    <qemu:arg value='vfio-pci,host=01:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on,romfile=/home/buraktamturk/rom.rom'/>
    <qemu:arg value='-device'/>
    <qemu:arg value='vfio-pci,host=01:00.1,bus=root.1,addr=00.1'/>

Now guest starts with no video. No invalid rom contents error either.

NOTE: with old configuration, gpu passthrought was working without romfile, and after appending romfile to old working configuration, no video shows up too, it's was just fine without romfile.

Any throughts?

Thank you.

Nvidia vga arbiter patch as mentioned in the original post?

---

i'm sorry for my poor english wirting skills…

buraktamturk

Member

Registered: 2014-05-28

Posts: 13

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Since i can't use DRI with vga arb patch, I attached a nvidia gpu to the host, and installed non-free nvidia drivers from nvidia website. (340.24)

01:00.0 VGA compatible controller: Advanced Micro Devices, Inc. [AMD/ATI] Device 68f0
01:00.1 Audio device: Advanced Micro Devices, Inc. [AMD/ATI] Cedar HDMI Audio [Radeon HD 5400/6300 Series]
02:00.0 VGA compatible controller: NVIDIA Corporation GF114 [GeForce GTX 560] (rev a1)
02:00.1 Audio device: NVIDIA Corporation GF114 HDMI Audio Controller (rev a1)

But when i try to attach ATI to guest. I get

[  710.872104] vfio-pci 0000:01:00.0: Invalid ROM contents

I have dumped ati card rom, and appened romfile=/home/buraktamturk/rom.rom to qemu command line.

    <qemu:arg value='-device'/>
    <qemu:arg value='ioh3420,bus=pcie.0,addr=1c.0,multifunction=on,port=1,chassis=1,id=root.1'/>
    <qemu:arg value='-device'/>
    <qemu:arg value='vfio-pci,host=01:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on,romfile=/home/buraktamturk/rom.rom'/>
    <qemu:arg value='-device'/>
    <qemu:arg value='vfio-pci,host=01:00.1,bus=root.1,addr=00.1'/>

Now guest starts with no video. No invalid rom contents error either.

NOTE: with old configuration, gpu passthrought was working without romfile, and after appending romfile to old working configuration, no video shows up too, it's was just fine without romfile.

Any throughts?

Thank you.

Nvidia vga arbiter patch as mentioned in the original post?

Thank you, i didn't applied such thing.

How and where i can apply it? Should i apply it to the linux kernel or nvidia driver installer? How i can recompile kernel module to patch it?

Thank you.

buraktamturk

Member

Registered: 2014-05-28

Posts: 13

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Since i can't use DRI with vga arb patch, I attached a nvidia gpu to the host, and installed non-free nvidia drivers from nvidia website. (340.24)

01:00.0 VGA compatible controller: Advanced Micro Devices, Inc. [AMD/ATI] Device 68f0
01:00.1 Audio device: Advanced Micro Devices, Inc. [AMD/ATI] Cedar HDMI Audio [Radeon HD 5400/6300 Series]
02:00.0 VGA compatible controller: NVIDIA Corporation GF114 [GeForce GTX 560] (rev a1)
02:00.1 Audio device: NVIDIA Corporation GF114 HDMI Audio Controller (rev a1)

But when i try to attach ATI to guest. I get

[  710.872104] vfio-pci 0000:01:00.0: Invalid ROM contents

I have dumped ati card rom, and appened romfile=/home/buraktamturk/rom.rom to qemu command line.

    <qemu:arg value='-device'/>
    <qemu:arg value='ioh3420,bus=pcie.0,addr=1c.0,multifunction=on,port=1,chassis=1,id=root.1'/>
    <qemu:arg value='-device'/>
    <qemu:arg value='vfio-pci,host=01:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on,romfile=/home/buraktamturk/rom.rom'/>
    <qemu:arg value='-device'/>
    <qemu:arg value='vfio-pci,host=01:00.1,bus=root.1,addr=00.1'/>

Now guest starts with no video. No invalid rom contents error either.

NOTE: with old configuration, gpu passthrought was working without romfile, and after appending romfile to old working configuration, no video shows up too, it's was just fine without romfile.

Any throughts?

Thank you.

Nvidia vga arbiter patch as mentioned in the original post?

Now i applied this patch and i am getting the same error. "Invalid ROM contents" error without romfile, no video with romfile.

Old configuration was working without "romfile", when i give "romfile" to old configuration, no video again. I dumped romfile with windows host and GPU-Z program, may be i got wrong dump or the romfile that program generated isn't compatible with seabios?

Any throughts?

EDIT: The ATI card was half-connected to the motherboard, i didn't think such a situation because lspci showed it up. Now everything is working fine. Thanks!

Last edited by buraktamturk (2014-07-15 07:54:09)

_pheinrich_

Member

Registered: 2014-05-26

Posts: 53

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Hi,

I got it to work, but only if I am using xf86-video-vesa for my host system.
Otherwise if I am using nouveau I get a mass of DMAR Read errors.

DMAR:[fault reason 06] PTE Read access is not set
[   81.044278] dmar: DRHD: handling fault status reg 3
[   81.044279] dmar: DMAR:[DMA Read] Request device [01:00.0] fault addr 0 
DMAR:[fault reason 06] PTE Read access is not set
[   81.044921] dmar: DRHD: handling fault status reg 3
[   81.044923] dmar: DMAR:[DMA Read] Request device [01:00.0] fault addr 0 
DMAR:[fault reason 06] PTE Read access is not set
[   81.054085] dmar: DRHD: handling fault status reg 3
[   81.054086] dmar: DMAR:[DMA Read] Request device [01:00.0] fault addr 0 
DMAR:[fault reason 06] PTE Read access is not set
[   81.055395] dmar: DRHD: handling fault status reg 3
[   81.055398] dmar: DMAR:[DMA Read] Request device [01:00.0] fault addr 0 
DMAR:[fault reason 06] PTE Read access is not set
[   81.058005] dmar: DRHD: handling fault status reg 3
[   81.058007] dmar: DMAR:[DMA Read] Request device [01:00.0] fault addr 0 
DMAR:[fault reason 06] PTE Read access is not set
[   81.058659] dmar: DRHD: handling fault status reg 3
[   81.058661] dmar: DMAR:[DMA Read] Request device [01:00.0] fault addr 0 
DMAR:[fault reason 06] PTE Read access is not set
[   81.061882] dmar: DRHD: handling fault status reg 3
[   81.061883] dmar: DMAR:[DMA Read] Request device [01:00.0] fault addr 0 
DMAR:[fault reason 06] PTE Read access is not set
[  118.857410] dmar: DRHD: handling fault status reg 3
[  118.857425] dmar: DMAR:[DMA Write] Request device [07:00.0] fault addr a7aba000 
DMAR:[fault reason 05] PTE Write access is not set
[  118.876399] dmar: DRHD: handling fault status reg 3
[  118.876405] dmar: DMAR:[DMA Write] Request device [07:00.0] fault addr c9cdc000 
DMAR:[fault reason 05] PTE Write access is not set
[  118.876413] dmar: DRHD: handling fault status reg 3
[  118.876415] dmar: DMAR:[DMA Write] Request device [07:00.0] fault addr b1b5b000 
DMAR:[fault reason 05] PTE Write access is not set
[  118.876419] dmar: DRHD: handling fault status reg 3
[  118.876422] dmar: DMAR:[DMA Write] Request device [07:00.0] fault addr c1c5c000 
DMAR:[fault reason 05] PTE Write access is not set
[  158.195280] dmar: DRHD: handling fault status reg 3
[  158.195288] dmar: DMAR:[DMA Write] Request device [07:00.0] fault addr 1f472000

01:00.0 VGA compatible controller: NVIDIA Corporation NV43 [GeForce 6600 GT] (rev a2)
        Kernel driver in use: nouveau
        Kernel modules: nouveau
07:00.0 Multimedia controller: Twinhan Technology Co. Ltd Mantis DTV PCI Bridge Controller [Ver 1.0] (rev 01)
        Subsystem: TERRATEC Electronic GmbH Device 1178
        Kernel driver in use: Mantis
        Kernel modules: mantis

I have seen in this thread on page 63 some discussion about that problem to use intel_iommu=iglx_off but I think I am not able to passthrough VGA controllers with that option.

Is there a option to exclude devices from iommu since I only want to passthrouch pci 02:00:00?

d3lxa

Member

Registered: 2014-07-16

Posts: 4

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Hey guys, thanks for the great post! I would like to contribute what is working for me (everything for now, that's awesome). That's my personal notes:

Hardware:
    CPU: Intel i7-4770S
    MB : GA Z87N-WIFI
    GPU: Intel HD 4600 (linux, integrated)
    GPU: Asus HD 6850 (win7, dedicated)
        romfile (VERY IMPORTANT): Asus.HD6850.1024.110616.rom
        2011-06-16 22:24:00     013.012.000.021.000000  PCI-E   790 / 1000
        ^ http://www.techpowerup.com/vgabios/inde … =&memSize=
    require (1 for linux, 1 for windows):
        2 monitors + 2 keyboards + 2 mouses
        (^ or 1 monitor with two inputs)

Linux:
Debian Testing, linux-image-3.14-1-amd64=3.14.12-1 (compiled myself with CONFIG_VFIO_PCI_VGA=y CONFIG_HZ_1000=y)
    tested too: 3.15.5 (trunk + acs_override)
    bootline: quiet intel_iommu=on
        not necessary: vfio_iommu_type1.allow_unsafe_interrupts=1 pcie_acs_override=downstream i915.enable_hd_vgaarb=1 (corresponding patch not applied anyway)
    qemu: qemu-kvm=2.0.0+dfsg-6+b1
       or from git ab6d3749c4915cd5692633e321f7745dce06fe77
    xserver-xorg 1:7.7+7
    xserver-xorg-video-intel 2:2.21.15-2+b1
    libegl1-mesa-drivers 10.2.2-1
    firmware-linux 0.43
    firmware-linux-free 3.3

qemu line:
    QEMU_AUDIO_DRV=alsa QEMU_AUDIO_TIMER_PERIOD=0 ~/qemu-git/x86_64-softmmu/qemu-system-x86_64 \
        -enable-kvm -M q35 -m 1024 -cpu host -smp 4,sockets=1,cores=4,threads=1 \
        -bios /usr/share/qemu/bios.bin -vga none \
        -device ioh3420,bus=pcie.0,addr=1c.0,multifunction=on,port=1,chassis=1,id=root.1 \
        -device piix4-ide,bus=pcie.0,id=piix4-ide \
        -device vfio-pci,host=01:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on,romfile=$HOME/Asus.HD6850.1024.110616.rom \
        -device vfio-pci,host=01:00.1,bus=pcie.0 \
        -usb -usbdevice host:0603:00f2 -usbdevice host:046d:c01b \
        -soundhw ac97 \
        -drive file=$HOME/win7_rootfs.img,id=disk,format=raw -device ide-hd,bus=piix4-ide.0,drive=disk \
    ;

audio
    -soundhw ac97
        nearly perfect (some glitches)
        driver: http://www.realtek.com.tw/downloads/dow … Down=false
    -device usb-audio and QEMU_AUDIO_TIMER_PERIOD=10
        better but high CPU, may glitch under heavy load (or after closing program)
    -device ich9-intel-hda,bus=pcie.0,addr=1b.0,id=sound0 -device hda-duplex,id=sound0-codec0,bus=sound0.0,cad=0 \
        audio glitches (even with timer period to 0)

cdrom
    for installation (OS or drivers):
    -drive file=$HOME/win7.iso,id=isocd -device ide-cd,bus=piix4-ide.1,drive=isocd \
    can be removed safely after reboot (install)

disks
    virtio: can be switched to *after* fedora driver install
    https://alt.fedoraproject.org/pub/alt/v … mages/bin/
    install manually in windows by "updating" the driver of the unknown disk device

software tested:
    Firefox WebGL (google map)
    Pangya
    Natural Selection 2 (Steam)
    Media Player Classic (h264 video)

bugs:
    each time the VM boots, my linux display has color glitches (partial inversion), need to switch TTY (ctrl+alt+X) to reset colors (framebuffer)
    sound may glitch a bit in video playback

Windows 7 SP1 Ultimate (64 Bit)
    torrent: 7ef0145bc19f53251df9ff7e11a795e9e088b65c
    drivers: 14.10.1006-140417a-171099C (Overdrive works)
          or amd_13_9_win7_win8_64_dd_ccc_whql.exe
            ^ http://www.driversdown.com/drivers.asp? … m=3&show=0

Before starting qemu, I've to switch to the vfio driver for my two devices (GPU+HDMI sound card, the readeon driver is *not* loaded it's blacklisted):

# $ lspci | grep AMD
#01:00.0 VGA compatible controller [0300]: Advanced Micro Devices, Inc. [AMD/ATI] Barts PRO [Radeon HD 6850] [1002:6739]
#01:00.1 Audio device [0403]: Advanced Micro Devices, Inc. [AMD/ATI] Barts HDMI Audio [Radeon HD 6800 Series] [1002:aa88]
sudo modprobe vfio
sudo modprobe vfio_pci

echo '0000:01:00.1' | sudo tee /sys/bus/pci/devices/0000:01:00.1/driver/unbind
echo 1002 6739 | sudo tee /sys/bus/pci/drivers/vfio-pci/new_id
echo 1002 aa88 | sudo tee /sys/bus/pci/drivers/vfio-pci/new_id

One thing that would be more convenient is to share the keyboard+mouse between the two OSes with a software multiplexer: that would create two virtual devices, one for linux + one for windows and you could choose to what it's connected. For the monitors that's fine, one way would be to use VNC inside windows but I'm not sure of the overhead, wouldn't be nice. The network seems fine, I don't know if the default nic induce an important overhead, I didn't notice in game, seems fine.

I have tested the sound and it works (my microphone too! This is useful in NS2). The only problem of the whole is a minor bug when Windows init my card, it touches my linux buffer and changes some colors (kind of psychedelic) but that's once at boot, not that annoying. This looks a bit like what this guy had:

After a few seconds my screen looks like this http://imgur.com/G7yT4uZ.

You can see the harmless messages in my kern.log while the VM was running here (unsorted) http://paste.debian.net/110091/ .

Hope this helps!

Last edited by d3lxa (2014-07-16 13:37:55)

d3lxa

Member

Registered: 2014-07-16

Posts: 4

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

I am trying to create an audio infterface on the VM, but I end up with the scratchy / crackling sound problem that some others have come across in this thread.

I tried to set the sample rate and driver for QEMU, but that resulted in the problem becomming 100 times worse so I commented it out.
[…]
Has anyone been able to solve this problem when those two things didn't help?

Edit:
I'm becomming unsure if the nested option is refering to the page tables or something else.

I had this with the same settings you were using, try me version with -soundhw ac97 (remove the two "sound" lines) and don't forget to export the delay, not sure it works for PulseAudio though.

aw

Member

Registered: 2013-10-04

Posts: 921

Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Hey folks,

I've got some reset tweaks up for testing that might be useful to a number of you:

https://lkml.org/lkml/2014/7/16/562 (patch 1, patch 2, patch 3)

https://lkml.org/lkml/2014/7/16/567 (patch)

With these, the GPU should be reset regardless of how clean the guest was shutdown.  Note that all devices on the bus need to be bound to vfio-pci for this to work.  That means that if you're using pci-stub to sequester unused devices (perhaps the audio function), bind them to vfio-pci or there won't be any change.  The second set of links is a patch necessary for AMD GPU users only and I'd appreciate feedback on whether it solves and of the other reset issues (lack of reset) found on these GPUs.  Thanks

PS - these are kernel patches against the latest development kernel, 3.16-rc5.  They may or may not apply cleanly to older kernels.

---

http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

aw

Member

Registered: 2013-10-04

Posts: 921

Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

One thing that would be more convenient is to share the keyboard+mouse between the two OSes with a software multiplexer: that would create two virtual devices, one for linux + one for windows and you could choose to what it's connected. For the monitors that's fine, one way would be to use VNC inside windows but I'm not sure of the overhead, wouldn't be nice. The network seems fine, I don't know if the default nic induce an important overhead, I didn't notice in game, seems fine.

Synergy?

I have tested the sound and it works (my microphone too! This is useful in NS2). The only problem of the whole is a minor bug when Windows init my card, it touches my linux buffer and changes some colors (kind of psychedelic) but that's once at boot, not that annoying.

This is what the i915 patch and kernel option fixes that you've decided is not necessary.  It means that you won't get any VGA output on the guest, such as seabios or maybe Windows prompting how to startup after a crash.

---

http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Val532

Member

Registered: 2013-11-13

Posts: 35

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Hi,

@aw Thanks for the new patch, i will test for AsMedia Sata controler.
But i found a new strange bug with Renesas USB 3.0 controler, when it use with vfio-pci the device is activate with a lot of new msi/msi-x like 43, 44, 45, 46 ... around 20 new irq adresse.

mbroemme

Member

From: Cologne

Registered: 2014-02-05

Posts: 40

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Hey folks,

I've got some reset tweaks up for testing that might be useful to a number of you:

https://lkml.org/lkml/2014/7/16/562 (patch 1, patch 2, patch 3)

https://lkml.org/lkml/2014/7/16/567 (patch)

With these, the GPU should be reset regardless of how clean the guest was shutdown.  Note that all devices on the bus need to be bound to vfio-pci for this to work.  That means that if you're using pci-stub to sequester unused devices (perhaps the audio function), bind them to vfio-pci or there won't be any change.  The second set of links is a patch necessary for AMD GPU users only and I'd appreciate feedback on whether it solves and of the other reset issues (lack of reset) found on these GPUs.  Thanks

PS - these are kernel patches against the latest development kernel, 3.16-rc5.  They may or may not apply cleanly to older kernels.

I've tested them against 3.16-rc5 kernel. They apply clean, I've rebooted and re-tested my Linux + fglrx and Windows + catalyst with my R9 290X. It still suffers from the reset problem. On Linux + fglrx I get oops (inside VM) after reboot of VM and X start. On Windows + catalyst I still get the BSOD PAGE_FAULT_IN_NONPAGED_AREA.

Last edited by mbroemme (2014-07-16 23:30:29)

shrubuntu

Member

Registered: 2012-12-26

Posts: 19

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

I ran

dmesg | grep -e DMAR -e IOMMU

and got:

[    0.056332] IOAPIC id 8 under DRHD base  0xfed90000 IOMMU 0
rishub@debian:~$ dmesg | grep -e DMAR -e IOMMU 
[    0.000000] ACPI: DMAR 00000000de555810 000080 (v01 INTEL      BDW  00000001 INTL 00000001)
[    0.056269] dmar: IOMMU 0: reg_base_addr fed90000 ver 1:0 cap d2008c20660462 ecap f010da
[    0.056332] IOAPIC id 8 under DRHD base  0xfed90000 IOMMU 0
[    0.537854] AMD IOMMUv2 driver by Joerg Roedel <joerg.roedel@amd.com>
[    0.537854] AMD IOMMUv2 functionality not available on this system

I am running i7 4790k with Msi Gaming 5
In my bios, there is a vt-d option and the processor supports it

Do I have it running on my system? Is this output correct?

aw

Member

Registered: 2013-10-04

Posts: 921

Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

I've tested them against 3.16-rc5 kernel. They apply clean, I've rebooted and re-tested my Linux + fglrx and Windows + catalyst with my R9 290X. It still suffers from the reset problem. On Linux + fglrx I get oops (inside VM) after reboot of VM and X start. On Windows + catalyst I still get the BSOD PAGE_FAULT_IN_NONPAGED_AREA.

Bummer.  Is the oops a regression or is that something you see already?

---

http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users