Arch Linux

ttouch

Member

From: /dev/null

Registered: 2012-05-27

Posts: 130

Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Seems that I get "Windows has stopped this device because it has reported problems. (Code 43)". That may be the cause of the underscan problem.
I searched an other users also have this problem but I can't figure out how to solve it...
Any help appreciated

---

GitHub | Git Server | Blog

flack

Member

Registered: 2014-06-12

Posts: 7

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Hi I am here new in forum. Why you dont use somethink like this. In lsusb you can find your device vendorID and productID.

back@KVM-Ubuntu:~$ lsusb
Bus 002 Device 004: ID 058f:6362 Alcor Micro Corp. Flash Card Reader/Writer
Bus 002 Device 003: ID 9710:7830 MosChip Semiconductor MCS7830 10/100 Mbps Ethernet adapter
Bus 002 Device 002: ID 8087:0024 Intel Corp. Integrated Rate Matching Hub
Bus 002 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub
Bus 006 Device 001: ID 1d6b:0003 Linux Foundation 3.0 root hub
Bus 005 Device 003: ID 13ba:0018 PCPlay Barcode PCP-BCG4209
Bus 005 Device 002: ID 2109:0811  
Bus 005 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub
Bus 001 Device 006: ID 04b3:310b IBM Corp. Red Wheel Mouse

my mouse is 04b3:310b then i add this parameter to qemu.

-usb -usbdevice host:04b3:310b

I pass both my mouse and keyboard to VM in the same way as you, it's very robust. But if you want to use host keyboard at the same time, you need a second one with a different device ID (this probably means different model or vendor). For me it does not matter, I only ssh to host anyway (when running a VM).

I wonder if a USB hub can be passed through this way (taking with it all the attached devices), didn't have time to check yet. Even better if this also worked for USB3.0 hub ...

I know what you think. But i have KVM switch like this http://www.ebay.com/itm/BELKIN-PS-2-Por … 0710063511
It is PS/2 switch but for Host i use PS2 input. And for Quest i use adapter ps/2-->USB this one http://www.cablewholesale.com/products/ … c-451g.php
I can use one monitor one keyboard and one mouse together. USB adapter have own vID and pID. But this is older method with PS/2 today all people have USB mouses and keyboards.

Jdioutkast

Member

Registered: 2014-06-14

Posts: 1

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Hey aw,

I was wondering if you had any insight into this issue,

I'm using a XFX R7 260X and I can get GPU passthrough working fine, but as soon as I either attempt to install the ATI driver(Windows 7) or the driver installs(Windows 8) I get an immediate BSOD, or the screen just goes black and the CPU usage of the guest goes to 100%.  I also have to reboot the host to get the card working again.

I am running Fedora virt-preview with kernel 3.15.

I have tried doing the bus reset test you posted a few pages back and the screen did not clear, so I am guessing it is a bus reset issue.  Any ideas on what to do next?  Anything I can try to get around the issue?

Thanks!

vais

Member

Registered: 2014-06-07

Posts: 2

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

In the end I believe all I need is the VGA arbiter patch: https://lkml.org/lkml/2014/5/9/517 but the host graphics issue persists.

Using this version of the patch, make sue you boot with i915.enable_hd_vgaarb=1

Yes, I pass this both as kernel parameter and in /etc/modprobe.d as an option, just in case. Still, it doesn't seem to work

spstarr

Member

Registered: 2014-06-15

Posts: 1

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Hello folks,

I have a Lenovo W500 laptop that has an Intel 4gen GPU (HD graphics). I'm trying to pass the discrete Radeon HD3650 GPU to a WIndows 7 guest here is my configuration:

grub options:

vconsole.keymap=us rhgb slub_debug=- cgroup_disable=memory console=tty0 console=ttyS0,115200n8 radeon.blacklist=1 intel_iommu=on pci-stub.ids=1002:9591 vfio_iommu_type1.allow_unsafe_interrupts=1

/etc/modprobe.d/kvm.conf:

options kvm allow_unsafe_assigned_interrupts=1
options kvm ignore_msrs=1

Kernel sub

[    0.668873] pci-stub: add 1002:9591 sub=FFFFFFFF:FFFFFFFF cls=00000000/00000000
[    0.669199] pci-stub 0000:01:00.0: claimed by stub
[  691.577263] pci-stub 0000:01:00.0: claimed by stub

Binded it with the script

vfio-bind 0000:01:00.0

PCI Information

01:00.0 VGA compatible controller: Advanced Micro Devices, Inc. [AMD/ATI] RV635/M86 [Mobility Radeon HD 3650] (prog-if 00 [VGA controller])
        Subsystem: Advanced Micro Devices, Inc. [AMD/ATI] Mobility Radeon HD 3650
        Physical Slot: 1-1
        Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx+
        Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
        Interrupt: pin A routed to IRQ 16
        Region 0: Memory at c0000000 (32-bit, prefetchable) [disabled] [size=256M]
        Region 1: I/O ports at 2000 [disabled] [size=256]
        Region 2: Memory at bfff0000 (32-bit, non-prefetchable) [disabled] [size=64K]
        [virtual] Expansion ROM at bff00000 [disabled] [size=128K]
        Capabilities: [50] Power Management version 3
                Flags: PMEClk- DSI- D1+ D2+ AuxCurrent=0mA PME(D0-,D1-,D2-,D3hot-,D3cold-)
                Status: D0 NoSoftRst- PME-Enable- DSel=0 DScale=0 PME-
        Capabilities: [58] Express (v2) Legacy Endpoint, MSI 00
                DevCap: MaxPayload 128 bytes, PhantFunc 0, Latency L0s <4us, L1 unlimited
                        ExtTag+ AttnBtn- AttnInd- PwrInd- RBE+ FLReset-
                DevCtl: Report errors: Correctable- Non-Fatal- Fatal- Unsupported-
                        RlxdOrd+ ExtTag- PhantFunc- AuxPwr- NoSnoop+
                        MaxPayload 128 bytes, MaxReadReq 128 bytes
                DevSta: CorrErr- UncorrErr- FatalErr- UnsuppReq- AuxPwr- TransPend-
                LnkCap: Port #0, Speed 2.5GT/s, Width x16, ASPM L0s L1, Exit Latency L0s <64ns, L1 <1us
                        ClockPM- Surprise- LLActRep- BwNot-
                LnkCtl: ASPM Disabled; RCB 64 bytes Disabled- CommClk+
                        ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
                LnkSta: Speed 2.5GT/s, Width x16, TrErr- Train- SlotClk+ DLActive- BWMgmt- ABWMgmt-
                DevCap2: Completion Timeout: Range A, TimeoutDis+, LTR-, OBFF Not Supported
                DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis-, LTR-, OBFF Disabled
                LnkCtl2: Target Link Speed: 2.5GT/s, EnterCompliance- SpeedDis-
                         Transmit Margin: Normal Operating Range, EnterModifiedCompliance- ComplianceSOS-
                         Compliance De-emphasis: -6dB
                LnkSta2: Current De-emphasis Level: -3.5dB, EqualizationComplete-, EqualizationPhase1-
                         EqualizationPhase2-, EqualizationPhase3-, LinkEqualizationRequest-
        Capabilities: [a0] MSI: Enable- Count=1/1 Maskable- 64bit+
                Address: 0000000000000000  Data: 0000
        Capabilities: [100 v1] Vendor Specific Information: ID=0001 Rev=1 Len=010 <?>
        Kernel driver in use: vfio-pci
        Kernel modules: radeon

libvirt guest XML config:

<!--
WARNING: THIS IS AN AUTO-GENERATED FILE. CHANGES TO IT ARE LIKELY TO BE
OVERWRITTEN AND LOST. Changes to this xml configuration should be made using:
  virsh edit Windows7
or other application using the libvirt API.
-->

<domain type='kvm' xmlns:qemu='http://libvirt.org/schemas/domain/qemu/1.0'>
  <name>Windows7</name>
  <uuid>XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX</uuid>
  <memory unit='KiB'>1048576</memory>
  <currentMemory unit='KiB'>1048576</currentMemory>
  <vcpu placement='static'>1</vcpu>
  <os>
    <type arch='x86_64' machine='pc-q35-2.0'>hvm</type>
    <boot dev='hd'/>
  </os>
  <features>
    <acpi/>
    <apic/>
    <pae/>
  </features>
  <cpu mode='custom' match='exact'>
    <model fallback='allow'>Penryn</model>
  </cpu>
  <clock offset='localtime'>
    <timer name='rtc' tickpolicy='catchup'/>
    <timer name='pit' tickpolicy='delay'/>
    <timer name='hpet' present='no'/>
  </clock>
  <on_poweroff>destroy</on_poweroff>
  <on_reboot>restart</on_reboot>
  <on_crash>restart</on_crash>
  <pm>
    <suspend-to-mem enabled='no'/>
    <suspend-to-disk enabled='no'/>
  </pm>
  <devices>
    <emulator>/usr/bin/qemu-kvm</emulator>
    <disk type='block' device='disk'>
      <driver name='qemu' type='raw' cache='none' io='native'/>
      <source dev='/dev/sda'/>
      <target dev='vda' bus='virtio'/>
      <address type='pci' domain='0x0000' bus='0x02' slot='0x05' function='0x0'/>
    </disk>
    <controller type='usb' index='0' model='ich9-ehci1'>
      <address type='pci' domain='0x0000' bus='0x02' slot='0x03' function='0x7'/>
    </controller>
    <controller type='usb' index='0' model='ich9-uhci1'>
      <master startport='0'/>
      <address type='pci' domain='0x0000' bus='0x02' slot='0x03' function='0x0' multifunction='on'/>
    </controller>
    <controller type='usb' index='0' model='ich9-uhci2'>
      <master startport='2'/>
      <address type='pci' domain='0x0000' bus='0x02' slot='0x03' function='0x1'/>
    </controller>
    <controller type='usb' index='0' model='ich9-uhci3'>
      <master startport='4'/>
      <address type='pci' domain='0x0000' bus='0x02' slot='0x03' function='0x2'/>
    </controller>
    <controller type='pci' index='0' model='pcie-root'/>
    <controller type='pci' index='1' model='dmi-to-pci-bridge'>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x1e' function='0x0'/>
    </controller>
    <controller type='pci' index='2' model='pci-bridge'>
      <address type='pci' domain='0x0000' bus='0x01' slot='0x01' function='0x0'/>
    </controller>
    <controller type='sata' index='0'>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x1f' function='0x2'/>
    </controller>
    <controller type='virtio-serial' index='0'>
      <address type='pci' domain='0x0000' bus='0x02' slot='0x04' function='0x0'/>
    </controller>
    <interface type='bridge'>
      <mac address='XXXXXXXXXXXXXXX'/>
      <source bridge='Bridge'/>
      <model type='virtio'/>
      <address type='pci' domain='0x0000' bus='0x02' slot='0x01' function='0x0'/>
    </interface>
    <serial type='pty'>
      <target port='0'/>
    </serial>
    <console type='pty'>
      <target type='serial' port='0'/>
    </console>
    <channel type='spicevmc'>
      <target type='virtio' name='com.redhat.spice.0'/>
      <address type='virtio-serial' controller='0' bus='0' port='1'/>
    </channel>
    <input type='tablet' bus='usb'/>
    <input type='mouse' bus='ps2'/>
    <input type='keyboard' bus='ps2'/>
    <graphics type='spice' autoport='yes'/>
    <sound model='ich6'>
      <address type='pci' domain='0x0000' bus='0x02' slot='0x02' function='0x0'/>
    </sound>
    <video>
      <model type='qxl' ram='65536' vram='65536' heads='1'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x0'/>
    </video>
    <memballoon model='virtio'>
      <address type='pci' domain='0x0000' bus='0x02' slot='0x06' function='0x0'/>
    </memballoon>
  </devices>
  <qemu:commandline>
    <qemu:arg value='-vga'/>
    <qemu:arg value='none'/>
    <qemu:arg value='-bios'/>
    <qemu:arg value='/usr/share/qemu/bios.bin'/> 
    <qemu:arg value='-device'/>
    <qemu:arg value='ioh3420,bus=pcie.0,addr=1c.0,multifunction=on,port=1,chassis=1,id=root.1'/>
    <qemu:arg value='-device'/>
    <qemu:arg value='vfio-pci,host=01:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on,rombar=1,romfile=/home/spstarr/radeon-vbios.rom'/>
  </qemu:commandline>
</domain>

I've tried with rombar=0, no romfile specified, none work.

- If I use virt-managet and do PCI passing I get code 43
- If I use my QEMU embedded flag options I get code 12.

I've tried

modprobe kvm_intel emulate_invalid_guest_state=0

But this didn't work.

Anyone have tips? I saw someone mentioning to use multiple monitors, but when I try, I get VGA conflict errors with I start with virsh start.

additional info:
* I am using kernel 3.15 final
* QEMU 2.0.0 final release

Any help would be great!

** Update: So I'm told on IRC, you *CANNOT* pass a secondary GPU to QEMU VM window or full-screen directly, there is no such way to get the GPU display to another overlay.

Last edited by spstarr (2014-06-15 21:05:07)

ttouch

Member

From: /dev/null

Registered: 2012-05-27

Posts: 130

Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Seems that I get "Windows has stopped this device because it has reported problems. (Code 43)". That may be the cause of the underscan problem.
I searched an other users also have this problem but I can't figure out how to solve it...
Any help appreciated

Solved with Nvidia 335
(Advanced search in nvidia page)

---

GitHub | Git Server | Blog

syndtr

Member

Registered: 2014-05-18

Posts: 4

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Seems that I get "Windows has stopped this device because it has reported problems. (Code 43)". That may be the cause of the underscan problem.
I searched an other users also have this problem but I can't figure out how to solve it...
Any help appreciated

Solved with Nvidia 335
(Advanced search in nvidia page)

Or you can hide KVM hypervisor signature and continue using latest nvidia driver. You can add '-cpu kvm=off' to hide the kvm signature if you use current qemu git.

http://lists.gnu.org/archive/html/qemu- … 00808.html

Last edited by syndtr (2014-06-15 11:33:03)

sinny

Member

Registered: 2014-05-24

Posts: 75

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Yes.

i'm getting this as well.
my setup is one monitor (benq g2420hd) with connected:
- DVI : host (fedora 20, i7-4770 IGP)
- HDMI : guest (win 7, radeon HD 6450)

i just switch displayed input on monitor when needed.
host video (DVI) works just fine - native 1920x1080, no problems at all
but when i switch to guest video (HDMI) using 1920x1080 - i get pretty large black frame visible on the screen. so far, i "solved" it by decreasing screen resolution in guest

but if there is some sort of solution (e.g. it is not a problem with the monitor not being able to handle dual input correctly) - that would be great

I had exactly same problem with my Geforce 660 over HDMI when i installed newest drivers (version 337.88) in client.
It went away after i deleted those drivers and installed older ones (version 335 i belive) and now its working in full screen without that anoying black frame around it.
It was also working without any problems when i used default drivers that windows (win8) found on its own.

ok, found solution to hdmi underscan for radeon cards - works for me: link

ttouch

Member

From: /dev/null

Registered: 2012-05-27

Posts: 130

Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Seems that I get "Windows has stopped this device because it has reported problems. (Code 43)". That may be the cause of the underscan problem.
I searched an other users also have this problem but I can't figure out how to solve it...
Any help appreciated

Solved with Nvidia 335
(Advanced search in nvidia page)

Or you can hide KVM hypervisor signature and continue using latest nvidia driver. You can add '-cpu kvm=off' to hide the kvm signature if you use current qemu git.

http://lists.gnu.org/archive/html/qemu- … 00808.html

Is this the cause of the lowered FPS I get?
I tested battlefield 3 and I get a bit laggy gameplay. Just less FPS.
Will this solve it?
Also do I pass "-cpu host", or just "-cpu kvm=off"?

Thank you.

---

GitHub | Git Server | Blog

ttouch

Member

From: /dev/null

Registered: 2012-05-27

Posts: 130

Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

I get "Unable to find CPU definition: kvm=off"
Also which are the possible optimizations I could do? (I can not find a way to search only in this topic)
My system:
Intel i7-3820
16GB ram
Asus P9X79 Deluxe
Asus GTX 660 Ti DirectCU II TOP

I use kernel 3.15.0-1-mainline provided by nbhs

Last edited by ttouch (2014-06-15 13:53:24)

---

GitHub | Git Server | Blog

siddharta

Member

Registered: 2014-05-03

Posts: 30

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Hi I am here new in forum. Why you dont use somethink like this. In lsusb you can find your device vendorID and productID.

back@KVM-Ubuntu:~$ lsusb
Bus 002 Device 004: ID 058f:6362 Alcor Micro Corp. Flash Card Reader/Writer
Bus 002 Device 003: ID 9710:7830 MosChip Semiconductor MCS7830 10/100 Mbps Ethernet adapter
Bus 002 Device 002: ID 8087:0024 Intel Corp. Integrated Rate Matching Hub
Bus 002 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub
Bus 006 Device 001: ID 1d6b:0003 Linux Foundation 3.0 root hub
Bus 005 Device 003: ID 13ba:0018 PCPlay Barcode PCP-BCG4209
Bus 005 Device 002: ID 2109:0811  
Bus 005 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub
Bus 001 Device 006: ID 04b3:310b IBM Corp. Red Wheel Mouse

my mouse is 04b3:310b then i add this parameter to qemu.

-usb -usbdevice host:04b3:310b

I pass both my mouse and keyboard to VM in the same way as you, it's very robust. But if you want to use host keyboard at the same time, you need a second one with a different device ID (this probably means different model or vendor). For me it does not matter, I only ssh to host anyway (when running a VM).

I wonder if a USB hub can be passed through this way (taking with it all the attached devices), didn't have time to check yet. Even better if this also worked for USB3.0 hub ...

I know what you think. But i have KVM switch like this http://www.ebay.com/itm/BELKIN-PS-2-Por … 0710063511
It is PS/2 switch but for Host i use PS2 input. And for Quest i use adapter ps/2-->USB this one http://www.cablewholesale.com/products/ … c-451g.php
I can use one monitor one keyboard and one mouse together. USB adapter have own vID and pID. But this is older method with PS/2 today all people have USB mouses and keyboards.

Having multiple USB devices with same vendor and device ID is possible with libvirt if you address them by the device address on the host (<address> element in <source> rather than <vendor> and <product>), have a look at http://libvirt.org/formatdomain.html#el … tDevSubsys. It's a bit more limiting as the address of the device will change if you plug it into a different USB port for example.

penetal

Member

Registered: 2014-06-15

Posts: 10

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

I was planing to get a setup so I can get passthrough for the games that needs Windows to run.

My plan is to get:
-ASRock z97 extreme 6
-i7 4790

The reason I'm going for that motherboard is the M.2 slot that has a speed of 4x PCIe aswell as the more common 2x slot, so I can use faster M.2 cards (not many available yet but there more might come to market later)
Now that will cut the dGPU to 8x lanes as the fast M.2 slot uses the CPU lanes aswell, would this in any way create a problem for the dGPU passthrough?

Also, I am really lost on memory, I have selected:
G.Skill TridentX DDR3 PC21300/2666MHz CL12 2x8GB
if anyone wish to comment one way or another on that it would be great.

shawly

Member

Registered: 2014-05-24

Posts: 22

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Alright, since I'm sick of not getting my GTX 770 performance problem fixed, I built in my AMD HD 6950.
Driver install worked and the short freezes are finally gone. <3
BUT the Benchmark is still pretty low.. http://www.3dmark.com/3dm11/8432530
Since there is no user with this setup, I searched for users with the i7 4770 which is pretty much the same like my Xeon E3-1245v3 and these guys have an average of 9000 in physics and 5500 in graphics score.. which means that my performance sucks and isn't even above 90% of the native performance..

As you can see it doesn't even push my Xeon into 3.8Ghz turbomode and my GPU isn't very fast aswell, with my GTX 770 I had the average of 11000 graphics score which was higher than some native users benchmarks, only the cpu slowed down. But what could cause this now?

Another little question, I'm using libvirt but can't get the emulated sound working, passthrough with libvirt works fine, only sound causes me trouble... I used the parameter in the first post, sound device is shown in windows, but I can't hear anything.

Last edited by shawly (2014-06-15 18:12:47)

buraktamturk

Member

Registered: 2014-05-28

Posts: 13

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

I can't bind HDMI audio device to vfio-pci after compiling new 3.15 kernel.

01:00.0 VGA compatible controller: Advanced Micro Devices, Inc. [AMD/ATI] Cedar [Radeon HD 5000/6000/7350/8350 Series] (prog-if 00 [VGA controller])
        Subsystem: Hightech Information System Ltd. Device 2291
        Flags: fast devsel, IRQ 11
        Memory at e0000000 (64-bit, prefetchable) [disabled] [size=256M]
        Memory at f7e20000 (64-bit, non-prefetchable) [disabled] [size=128K]
        I/O ports at e000 [disabled] [size=256]
        Expansion ROM at f7e00000 [disabled] [size=128K]
        Capabilities: [50] Power Management version 3
        Capabilities: [58] Express Legacy Endpoint, MSI 00
        Capabilities: [a0] MSI: Enable- Count=1/1 Maskable- 64bit+
        Capabilities: [100] Vendor Specific Information: ID=0001 Rev=1 Len=010 <?>
        Capabilities: [150] Advanced Error Reporting
        Kernel driver in use: vfio-pci

01:00.1 Audio device: Advanced Micro Devices, Inc. [AMD/ATI] Cedar HDMI Audio [Radeon HD 5400/6300 Series]
        Subsystem: Hightech Information System Ltd. Device aa68
        Flags: bus master, fast devsel, latency 0, IRQ 46
        Memory at f7e40000 (64-bit, non-prefetchable) [size=16K]
        Capabilities: [50] Power Management version 3
        Capabilities: [58] Express Legacy Endpoint, MSI 00
        Capabilities: [a0] MSI: Enable+ Count=1/1 Maskable- 64bit+
        Capabilities: [100] Vendor Specific Information: ID=0001 Rev=1 Len=010 <?>
        Capabilities: [150] Advanced Error Reporting

As you can see "vfio-bind 01:00.0" works good, the device claimed by vfio driver. But "vfio-bind 01:00.1" just freezes, this device was claimed by snd_hda_intel, whenever i try to bind to vfio, the vfio-bind script just hangs (Ctrl+C does not stop the script). This device is claimed by snd_hda_intel even my cmdline.

root=UUID=94c11fcb-ec91-4b49-b21d-d2cb08d5929e ro quiet rootfstype=xfs add_efi_memmap intel_iommu=on initrd=\EFI\debian\initrd.img vfio_iommu_type1.allow_unsafe_interrupts=1 kvm.ignore_msrs=1 pci-stub.ids=1002:9505,1002:aa18 pcie_acs_override=downstream i915.enable_hd_vgaarb=1

vfio-bind code, (i get from somewhere at the forum)

#!/bin/bash

modprobe vfio-pci

for dev in "$@"; do
    read -r vendor </sys/bus/pci/devices/$dev/vendor
    read -r device </sys/bus/pci/devices/$dev/device
    if [ -e /sys/bus/pci/devices/$dev/driver ]; then
        echo $dev > /sys/bus/pci/devices/$dev/driver/unbind
    fi
    echo $vendor $device > /sys/bus/pci/drivers/vfio-pci/new_id
done

(despite the script, i did not compile the vfio-pci as a module, its bundled on the kernel)

Any throughts?
Thanks.

sinny

Member

Registered: 2014-05-24

Posts: 75

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

...

As you can see "vfio-bind 01:00.0" works good, the device claimed by vfio driver. But "vfio-bind 01:00.1" just freezes, this device was claimed by snd_hda_intel, whenever i try to bind to vfio, the vfio-bind script just hangs (Ctrl+C does not stop the script). This device is claimed by snd_hda_intel even my cmdline.

root=UUID=94c11fcb-ec91-4b49-b21d-d2cb08d5929e ro quiet rootfstype=xfs add_efi_memmap intel_iommu=on initrd=\EFI\debian\initrd.img vfio_iommu_type1.allow_unsafe_interrupts=1 kvm.ignore_msrs=1 pci-stub.ids=1002:9505,1002:aa18 pcie_acs_override=downstream i915.enable_hd_vgaarb=1

vfio-bind code, (i get from somewhere at the forum)
...

(despite the script, i did not compile the vfio-pci as a module, its bundled on the kernel)

Any throughts?
Thanks.

if it is claimed, i guess it's id is wrong
i'd first look at dmesg:

[    0.000000] Command line: BOOT_IMAGE=/boot/vmlinuz-3.15.0-1.vanilla.mainline.knurd.1.fc20.x86_64 root=UUID=588911d3-23f7-4544-9d46-0b06b85f287c ro pci-stub.ids=1002:6779,1002:aa98,10de:0de1,10de:0bea intel_iommu=on isolcpus=1,2,3,5,6,7 LANG=en_US.UTF-8
...
[    0.402432] pci-stub: add 1002:6779 sub=FFFFFFFF:FFFFFFFF cls=00000000/00000000
[    0.402578] pci-stub 0000:01:00.0: claimed by stub
[    0.402669] pci-stub: add 1002:AA98 sub=FFFFFFFF:FFFFFFFF cls=00000000/00000000
[    0.402813] pci-stub 0000:01:00.1: claimed by stub
[    0.402904] pci-stub: add 10DE:0DE1 sub=FFFFFFFF:FFFFFFFF cls=00000000/00000000
[    0.403058] pci-stub: add 10DE:0BEA sub=FFFFFFFF:FFFFFFFF cls=00000000/00000000
...

it is clearly seen here, that radeon HD 6450 is present, while nvidia gt 430 - not (10DE:* reference missing hardware)

how about "lspci -nn" output for additional clarity?

p.s. you cut your "lspci -v" output too early for others to see which kernel driver actually is used for amd audio

Last edited by sinny (2014-06-16 04:42:34)

sedstr

Member

Registered: 2014-06-16

Posts: 2

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

I've been banging away at this, trying to achieve what may be impossible, please someone tell me.

I have an asus g74vw, it only has a single nvidia gtx670m.

What I want to do:
Install linux as the host, configure the network, then disable host access to basically everything but the network, I can forward X11 to the guest if I really wanted to, everything else can be done on the command line.
Install one windows VM and have it directly access the video card/keyboard/mouse/USB/CD, everything but the network.

I've put all my efforts into a deb installation, I've compiled kernels, updated qemu, but after detaching the adapter, I keep getting to...
ERROR    Unable to read from monitor: Connection reset by peer.
Only a days efforts, but its tiring.

I'll have another go with archlinux, if there is actually a possibility of getting to where I want, but if someone can tell me if its achievable, and possibly some tips, I'll keep bashing away at it.

This has been one of my technical wants for like 10 years! Main reason I want things setup this way, is I want to isolate windows but still be able to play a game. else I'd just run windows with linux in vmware.

TIA for any advice.

sinny

Member

Registered: 2014-05-24

Posts: 75

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

I've been banging away at this, trying to achieve what may be impossible, please someone tell me.

I have an asus g74vw, it only has a single nvidia gtx670m.

What I want to do:
Install linux as the host, configure the network, then disable host access to basically everything but the network, I can forward X11 to the guest if I really wanted to, everything else can be done on the command line.
Install one windows VM and have it directly access the video card/keyboard/mouse/USB/CD, everything but the network.

I've put all my efforts into a deb installation, I've compiled kernels, updated qemu, but after detaching the adapter, I keep getting to...
ERROR    Unable to read from monitor: Connection reset by peer.
Only a days efforts, but its tiring.

I'll have another go with archlinux, if there is actually a possibility of getting to where I want, but if someone can tell me if its achievable, and possibly some tips, I'll keep bashing away at it.

This has been one of my technical wants for like 10 years! Main reason I want things setup this way, is I want to isolate windows but still be able to play a game. else I'd just run windows with linux in vmware.

TIA for any advice.

the only "technical" details provided are - motherboard and video card models and a single log line (not even specifying when it happens - e.g. in response to what action(-s)?)
imo, you are pretty much limiting the amount of help you may receive yourself

sedstr

Member

Registered: 2014-06-16

Posts: 2

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

the only "technical" details provided are - motherboard and video card models and a single log line (not even specifying when it happens - e.g. in response to what action(-s)?)
imo, you are pretty much limiting the amount of help you may receive yourself

The issue was presenting on trying to define a new VM with virt-install.
I'd nodedev-detach the gpu and the audio successfully, the virt-install command would cause the display to reset, then I'd get that message ERROR    Unable to read from monitor: Connection reset by peer.

All the examples given have one adapter assigned to the host, and one to the guest, I'm trying to run the host without a graphics adapter at all...
I was mostly interested in the 'is it possible' question, obviously, if what I'm trying to do is impossible due to some technical restriction, then I'll stop trying till I have another system more suitable.

Having said that, a few hours ago, I upgraded qemu to a later version via backports, and it started saying the hardware wasn't compatible.
Installed Arch, and well, learning curve, it'll take me a while to get to the stage I was at with debian ....AIX is my regular *nix.

Last edited by sedstr (2014-06-16 17:05:17)

lojsar

Member

Registered: 2014-06-16

Posts: 7

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

The problem, RmInitAdapter failed! (0x23:0x2f:*LineNumber*), was fixed with the kvm=off patch!

Last edited by lojsar (2014-06-17 20:00:51)

shawly

Member

Registered: 2014-05-24

Posts: 22

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Alright, since I'm sick of not getting my GTX 770 performance problem fixed, I built in my AMD HD 6950.
Driver install worked and the short freezes are finally gone. <3
BUT the Benchmark is still pretty low.. http://www.3dmark.com/3dm11/8432530
Since there is no user with this setup, I searched for users with the i7 4770 which is pretty much the same like my Xeon E3-1245v3 and these guys have an average of 9000 in physics and 5500 in graphics score.. which means that my performance sucks and isn't even above 90% of the native performance..

As you can see it doesn't even push my Xeon into 3.8Ghz turbomode and my GPU isn't very fast aswell, with my GTX 770 I had the average of 11000 graphics score which was higher than some native users benchmarks, only the cpu slowed down. But what could cause this now?

Another little question, I'm using libvirt but can't get the emulated sound working, passthrough with libvirt works fine, only sound causes me trouble... I used the parameter in the first post, sound device is shown in windows, but I can't hear anything.

/bump

I saw the OP has a similar score with the same GPU, but that post was also years ago, so I dunno anymore.. tried with so many different CPU options in libvirt.. I even tried cpu mode host-passthrough, the benchmark said the CPU had 4GHz in turbomode which can't be possible since the Xeon only has 3,8 GHz in turbomode.. the score was even lower.

deniv

Member

Registered: 2013-10-16

Posts: 27

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

BUT the Benchmark is still pretty low.. http://www.3dmark.com/3dm11/8432530
Since there is no user with this setup, I searched for users with the i7 4770 which is pretty much the same like my Xeon E3-1245v3 and these guys have an average of 9000 in physics and 5500 in graphics score.. which means that my performance sucks and isn't even above 90% of the native performance..

Your results aren't that bad. The average for your graphics card is only 5240 (with 4900 you get 93% of that). The CPU score could be better, though (since the average is 8960).
Anyway, if you haven't done this yet, you can try:
1. Backing memory with hugepages.
2. Pinning vcpus.
3. Adding hyper-v enlightenments (hv-time, hv-relaxed, hv-vapic, hv-spinlocks=0x1000).
4. Kernel 3.15 since it has the debug registers patch.
5. Let libvirt set the CPU model without host passthrough.

While I was at it, I tried this benchmark and was quite pleased with what I got: http://www.3dmark.com/3dm11/8436872 The time measurement error is probably because of hv-time. My grapics card score is 14 points above the average.  As for my CPU (E3-1245v2), it supports HT but for this win7 vm I use only the four main cores because most games don't profit much from HT and the "emulated" cores are kind of dedicated to kvm and other vms. So for 3DMark needs I have i5-3570 with the average score of 6420 while my score is 7165.

lojsar

Member

Registered: 2014-06-16

Posts: 7

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

I have a Dell XPS-15 (L502X) laptop with nVidia GT525M in an nVidia Optimus configuration.
The Intel GPU is connected to the internal display.
The nVidia GPU is connected to a HDMI port, which to I connected a monitor.
VBIOS is stored in ACPI. I extracted it from nouveau's debugfs.

After some fiddling, I now have it booting:

seabios runs the VGA BIOS, but nothing is displayed on the monitor.

When I open the nVidia device, the driver complains that it can't load the VBIOS. I suspected that it wanted to find it in ACPI, so I modified the nvidia driver, replacing the function that reads VBIOS from ACPI with one that reads it from my dumpfile. This worked!
Is there a way I can avoid modifying the driver? I would like to launch Windows too! Is it "written" somewhere on the GPU that VBIOS is located in ACPI?

I see that the nVidia driver thinks CRT-0 is connected. Could this be why I don't see anything from BIOS? I was thinking maybe the card is using the wrong output. (DFP-0/HDMI is working! I just have to disable CRT-0 with xrandr to get a sane resolution)

qemu command

exec qemu-system-x86_64 \
        -enable-kvm \
        -M q35 \
        -m 2048 \
        -cpu host,kvm=off \
        -vga none \
        -chardev stdio,id=seabios \
        -device isa-debugcon,iobase=0x402,chardev=seabios \
        -drive if=virtio,file=debian.qcow2,id=disk,format=qcow2,cache=off \
        -device ioh3420,bus=pcie.0,addr=1c.0,multifunction=on,port=1,chassis=1,id=root.1 \
        -device vfio-pci,host=01:00.0,bus=root.1,addr=00.0,id=gt525m,romfile=nouveau-vbios.rom,x-vga=on \

host: linux-3.15.0 (i915.enable_hd_vgaarb=1)
qemu: 2.0.0 with kvm=off patch
seabios: 1.7.4
guest: Debian Sid /w linux-3.14.1 and nvidia-340.17

Last edited by lojsar (2014-06-17 20:53:02)

aw

Member

Registered: 2013-10-04

Posts: 921

Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

When I open the nVidia device, the driver complains that it can't load the VBIOS. I suspected that it wanted to find it in ACPI, so I modified the nvidia driver, replacing the function that reads VBIOS from ACPI with one that reads it from my dumpfile. This worked!
Is there a way I can avoid modifying the driver? I would like to launch Windows too! Is it "written" somewhere on the GPU that VBIOS is located in ACPI?

Is the VBIOS you extracted actually a PCI ROM?  Run 'xxd nouveau-vbios.rom | less' and you should see these important markers:

0000000: 55aa 7eeb 4b37 3430 30e9 4c19 77cc 5649  U.~.K7400.L.w.VI
0000010: 4445 4f20 0d00 0000 8801 cf18 0000 4942  DEO ..........IB
0000020: 4d20 5647 4120 436f 6d70 6174 6962 6c65  M VGA Compatible
0000030: 0100 0000 8000 58be 3130 2f32 302f 3039  ......X.10/20/09

The first two bytes should always be 55aa

0000180: 91df aa8c 9af2 f5ff 5043 4952 de10 740a  ........PCIR..t.

Somewhere not too far in, you should be able to find "PCIR" and following it will be the device it's for in little endian, ex: de10 740a = 0x10de 0x0a74, which translates to NVIDIA Vendor ID, G210M device ID (http://pci-ids.ucw.cz/read/PC/10de/0a74).  I just happen to have this ROM sitting around, the device itself only works with the basic VGA driver in a VM.

---

http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

lojsar

Member

Registered: 2014-06-16

Posts: 7

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

When I open the nVidia device, the driver complains that it can't load the VBIOS. I suspected that it wanted to find it in ACPI, so I modified the nvidia driver, replacing the function that reads VBIOS from ACPI with one that reads it from my dumpfile. This worked!
Is there a way I can avoid modifying the driver? I would like to launch Windows too! Is it "written" somewhere on the GPU that VBIOS is located in ACPI?

Is the VBIOS you extracted actually a PCI ROM?  Run 'xxd nouveau-vbios.rom | less' and you should see these important markers:

0000000: 55aa 7eeb 4b37 3430 30e9 4c19 77cc 5649  U.~.K7400.L.w.VI
0000010: 4445 4f20 0d00 0000 8801 cf18 0000 4942  DEO ..........IB
0000020: 4d20 5647 4120 436f 6d70 6174 6962 6c65  M VGA Compatible
0000030: 0100 0000 8000 58be 3130 2f32 302f 3039  ......X.10/20/09

The first two bytes should always be 55aa

0000180: 91df aa8c 9af2 f5ff 5043 4952 de10 740a  ........PCIR..t.

Somewhere not too far in, you should be able to find "PCIR" and following it will be the device it's for in little endian, ex: de10 740a = 0x10de 0x0a74, which translates to NVIDIA Vendor ID, G210M device ID (http://pci-ids.ucw.cz/read/PC/10de/0a74).  I just happen to have this ROM sitting around, the device itself only works with the basic VGA driver in a VM.

I think it is. seabios seemed to run it at least (I enabled seabios debug output).

0000000: 55aa 7feb 4b37 3430 30e9 4c19 77cc 5649  U...K7400.L.w.VI
0000010: 4445 4f20 0d00 0000 8801 3c19 0000 4942  DEO ......<...IB
0000020: 4d20 5647 4120 436f 6d70 6174 6962 6c65  M VGA Compatible
0000030: 0100 0000 c000 3ab6 3032 2f31 362f 3131  ......:.02/16/11
0000040: 0000 0000 0000 0000 0010 4000 51fd 24fd  ..........@.Q.$.
0000050: e996 2a00 2810 6704 ff3f fc7f 0040 0000  ..*.(.g..?...@..
0000060: ffff ff7f 0000 0080 493f a5de e9fe 46e9  ........I?....F.
0000070: 0547 504d 4944 6c00 6f00 0000 00a0 00b0  .GPMIDl.o.......
0000080: 00b8 00c0 0033 4746 3130 3820 4531 3037  .....3GF108 E107
0000090: 3920 534b 5520 3030 3036 2056 4741 2042  9 SKU 0006 VGA B
00000a0: 494f 530d 0a00 0000 0000 0000 0000 0000  IOS.............
00000b0: 0000 0000 0000 0000 0000 0000 0000 0000  ................
00000c0: 0000 0000 0000 0000 0000 0000 0000 0000  ................
00000d0: 0000 0000 0000 0056 6572 7369 6f6e 2037  .......Version 7
00000e0: 302e 3038 2e35 332e 3030 2e30 3420 0d0a  0.08.53.00.04 ..
00000f0: 0043 6f70 7972 6967 6874 2028 4329 2031  .Copyright (C) 1
0000100: 3939 362d 3230 3131 204e 5649 4449 4120  996-2011 NVIDIA 
0000110: 436f 7270 2e0d 0a00 0000 ffff 0000 0000  Corp............
0000120: ffff 4746 3130 3820 426f 6172 6420 2d20  ..GF108 Board - 
0000130: 3130 3739 6466 3530 0000 0000 0000 0000  1079df50........
0000140: 0000 0000 0043 6869 7020 5265 7620 2020  .....Chip Rev   
0000150: 0000 0000 0000 0000 00ba 9198 9691 9a9a  ................
0000160: 8d96 9198 dfad 9a93 9a9e 8c9a dfd2 dfb1  ................
0000170: 908b dfb9 908d dfaf 8d90 9b8a 9c8b 9690  ................
0000180: 91df aa8c 9af2 f5ff 5043 4952 de10 f50d  ........PCIR....
0000190: 0000 1800 0000 0003 7f00 0100 0080 0000  ................
00001a0: 4859 4224 ffb8 4249 5400 0001 0c06 1245  HYB$..BIT......E
00001b0: 3201 0400 2c02 4202 2100 3802 4301 0e00  2...,.B.!.8.C...
00001c0: 5902 4401 0400 6702 4101 0300 6b02 4901  Y.D...g.A...k.I.
00001d0: 1200 6e02 4c01 0200 8002 4d02 1100 8202  ..n.L.....M.....
00001e0: 4e00 0000 0000 5002 3000 9302 5302 1500  N.....P.0...S...
00001f0: c302 5401 0200 d802 5501 0300 da02 5601  ..T.....U.....V.
0000200: 0600 dd02 7801 0800 e302 6401 0200 eb02  ....x.....d.....
0000210: 7001 0e00 ed02 6902 4200 fc02 0000 3e03  p.....i.B.....>.
0000220: 81b0 2dc5 46cd b5ef 0000 0000 0000 0000  ..-.F...........
0000230: 0000 0000 0000 0000 0053 0870 0400 0000  .........S.p....
0000240: 0000 a807 0000 0000 0000 0000 0200 5c5c  ..............\\
0000250: 1c02 0000 3002 0400 0000 0000 0000 0000  ....0...........
0000260: 0065 b1c6 8700 0059 b4d8 4100 0000 94b2  .e.....Y..A.....
0000270: a0b2 a6b2 beb2 0eb4 59b4 a0b2 8aef 59b4  ........Y.....Y.
0000280: 10b9 084e b113 b2b9 d200 00dc d200 008a  ...N............
0000290: b200 0047 cd00 0072 ce00 007a d000 00b9  ...G...r...z....
00002a0: d100 0061 d200 0000 0000 00a5 d200 0000  ...a............
00002b0: 0000 00e8 d100 00ac d100 003e d200 0042  ...........>...B
00002c0: d200 0086 0050 d700 19f1 0028 81b0 1495  .....P.....(....
00002d0: b023 2201 2345 0114 24b5 b8b0 0033 b1ef  .#".#E..$....3..
00002e0: b000 0000 0001 0100 0000 00dd 4300 0000  ............C...
00002f0: 0000 0000 0000 0000 0000 0000 0053 0870  .............S.p
0000300: 0410 003f 0f85 001e ce00 0030 322f 3033  ...?.......02/03
0000310: 2f31 3100 0000 0000 0000 0000 008e ef01  /11.............
0000320: 0100 0000 0000 0000 0000 0000 3730 3000  ............700.
0000330: 3130 3739 3030 3030 0000 0000 0000 1000  10790000........

I uploaded the whole file here: http://cnu.no-ip.com/u/KQuzzYhbdKzG2WMI … -vbios.rom

#nouveau wasn't sure if the binary driver checked PCIROM at all. They suggested I try a tool called "nvafakebios". I will do that now!

Thank you!

belliash

Member

Registered: 2014-04-25

Posts: 16

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

I have a Dell XPS-15 (L502X) laptop with nVidia GT525M in an nVidia Optimus configuration.
The Intel GPU is connected to the internal display.
The nVidia GPU is connected to a HDMI port, which to I connected a monitor.
VBIOS is stored in ACPI. I extracted it from nouveau's debugfs.

After some fiddling, I now have it booting:

seabios runs the VGA BIOS, but nothing is displayed on the monitor.

When I open the nVidia device, the driver complains that it can't load the VBIOS. I suspected that it wanted to find it in ACPI, so I modified the nvidia driver, replacing the function that reads VBIOS from ACPI with one that reads it from my dumpfile. This worked!
Is there a way I can avoid modifying the driver? I would like to launch Windows too! Is it "written" somewhere on the GPU that VBIOS is located in ACPI?

I see that the nVidia driver thinks CRT-0 is connected. Could this be why I don't see anything from BIOS? I was thinking maybe the card is using the wrong output.

qemu command

exec qemu-system-x86_64 \
        -enable-kvm \
        -M q35 \
        -m 2048 \
        -cpu host,kvm=off \
        -vga none \
        -chardev stdio,id=seabios \
        -device isa-debugcon,iobase=0x402,chardev=seabios \
        -drive if=virtio,file=debian.qcow2,id=disk,format=qcow2,cache=off \
        -device ioh3420,bus=pcie.0,addr=1c.0,multifunction=on,port=1,chassis=1,id=root.1 \
        -device vfio-pci,host=01:00.0,bus=root.1,addr=00.0,id=gt525m,romfile=nouveau-vbios.rom,x-vga=on \

host: linux-3.15.0 (i915.enable_hd_vgaarb=1)
qemu: 2.0.0 with kvm=off patch
seabios: 1.7.4
guest: Debian Sid /w linux-3.14.1 and nvidia-340.17

Most probably you nvidia gpu is not connected to HDMI, but to igp. Thats why it thinks it has CRT-0 connected.

BTW: I'm also fighting with passing through my nVidia GPU, but I always got error 43 and 10/12 (cant remember) before installing nvidia driver in windows.
Can you tell sth more about what you done? Maybe some instruction please?