Arch Linux

aw

Member

Registered: 2013-10-04

Posts: 921

Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Hello,

I have been trying to get this VGA Passthrough to work now for some time but seems like I am stuck now

Your kernel version indicates that you're attempting to use stock Ubuntu 13.04 and we can only assume stock QEMU from that release too.  VGA assignment did not work at the time that release.  I'd suggest at least kernel 3.16 and QEMU 2.1, but you may still need patches beyond that depending on your host graphics.

---

http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

stokbaek

Member

Registered: 2014-09-20

Posts: 3

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Hello,

I have been trying to get this VGA Passthrough to work now for some time but seems like I am stuck now

Your kernel version indicates that you're attempting to use stock Ubuntu 13.04 and we can only assume stock QEMU from that release too.  VGA assignment did not work at the time that release.  I'd suggest at least kernel 3.16 and QEMU 2.1, but you may still need patches beyond that depending on your host graphics.

Sorry forgot to mention kernel and QEMU version

uname -r

3.9.3-030903-generic

kvm -version

QEMU emulator version 2.0.0 (Debian 2.0.0+dfsg-2ubuntu1.3), Copyright (c) 2003-2008 Fabrice Bellard

0011001011

Member

Registered: 2014-09-20

Posts: 3

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Hi everybody, and thank you for this amazing thread ! I was able to make my virtual gaming machine and definitively erase my dualboot. I am almost free from windows (and with steam on linux things will only get better). Information on internet is not clear so I want to confirm that GTX 570 works well with VGA passthrough on HD4000 with CGI working on arch host (don't know why btw)/

However.. my guest sometimes violently crashes : impossible to boot and I have to restore it from a copy of last working raw.img, it seems to be caused by updates but dont know which one (300+ to check) on w7 64bits ultimate. I checked to disable driver integrity check and stuff, run a chkdsk bootrec and bcdedit but nothing works. Does anybody has a solution or plan of solution ? Impossible to debug since windows repair tool thinks that a number of things are broken, not only the boot (and I think it broke it itself while trying to repair)

aw

Member

Registered: 2013-10-04

Posts: 921

Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Hi everybody, and thank you for this amazing thread ! I was able to make my virtual gaming machine and definitively erase my dualboot. I am almost free from windows (and with steam on linux things will only get better). Information on internet is not clear so I want to confirm that GTX 570 works well with VGA passthrough on HD4000 with CGI working on arch host (don't know why btw)/

However.. my guest sometimes violently crashes : impossible to boot and I have to restore it from a copy of last working raw.img, it seems to be caused by updates but dont know which one (300+ to check) on w7 64bits ultimate. I checked to disable driver integrity check and stuff, run a chkdsk bootrec and bcdedit but nothing works. Does anybody has a solution or plan of solution ? Impossible to debug since windows repair tool thinks that a number of things are broken, not only the boot (and I think it broke it itself while trying to repair)

Are you running virtio or AHCI for the guest disk?  Q35 or 440FX?  virtio + 440FX are likely to be the best performing and best supported combo.

---

http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

0011001011

Member

Registered: 2014-09-20

Posts: 3

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Owh i didn't know about that, I use Q35 and for the disk I typed the following :

drive file=/home/sofian/vm/windows.img,id=disk,format=raw -device ide-hd,bus=ide.0,drive=disk

So I used q35.. I really thought it was superior to 440fx.
It might be uninteresting to notice but I never installed virtio driver for windows because I wanted to update the system first (and thought it would crash the system because it may change the device windows detect)

Last edited by 0011001011 (2014-09-20 17:04:54)

aw

Member

Registered: 2013-10-04

Posts: 921

Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Owh i didn't know about that, I use Q35 and for the disk I typed the following :

drive file=/home/sofian/vm/windows.img,id=disk,format=raw -device ide-hd,bus=ide.0,drive=disk

So I used q35.. I really thought it was superior to 440fx.
It might be uninteresting to notice but I never installed virtio driver for windows because I wanted to update the system first (and thought it would crash the system because it may change the device windows detect)

You can find an example of a 440FX+virtio config here

For installation to virtio you can configure the VM with 2 CD images, the second being the virtio ISO image you can find here.  When discovering the disk the Windows installer will prompt to load external drivers.  It's also recommended to use virtio for the NIC rather than an emulated device.

Q35 is largely overrated.  There are some configurations of Linux that need it, but Windows is fine on 440FX.

Last edited by aw (2014-09-20 17:16:44)

---

http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

0011001011

Member

Registered: 2014-09-20

Posts: 3

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Right I will test it right now and try my 5th win installation (love you SSDs)

I know that vrtio is better for NIC however i was not using it, I already have a dedicated wifi PCi card but it disturb iommu at boot.
So you think Q35 and drivers might be the explanation of these crashes ? In fact I absolutely don't know why win update fail at some point. Assuming that it is the root of the error.

winie

Member

Registered: 2014-03-01

Posts: 17

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

So... just happened to get an update for Nvidia 344.11... Code 43 again

I have no words...

"bluebird" reported on oftc/#qemu that 344.11 is now checking hyper-v extensions, so if you remove all those hv-foo cpu parameters and leave kvm=off, 344.11 works.  I've confirmed this on my VM.  Unfortunately some of those hypver-v extensions are fairly useful and have shown to help performance, so NVIDIA appears to be intentionally crippling GeForce, possibly to make Quadro appear to be a better choice for a VM (as if official support isn't enough).  NVIDIA already seems to disable MSI on GeForce, so it's not the first time GeForce has been intentionally crippled to boost the professional cards.

After 4 months I tried vga passthrough again and succeeded this time and now this happens  :X

What were they thinking? disabling the product i own?
I will make it my mission to find a way to get back at them!

Do you think it can be solved like the kvm flag check or we can no more use hyper-v enlightments?

by the way my setup: i5-4670, Asrock Z87 Extreme4, 2x MSI Geforce 770
i hope i can make an awesome permanent setup. using onboard vga for xbmc mediacenter, one 770 for my little brother, one vm with 770 for me streamed  to my tablet, and automatic tiered storage in the virtual machines

Chetyre

Member

Registered: 2013-03-27

Posts: 34

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

I made the switch from legacy VGA to libvirt + OVMF and I report success with some small issues.

My card (Sapphire 7970 Vapor-X) didn't have UEFI, so I flashed it with the Sapphire x280 Vapor-X bios (basically the same card, but it has UEFI). I left it like that for a week to see if I'd run into any problems and I didn't.

So today I set it up with OVMF (libvirt-git, ovmf-svn) and I got it to boot and work, but only as a secondary passthrough. Even if I remove the video devices I still don't get any output before windows loads the AMD drivers

Also, I need to use the audio device on the card because it is how I take sound to the TV, but it is crackling and lagging. It didn't do that before on legacy. And when I tried to play a game the card was stuttering in places where it didn't before.

Here is my xml:

<domain type='kvm'>
  <name>windows</name>
  <uuid>298e3f5f-1546-424c-b8f8-1225241eb5d0</uuid>
  <memory unit='KiB'>16777216</memory>
  <currentMemory unit='KiB'>16777216</currentMemory>
  <memoryBacking>
    <hugepages/>
  </memoryBacking>
  <vcpu placement='static'>6</vcpu>
  <os>
    <type arch='x86_64' machine='pc-i440fx-2.1'>hvm</type>
    <loader type='pflash'>/var/lib/libvirt/images/ovmf_x64.bin</loader>
  </os>
  <features>
    <acpi/>
    <apic/>
    <pae/>
    <hyperv>
      <relaxed state='on'/>
      <vapic state='on'/>
      <spinlocks state='on' retries='8191'/>
    </hyperv>
  </features>
  <cpu mode='host-passthrough'>
    <topology sockets='1' cores='3' threads='2'/>
  </cpu>
  <clock offset='localtime'>
    <timer name='rtc' tickpolicy='catchup'/>
    <timer name='pit' tickpolicy='delay'/>
    <timer name='hpet' present='no'/>
    <timer name='hypervclock' present='yes'/>
  </clock>
  <on_poweroff>destroy</on_poweroff>
  <on_reboot>restart</on_reboot>
  <on_crash>restart</on_crash>
  <pm>
    <suspend-to-mem enabled='no'/>
    <suspend-to-disk enabled='no'/>
  </pm>
  <devices>
    <emulator>/usr/sbin/qemu-system-x86_64</emulator>
    <disk type='block' device='disk'>
      <driver name='qemu' type='raw' cache='none' io='native'/>
      <source dev='/dev/virtual-machines/windows'/>
      <target dev='vda' bus='virtio'/>
      <boot order='2'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x07' function='0x0'/>
    </disk>
    <disk type='file' device='disk'>
      <driver name='qemu' type='raw'/>
      <source file='/home/lucas/media/windows.img'/>
      <target dev='vdb' bus='virtio'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x0c' function='0x0'/>
    </disk>
    <controller type='usb' index='0' model='ich9-ehci1'>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x05' function='0x7'/>
    </controller>
    <controller type='usb' index='0' model='ich9-uhci1'>
      <master startport='0'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x05' function='0x0' multifunction='on'/>
    </controller>
    <controller type='usb' index='0' model='ich9-uhci2'>
      <master startport='2'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x05' function='0x1'/>
    </controller>
    <controller type='usb' index='0' model='ich9-uhci3'>
      <master startport='4'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x05' function='0x2'/>
    </controller>
    <controller type='pci' index='0' model='pci-root'/>
    <controller type='virtio-serial' index='0'>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x06' function='0x0'/>
    </controller>
    <interface type='bridge'>
      <mac address='52:54:00:5a:02:e8'/>
      <source bridge='br0'/>
      <model type='virtio'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x0'/>
    </interface>
    <serial type='pty'>
      <target port='0'/>
    </serial>
    <console type='pty'>
      <target type='serial' port='0'/>
    </console>
    <input type='mouse' bus='ps2'/>
    <input type='keyboard' bus='ps2'/>
    <graphics type='spice' autoport='yes'/>
    <video>
      <model type='qxl' ram='65536' vram='65536' heads='1'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x0b' function='0x0'/>
    </video>
    <hostdev mode='subsystem' type='pci' managed='yes'>
      <source>
        <address domain='0x0000' bus='0x04' slot='0x00' function='0x0'/>
      </source>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x0a' function='0x0'/>
    </hostdev>
    <hostdev mode='subsystem' type='pci' managed='yes'>
      <source>
        <address domain='0x0000' bus='0x02' slot='0x00' function='0x0'/>
      </source>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x0'/>
    </hostdev>
    <hostdev mode='subsystem' type='pci' managed='yes'>
      <source>
        <address domain='0x0000' bus='0x02' slot='0x00' function='0x1'/>
      </source>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x04' function='0x0'/>
    </hostdev>
    <hostdev mode='subsystem' type='pci' managed='yes'>
      <source>
        <address domain='0x0000' bus='0x08' slot='0x00' function='0x0'/>
      </source>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x09' function='0x0'/>
    </hostdev>
    <memballoon model='virtio'>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x08' function='0x0'/>
    </memballoon>
  </devices>
</domain>

kevinxucs

Member

Registered: 2014-09-21

Posts: 2

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

For anyone who owns a Radeon HD 7850 (made by MSI), and having no output on Windows 8.1 after installing catalyst (or BSoD on Windows 7).

Try not passing through the HDMI audio controller inside the graphic card (address should be 01:00.1), it works for me.

Last edited by kevinxucs (2014-09-21 07:53:59)

TripleSpeeder

Member

Registered: 2011-05-02

Posts: 47

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Question on USB:
I have passed through one of my USB controllers, so all attached devices are directly recognized by the VM. This works fine for all kinds of devices, but one problem exists: It seems Seabios does not support USB keyboards, so I can not use the keyboard untill the VM has booted up. This is mostly not a problem, but I can't influence the boot process of Seabios (boot order etc.) and in case of unclean Windows guest shutdown I can't move the cursor in the "start windows in safe mode or boot normally"-screen...
I suspect this is a limitation of Seabios, but I did not find any confirmation on this. Any advice?

anickname

Member

Registered: 2014-01-07

Posts: 23

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Question on USB:
I have passed through one of my USB controllers, so all attached devices are directly recognized by the VM. This works fine for all kinds of devices, but one problem exists: It seems Seabios does not support USB keyboards, so I can not use the keyboard untill the VM has booted up. This is mostly not a problem, but I can't influence the boot process of Seabios (boot order etc.) and in case of unclean Windows guest shutdown I can't move the cursor in the "start windows in safe mode or boot normally"-screen...
I suspect this is a limitation of Seabios, but I did not find any confirmation on this. Any advice?

Use : -bios bios-256k.bin

Arakatak

Member

Registered: 2014-09-12

Posts: 4

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Seems like the problem in black screen after driver setup (VGA AMD R7 260x) is actually in lack of reset capability.
I got working passthrough with the following workarounds:
Passthrough only VGA, not its audio device
Cut power to VGA on every VM restart (possible with shutdown or suspend, not with reboot)

Thank you for all suggestions!

Q35 is largely overrated.  There are some configurations of Linux that need it, but Windows is fine on 440FX.

Will you explain the advantages of Q35 over 440FX, please? I know that Q35 supports pci-e, and 440FX doesn't. Are there any performance penalties for passing through pci-e card to virtual pci bus?

aw

Member

Registered: 2013-10-04

Posts: 921

Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

I made the switch from legacy VGA to libvirt + OVMF and I report success with some small issues.

My card (Sapphire 7970 Vapor-X) didn't have UEFI, so I flashed it with the Sapphire x280 Vapor-X bios (basically the same card, but it has UEFI). I left it like that for a week to see if I'd run into any problems and I didn't.

So today I set it up with OVMF (libvirt-git, ovmf-svn) and I got it to boot and work, but only as a secondary passthrough. Even if I remove the video devices I still don't get any output before windows loads the AMD drivers

Also, I need to use the audio device on the card because it is how I take sound to the TV, but it is crackling and lagging. It didn't do that before on legacy. And when I tried to play a game the card was stuttering in places where it didn't before.

Hmm, the setup is a little suspect since we're using a ROM image from a different card, even though I agree they seem like they should be pretty much the same.  Not getting any output until the driver loads sounds like the UEFI ROM isn't working.  Maybe try this ovmf image, which is just a copy of OVMF-pure-efi.fd from Gerd Hoffmann's Fedora build.  Maybe your OVMF image includes the CSM and is trying to use the VGA BIOS, I don't know.

---

http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

aw

Member

Registered: 2013-10-04

Posts: 921

Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Q35 is largely overrated.  There are some configurations of Linux that need it, but Windows is fine on 440FX.

Will you explain the advantages of Q35 over 440FX, please? I know that Q35 supports pci-e, and 440FX doesn't. Are there any performance penalties for passing through pci-e card to virtual pci bus?

Two things that Q35 does:

1) The guest PCI topology is closer to the host's.  IOW, you can put your assigned graphics card functions behind an emulated root port.

2) It provides access to extended PCIe config space of the device.

AFAIK, 1) is the main reason it gets used, but the topology difference is largely FUD.  There are some cases where Linux drivers find a PCIe device and assume they can poke the downstream port above it without testing whether it actually exists.  If you're attempting to do SLI, it might also be useful since the docs seem to be rather particular about the topology of the hardware for that use case.

For everything else, I don't expect it makes much difference and I'm not aware of any performance difference (if there is one, please share).  Having a Q35 model does not enable any sort of link negotiation; the link registers on the root port is emulated and fixed.  If anything I'd be afraid that a negotiation would result in lowering the speed of the device to match the root port (but I assume that's not happening because then the device and physical root port would be out of sync and probably stop working altogether).  The extended config space is potentially an area where there could be some added performance, manipulating the device IOTLB for instance, but I have doubts whether it's actually used and if it is, how well it works on Q35 anyway.

---

http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

TripleSpeeder

Member

Registered: 2011-05-02

Posts: 47

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Question on USB:
I have passed through one of my USB controllers, so all attached devices are directly recognized by the VM. This works fine for all kinds of devices, but one problem exists: It seems Seabios does not support USB keyboards, so I can not use the keyboard untill the VM has booted up. This is mostly not a problem, but I can't influence the boot process of Seabios (boot order etc.) and in case of unclean Windows guest shutdown I can't move the cursor in the "start windows in safe mode or boot normally"-screen...
I suspect this is a limitation of Seabios, but I did not find any confirmation on this. Any advice?

Use : -bios bios-256k.bin

Cool, works

If you are using libvirt to manage your VMs you need to

virsh edit <vmname>

and change the "loader" line to point to bios-256k.bin:

...
<os>
  <type arch='x86_64' machine='pc-q35-2.0'>hvm</type>
  <loader>/usr/share/qemu/bios-256k.bin</loader>
  <bootmenu enable='yes' timeout='3000'/>
</os>
...

(At least I did not find a way to change this setting with the virt-manager GUI)

Chetyre

Member

Registered: 2013-03-27

Posts: 34

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

I made the switch from legacy VGA to libvirt + OVMF and I report success with some small issues.

My card (Sapphire 7970 Vapor-X) didn't have UEFI, so I flashed it with the Sapphire x280 Vapor-X bios (basically the same card, but it has UEFI). I left it like that for a week to see if I'd run into any problems and I didn't.

So today I set it up with OVMF (libvirt-git, ovmf-svn) and I got it to boot and work, but only as a secondary passthrough. Even if I remove the video devices I still don't get any output before windows loads the AMD drivers

Also, I need to use the audio device on the card because it is how I take sound to the TV, but it is crackling and lagging. It didn't do that before on legacy. And when I tried to play a game the card was stuttering in places where it didn't before.

Hmm, the setup is a little suspect since we're using a ROM image from a different card, even though I agree they seem like they should be pretty much the same.  Not getting any output until the driver loads sounds like the UEFI ROM isn't working.  Maybe try this ovmf image, which is just a copy of OVMF-pure-efi.fd from Gerd Hoffmann's Fedora build.  Maybe your OVMF image includes the CSM and is trying to use the VGA BIOS, I don't know.

I have tried it just now, but it still didn't work. Still only showing an image when windows loads the driver.

But what really worries me is the performance issue, I tried it with Resident Evil 6 which shouldn't be a really demanding game for this card (and it wasn't when I was using qemu directly with legacy), but I couldn't even get it to the menu, it was rendering things at ~13 FPS.

Any idea on what may be wrong? I'm thinking about just returning my setup to how it was (might keep the new bios though, it wasn't causing any problems).

EDIT: Ok... Afterburner says my GPU clock is at 300 MHz when the game starts. This might not be a passthrough problem afterall.

Last edited by Chetyre (2014-09-21 16:27:09)

nose

Member

Registered: 2014-09-22

Posts: 5

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Hey i want to pass through my dual graphic card! (PowerColor HD6870X2, dualcore CrossFire)

i did everything as told in post #1,
but if i want to start a test VM with these settings:

# qemu-system-x86_64 -enable-kvm -M q35 -m 1024 -cpu host \
-smp 6,sockets=1,cores=6,threads=1 \
-bios /usr/share/qemu/bios.bin -vga none \
-device ioh3420,bus=pcie.0,addr=1c.0,multifunction=on,port=1,chassis=1,id=root.1 \
-device vfio-pci,host=03:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on \
-device vfio-pci,host=04:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on \
-device vfio-pci,host=04:00.1,bus=root.1,addr=00.1

i get this error:

qemu-system-x86_64: -device vfio-pci,host=03:00.0,bus=root.1,addr=00.0,multifunction=on: vfio: error opening /dev/vfio/1: No such file or directory
qemu-system-x86_64: -device vfio-pci,host=03:00.0,bus=root.1,addr=00.0,multifunction=on: vfio: failed to get group 1
qemu-system-x86_64: -device vfio-pci,host=03:00.0,bus=root.1,addr=00.0,multifunction=on: Device initialization failed.
qemu-system-x86_64: -device vfio-pci,host=03:00.0,bus=root.1,addr=00.0,multifunction=on: Device 'vfio-pci' could not be initialized

Additional info:

uname -r
3.16.0-1-mainline

The patched one from post #1

pci groups:

sh ./lsgroup
### Group 0 ###
    00:00.0 Host bridge: Intel Corporation Xeon E3-1200 v2/Ivy Bridge DRAM Controller (rev 09)
### Group 1 ###
    00:01.0 PCI bridge: Intel Corporation Xeon E3-1200 v2/3rd Gen Core processor PCI Express Root Port (rev 09)
    01:00.0 PCI bridge: Device 1fc8:0910 (rev 80)
    02:08.0 PCI bridge: Device 1fc8:0810
    02:09.0 PCI bridge: Device 1fc8:0890
    03:00.0 Display controller: Advanced Micro Devices, Inc. [AMD/ATI] Barts XT [Radeon HD 6870]
    04:00.0 VGA compatible controller: Advanced Micro Devices, Inc. [AMD/ATI] Barts XT [Radeon HD 6870]
    04:00.1 Audio device: Advanced Micro Devices, Inc. [AMD/ATI] Barts HDMI Audio [Radeon HD 6800 Series]
### Group 2 ###
    00:14.0 USB controller: Intel Corporation 7 Series/C210 Series Chipset Family USB xHCI Host Controller (rev 04)
### Group 3 ###
    00:16.0 Communication controller: Intel Corporation 7 Series/C210 Series Chipset Family MEI Controller #1 (rev 04)
### Group 4 ###
    00:1a.0 USB controller: Intel Corporation 7 Series/C210 Series Chipset Family USB Enhanced Host Controller #2 (rev 04)
### Group 5 ###
    00:1b.0 Audio device: Intel Corporation 7 Series/C210 Series Chipset Family High Definition Audio Controller (rev 04)
### Group 6 ###
    00:1c.0 PCI bridge: Intel Corporation 7 Series/C210 Series Chipset Family PCI Express Root Port 1 (rev c4)
### Group 7 ###
    00:1c.1 PCI bridge: Intel Corporation 7 Series/C210 Series Chipset Family PCI Express Root Port 2 (rev c4)
### Group 8 ###
    00:1c.2 PCI bridge: Intel Corporation 7 Series/C210 Series Chipset Family PCI Express Root Port 3 (rev c4)
### Group 9 ###
    00:1c.4 PCI bridge: Intel Corporation 7 Series/C210 Series Chipset Family PCI Express Root Port 5 (rev c4)
### Group 10 ###
    00:1d.0 USB controller: Intel Corporation 7 Series/C210 Series Chipset Family USB Enhanced Host Controller #1 (rev 04)
### Group 11 ###
    00:1e.0 PCI bridge: Intel Corporation 82801 PCI Bridge (rev a4)
### Group 12 ###
    00:1f.0 ISA bridge: Intel Corporation B75 Express Chipset LPC Controller (rev 04)
    00:1f.2 SATA controller: Intel Corporation 7 Series/C210 Series Chipset Family 6-port SATA Controller [AHCI mode] (rev 04)
    00:1f.3 SMBus: Intel Corporation 7 Series/C210 Series Chipset Family SMBus Controller (rev 04)
### Group 13 ###
    05:00.0 Ethernet controller: Realtek Semiconductor Co., Ltd. RTL8111/8168/8411 PCI Express Gigabit Ethernet Controller (rev 01)
### Group 14 ###
    06:00.0 SATA controller: ASMedia Technology Inc. ASM1062 Serial ATA Controller (rev 01)
### Group 15 ###
    07:00.0 Ethernet controller: Realtek Semiconductor Co., Ltd. RTL8111/8168/8411 PCI Express Gigabit Ethernet Controller (rev 06)
### Group 16 ###
    08:00.0 VGA compatible controller: NVIDIA Corporation G84 [GeForce 8600 GT] (rev a1)

can anyone help me or tell me where to read?
regards!

mauorrizze

Member

Registered: 2014-08-22

Posts: 18

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

@nose, I can't tell how well the pass-through of a dual graphic card does work, but this error:

vfio: error opening /dev/vfio/1: No such file or directory

tells me that the vfio module might be missing. Look at the starting post, "Binding a device to vfio-pci". Within the little script:

modprobe vfio-pci

But the rest of the script is important, too, to bind your graphics card(s) to the vfio driver.

aw

Member

Registered: 2013-10-04

Posts: 921

Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Yes, what mauorrizze says, additionallly...

Hey i want to pass through my dual graphic card! (PowerColor HD6870X2, dualcore CrossFire)

i did everything as told in post #1,
but if i want to start a test VM with these settings:

# qemu-system-x86_64 -enable-kvm -M q35 -m 1024 -cpu host \
-smp 6,sockets=1,cores=6,threads=1 \
-bios /usr/share/qemu/bios.bin -vga none \
-device ioh3420,bus=pcie.0,addr=1c.0,multifunction=on,port=1,chassis=1,id=root.1 \
-device vfio-pci,host=03:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on \
-device vfio-pci,host=04:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on \
-device vfio-pci,host=04:00.1,bus=root.1,addr=00.1

This is an invalid config, you're assigning both 3:00.0 and 4:00.0 to the same address on the guest.  Also note that only one of the GPUs is identified as VGA (4:00.0), only that one should use x-vga.  I would start with just the VGA GPU + audio (4:00.0 & 4:00.1), bind 3:00.0 to vfio-pci, but don't expose it to the guest.  See if you can get that working first.  Then add the 2nd GPU.  There are several ways you could expose it, the easiest might be to try to expose it as function 2 on the same device.  You could also add a 2nd root port and expose it there.  QEMU also has support for PCIe switches, so you could add an upstream switch port with 2 downstream ports, just like you have on real hardware and expose VGA+audio and GPU.

Now the bad news; there's a very real possibility that the switches on this card are designed to re-route DMA between the endpoints so that peer-to-peer DMA between the GPUs never leaves the card.  That means the DMA might not be seen by the IOMMU and without ACS on the switches we don't have the ability to configure it for upstream forwarding.  If that's the case, then the only way these devices could work in a VM is if all of their address ranges were identity mapped into the guest, which is not something we have any ability to do.  Try it and see if it works, but know that it's a gamble.

---

http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

nose

Member

Registered: 2014-09-22

Posts: 5

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Thanks for your replay, mauorrizze, i fixed my scripts!

Yes, what mauorrizze says, additionallly...

Hey i want to pass through my dual graphic card! (PowerColor HD6870X2, dualcore CrossFire)

i did everything as told in post #1,
but if i want to start a test VM with these settings:

# qemu-system-x86_64 -enable-kvm -M q35 -m 1024 -cpu host \
-smp 6,sockets=1,cores=6,threads=1 \
-bios /usr/share/qemu/bios.bin -vga none \
-device ioh3420,bus=pcie.0,addr=1c.0,multifunction=on,port=1,chassis=1,id=root.1 \
-device vfio-pci,host=03:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on \
-device vfio-pci,host=04:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on \
-device vfio-pci,host=04:00.1,bus=root.1,addr=00.1

This is an invalid config, you're assigning both 3:00.0 and 4:00.0 to the same address on the guest.  Also note that only one of the GPUs is identified as VGA (4:00.0), only that one should use x-vga.  I would start with just the VGA GPU + audio (4:00.0 & 4:00.1), bind 3:00.0 to vfio-pci, but don't expose it to the guest.  See if you can get that working first.  Then add the 2nd GPU.  There are several ways you could expose it, the easiest might be to try to expose it as function 2 on the same device.  You could also add a 2nd root port and expose it there.  QEMU also has support for PCIe switches, so you could add an upstream switch port with 2 downstream ports, just like you have on real hardware and expose VGA+audio and GPU.

Now the bad news; there's a very real possibility that the switches on this card are designed to re-route DMA between the endpoints so that peer-to-peer DMA between the GPUs never leaves the card.  That means the DMA might not be seen by the IOMMU and without ACS on the switches we don't have the ability to configure it for upstream forwarding.  If that's the case, then the only way these devices could work in a VM is if all of their address ranges were identity mapped into the guest, which is not something we have any ability to do.  Try it and see if it works, but know that it's a gamble.

Tannks aw,
i tried what you said:

# qemu-system-x86_64 -enable-kvm -M q35 -m 1024 -cpu host \
> -smp 6,sockets=1,cores=6,threads=1 \
> -bios /usr/share/qemu/bios.bin -vga none \
> -device ioh3420,bus=pcie.0,addr=1c.0,multifunction=on,port=1,chassis=1,id=root.1 \
> -device vfio-pci,host=04:00.0,bus=root.1,multifunction=on,x-vga=on \
> -device vfio-pci,host=04:00.1,bus=root.1

gives me:

qemu-system-x86_64: -device vfio-pci,host=04:00.0,bus=root.1,multifunction=on,x-vga=on: VFIO 0000:04:00.0 BAR 0 mmap unsupported. Performance may be slow

and a QEMU console window! but sadly the screens attached to the passthoughed card just tun off and stay black.

but at least some vm starts..

aw, in Windows the card appears as two graphic cards, but only one appears to have connectors. And i have no idea if they redirect any DMA traffic.

UPDATE:
still no luck to get the screen on the passtroughed card working...
i noticed that in the group of the graphiccard:

....
....
### Group 1 ###
    00:01.0 PCI bridge: Intel Corporation Xeon E3-1200 v2/3rd Gen Core processor PCI Express Root Port (rev 09)
    01:00.0 PCI bridge: Device 1fc8:0910 (rev 80)
    02:08.0 PCI bridge: Device 1fc8:0810
    02:09.0 PCI bridge: Device 1fc8:0890
    03:00.0 Display controller: Advanced Micro Devices, Inc. [AMD/ATI] Barts XT [Radeon HD 6870]
    04:00.0 VGA compatible controller: Advanced Micro Devices, Inc. [AMD/ATI] Barts XT [Radeon HD 6870]
    04:00.1 Audio device: Advanced Micro Devices, Inc. [AMD/ATI] Barts HDMI Audio [Radeon HD 6800 Series]
....
....

are alot of pci bridges which surely belong to the integrated second GPU and its Crossfire ability.
better view:

# lspci -t
-[0000:00]-+-00.0
           +-01.0-[01-04]----00.0-[02-04]--+-[0000:04]-+-00.0
           |                               |           \-00.1
           |                               +-[0000:03]---00.0
           |                               \-[0000:02]-+-08.0-[00]--
           |                                           \-09.0-[00]--
           +-14.0
           +-16.0
           +-1a.0
           +-1c.0-[05]----00.0
           +-1c.1-[06]----00.0
           +-1c.2-[07]----00.0
           +-1c.4-[08]----00.0
           +-1d.0
           +-1e.0-[09]--
           +-1f.0
           +-1f.2
           \-1f.3

I tried to pass trough the whole group, but i get errors for example:

-device vfio-pci,host=01:00.0,bus=root.1: vfio: Assignment of PCIe type 0x5 devices is not currently supported

i dont want to buy a new graphiccard

if anybody has suggestions, feel free!

Last edited by nose (2014-09-22 15:34:47)

Duelist

Member

Registered: 2014-09-22

Posts: 358

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Thanks for your replay, mauorrizze, i fixed my scripts!

Yes, what mauorrizze says, additionallly...

Hey i want to pass through my dual graphic card! (PowerColor HD6870X2, dualcore CrossFire)

i did everything as told in post #1,
but if i want to start a test VM with these settings:

# qemu-system-x86_64 -enable-kvm -M q35 -m 1024 -cpu host \
-smp 6,sockets=1,cores=6,threads=1 \
-bios /usr/share/qemu/bios.bin -vga none \
-device ioh3420,bus=pcie.0,addr=1c.0,multifunction=on,port=1,chassis=1,id=root.1 \
-device vfio-pci,host=03:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on \
-device vfio-pci,host=04:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on \
-device vfio-pci,host=04:00.1,bus=root.1,addr=00.1

This is an invalid config, you're assigning both 3:00.0 and 4:00.0 to the same address on the guest.  Also note that only one of the GPUs is identified as VGA (4:00.0), only that one should use x-vga.  I would start with just the VGA GPU + audio (4:00.0 & 4:00.1), bind 3:00.0 to vfio-pci, but don't expose it to the guest.  See if you can get that working first.  Then add the 2nd GPU.  There are several ways you could expose it, the easiest might be to try to expose it as function 2 on the same device.  You could also add a 2nd root port and expose it there.  QEMU also has support for PCIe switches, so you could add an upstream switch port with 2 downstream ports, just like you have on real hardware and expose VGA+audio and GPU.

Now the bad news; there's a very real possibility that the switches on this card are designed to re-route DMA between the endpoints so that peer-to-peer DMA between the GPUs never leaves the card.  That means the DMA might not be seen by the IOMMU and without ACS on the switches we don't have the ability to configure it for upstream forwarding.  If that's the case, then the only way these devices could work in a VM is if all of their address ranges were identity mapped into the guest, which is not something we have any ability to do.  Try it and see if it works, but know that it's a gamble.

Tannks aw,
i tried what you said:

# qemu-system-x86_64 -enable-kvm -M q35 -m 1024 -cpu host \
> -smp 6,sockets=1,cores=6,threads=1 \
> -bios /usr/share/qemu/bios.bin -vga none \
> -device ioh3420,bus=pcie.0,addr=1c.0,multifunction=on,port=1,chassis=1,id=root.1 \
> -device vfio-pci,host=04:00.0,bus=root.1,multifunction=on,x-vga=on \
> -device vfio-pci,host=04:00.1,bus=root.1

gives me:

qemu-system-x86_64: -device vfio-pci,host=04:00.0,bus=root.1,multifunction=on,x-vga=on: VFIO 0000:04:00.0 BAR 0 mmap unsupported. Performance may be slow

and a QEMU console window! but sadly the screens attached to the passthoughed card just tun off and stay black.

but at least some vm starts..

aw, in Windows the card appears as two graphic cards, but only one appears to have connectors. And i have no idea if they redirect any DMA traffic.

UPDATE:
still no luck to get the screen on the passtroughed card working...
i noticed that in the group of the graphiccard:

....
....
### Group 1 ###
    00:01.0 PCI bridge: Intel Corporation Xeon E3-1200 v2/3rd Gen Core processor PCI Express Root Port (rev 09)
    01:00.0 PCI bridge: Device 1fc8:0910 (rev 80)
    02:08.0 PCI bridge: Device 1fc8:0810
    02:09.0 PCI bridge: Device 1fc8:0890
    03:00.0 Display controller: Advanced Micro Devices, Inc. [AMD/ATI] Barts XT [Radeon HD 6870]
    04:00.0 VGA compatible controller: Advanced Micro Devices, Inc. [AMD/ATI] Barts XT [Radeon HD 6870]
    04:00.1 Audio device: Advanced Micro Devices, Inc. [AMD/ATI] Barts HDMI Audio [Radeon HD 6800 Series]
....
....

are alot of pci bridges which surely belong to the integrated second GPU and its Crossfire ability.
better view:

# lspci -t
-[0000:00]-+-00.0
           +-01.0-[01-04]----00.0-[02-04]--+-[0000:04]-+-00.0
           |                               |           \-00.1
           |                               +-[0000:03]---00.0
           |                               \-[0000:02]-+-08.0-[00]--
           |                                           \-09.0-[00]--
           +-14.0
           +-16.0
           +-1a.0
           +-1c.0-[05]----00.0
           +-1c.1-[06]----00.0
           +-1c.2-[07]----00.0
           +-1c.4-[08]----00.0
           +-1d.0
           +-1e.0-[09]--
           +-1f.0
           +-1f.2
           \-1f.3

I tried to pass trough the whole group, but i get errors for example:

-device vfio-pci,host=01:00.0,bus=root.1: vfio: Assignment of PCIe type 0x5 devices is not currently supported

i dont want to buy a new graphiccard

if anybody has suggestions, feel free!

"BAR 0 mmap unsupported. Performance may be slow"

Get lspci -vvnn, find your gpu, find it's region 10 memory range
It'll look something like this:

01:00.0 VGA compatible controller [0300]: Advanced Micro Devices, Inc. [AMD/ATI] Cape Verde PRO [Radeon HD 7750 / R7 250E] [1002:683f] (prog-if 00 [VGA controller])
        Subsystem: ASUSTeK Computer Inc. Device [1043:0459]
        Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx+
        Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
        Interrupt: pin A routed to IRQ 18
        Region 0: Memory at c0000000 (64-bit, prefetchable) [disabled] [size=256M]
        Region 2: Memory at fe300000 (64-bit, non-prefetchable) [disabled] [size=256K]
        Region 4: I/O ports at e000 [disabled] [size=256]
        Expansion ROM at fe340000 [disabled] [size=128K]
        Capabilities: <access denied>
        Kernel driver in use: vfio-pci
        Kernel modules: radeon

Then, after acknowledging region 10 memory range, you go cat /proc/iomem
and look for that range, it'll look like this:

  c0000000-cfffffff : PCI Bus 0000:01
    c0000000-cfffffff : 0000:01:00.0
AND HERE HIDES THE VESAFB!

(i've disabled vesa already)

I guess the GPU you're trying to pass is the first one in the system, and various uefi/bios output data on screen using it. Vesa does the same. Some motherboards have the option to change the default vga, but i don't have on, so...

To turn it off, assuming you have GRUB2:
edit /etc/default/grub
change GRUB_GFXPAYLOAD_LINUX="keep" to "text"
and append nofb to GRUB_CMDLINE_LINUX="..."
And then run grub2-mkconfig -o /boot/grub2/grub.cfg

Then reboot and check iomem again, vesa should be gone.

Vesa is the framebuffer, check the archwiki for more info on why you would or would not need it.

P.S. now there is two of us, trying to get crossfire working inside the vm.

Last edited by Duelist (2014-09-22 21:36:23)

---

The forum rules prohibit requesting support for distributions other than arch.
I gave up. It was too late.
What I was trying to do.
The reference about VFIO and KVM VGA passthrough.

sinny

Member

Registered: 2014-05-24

Posts: 75

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

seems like no more nvidia for me (both for ideology and practical aspects)

gigabyte r7 240 working using ovmf (or at least seem to - will stick with win7/vga for now)

TripleSpeeder

Member

Registered: 2011-05-02

Posts: 47

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Can someone give a few pointer what benefits OVMF would give? I understand that it is basically providing an UEFI Bios to the VM, enabling stuff like secureboot etc. But if I have a working setup with the old Seabios - will using OVMF give me any advantage?

sinny

Member

Registered: 2014-05-24

Posts: 75

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Can someone give a few pointer what benefits OVMF would give? I understand that it is basically providing an UEFI Bios to the VM, enabling stuff like secureboot etc. But if I have a working setup with the old Seabios - will using OVMF give me any advantage?

i wonder if you have actually read/understood anything at all...

it's gpu passthrough we are talking about here - not virtualising uefi boot process in general.
the core advantage ovmf gives is no vga legacy mess (kernel patching, etc).

p.s. i may be wrong here, but i really doubt anyone cares about secureboot (or any other weird stuff) when running 3d-accelerated vm.