You are not logged in.

#1451 2014-03-28 17:18:36

TripleSpeeder
Member
Registered: 2011-05-02
Posts: 47

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

TripleSpeeder wrote:

Okay, I am out of ideas now. I got everything running perfectly when starting qemu manual. Now i want to move to a libvirt xml file, so i can start/stop/manage my VMs with virtmanager. But no matter what I do, i don't get the card passed through.
Here is a minimal xml config to reproduce:

When starting this instance e.g. through "virsh start win7" i get the following error:

# virsh start win7
error: Failed to start domain win7
error: internal error: early end of file from monitor: possible problem:
qemu-system-x86_64: -device vfio-pci,host=01:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on: vfio: error opening /dev/vfio/13: Operation not permitted
qemu-system-x86_64: -device vfio-pci,host=01:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on: vfio: failed to get group 13
qemu-system-x86_64: -device vfio-pci,host=01:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on: Device initialization failed.
qemu-system-x86_64: -device vfio-pci,host=01:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on: Device 'vfio-pci' could not be initialized

What I did:

  • Make sure that AMD-card is assigned to vfio

  • Edit /etc/libvirt/qemu.conf and set user/group qemu runs as to "root"

  • Getting desperate: chmod a+rw /dev/vfio/13

But no luck. Everytime the same error :-(

Does anyone have a clue what I am missing? As I said, when I launch qemu manually (as root) everything works fine...

Problem solved!

I missed one part of the virt-manager configuration to fix the permissions issue: You need to edit /etc/libvirt/qemu.conf and explicitly allow access to the /dev/vfio/<group>:

...
cgroup_device_acl = [
    "/dev/null", "/dev/full", "/dev/zero",
    "/dev/random", "/dev/urandom",
    "/dev/ptmx", "/dev/kvm", "/dev/kqemu",
    "/dev/rtc","/dev/hpet", "/dev/vfio/vfio", 
    "dev/vfio/3",
    "dev/vfio/13",
    "dev/vfio/16",
]
...

As you can see I added three groups that i am passing through.

With this change I everything was fine and I can now happily start/stop/restart etc. my VM through virt-manager. By the way, there is a tool included with libvirt that supports you with converting a qemu-command-line to xml config: virsh domxml-from-native. It even takes care of adding the extra namespace to the config file to support the quemu-commandline directive. I had to remove one or two items from my commandline to get it running, but still it was a great help.

Another big hurdle was getting sound via pulseaudio to work. As a vm started by virt-manager is not aware of the current login session it has no access to your normal pulseaudio instance. In order to get this working you need to have pulseaudio running in system mode. Note that this is in general considered a security risk and not recomended, however it is the only way (to my knowledege) to get the pulseaudio integration to work.
You need to do the following steps:

  • Create a user "pulse" with homedir "/var/run/pulse"

  • Create a group "pulse", add user pulse to this group

  • Add user pulse to group "audio" so it is allowed to access the audio hardware

Now you can already try to start pulseaudio in system mode (make sure to stop any running pulseaudio instance from your login session). As root, type:

# pulseaudio --system -v

If you see some error about problems connecting to DBus: Check https://bbs.archlinux.org/viewtopic.php … 81#p563481

Now you should already be able to startup your vm and the sound should work. If everything is working you can create a systemd service to start pulseaudio on boot in system mode:

# systemd service spec for pulseaudio running in system mode -- not recommended though!
# on arch, put it under /etc/systemd/system/pulseaudio.service
# start with: systemctl start pulseaudio.service
# enable on boot: systemctl enable pulseaudio.service 
[Unit]
Description=Pulseaudio sound server
After=avahi-daemon.service network.target
 
[Service]
ExecStart=/usr/bin/pulseaudio --system --disallow-exit --disallow-module-loading
ExecReload=/bin/kill -HUP $MAINPID
 
[Install]
WantedBy=multi-user.target

(Source: https://gist.github.com/awidegreen/6003640)

And finally you need to prevent autostarting a pulseaudio instance at login. I achieved this by removing /etc/xdg/autostart/pulseaudio-kde.desktop.

My current config:

# virsh dumpxml windows
<domain type='kvm' xmlns:qemu='http://libvirt.org/schemas/domain/qemu/1.0'>
  <name>windows</name>
  <uuid>f702c5eb-83d4-4b50-b909-8da878e08d36</uuid>
  <memory unit='KiB'>8290304</memory>
  <currentMemory unit='KiB'>4145152</currentMemory>
  <vcpu placement='static'>6</vcpu>
  <os>
    <type arch='x86_64' machine='pc-q35-2.0'>hvm</type>
    <loader>/usr/share/qemu/bios.bin</loader>
    <boot dev='hd'/>
    <bootmenu enable='no'/>
  </os>
  <features>
    <acpi/>
  </features>
  <cpu mode='custom' match='exact'>
    <model fallback='allow'>SandyBridge</model>
    <vendor>Intel</vendor>
    <feature policy='require' name='vme'/>
    <feature policy='require' name='dtes64'/>
    <feature policy='require' name='vmx'/>
    <feature policy='require' name='erms'/>
    <feature policy='require' name='xtpr'/>
    <feature policy='require' name='smep'/>
    <feature policy='require' name='pcid'/>
    <feature policy='require' name='est'/>
    <feature policy='require' name='monitor'/>
    <feature policy='require' name='smx'/>
    <feature policy='require' name='tm'/>
    <feature policy='require' name='acpi'/>
    <feature policy='require' name='osxsave'/>
    <feature policy='require' name='ht'/>
    <feature policy='require' name='pdcm'/>
    <feature policy='require' name='fsgsbase'/>
    <feature policy='require' name='f16c'/>
    <feature policy='require' name='ds'/>
    <feature policy='require' name='tm2'/>
    <feature policy='require' name='ss'/>
    <feature policy='require' name='pbe'/>
    <feature policy='require' name='ds_cpl'/>
    <feature policy='require' name='rdrand'/>
  </cpu>
  <clock offset='localtime'/>
  <on_poweroff>destroy</on_poweroff>
  <on_reboot>restart</on_reboot>
  <on_crash>destroy</on_crash>
  <devices>
    <emulator>/usr/bin/qemu-system-x86_64</emulator>
    <disk type='block' device='disk'>
      <driver name='qemu' type='raw'/>
      <source dev='/dev/sdc'/>
      <target dev='vda' bus='virtio'/>
      <address type='pci' domain='0x0000' bus='0x02' slot='0x01' function='0x0'/>
    </disk>
    <disk type='file' device='cdrom'>
      <driver name='qemu' type='raw'/>
      <source file='/home/michael/hdd/isos/virtio-win-0.1-74.iso'/>
      <target dev='hda' bus='ide'/>
      <readonly/>
      <address type='drive' controller='0' bus='0' target='0' unit='0'/>
    </disk>
    <controller type='sata' index='0'>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x1f' function='0x2'/>
    </controller>
    <controller type='pci' index='0' model='pcie-root'/>
    <controller type='pci' index='1' model='dmi-to-pci-bridge'>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x1e' function='0x0'/>
    </controller>
    <controller type='pci' index='2' model='pci-bridge'>
      <address type='pci' domain='0x0000' bus='0x01' slot='0x01' function='0x0'/>
    </controller>
    <controller type='ide' index='0'/>
    <interface type='bridge'>
      <mac address='52:54:00:29:ae:89'/>
      <source bridge='xenbr0'/>
      <model type='virtio'/>
      <address type='pci' domain='0x0000' bus='0x02' slot='0x03' function='0x0'/>
    </interface>
    <memballoon model='virtio'>
      <address type='pci' domain='0x0000' bus='0x02' slot='0x02' function='0x0'/>
    </memballoon>
  </devices>
  <qemu:commandline>
    <qemu:arg value='-device'/>
    <qemu:arg value='ioh3420,bus=pcie.0,addr=1c.0,multifunction=on,port=1,chassis=1,id=root.1'/>
    <qemu:arg value='-device'/>
    <qemu:arg value='ich9-intel-hda,bus=pcie.0,addr=1b.0,id=sound0'/>
    <qemu:arg value='-device'/>
    <qemu:arg value='hda-duplex,id=sound0-codec0,bus=sound0.0,cad=0'/>
    <qemu:arg value='-device'/>
    <qemu:arg value='vfio-pci,host=00:1a.0,bus=pcie.0'/>
    <qemu:arg value='-device'/>
    <qemu:arg value='vfio-pci,host=04:00.0,bus=pcie.0'/>
    <qemu:arg value='-device'/>
    <qemu:arg value='vfio-pci,host=01:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on'/>
    <qemu:env name='QEMU_AUDIO_DRV' value='pa'/>
  </qemu:commandline>
</domain>

Everything is working perfectly, I can restart the VM as often as I like, played several hours iRacing with framerates in the range of 180-220 FPS (full HD with max quality) and shortly tried several other games which all seem to run fine with top performance. Also for the sound there is no noticeable delay using pulseaudio.

I finally deleted my windows partition and removed all dualboot settings from grub :-)

Only thing to keep in mind is you need to manually set up the vfio groups before trying to start the VM. But somewhere in this thread there was already information how to set this up to happen automatically at boot time, so this should also be solvable smile

Offline

#1452 2014-03-29 23:26:19

apex8
Member
Registered: 2014-03-29
Posts: 60

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Hi together,
I`m trying to setup a Windows 7 guest and was following this thread. I successfully created a q35 guest with kvm enabled using

-vga std \
-mem-path /dev/hugepages \
-m 4096
-device ioh3420,bus=pcie.0,addr=1c.0,multifunction=on,port=1,chassis=1,id=root.1 \
-device vfio-pci,host=01:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on \
-device vfio-pci,host=01:00.1,bus=root.1,addr=00.1 \

When I finished the initial guest setup, which included installing nvidia drivers for my GTX 760, I tried to switch to

-vga none

Doing so I can see the initial bios screen and the OS is starting to boot up. However during that, my mouse cursor gets laggy and after a couple of seconds the whole host crashes. I read that the nouveau module could cause such things, so that module is already blacklisted, which does not solve the problem. I`m using an Intel IGP as primary graphics device and the compiled the kernel, qemu and seabios from the packages provided here.

At the moment I`m out of ideas.
My current start script is:

qemu-system-x86_64 \
-nodefaults \
-name win7 \
-enable-kvm \
-cpu host \
-smp 6,sockets=1,cores=6,threads=1 \
-machine type=q35,accel=kvm \
-m 4096 \
-mem-path /dev/hugepages \
-mem-prealloc \
-k de \
-bios /usr/share/qemu/bios.bin \
-vga none \
-drive if=none,file=drive/win_drive.img,id=vdisk1,cache=none,format=raw,aio=native \
-device virtio-blk,scsi=off,config-wce=off,drive=vdisk1,id=disk1 \
-boot order=c \
-rtc base=utc \
-device ioh3420,bus=pcie.0,addr=1c.0,multifunction=on,port=1,chassis=1,id=root.1 \
-device vfio-pci,host=01:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on \
-device vfio-pci,host=01:00.1,bus=root.1,addr=00.1 \
-usb \
-device ich9-usb-uhci3,id=uhci \
-device usb-ehci,id=ihci \
-usbdevice tablet \

Is there any way, how I could track down, what I may have forgotten to configure, or whats going wrong? Reading through the whole thread didn't help me so far. Its rather confusing, because there is so much information here.
Any hints would be helpful.
Thanks in advance
apex

Offline

#1453 2014-03-29 23:55:31

dwe11er
Member
Registered: 2014-03-18
Posts: 73

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

What motherboard and CPU do you have? I've got i7-4771, ASRock Z87E-ITX with Intel as host's graphics and GTX 760 for Windows 8.1 and it's working flawlessly.

Last edited by dwe11er (2014-03-31 17:32:10)

Offline

#1454 2014-03-30 00:45:19

etcet
Member
Registered: 2010-02-15
Posts: 3

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Thanks to nbhs and everyone else involved in this thread, I've got VGA passthrough working using this guide with the following setup:

Intel 4770
Asus Z87-Plus
Gigabyte "Windforce" GTX 770 GV-N770OC-2GD

I'm starting X with the onboard Intel GPU and getting near native Windows performance on a passthroughed 770.

For others and my future self, here are the steps involved. I think this is everything:

Install qemu-git
Install seabios-git

The i915 kernel patch is required. I couldn't get the provided mainline kernel to work. I built the latest kernel and added the i915 patch with the following:

pacman -S abs base-devel
ABSROOT=. abs core/linux
cd core/linux/
wget https://gist.githubusercontent.com/anonymous/732acf9551136a78e91b/raw/5b97d50e7fb7faced258ad8948b7a82127d25f31/i915_313rc4.patch
wget https://gist.githubusercontent.com/etcet/9864922/raw/e49d204ddaf2d6130c91d94d0ce6a9123c6197cb/i915-pkgbuild.patch
patch -p3 < i915-pkgbuild.patch
makepkg
pacman -U linux-i915-headers-3.13.7-1-x86_64.pkg.tar.xz
pacman -U linux-i915-3.13.7-1-x86_64.pkg.tar.xz

More: https://wiki.archlinux.org/index.php/Ke … ild_System

Get PCI stub ids for passthrough devices:

[chris@archserver linux]$ lspci | grep VGA
00:02.0 VGA compatible controller: Intel Corporation Xeon E3-1200 v3/4th Gen Core Processor Integrated Graphics Controller (rev 06)
01:00.0 VGA compatible controller: NVIDIA Corporation GK104 [GeForce GTX 770] (rev a1)

[chris@archserver src]$ lspci -n | grep "01:00.0"
01:00.0 0300: 10de:1184 (rev a1)

Add new kernel to grub, /etc/grub.d/40_custom:

menuentry "Arch Linux i915 kernel w/ iommu" {
    set gfxpayload=keep
    insmod ext2
    echo 'Loading Arch Linux i915 kernel w/ iommu ...'
    linux /boot/vmlinuz-linux-i915 root=UUID=f4af663b-5407-498c-b46d-d3e2b9b6bc64 rw intel_iommu=on pci-stub.ids=10de:1184
    echo 'Loading Arch Linux i915 kernel w/ iommu initramfs ...'
    initrd /boot/initramfs-linux-i915.img
}

Build grub conf:

sudo grub-mkconfig -o /boot/grub/grub.cfg

Miscellaneous:

[chris@archserver src]$ cat /etc/modprobe.d/blacklist.conf 
blacklist nouveau

[chris@archserver src]$ cat /etc/modprobe.d/vfio_iommu_type1.conf
options vfio_iommu_type1 allow_unsafe_interrupts=1

Simple xorg.conf for running host with onboard Intel GPU: https://gist.github.com/etcet/9865391

Reboot into new kernel and take the oppurtunity to check/change BIOS settings:

Advanced
-- System Agent Configuration
---- Vt-d: Enabled
---- Graphics Configuration
------ Primary Display: iGPU

Test it:

sudo /usr/bin/vfio-bind 0000:01:00.0

sudo qemu-system-x86_64 -enable-kvm -M q35 -m 1024 -cpu host \
-smp 6,sockets=1,cores=6,threads=1 \
-bios /usr/share/qemu/bios.bin -vga none \
-device ioh3420,bus=pcie.0,addr=1c.0,multifunction=on,port=1,chassis=1,id=root.1 \
-device vfio-pci,host=01:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on

Last edited by etcet (2014-03-30 00:46:12)

Offline

#1455 2014-03-30 07:28:04

jonascj
Member
Registered: 2014-02-28
Posts: 20

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Preparing the GPU so we can bind it to vfio

For the next step we need to:

  1. Blacklist radeon or nouveau or nvidia or fglrx on /etc/modprobe.d/blacklist.conf

    Example, blacklisting the opensource radeon module:

    echo "blacklist radeon" >> /etc/modprobe.d/blacklist.conf 
  2. Use pci-stub

    In my case since i have 2 radeon cards blacklisting the radeon module is not an option, so i use pci-stub.

    NOTE: If pci-stub was built as a module, you'll need to modify /etc/mkinitcpio.conf and add pci-stub in the MODULES section, after that you need to update your initramfs like this.

    mkinitcpio -p linux-mainline

    lspci

    07:00.0 VGA compatible controller: Advanced Micro Devices, Inc. [AMD/ATI] Cayman PRO [Radeon HD 6950] <-- radeon 6950
    07:00.1 Audio device: Advanced Micro Devices, Inc. [AMD/ATI] Cayman/Antilles HDMI Audio [Radeon HD 6900 Series] <-- radeon 6950 audio

    lspci -n

    07:00.0 0300: 1002:6719 <-- radeon 6950
    07:00.1 0403: 1002:aa80 <-- radeon 6950 audio

    Now add this to your grub cfg file:

    pci-stub.ids=1002:6719,1002:aa80

    dmesg | grep pci-stub

    [    2.096151] pci-stub: add 1002:6719 sub=FFFFFFFF:FFFFFFFF cls=00000000/00000000
    [    2.096160] pci-stub 0000:07:00.0: claimed by stub
    [    2.096165] pci-stub: add 1002:AA80 sub=FFFFFFFF:FFFFFFFF cls=00000000/00000000
    [    2.096174] pci-stub 0000:07:00.1: claimed by stub
    [    2.096178] pci-stub: add 1B21:1042 sub=FFFFFFFF:FFFFFFFF cls=00000000/00000000

Hi all.

I am confused about whether I should do both 1 and 2, or only 1 if I am okay with blacklisting the Radeon module.

The wording "For the next step we need to: 1)    2)" and "In my case since i have 2 radeon cards blacklisting the radeon module is not an option, so i use pci-stub." suggests it is an choice between the two, but I am not so sure. Everywhere I see people looking at the pci-stub grep's. So it seems everybody is doing pci-stub...

My hoped setup is with a single R9 280x (kernel calls it Radeon 79something) for the Windows guest and Intel built-in for Linux host. So as far as I can figure out I would be fine with black listing the radeon module.

Last edited by jonascj (2014-03-30 07:36:00)

Offline

#1456 2014-03-30 07:59:38

apex8
Member
Registered: 2014-03-29
Posts: 60

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

dwe11er wrote:

What motherboard and CPU do you have? I've got i7-4771, ASRock Z86E-ITX with Intel as host's graphics and GTX 760 for Windows 8.1 and it's working flawlessly.

I`ve got an Core i7-3770 and an Asrock Z68 Pro3 Gen3
I`m booting with

BOOT_IMAGE=/vmlinuz-linux-mainline root=UUID=0bf391df-8b9d-4a36-9961-9cc54bcd24cd rw intel_iommu=on hugepagesz=2MB hugepages=2100 pci-stub.ids=10de:1187,10de:0e0a,1102:0012,8086:1c2d

binding the graphics adapter works without reporting any errors

EDIT

dmesg | grep IOMMU                                                                                                ⏎
[    0.000000] Intel-IOMMU: enabled
[    0.089982] dmar: IOMMU 0: reg_base_addr fed90000 ver 1:0 cap c0000020e60262 ecap f0101a
[    0.089986] dmar: IOMMU 1: reg_base_addr fed91000 ver 1:0 cap c9008020660262 ecap f0105a
[    0.090057] IOAPIC id 2 under DRHD base  0xfed91000 IOMMU 1
[    0.470765] IOMMU 0 0xfed90000: using Queued invalidation
[    0.470766] IOMMU 1 0xfed91000: using Queued invalidation
[    0.470767] IOMMU: Setting RMRR:
[    0.470776] IOMMU: Setting identity map for device 0000:00:02.0 [0xbf800000 - 0xcf9fffff]
[    0.471865] IOMMU: Setting identity map for device 0000:00:1d.0 [0xbddcd000 - 0xbddf7fff]
[    0.471880] IOMMU: Setting identity map for device 0000:00:1a.0 [0xbddcd000 - 0xbddf7fff]
[    0.471889] IOMMU: Prepare 0-16MiB unity mapping for LPC
[    0.471894] IOMMU: Setting identity map for device 0000:00:1f.0 [0x0 - 0xffffff]

Last edited by apex8 (2014-03-30 08:07:56)

Offline

#1457 2014-03-30 13:21:20

Blind Tree Frog
Member
Registered: 2013-12-31
Posts: 27

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

jonascj wrote:

Hi all.

I am confused about whether I should do both 1 and 2, or only 1 if I am okay with blacklisting the Radeon module.

The wording "For the next step we need to: 1)    2)" and "In my case since i have 2 radeon cards blacklisting the radeon module is not an option, so i use pci-stub." suggests it is an choice between the two, but I am not so sure. Everywhere I see people looking at the pci-stub grep's. So it seems everybody is doing pci-stub...

My hoped setup is with a single R9 280x (kernel calls it Radeon 79something) for the Windows guest and Intel built-in for Linux host. So as far as I can figure out I would be fine with black listing the radeon module.

If blacklisting works for you, then you only need to use step 1 and can skip step 2.

Offline

#1458 2014-03-30 19:01:43

apex8
Member
Registered: 2014-03-29
Posts: 60

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

I went through my qemu configuration and gave it a try with a whole new guest installation. I used the following script

sudo qemu-system-x86_64 \
-nodefaults \
-name win7 \
-enable-kvm \
-machine type=q35,accel=kvm \
-m 4096 \
-mem-path /dev/hugepages \
-mem-prealloc \
-k de \
-cpu host \
-smp 6,sockets=1,cores=6,threads=1 \
-bios /usr/share/qemu/bios.bin \
-vga none \
-device ioh3420,bus=pcie.0,addr=1c.0,multifunction=on,port=1,chassis=1,id=root.1 \
-device vfio-pci,host=01:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on \
-device vfio-pci,host=01:00.1,bus=root.1,addr=00.1 \
-drive if=none,file=drive/testdrive.img,id=vdisk1,cache=none,format=raw,aio=native \
-device virtio-blk,scsi=off,config-wce=off,drive=vdisk1,id=disk1 \
-boot order=c \
-drive file=drive/win_7_sp1.iso,id=winimg \
-device ide-cd,bus=ide.0,drive=winimg \
-drive file=drive/virtio_drv.iso,id=isocd \
-device ide-cd,bus=ide.1,drive=isocd \
-boot menu=on \
-usb -usbdevice tablet \
-spice port=5930,disable-ticketing \
-net nic -net bridge,br=kvmbr0 \

So, from the very first moment I used my gtx760. The setup was totally fine, but once I installed the recent nvidia drivers I end up in host a crash during guest boot.
Any ideas what could be the problem? I read I shouldn't run qemu using sudo, but without it I get errors when starting qemu. Anyway, until the nvidia installation the guest runs fine.
So would compiling a newer qemu version a possible solution?

Offline

#1459 2014-04-01 14:03:43

SXX
Member
Registered: 2014-03-31
Posts: 15

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Hi everyone. Is there anybody who successfully configured VGA Passthough on ASRock Z87 Extreme6? Is it possible to leave integrated graphics for host and passthough discrete one into VM?

I'm looking for motherboard where both discrete and integrated graphics would work so it's would be helpful if somebody give advice on mobo I should buy. There is person who using this mobo (Evonat), but he have no contact in profile so I decide to ask it on forums.

Offline

#1460 2014-04-01 14:17:09

andy123
Member
Registered: 2011-11-04
Posts: 169
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

SXX wrote:

Hi everyone. Is there anybody who successfully configured VGA Passthough on ASRock Z87 Extreme6? Is it possible to leave integrated graphics for host and passthough discrete one into VM?

I'm looking for motherboard where both discrete and integrated graphics would work so it's would be helpful if somebody give advice on mobo I should buy. There is person who using this mobo (Evonat), but he have no contact in profile so I decide to ask it on forums.

Here's a list. Seems like there are some people with that board on there.


i'm sorry for my poor english wirting skills…

Offline

#1461 2014-04-01 16:42:26

SXX
Member
Registered: 2014-03-31
Posts: 15

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

andy123 wrote:

Here's a list. Seems like there are some people with that board on there.

Thanks a lot for link, this one is really useful.

One more question. As I understand from reading forums at moment it's impossible to use PCIe device on host again it's assigned to guest even after guest shutdown. E.g you have to restart host to use GPU on host again. Is it hardware or software limitation? Is there any information about that?

Just want to understand what hardware I should get to avoid as many problems as possible.

Last edited by SXX (2014-04-01 16:44:28)

Offline

#1462 2014-04-01 17:02:50

andy123
Member
Registered: 2011-11-04
Posts: 169
Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

SXX wrote:
andy123 wrote:

Here's a list. Seems like there are some people with that board on there.

Thanks a lot for link, this one is really useful.

One more question. As I understand from reading forums at moment it's impossible to use PCIe device on host again it's assigned to guest even after guest shutdown. E.g you have to restart host to use GPU on host again. Is it hardware or software limitation? Is there any information about that?

Just want to understand what hardware I should get to avoid as many problems as possible.

Yeah, that list should probably linked to in the first post, if it isn't already.

Using devices without a host reboot, after shutdown of the guest works with PCI devices for me. I simply unbind the device from vfio-pci and "assign" it to another driver. (tested with my Marvell (SATA/IDE) and AMD USB controllers)

And by "assign the device to another driver" I mean something like this:

cd /sys/bus/pci/drivers/vfio-pci
echo 0000:02:00.0 > unbind
cd ../ahci
echo 1b4b 91a0 > new_id

As for doing this with GPUs: I have no idea. Maybe some other people in this thread?

PS: I have no idea if any of this should be done the way I do it or if it's a good idea in any way, but it WORKSFORME™


i'm sorry for my poor english wirting skills…

Offline

#1463 2014-04-01 17:27:17

SXX
Member
Registered: 2014-03-31
Posts: 15

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Thanks for more information.

andy123 wrote:

As for doing this with GPUs: I have no idea. Maybe some other people in this thread?

GPU was my main concern because I want to use discrete GPU on host (Radeon + DRI_PRIME) when guest is off.

After many hours of search in internet I suppose it's not possible at moment, but I wonder if this software or hardware limitation.

Last edited by SXX (2014-04-01 17:28:03)

Offline

#1464 2014-04-01 18:02:48

Flyser
Member
Registered: 2013-12-19
Posts: 29

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

SXX wrote:

GPU was my main concern because I want to use discrete GPU on host (Radeon + DRI_PRIME) when guest is off.

After many hours of search in internet I suppose it's not possible at moment, but I wonder if this software or hardware limitation.

Actually it is possible. I do that on my system (dh87rl, haswell i7, nvidia gtx660). No idea if it works with amd gpus though.

Last edited by Flyser (2014-04-01 18:02:59)

Offline

#1465 2014-04-01 18:08:40

SXX
Member
Registered: 2014-03-31
Posts: 15

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Flyser wrote:

Actually it is possible. I do that on my system (dh87rl, haswell i7, nvidia gtx660). No idea if it works with amd gpus though.

Interesting... What driver you using on host for Nvidia? It's loading kernel module after GPU used in guest?

Last edited by SXX (2014-04-01 18:12:18)

Offline

#1466 2014-04-01 19:11:42

blacky
Member
Registered: 2012-10-26
Posts: 14

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

andy123 wrote:
SXX wrote:

Hi everyone. Is there anybody who successfully configured VGA Passthough on ASRock Z87 Extreme6? Is it possible to leave integrated graphics for host and passthough discrete one into VM?

I'm looking for motherboard where both discrete and integrated graphics would work so it's would be helpful if somebody give advice on mobo I should buy. There is person who using this mobo (Evonat), but he have no contact in profile so I decide to ask it on forums.

Here's a list. Seems like there are some people with that board on there.

Heya, this list (and my entry on it) reminded me I wanted to tell you a funny story about my R290X.

The most important thing first: It’s working. Kinda.

First of all I see to have some bus reset issue because it’s only working after a reboot or stand-by. But what’s *really* astonishing is that it *only* works as long as the old HD4850 is present in the Arch host system and assigned to vfio-pci (IOMMU group), too. I tried it multiple times: Removing the HD4850 (including kernel boot parameters and vfio-pci.cfg) results in a BSoD of atikmpag.sys (I think). A similar problem occurs if I actually assign the HD4850 as *secondary* device to the VM; putting it as the *primary* device gives me a Error 43 for the 290X. The only way to make my 290X work properly is to have the HD4850 present on the host system and assigned to vfio-pci but not actually hand it to the VM.

Oh and I load the vbios image.

PS:

@SXX: Yes.

But you may hit some bumps on the road. E.g. I reverted to a specific git revision of qemu-git and libcacard multiple times already because newer versions cause weird graphic bugs on the host on launch (and on launch of the VM only).

Last edited by blacky (2014-04-01 19:14:15)

Offline

#1467 2014-04-01 19:37:14

Flyser
Member
Registered: 2013-12-19
Posts: 29

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

SXX wrote:

Interesting... What driver you using on host for Nvidia? It's loading kernel module after GPU used in guest?

I am using the nvidia binary driver and haven't tried nouveau so far. As long as the device is no longer bound to vfio *and* I execute the following script after the actual qemu command, it loads fine:

qemu-system-x86_64 -name bus-reset-only -enable-kvm -boot order=c,menu=off,reboot-timeout=0 -machine type=q35,accel=kvm -cpu host -smp 1 -m 32 -k de -vga none -display none -device ioh3420,bus=pcie.0,addr=1c.0,multifunction=on,port=1,chassis=1,id=root.1 -device vfio-pci,host=01:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on -device vfio-pci,host=01:00.1,bus=root.1,addr=00.1 -net none -no-reboot

This simply issues a bus reset (afaik).

Offline

#1468 2014-04-01 21:14:32

SXX
Member
Registered: 2014-03-31
Posts: 15

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

blacky wrote:

@SXX: Yes.

But you may hit some bumps on the road. E.g. I reverted to a specific git revision of qemu-git and libcacard multiple times already because newer versions cause weird graphic bugs on the host on launch (and on launch of the VM only).

I think it's fine. Didn't you tried to return GPU to host and use it with fglrx?

Flyser wrote:

I am using the nvidia binary driver and haven't tried nouveau so far. As long as the device is no longer bound to vfio *and* I execute the following script after the actual qemu command, it loads fine

Thanks for your answers. About two years ago I already tried to setup VGA passthough using XEN and in result I wasted money and time with no success so now I want to choose hardware carefully.

I'll have some hardware in few weeks and post here about results.

Last edited by SXX (2014-04-01 21:15:54)

Offline

#1469 2014-04-01 21:38:04

blacky
Member
Registered: 2012-10-26
Posts: 14

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

SXX wrote:
blacky wrote:

@SXX: Yes.

But you may hit some bumps on the road. E.g. I reverted to a specific git revision of qemu-git and libcacard multiple times already because newer versions cause weird graphic bugs on the host on launch (and on launch of the VM only).

I think it's fine. Didn't you tried to return GPU to host and use it with fglrx?

I meant the Intel on-board graphic I use for the host system. I don’t use the dedicated GPU for the host system, also because I hope I can basically completely shut it down (and save power) unless the related VM is running.

Flyser wrote:

This simply issues a bus reset (afaik).

Ahh … thanks, I was too lazy so far to look for this command in the thread. Maybe now I can start my VM without reboot / standby.

Offline

#1470 2014-04-02 00:57:37

dwe11er
Member
Registered: 2014-03-18
Posts: 73

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

andy123 wrote:
SXX wrote:

Hi everyone. Is there anybody who successfully configured VGA Passthough on ASRock Z87 Extreme6? Is it possible to leave integrated graphics for host and passthough discrete one into VM?

I'm looking for motherboard where both discrete and integrated graphics would work so it's would be helpful if somebody give advice on mobo I should buy. There is person who using this mobo (Evonat), but he have no contact in profile so I decide to ask it on forums.

Here's a list. Seems like there are some people with that board on there.

AFAIK every ASRock mobo based on Z87 (H87 should have too) has VT-d support.

Offline

#1471 2014-04-02 03:14:04

lildigiman
Member
Registered: 2014-04-02
Posts: 3

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Hi everyone! Just today I finally got my Asrock Z77 Extreme4 to replace my P8Z77-V LK because I wanted IOMMU support.

I'm reading back on an earlier post made by GizmoChicken who has successfully utilized VGA passthrough on a stock Ubuntu Kernel and stock qemu 1.5.

My setup:
Intel 3570 (non-k)
Asrock Z77 Extreme4
GTX 660 ti for Guest and iGPU for Host

Software (which should/could work according to the post linked above):
Ubuntu 13.10 (stock kernel + latest updates)
Qemu 1.5 from repositories

My problem occurs when I try to start the VM and get:

qemu-system-x86_64: -device vfio-pci,host=01:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on: vfio: error no iommu_group for device
qemu-system-x86_64: -device vfio-pci,host=01:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on: Device initialization failed.
qemu-system-x86_64: -device vfio-pci,host=01:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on: Device 'vfio-pci' could not be initialized

Google searches for "vfio: error no iommu_group for device" hasn't brought me anywhere except one user's post that had never properly answered what this error suggests.

My dmesg | grep -e DMAR -e IOMMU seems to suggest that VT-d is properly enabled (I think):

[    0.000000] ACPI: DMAR 00000000bd969280 000B8 (v01 INTEL      SNB  00000001 INTL 00000001)
[    0.086615] dmar: IOMMU 0: reg_base_addr fed90000 ver 1:0 cap c0000020e60262 ecap f0101a
[    0.086619] dmar: IOMMU 1: reg_base_addr fed91000 ver 1:0 cap c9008020660262 ecap f0105a
[    0.086690] IOAPIC id 2 under DRHD base  0xfed91000 IOMMU 1
[   21.194509] vboxpci: IOMMU not found (not registered)

I also added the following to /etc/modules

pci_stub
vfio
vfio_iommu_type1
vfio_pci
kvm
kvm_amd

So basically my question is what can I do from here with that error? I'm happy to provide more information if necessary.

Offline

#1472 2014-04-02 19:50:03

gneville
Member
Registered: 2014-04-02
Posts: 27

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Hi All,

I'm hoping someone will be able to help me out please. I've been trying to get vga passthrough working with qemu and vfio for a few months on and off but having no luck. I'm getting similar problems to users redger & DanaGoyette.

My hardware: CPU: i7 4770s, MB: Jetway NF9J-Q87 GFX: AMD R9 290

I used to use Xen 4.3 and had vga passthrough working just passing through the intel hd4600 controller to hosts, but since I got the AMD R9 290 card Xen doesn't seem to passthrough the card whatsoever, so thought I would try qemu and vfio.

I've tried using different linux versions and qemu versions but not having much luck. I am able to successfully vfio-bind my gfx card and I can start QEMU and get video via the GFX Card HDMI cable, I'm also able to successfully install Windows 8.1. However after installation I'm getting problems with either windows installing the amd drivers or using the catalyst driver installer, the host just reboots.

My latest setup:

linux 3.13.7-1 + i915 patch
Seabios git 1.7.4-66 (31Mar14)
Qemu 2.0.0 RC0 + patched pcibus_reset & assertion (https://lists.nongnu.org/archive/html/q … 05901.html & https://lists.gnu.org/archive/html/qemu … 00767.html)

Before the pcibus_reset patch I kept getting the following messages when the host rebooted while trying to install the drivers:

"qemu-system-x86_64 shows: hw/pci/pci.c:250: pcibus_reset: Assertion `bus->irq_count[\i] == 0' failed."

dmesg shows:  2805.839608 vfio-pci 0000:01:00.0: irq 46 for MSI/MSI-X


After applying the pcibust_reset patch the host no longer reboots after installing the drivers however the screen just goes blank so I have to manually kill the host. I do see the following messages:

qemu-system-x86_64 shows: ehci warning: guest updated active QH

dmesg shows:  4681.189513 vfio-pci 0000:01:00.0: irq 46 for MSI/MSI-X

This is my QEMU config:

/usr/local/qemu-2-0-0-rc0-patched-pci/bin/qemu-system-x86_64 -name win8 -enable-kvm -M q35 -m 6144  \
-cpu Haswell,hv-time,hv_relaxed,hv_vapic,hv_spinlocks=0x1000 \
-boot order=dc \
-smp 2,sockets=1,cores=2,threads=1 \
-bios /root/vfio/bios.bin -vga none \
-usb -usbdevice tablet \
-spice port=5902,disable-ticketing \
-vnc 0.0.0.0:0 \
-device ioh3420,bus=pcie.0,addr=1c.0,multifunction=on,port=1,chassis=1,id=root.1 \
-device ahci,bus=pcie.0,id=ahci \
-net nic \
-net tap,ifname=tap0,script=no,downscript=no \
-device vfio-pci,host=01:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on \
-device vfio-pci,host=01:00.1,bus=root.1,addr=00.1 \
-device vfio-pci,host=00:14.0,bus=pcie.0 \
-drive file=/dev/mapper/vg1-winv1,media=disk,index=1,id=disk,cache=none,if=virtio \
-drive file=/dev/mapper/vg1-winv1data,media=disk,index=2,id=diskb,cache=none,if=virtio \
-drive file=/home/xen/iso/windows/en-gb_windows_8_1_enterprise_x64_dvd_2971910.iso,boot=on,media=cdrom,index=3,id=isocd  \
-device ide-cd,bus=ahci.1,drive=isocd \
-drive file=/root/virtio-win-0.1-74.iso,media=cdrom,index=4,id=virtio  \
-device ide-cd,bus=ahci.2,drive=virtio

If I enable "-vga cirrus" and VNC to the host I can install the AMD drivers successfully and after a reboot it shows in device manager the AMD card but with a yellow exclamation and code 43. If I then go back to "-vga none" windows no longer wants to boot and I need up in a boot loop (seabios>windows booting>seabios>windows booting).

Any ideas please?

Offline

#1473 2014-04-02 19:57:52

mv0lnicky
Member
Registered: 2014-03-05
Posts: 16

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

lildigiman wrote:

qemu-system-x86_64: -device vfio-pci,host=01:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on: vfio: error no iommu_group for device
qemu-system-x86_64: -device vfio-pci,host=01:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on: Device initialization failed.
qemu-system-x86_64: -device vfio-pci,host=01:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on: Device 'vfio-pci' could not be initialized

Google searches for "vfio: error no iommu_group for device" hasn't brought me anywhere except one user's post that had never properly answered what this error suggests.

So basically my question is what can I do from here with that error? I'm happy to provide more information if necessary.

Check if your device is bound to vfio-pci.

ls /sys/bus/pci/drivers/vfio-pci | grep 0000

Offline

#1474 2014-04-02 20:22:59

lildigiman
Member
Registered: 2014-04-02
Posts: 3

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

mv0lnicky wrote:

Check if your device is bound to vfio-pci.

ls /sys/bus/pci/drivers/vfio-pci | grep 0000

Thanks for the response, mv0lnicky. The output is as follows:
0000:00:1d.0
0000:01:00.0
0000:01:00.1

Device 1d.0 is a USB 2.0 Controller where I have a keyboard and mouse connected.

I might also add that it appears that my Kernel was slightly older than I had thought, so I updated to 3.12.6-031206-generic.

When I now try to start the VM my error is no longer vfio: error no iommu_group for device, but this:

qemu-system-x86_64: -device vfio-pci,host=01:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on: Warning, device 0000:01:00.0 does not support reset
qemu-system-x86_64: -device vfio-pci,host=01:00.1,bus=root.1,addr=00.1: Warning, device 0000:01:00.1 does not support reset

A QEMU window then opens, but the VM stops instantly only outputting what is written above. The guest display (a third monitor) never displays anything anything at all.

On a side note (and maybe irrelevant?):
I also have display issues on the HOST when I try to start the VM (possibly compiz) causing my Host's secondary screen to display artifacts and inverted colors... not sure why or how this is happening. Note that this does not happen every time I try to start the VM, only maybe a third of the time.

Offline

#1475 2014-04-02 21:33:55

gneville
Member
Registered: 2014-04-02
Posts: 27

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

On top of this I have just installed the OP linux-mainline (3.13.6-1-mainline),seabios (1.7.4.r1788.g8abc6a8-1) & qemu (2.0.r31714.g03d5142-1) versions (Just added reset patch and manually compiled (https://lists.gnu.org/archive/html/qemu … 0767.html))

and still no change sad



gneville wrote:

Hi All,

I'm hoping someone will be able to help me out please. I've been trying to get vga passthrough working with qemu and vfio for a few months on and off but having no luck. I'm getting similar problems to users redger & DanaGoyette.

My hardware: CPU: i7 4770s, MB: Jetway NF9J-Q87 GFX: AMD R9 290

I used to use Xen 4.3 and had vga passthrough working just passing through the intel hd4600 controller to hosts, but since I got the AMD R9 290 card Xen doesn't seem to passthrough the card whatsoever, so thought I would try qemu and vfio.

I've tried using different linux versions and qemu versions but not having much luck. I am able to successfully vfio-bind my gfx card and I can start QEMU and get video via the GFX Card HDMI cable, I'm also able to successfully install Windows 8.1. However after installation I'm getting problems with either windows installing the amd drivers or using the catalyst driver installer, the host just reboots.

My latest setup:

linux 3.13.7-1 + i915 patch
Seabios git 1.7.4-66 (31Mar14)
Qemu 2.0.0 RC0 + patched pcibus_reset & assertion (https://lists.nongnu.org/archive/html/q … 05901.html & https://lists.gnu.org/archive/html/qemu … 00767.html)

Before the pcibus_reset patch I kept getting the following messages when the host rebooted while trying to install the drivers:

"qemu-system-x86_64 shows: hw/pci/pci.c:250: pcibus_reset: Assertion `bus->irq_count[\i] == 0' failed."

dmesg shows:  2805.839608 vfio-pci 0000:01:00.0: irq 46 for MSI/MSI-X


After applying the pcibust_reset patch the host no longer reboots after installing the drivers however the screen just goes blank so I have to manually kill the host. I do see the following messages:

qemu-system-x86_64 shows: ehci warning: guest updated active QH

dmesg shows:  4681.189513 vfio-pci 0000:01:00.0: irq 46 for MSI/MSI-X

This is my QEMU config:

/usr/local/qemu-2-0-0-rc0-patched-pci/bin/qemu-system-x86_64 -name win8 -enable-kvm -M q35 -m 6144  \
-cpu Haswell,hv-time,hv_relaxed,hv_vapic,hv_spinlocks=0x1000 \
-boot order=dc \
-smp 2,sockets=1,cores=2,threads=1 \
-bios /root/vfio/bios.bin -vga none \
-usb -usbdevice tablet \
-spice port=5902,disable-ticketing \
-vnc 0.0.0.0:0 \
-device ioh3420,bus=pcie.0,addr=1c.0,multifunction=on,port=1,chassis=1,id=root.1 \
-device ahci,bus=pcie.0,id=ahci \
-net nic \
-net tap,ifname=tap0,script=no,downscript=no \
-device vfio-pci,host=01:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on \
-device vfio-pci,host=01:00.1,bus=root.1,addr=00.1 \
-device vfio-pci,host=00:14.0,bus=pcie.0 \
-drive file=/dev/mapper/vg1-winv1,media=disk,index=1,id=disk,cache=none,if=virtio \
-drive file=/dev/mapper/vg1-winv1data,media=disk,index=2,id=diskb,cache=none,if=virtio \
-drive file=/home/xen/iso/windows/en-gb_windows_8_1_enterprise_x64_dvd_2971910.iso,boot=on,media=cdrom,index=3,id=isocd  \
-device ide-cd,bus=ahci.1,drive=isocd \
-drive file=/root/virtio-win-0.1-74.iso,media=cdrom,index=4,id=virtio  \
-device ide-cd,bus=ahci.2,drive=virtio

If I enable "-vga cirrus" and VNC to the host I can install the AMD drivers successfully and after a reboot it shows in device manager the AMD card but with a yellow exclamation and code 43. If I then go back to "-vga none" windows no longer wants to boot and I need up in a boot loop (seabios>windows booting>seabios>windows booting).

Any ideas please?

Offline

Board footer

Powered by FluxBB