Arch Linux

Wimma77

Member

Registered: 2014-08-12

Posts: 15

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

I'm having issues too, and hoping someone could help please.
I've tried lots of combinations, so probably screwed something simple up in the process.
I followed the ubuntu 14.04 guide here:
http://www.pugetsystems.com/labs/articl … 4-KVM-585/
That is based on this arch thread, and adjusted for Ubuntu.
I can pass through the HD7770 and audio, and both show up in device manager.
But once I install the 14.4 ATI drivers (either catalyst custom install or pointing Windows driver update to extracted files) I cannot reboot unless I use safe mode. And my keyboard is not always recognised, which makes it even more difficult.

I have the following hardware (been proven to work in EXSi)
GA-X58-UD5 board
E5520 cpu
BIOS supports passthrough also
primary card HD5450
passthrough HD7770
Ubuntu 14.04 x64 server
Win7x64 kvm VM

simon@cavetroll:~$ qemu-system-x86_64 -version
QEMU emulator version 2.0.0 (Debian 2.0.0+dfsg-2ubuntu1.2), Copyright (c) 2003-2008 Fabrice Bellard

simon@cavetroll:~$ cat /etc/initramfs-tools/modules
...
pci_stub ids=1002:683d,1002:aab0

simon@cavetroll:~$ cat /proc/cmdline
BOOT_IMAGE=/boot/vmlinuz-3.16.0-031600-generic root=UUID=a03285ef-5e8c-4182-b517-34f9717cbd18 ro quiet splash intel_iommu=on vfio_iommu_type1.allow_unsafe_interrupts=1 vt.handoff=7

I tried on the vanilla 3.12 kernal, then also tried installing the 3.16 as noted on page 1, no difference

HD7770 is card captured by pci-stub

simon@cavetroll:~$ dmesg | grep pci-stub
[    2.064508] pci-stub: add 1002:683D sub=FFFFFFFF:FFFFFFFF cls=00000000/00000000
[    2.064524] pci-stub 0000:03:00.0: claimed by stub
[    2.064534] pci-stub: add 1002:AAB0 sub=FFFFFFFF:FFFFFFFF cls=00000000/00000000
[    2.064543] pci-stub 0000:03:00.1: claimed by stub

simon@cavetroll:~$ lspci -nn |grep ATI
03:00.0 VGA compatible controller [0300]: Advanced Micro Devices, Inc. [AMD/ATI] Cape Verde XT [Radeon HD 7770/8760 / R7 250X] [1002:683d]
03:00.1 Audio device [0403]: Advanced Micro Devices, Inc. [AMD/ATI] Cape Verde/Pitcairn HDMI Audio [Radeon HD 7700/7800 Series] [1002:aab0]
04:00.0 VGA compatible controller [0300]: Advanced Micro Devices, Inc. [AMD/ATI] Park [Mobility Radeon HD 5430] [1002:68e1]
04:00.1 Audio device [0403]: Advanced Micro Devices, Inc. [AMD/ATI] Cedar HDMI Audio [Radeon HD 5400/6300 Series] [1002:aa68]

simon@cavetroll:~$ ls /sys/bus/pci/devices/0000\:03\:00.0/iommu_group/devices/
0000:03:00.0  0000:03:00.1

Can then capture video card and usb manually (script fails for me, but no need yet)

sudo vfio-bind 0000:03:00.0 0000:03:00.1
sudo vfio-bind 0000:00:1a.0 0000:00:1a.1 0000:00:1a.2 0000:00:1a.7

And can start a VM with display on HD7770 as expected, boots & installs windows7, but after installing ATI driver it fails to reboot.
I get the error recovery screen but can log in (sometimes, when keyboard works) with safe mode, so assume driver issue?

sudo qemu-system-x86_64 -enable-kvm -M q35 -m 6144 -cpu host \
-smp 4,sockets=1,cores=2,threads=2 \
-bios /usr/share/qemu/bios.bin -vga none \
-device ioh3420,bus=pcie.0,addr=1c.0,multifunction=on,port=1,chassis=1,id=root.1 \
-device vfio-pci,host=03:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on \
-device vfio-pci,host=03:00.1,bus=root.1,addr=00.1 \
-device vfio-pci,host=00:1a.0,bus=pcie.0 \
-device vfio-pci,host=00:1a.1,bus=pcie.0 \
-device vfio-pci,host=00:1a.2,bus=pcie.0 \
-device vfio-pci,host=00:1a.7,bus=pcie.0 \
-drive file=/home/simon/win7.img,id=disk,format=raw -device ide-hd,bus=ide.0,drive=disk \
-drive file=/home/simon/Downloads/Win7_64.iso,id=isocd -device ide-cd,bus=ide.1,drive=isocd \
-boot menu=on

Also tried IDE and virtio, on image file and partition, same result.
I assume I'm missing something simple?

aldum

Member

Registered: 2012-03-14

Posts: 8

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

I can't get audio working right in Windows 8.1 as a guest, I get audio output but it is distorted. 
Any ideas?

What sinny said, try messing around with the parameters. It's a trial-and-error thing, can't really get it right otherwise.

Here's what's been working for me (Win7 guest):

QEMU_ALSA_DAC_BUFFER_SIZE=1024 QEMU_AUDIO_DAC_FIXED_FREQ=48000 \
QEMU_AUDIO_ADC_FIXED_FREQ=48000 QEMU_ALSA_DAC_BUFFER_SIZE=16384 \
QEMU_AUDIO_DRV=pa \
qemu-system-x86_64 -enable-kvm -M q35 -m 4G -cpu host,hv-time,kvm=off \
(...)
-device ich9-intel-hda,bus=pcie.0,addr=1b.0,id=sound0 \
-device hda-duplex,id=sound0-codec0,bus=sound0.0,cad=0 \
-soundhw hda,ac97 \

Fira

Member

Registered: 2011-09-24

Posts: 8

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Hey there
I recently broke my RAID when actually testing it (tip of the day: don't use FakeRAID, it can't do online recovery/degraded startup, at least at Intel's).

Reinstalled my whole Virt/Passthrough setup and I wanted to update on some things I figured out:
- I profited of the occasion to downgrade from Windows 8.1 to Windows 7, hoping things would turn out better. Bad idea. The bootloader kept breaking for random reasons, even once Windows Update was turned off. Tip for anyone experimenting with this kind of problem: use LVM, make snapshots. You'll gain a lot of time rather than reinstalling Win$ every time (in my case, restoring GPT, Bootloader, and Boot partition wasn't enough).
- Be careful with the versions of graphics drivers you run in your guest. For exemple i had a lucky streak with my first install because my GTX 750 Ti doesn't seem to run at all under passthrough when using versions above 335. It's probably always better to undershoot.
- I added kernel boot args to capture the card to PCI-Stub before passing it to VFIO to ensure nothing would get loaded. This is important for nVidia cards it seems because otherwise they can load an audio driver for the sound sub-card.
- A little important gotcha if you run PulseAudio and get connection issues, PulseAudio uses a cookie to authenticate users (though that might not be the case when running Policykit ?)... Because VFIO access requires root, and your pulseaudio server is probably running under a different user, the easiest workaround i found to that is to simply add a pulse input over TCP for 127.0.0.1 and add in /root/.pulse/client.conf the variable default-server=127.0.0.1 (or you could use PULSE_SERVER env variable).

Also a little errata. I said you need to lock your mouse to a screen when using Synergy with Mouse grabbing apps. That's true, but this will only work if you set the mouse movement type to relative rather than absolute in the Settings. Otherwise anything grabbing mouse expecting relative movements (hint: any FPS) will behave plain poorly...

Good luck !

Last edited by Fira (2014-08-19 15:36:50)

somearchfan

Member

Registered: 2014-08-18

Posts: 35

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Hi all

I need some help here. I've tried to set up hugepages but no luck. My computer crashes each and every time I start qemu w/ hpages enabled.

cat /etc/fstab

UUID=B039-B9D4 /boot vfat defaults,rw,relatime 0 0
UUID=9416fa53-b808-4b96-bd92-6839effde5c5 swap swap defaults 0 0
UUID=8038e49a-447d-4955-823f-803d6e0c3e32 /home ext4 defaults,rw,relatime,data=ordered 0 0
UUID=9fb159a7-84b2-401b-a745-8199d3b0339c / ext4 defaults,rw,relatime,data=ordered 0 1
tmpfs /tmp tmpfs defaults,noatime,mode=1777 0 0
hugetlbfs       /dev/hugepages  hugetlbfs       mode=1770,gid=78        0 0

mount | grep huge

hugetlbfs on /dev/hugepages type hugetlbfs (rw,relatime,mode=1770,gid=78)

grep HugePages_Total /proc/meminfo

HugePages_Total:    4096

My config:

QEMU_ALSA_DAC_BUFFER_SIZE=512 QEMU_ALSA_DAC_PERIOD_SIZE=170 QEMU_AUDIO_DRV=alsa qemu-system-x86_64 -enable-kvm -M q35 -m 8196 -mem-path /dev/hugepages -cpu host -boot d \
-smp 4,sockets=1,cores=4,threads=1 \
-bios /usr/share/qemu/bios.bin -vga none \
-device ioh3420,bus=pcie.0,addr=1c.0,multifunction=on,port=1,chassis=1,id=root.1 \
-device vfio-pci,host=01:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on,rombar=0,romfile=/home/john/Downloads/Asus.R9270.2048.131007.rom \
-device virtio-scsi-pci,id=scsi \
-drive file=/run/media/john/win/windows.img,id=disk,format=raw -device scsi-hd,drive=disk \
-drive file=/home/john/Downloads/virtio-win-0.1-81.iso,id=virtiocd -device ide-cd,bus=ide.2,drive=virtiocd \
-usb -usbdevice host:046d:c051 -usbdevice host:046a:0021 \
-net tap,vlan=0,ifname=tap0 -net nic,model=virtio,vlan=0 \
-device ich9-intel-hda,bus=pcie.0,addr=1b.0,id=sound0 \
-device hda-duplex,id=sound0-codec0,bus=sound0.0,cad=0

Any ideas?

Last edited by somearchfan (2014-08-20 00:06:01)

redger

Member

Registered: 2014-02-26

Posts: 15

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

@Wimma77
I run a similar setup (using kUbuntu, tried server and wanted to run desktop n an LXC container - without much success) .... and have just spent 2+ days resolving a similar issue. The vm was working fine till the end of last week
Anyway, what I found was that the most useful debugging information came from a Ubuntu vm which I created specially to test whetehr or not the problem was Windows specific.  Ubuntu startup information pointed to a sound problem, so I removed the AMD sound device passthrough (03:00.1 in your case) ... and then  itbooted ok Very odd since it had been working for a couple of months until then. The other consequence was that the dx9 graphics performance then improved by about 20% ... particularly strange since the card worked perfectly in another PC booting Windows directly, so unlikely to be a hardware issue

Anyway, if you have the disk space create a Linux (ubuntu ?) VM to see if it indicates where to look,  The other thing to be conscious of is the need to completely remove graphics drivers using something like http://www.guru3d.com/files-details/dis … nload.html before trying again

Wimma77

Member

Registered: 2014-08-12

Posts: 15

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

@redger
Thanks so much for the suggestion - will try that. But I thought you had to pass the whole iommu group through? Or just pass them both to vfio?
My card & audio share a group.
ie ls /sys/bus/pci/devices/0000\:03\:00.0/iommu_group/devices/
0000:03:00.0  0000:03:00.1
I will try though. Good idea on the ubuntu VM too - definitely worth checking.
Can I just ask - did you use just vanilla kernel and qemu?
I have a headless server, but did install xfce4 so I could VNC to it and use a GUI when required.
Would love to be able to start the vm from cli but have not researched enough yet on how to accomplish that.
Cheers.

Last edited by Wimma77 (2014-08-20 02:44:32)

redger

Member

Registered: 2014-02-26

Posts: 15

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

@Wimma77
you don't need to pass the whole group BUT you cannot pass parts of a group to fdifferent VMs
I start my VMs from the CLI using a simple script .. looks like this (ignore extra lines at the top, I keep them for debugging & experimentation)

#!/bin/sh
# Real command
# sudo sh /etc/init.d/vfio-bind-init.sh
# qemu-system-x86_64 \
# QEMU_PA_SAMPLES=128 QEMU_AUDIO_DRV=pa qemu-system-x86_64 \
QEMU_ALSA_DAC_BUFFER_SIZE=512 QEMU_ALSA_DAC_PERIOD_SIZE=170 QEMU_AUDIO_DRV=alsa qemu-system-x86_64 \
-name win7 -enable-kvm -M q35 -m 6144  -mem-path /dev/hugepages \
-cpu Haswell,hv-time,hv_relaxed,hv_vapic,hv_spinlocks=0x1000 -rtc base=localtime \
-boot menu=on \
-smp 2,sockets=1,cores=2,threads=1 \
-bios /usr/share/qemu/bios.bin -vga none \
-usbdevice tablet \
-spice port=5902,disable-ticketing \
-device ioh3420,bus=pcie.0,addr=1c.0,multifunction=on,port=1,chassis=1,id=root.1 \
-device ahci,bus=pcie.0,id=ahci \
-net nic,macaddr=00:00:00:0a:5b:2c,model=virtio \
-net user \
-device ioh3420,bus=pcie.0,addr=1c.1,multifunction=on,port=2,chassis=2,id=root.2 \
-device ich9-intel-hda,bus=root.2 \
-device hda-duplex,bus=hda.0 \
-device vfio-pci,host=01:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on \
-device vfio-pci,host=06:00.0,bus=pcie.0 \
-drive file=/dev/ssd-virt/lvwin7_kvm,id=disk1,format=raw,if=virtio \
-drive file=/mnt/programming_data/isos/virtio-win-0.1-74.iso,id=isocd2 \
-device ide-cd,bus=ahci.2,drive=isocd2 \
-drive file=/mnt/programming_data/isos/winfiles_2.iso,id=isocd3 \
-device ide-cd,bus=ahci.3,drive=isocd3

which I then added to the KDE menu - to be executed as root .... but you could easily adjust to enable direct execution. I have similar scripts fopr binding the VFIO devices and for strating spice (useful when debugging)

I used qemu etc. from the Ubuntu repos but I patched the kernel using the following script -

#!/bin/bash
# patch --dry-run --verbose -p 1 -i re_xxxxxxxxxxxxx

echo
echo ... PATCHING ... VGA Arbiter
patch -b -p 1 -i re_patch_01_i915_313rc4.patch
echo
echo ... PATCHING ... acs override
patch -b -p 1 -i re_patch_02_override_for_missing_acs_capabilities.patch
echo
echo ... PATCHING ... memleak
patch -b -p 1 -i re_patch_03_fix_memleak.patch
echo
echo ... PATCHING ... read DR6
patch -b -p 1 -i re_patch_04_fix_reading_of_DR6.patch
echo
echo ... PATCHING ... debug registers - has problem, needs to follow DR6 patch
patch -b -p 1 -i re_patch_05_debug_registers.patch
# patch -b -p 1 -i re_debug_registers_RE.patch   # Corrected to add additional lines before DR6 patch runs
echo
echo ... PATCHING ... kernel__gcc
patch -b -p 1 -i re_patch_06_kernel-38-gcc48-2.patch

Generally I copy the "mainline" file nbhs posts and extract the patches for each kernel generation, just be be sure. So I have the patches he posted for 3.13 and I have patched kernels for Ubuntu releases 24, 28, 32, 34 and I have copied patches for 3.13, 3.15, 3.15 (missed 3.14 somehow)

Here are my notes for patching the kernel, if they're of any use

To compile a new kernel
https://wiki.ubuntu.com/Kernel/BuildYourOwnKernel
https://help.ubuntu.com/community/Kernel/Compile
https://wiki.ubuntu.com/KernelTeam/KernelMaintenance
https://www.debian.org/releases/stable/i386/ch08s06.html.en

architecture is "amd64" or "x86_64"

1) download the source into the current directory using "apt-get source linux-image-xxxxx"  where xxxx is the name of the version eg. apt-get source linux-image-3.13.0-32-generic

This should download the tarball(s) and extract the source code into a directory (which should be renamed immediately because all versions use the same directory name !!!)

2) Open the new directory, clean up and prepare
  chmod a+x debian/scripts/*
  chmod a+x debian/scripts/misc/*
  fakeroot debian/rules clean

  and generate the required config by either -
  a) editing using "fakeroot debian/rules editconfigs"  (for all targets, one after another) ... or 
  b) "make menuconfig" and work through the various options. Remember to save before exit
  c) copy the current config from the boot directory as ".config" in the root direcotry for the new kernel and then use  "make oldconfig" ... which will ask a question for the value of each new config option
  
  Required config options are
    CONFIG_VFIO_IOMMU_TYPE1=y     in Device Drivers. 2 pages back from end
    CONFIG_VFIO=y
    CONFIG_VFIO_PCI=y
    CONFIG_VFIO_PCI_VGA=y
    CONFIG_PCI_STUB=y             in Bus Options, second page down
    HZ_1000=y                     in Processor Type & Features (last page)
    PREEMPT=voluntary

3) apply any patches /// remember to verify that they worked ok (look for fail)
   "sh re_apply_patches.sh > re_output_patch.txt"

4) "fakeroot debian/rules clean"   to remove any old build information / files

5) Ignore modules not created with new parameters ... copy re_modules.ignore to ...debian.master/abi/<previous-version>/modules.ignore

   and ignore other ABI errors by copying re_prevrelease_arch_ignore (rename to  "ignore") to debian.master/abi/<previous-version>/<arch>    eg. to debian.master/abi/3.13.0-32.56/amd64/

6) "DEB_BUILD_OPTIONS=parallel=3 skipmodule=true fakeroot debian/rules binary-headers binary-generic  > re_output_compile.txt"   to generate the deb files which can be installed (second thoughts don't pipe the poutput to a file - it will prevent selection of the CPU type)

7) Install all the Debs with command "sudo dpkg -i linux*<ver>*.deb
eg.   sudo dpkg -i linux*3.13.0-32_3.13.0-32.57*.deb
      sudo dpkg -i linux*3.13.0-32-generic_3.13.0-32.57*.deb

8) go into Synaptic and lock all the newly installed elements (linux-image*, linux-header*, linux-tool*, linx-cloud-tool*) - to prevent the new kernel and components being overwritten in the next upgrade

redger

Member

Registered: 2014-02-26

Posts: 15

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

@somearchfan
have you tried either (a) increasing the size of your hugepages or (b) decreasing the amount of memory allocated to the VM
The hugepages mechanism needs a small amount of RAM for management .. over and above what is allocated to the VM(s). For instance mine is set to

grep HugePages_Total /proc/meminfo
HugePages_Total:    3200

for a VM with memory allocation of 6144

qemu-system-x86_64 \
-name win7 -enable-kvm -M q35 -m 6144  -mem-path /dev/hugepages \

deivid

Member

Registered: 2014-08-13

Posts: 5

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

using
-device ich9-intel-hda,bus=pcie.0,addr=1b.0,id=sound0 \
-device hda-duplex,id=sound0-codec0,bus=sound0.0,cad=0

I get way lower performance in general and very choppy audio (using alsa). Any clue?

In particular, I get higher in-game ping (from 60ms to ~120ms) and also my network latency increases, the games drop from 60+ fps to ~12-20fps.. I'll keep looking later

Last edited by deivid (2014-08-20 04:14:30)

redger

Member

Registered: 2014-02-26

Posts: 15

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

@deivid
have a look at my command line a couple of posts above. Note the prefix to qemu right at the beginning and the 3 lines for the sound card ie. add another PCI slot for the card (ioh3420). You could also try Pulseaudio using the commented line at the top of what I posted - tho it performed poorly for me (lag), Alsa was much better performed and host tasks can still play sounds simultaneously via Pulse

deivid

Member

Registered: 2014-08-13

Posts: 5

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Basically copied the first line and last 3 lines from your config, but it's the same result (very choppy sound and poor graphics performance, didn't test network).. there must be something obvious I'm missing.
As 'recommended' above I removed my AMD sound device (01:00.1), which wasn't doing anything anyway

QEMU_ALSA_DAC_BUFFER_SIZE=512 QEMU_ALSA_DAC_PERIOD_SIZE=170 QEMU_AUDIO_DRV=alsa \
qemu-system-x86_64 -enable-kvm -M q35 -m 3200 -cpu host \
-smp 3,sockets=1,cores=3,threads=1 \
-bios /usr/share/qemu/bios.bin \
-device ioh3420,bus=pcie.0,addr=1c.0,multifunction=on,port=1,chassis=1,id=root.1 \
-device vfio-pci,host=01:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on \
-device piix4-ide,bus=pcie.0,id=piix4-ide \
-usb -usbdevice host:1bcf:0002 \
-usb -usbdevice host:04d9:1602 \
-drive file=/storage/virt/windows.qcow2,id=disk,format=qcow2 -device ide-hd,bus=ide.0,drive=disk \
-vga none -vnc :0 \
-net nic -net tap,ifname=tap0,script=no,downscript=no \
-mem-path /dev/hugepages \
-device ioh3420,bus=pcie.0,addr=1c.1,multifunction=on,port=2,chassis=2,id=root.2 \
-device ich9-intel-hda,bus=root.2 \
-device hda-duplex,bus=hda.0

Do you see what am  I doing wrong?

blu3bird

Member

Registered: 2014-08-20

Posts: 2

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

@somearchfan

It does not crash, it just get's very very very slow because you are ouf of memory ;-)

Using 2MB hugepages adds 16MB overhead per vm => If you want to assign 8192MB to your vm, you'll need 8208MB of hugepages.

PS: You can try 1GB hugepages to avoid this overhead. But that requires extra hardware support: grep pdpe1gb /proc/cpuinfo

Last edited by blu3bird (2014-08-20 05:29:24)

redger

Member

Registered: 2014-02-26

Posts: 15

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

@deivid
errrr I'm not an expert .... but

look at (a) your disk, it's on an IDE controller whereas it will perform much better using virtio (I create an LVM partition and passed that using virtio) (b) if running a 64 bit Windows it may be short of RAM - 3GB is a bit lean, can you allocate 4GB or more - paging could be killing the vm (c) you appear to have defined a VNC connection which may not help (shouldn't be necessary if you're passing a graphcis card, try using Spice instead). Presumably you're running this on a quad core Intel cpu / motherboard (guess) given that you're passing 3 cores, single thread each and no pinning. If this is an AMD you probably want to (i) allocate an even number of cores and (ii) pin them to avoid NUMA problems ie. cache build / rebuild
Have you set Hugepages appropriately ? See discussion up the page a bit

Also refer to NBHS's tuning comments on the first page. I have added hv-time,hv_relaxed,hv_vapic,hv_spinlocks=0x1000 to help improve Windows performance (refered to in early pages - maybe search for them).

perhaps one of the more capable folk on the forum could offer better / more advice

sinny

Member

Registered: 2014-05-24

Posts: 75

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Hi all

I need some help here. I've tried to set up hugepages but no luck. My computer crashes each and every time I start qemu w/ hpages enabled.

yep, hugepages is completely undocumented and almost not yet publically known kernel feature. thread on pci passthrough is a perfect place to whine about you not being able to use google

Basically copied the first line and last 3 lines from your config, but it's the same result (very choppy sound and poor graphics performance, didn't test network).. there must be something obvious I'm missing.
As 'recommended' above I removed my AMD sound device (01:00.1), which wasn't doing anything anyway

...

Do you see what am  I doing wrong?

i see what you are doing wrong (more like - it seems for me that it's the only thing you are doing) - being completely ignorant.

you got problems with sound - too high and mighty to pinpoint command line arguments used that are related to sound and read some manuals on their meaning?
got poor graphics performance - too busy to try tons of performance tips that are all over this thread (not to mention google)?
got any problems - too cute to at least try resolving the problem yourself (and i mean really trying to soil your hands digging around the issue)?

p.s. thanks for making the thread less usable with question about something pretty trivial that was already discussed quite a few times

deivid

Member

Registered: 2014-08-13

Posts: 5

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

@deivid
errrr I'm not an expert .... but

look at (a) your disk, it's on an IDE controller whereas it will perform much better using virtio (I create an LVM partition and passed that using virtio)

I'll look into it but I installed windows with the disk set to IDE, then installed drivers; trying to boot on virtio mode  gives me a BSOD and I'd have to format the virtual machine, which I might do anyway tomorrow.

(b) if running a 64 bit Windows it may be short of RAM - 3GB is a bit lean, can you allocate 4GB or more - paging could be killing the vm

I'm running this with only windows and a game (dota 2/half life) for now, so it's enough; didn't notice any difference between 3 or 4gb (I have 8 but I'm running a few things on the host)

(c) you appear to have defined a VNC connection which may not help (shouldn't be necessary if you're passing a graphcis card, try using Spice instead). Presumably you're running this on a quad core Intel cpu / motherboard (guess) given that you're passing 3 cores, single thread each and no pinning. If this is an AMD you probably want to (i) allocate an even number of cores and (ii) pin them to avoid NUMA problems ie. cache build / rebuild

VNC connection: I was getting a 'couldn't initialize SDL' on the machine, it's running headless, and this solved it. The vnc session only lets me manage the qemu console.

I'm running on an 8-code 8350, again, no difference with 3/4 cores, I'll look into pinning them.

Have you set Hugepages appropriately ? See discussion up the page a bit

Also refer to NBHS's tuning comments on the first page. I have added hv-time,hv_relaxed,hv_vapic,hv_spinlocks=0x1000 to help improve Windows performance (refered to in early pages - maybe search for them).

perhaps one of the more capable folk on the forum could offer better / more advice

I've set up 2MB hugepages but my mobo supports 1GB hugepages so I'll set it up later (didn't notice any difference with the hugepages on/off).
I'll look into the hv-* parameters.

Thank you

Last edited by deivid (2014-08-20 17:29:27)

deivid

Member

Registered: 2014-08-13

Posts: 5

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

you got problems with sound - too high and mighty to pinpoint command line arguments used that are related to sound and read some manuals on their meaning?
got poor graphics performance - too busy to try tons of performance tips that are all over this thread (not to mention google)?
got any problems - too cute to at least try resolving the problem yourself (and i mean really trying to soil your hands digging around the issue)?

p.s. thanks for making the thread less usable with question about something pretty trivial that was already discussed quite a few times

I'm guessing you didn't understand my post. All my problems start if I add the audio device. Without audio device, the VM works wonders, with it, I get poor network performance (LAN and WAN) and poor graphics/CPU performance, and the audio crackles like hell.

They are not separate issues and only happen with the audio device enabled; pci-passthrough is pretty obscure and most 'documentation' comes from guessing. I haven't found anything related to my problem so I posted here

devianceluka

Member

Registered: 2014-05-19

Posts: 44

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Could someone please help me with compiling kernel with acs override patch on Fedora? I want to compile the latest kernel from "yum update". I have those lines set up already and they are not the best:

yum install yum-utils
yumdownloader --source kernel
yum-builddep kernel-<version>.src.rpm
rpm -Uvh kernel-<version>.src.rpm
cd
cd rpmbuild/SOURCES
tar -xvf ....
cd rpmbuild/SOURCES/linux../
<copy acs.patch here>
patch -p1 < acs.patch
make oldconfig
make
make install

Please someone....

sinny

Member

Registered: 2014-05-24

Posts: 75

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

you got problems with sound - too high and mighty to pinpoint command line arguments used that are related to sound and read some manuals on their meaning?
got poor graphics performance - too busy to try tons of performance tips that are all over this thread (not to mention google)?
got any problems - too cute to at least try resolving the problem yourself (and i mean really trying to soil your hands digging around the issue)?

p.s. thanks for making the thread less usable with question about something pretty trivial that was already discussed quite a few times

I'm guessing you didn't understand my post. All my problems start if I add the audio device. Without audio device, the VM works wonders, with it, I get poor network performance (LAN and WAN) and poor graphics/CPU performance, and the audio crackles like hell.

They are not separate issues and only happen with the audio device enabled; pci-passthrough is pretty obscure and most 'documentation' comes from guessing. I haven't found anything related to my problem so I posted here

well, i had both choppy audio and poor 3d game fps. resolved this by using isolcpus/cpuset, hugetlbfs and playing around with audio parameters. everything was found in the thread - so you probably never searched or just stopped doing it after few minutes.

Could someone please help me with compiling kernel with acs override patch on Fedora? I want to compile the latest kernel from "yum update". I have those lines set up already and they are not the best:
yum install yum-utils
yumdownloader --source kernel
yum-builddep kernel-<version>.src.rpm
rpm -Uvh kernel-<version>.src.rpm
cd
cd rpmbuild/SOURCES
tar -xvf ....
cd rpmbuild/SOURCES/linux../
<copy acs.patch here>
patch -p1 < acs.patch
make oldconfig
make
make install
Please someone....

you have "lines" "set up"? i dont even want to think what kind of disorder it might be...
and the "lines" look very weird from fedora perspective - why use yum/rpmbuild at all if you are going to do everything (incorrectly, by the way) by hands later?

excuse me, but i hope noone will spoon-feed you with this info. it's god damned fedora - it even has its' own dedicated wiki with pretty good info (not to mention it being pretty popular/used distro).

why did you switch to fedora if you neither have an idea on its' infrastructure nor willing to use TONS of info on it in the net? search the damned web or stop begging for help (well, technically not for "help", but for someone to do everything for you)

dwe11er

Member

Registered: 2014-03-18

Posts: 73

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Hi all

I need some help here. I've tried to set up hugepages but no luck. My computer crashes each and every time I start qemu w/ hpages enabled.
http://www1.xup.to/tn/2014_08/63559622.jpeg

...

Any ideas?

Try to allocate bit more, like 5004 or something.

Last edited by dwe11er (2014-08-20 20:57:25)

starlays

Member

Registered: 2011-02-02

Posts: 12

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

If I have only one GPU available can I use  KVM VGA-Passthrough?
I have Gnome running on the host and I want to create a windows VM for gaming, I'm new to virtualization and I'm curious if It is possible to share the GPU between the host and the guest. All the scenarios that I found for now are involving the presence of 2 video cards, one on CPU and one PCI. My CPU has VT-d, it is  Intel Xeon 1230v3.
Thank you in advance.

somearchfan

Member

Registered: 2014-08-18

Posts: 35

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

You can't run both at the same time on one gpu. Just buy a cheap Radeon 5450 / 6450 for linux.

Wimma77

Member

Registered: 2014-08-12

Posts: 15

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

@Wimma77
you don't need to pass the whole group BUT you cannot pass parts of a group to fdifferent VMs
I start my VMs from the CLI using a simple script .. looks like this (ignore extra lines at the top, I keep them for debugging & experimentation)

which I then added to the KDE menu - to be executed as root .... but you could easily adjust to enable direct execution. I have similar scripts fopr binding the VFIO devices and for strating spice (useful when debugging)

I used qemu etc. from the Ubuntu repos but I patched the kernel using the following script -

Thanks so much for this.
Will try this weekend when I get some quiet time.
Got an nvidia card coming too, so might be a good time to start fresh.
Much appreciated.

erganzi

Member

Registered: 2014-07-09

Posts: 19

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Thank you for the guide! It finally work after I tried many times, changed many kernel and qemu versions.

So I want to share my experience.

Hardware Info：
Gigabyte Z97-D3H
i5-4570
i915 built-in card for host
Sapphire hd7850 for vm

Software Info:
linux-3.12.4 from kernel.org with following patchs.
pci/iommu: Quirk non-compliant PCIe-to-PCI bridges
i915_fixes[1]
i915_fixes[2]
Enable overrides for missing ACS capabilities

qemu-2.0.0 with patch
vfio-pci: Disallow device from using NoSnoop transactions

Kernel commandline:

linux /vmlinuz-3.12.4-VFIOPassthrough root=UUID=6b2e5409-8b73-40db-ad45-f4ec79cdc0cc quiet
intel_iommu=on vfio_iommu_type1.allow_unsafe_interrupts=1 pcie_acs_override=downstream
pci-stub.ids=1002:6819,1002:aab0

/etc/modprobe.d/kvm.conf

options kvm ignore_msrs=1
options kvm allow_unsafe_assigned_interrupts=1
options kvm_intel emulate_invalid_guest_state=0

Qemu commandline:

01:00.0 VGA compatible controller: ATI Technologies Inc Device 6819
01:00.1 Audio device: ATI Technologies Inc Device aab0

/root/vfio-bind 0000:01:00.0 0000:01:00.1

qemu-system-x86_64 -cpu SandyBridge -smp 2,sockets=2,cores=1,threads=1 \
-m 4096 -M q35 -enable-kvm \
-device piix4-ide,bus=pcie.0,id=piix4-ide \
-drive file=/var/lib/libvirt/images/fwq_win7_vfio.img,id=disk,format=raw \
-device ide-hd,bus=piix4-ide.0,drive=disk \
-drive file=/var/lib/libvirt/images/LENOVO_WIN7_64.iso,id=isocd \
-device ide-cd,bus=piix4-ide.1,drive=isocd \
-net nic,model=rtl8139,macaddr=52:54:00:1a:2b:3c \
-net tap,ifname=tap0,script=/etc/qemu-ifup,downscript=/etc/qemu-ifdown \
-vga none -display none \
-device ioh3420,bus=pcie.0,addr=1c.0,multifunction=on,port=1,chassis=1,id=root.1 \
-device vfio-pci,host=01:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on \
-device vfio-pci,host=01:00.1,bus=pcie.0 \     /* the audio bus must be pcie.0 or you will be failed. */
-usb -usbdevice host:17ef:6018 -usbdevice host:17ef:600e \
-boot order=cd -monitor stdio

PS: the graphic driver for hd7850 that I use [amd_13_9_win7_win8_64_dd_ccc_whql.exe]

Upon this, AMD FirePro R5000 works too, but there is another problem, if I start vm with VFIO passthrough, my host terminal has no response like screen freezed, only ssh works. give me some help, thanks!

erganzi

Member

Registered: 2014-07-09

Posts: 19

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

I found it seems like i915 driver. if I blacked i915 driver in blacklist.conf, the host is no respond. but canceled it, it normal.

BahbGTR

Member

Registered: 2013-07-12

Posts: 20

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

I got tired of manually patching and compiling the mainline kernel, so I used customizepkg to patch the default linux package automatically.  This may not be the proper way of doing things, but it's the lazy way!  This process would probably work with any kernel in AUR or any of the repositories.  I'm not sure if the patches will.  I would be surprised if they didn't, but you never know until you try. 

Sorry if this has been posted before, but it's super hard to search this thread.  My system specs are in my signature.  I passthrough my nvidia card to my VM, and I use the Intel P4600 graphics on my host.  I have been building the acs override patch, i915 VGA arbiter fix, and the VGA arbiter fix.  I boot using efi stubs and refind.

Please, make sure you understand what the following commands do.  Don't just copy and paste what I have here.  There is a serious risk it could break your install.  Don't drop your existing kernel until you know the patched one is working.  Don't blame me for any broken installs!

Requirements: yaourt, customizepkg, and Abs.

First of all, save the patches you need to apply to a location where the userid that will use yaourt can reach.  I have a build directory in my home folder, so I placed them there.

Second, take a look at How to use ABS.  Copy the kernel you want to patch and open up the PKGBUILD file in your text editor.  Look for the prepare() section.  Specifically where all the patch files are located.  You're going to use customizepkg to recognize where you want your patches to be inserted.  You'll need to copy one of the patch entries.  For the linux package I used the following as my entry point: 

patch -p1 -i "${srcdir}/change-default-console-loglevel.patch"

Open up your favorite text editor and create /etc/customizepkg.d/linux.  The file name should reflect the name of the package you are installing.  If it's linux-ck you should use /etc/customizepkg.d/linux-ck.

In that file, place the following on a single line (very important!): 

replace#global#patch -p1 -i \"${srcdir}\\/change-default-console-loglevel.patch\"#patch -p1 -i \"${srcdir}\\/change-default-console-loglevel.patch\"\\npatch -p1 -i \"\\/PATH_TO_PATCHES\\/i915_316.patch\"\\npatch -p1 -i \"\\/PATH_TO_PATCHES\\/override_for_missing_acs_capabilities.patch\"\\npatch -p1 -i \"\\PATH_TO_PATCHES\\/vga_arbitrer.patch\"

This code snippet will only for work for the linux package.  If you want to use a different kernel package, you're on your own to find the place to insert the patches.

A bit of an explanation of the snippet.  The below is the bit of the PKGBUILD customizepkg watches for.  When this point is reached, everything coming after this point on the same line is added to the PKGBUILD.

replace#global#patch -p1 -i \"${srcdir}\\/change-default-console-loglevel.patch\"

Below is what is being added to the PKGBUILD.  I added change-default-console-loglevel.patch back in because I still want that patch applied.  If you don't add it back in, that patch will be dropped. 

#patch -p1 -i \"${srcdir}\\/change-default-console-loglevel.patch\"\\npatch -p1 -i \"\\/PATH_TO_PATCHES\\/i915_316.patch\"\\npatch -p1 -i \"\\/PATH_TO_PATCHES\\/override_for_missing_acs_capabilities.patch\"\\npatch -p1 -i \"\\PATH_TO_PATCHES\\/vga_arbitrer.patch\"

Save the file, and now it's time test!

yaourt -S linux

Take a look at the output.  You should see customizepkg doing its thing.  Your three patches should show up a few lines above where yaourt asks if you wan to edit the PKGBUILD.  They probably wont be listed at the start where it receives the file list.  This is OK.  Also, it will tell you if there are any errors.  If there are errors from sed you'll need to tweak your file in customizepkg.d.  All slashes and double quotes need to be escaped.  That's a common error.  If there are no errors, continue the build and see if it completes.  If it compiles and installs properly, try booting from the new kernel.  Finally, try out your VM.

If all goes well, you should be able to update your whole system without having to do anything else manually using:

yaourt -Syua

It's worth watching the first page of this thread to see if any of the patches are changed every now and then.  Also, be careful if the kernel changes from one major kernel version to another (3.16 -> 3.17).  I just starting using this method, so I'm not sure if things will break at that point.

If there is a better, cleaner way of doing this, please let me know!  I recognize this is kind of hackish, but it works well for me so far.

Last edited by BahbGTR (2014-08-22 15:51:28)