Arch Linux

groggster

Member

Registered: 2014-08-22

Posts: 1

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Hi!

I need some help with getting this running properly. I am currently able to start up the VM with the graphics card passed along, everyting runs fine. I do however have problems getting virt-manager to boot this machine. How do i achieve that? I have tried manually configuring the .xml config files, but to no avail. This is my configuration file:

export QEMU_AUDIO_DRV=alsa QEMU_AUDIO_TIMER_PERIOD=0
qemu-system-x86_64 \
    -enable-kvm -M q35 -m 2048 -cpu host -smp 4 \
    -bios /usr/share/qemu/bios.bin -vga none -display none \
    -device ioh3420,bus=pcie.0,addr=1c.0,multifunction=on,port=1,chassis=1,id=root.1 \
    -device vfio-pci,host=07:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on,romfile=/mnt/vm/htpc/MSI.HD6450.1024.110407.rom \
    -device vfio-pci,host=07:00.1,bus=root.1,addr=00.1 \
    -usb -usbdevice host:046d:c52b \
    -drive file=/mnt/vm/htpc/htpc,id=disk,format=raw,index=0 -device ide-hd,bus=ide.0,drive=disk \
;

By executing this script file the VM starts and runs as intended. How to I get it to work in virt-manager?
I am able to boot the VM through virt-manager, and it registers the graphics card ("lspci" finds it), but no video is forwarded to it, all I get is video on the virt-manager console. Whats next?

This vm is running ubuntu for use as a htpc if that matters...

redger

Member

Registered: 2014-02-26

Posts: 15

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

@ groggster
see this post  https://bbs.archlinux.org/viewtopic.php … 6#p1438076

mauorrizze

Member

Registered: 2014-08-22

Posts: 18

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

edit: the following problem exists in current qemu git version (August 22th/23th), version from August 18th does work. Possibly related to q35*
*) https://www.mail-archive.com/qemu-devel … 51880.html

Hi, my first try on getting gpu passthrough to work ended with an segfaulting qemu, using the testing snippet from the first post. I tried to debug a little, but using valgrind the gpu does come to live (can see GPU Post-Screen and seabios looking for disks in slow-motion, but hey)!
So first I thought my hardware has some bugs or faulty firmware or something, but I hope it might be a software problem. Does anyone else has problems with the latest qemu git?

My hardware: Intel E3-1245 (V1), MSI Z77A-GD65 / ASRock H77 Pro4-m (the MSI has been successfully used in a home server, passing through a SAS controller), NVidia GTX 750 Ti
Software: arch linux with patched mainline 3.16 kernel from first post, qemu-git from AUR (and built and installed in home for debugging purpose)

% uname -a
Linux mustrum 3.16.0-1-mainline #1 SMP PREEMPT Fri Aug 22 01:35:09 CEST 2014 x86_64 GNU/Linux

kernel log

% dmesg | grep -i -e dmar -e iommu -e vfio -e arb -e vga -e drm
[    0.000000] Command line: \vmlinuz-linux-mainline root=PARTUUID=3520fde9-ca4e-4b5c-b48d-077964e89ef5 rootfstype=f2fs ro pci-stub.ids=10de:1380,10de:0fbc i915.enable_hd_vgaarb=1 intel_iommu=on initrd=initramfs-linux-mainline.img
[    0.000000] ACPI: DMAR 0x00000000C9435038 0000B8 (v01 INTEL  SNB      00000001 INTL 00000001)
[    0.000000] Kernel command line: \vmlinuz-linux-mainline root=PARTUUID=3520fde9-ca4e-4b5c-b48d-077964e89ef5 rootfstype=f2fs ro pci-stub.ids=10de:1380,10de:0fbc i915.enable_hd_vgaarb=1 intel_iommu=on initrd=initramfs-linux-mainline.img
[    0.000000] Intel-IOMMU: enabled
[    0.031799] dmar: Host address width 36
[    0.031801] dmar: DRHD base: 0x000000fed90000 flags: 0x0
[    0.031809] dmar: IOMMU 0: reg_base_addr fed90000 ver 1:0 cap c0000020e60262 ecap f0101a
[    0.031810] dmar: DRHD base: 0x000000fed91000 flags: 0x1
[    0.031814] dmar: IOMMU 1: reg_base_addr fed91000 ver 1:0 cap c9008020660262 ecap f0105a
[    0.031815] dmar: RMRR base: 0x000000ca48e000 end: 0x000000ca49cfff
[    0.031816] dmar: RMRR base: 0x000000cb800000 end: 0x000000cf9fffff
[    0.031885] IOAPIC id 2 under DRHD base  0xfed91000 IOMMU 1
[    0.299425] vgaarb: device added: PCI:0000:00:02.0,decodes=io+mem,owns=io+mem,locks=none
[    0.299428] vgaarb: device added: PCI:0000:01:00.0,decodes=io+mem,owns=none,locks=none
[    0.299430] vgaarb: loaded
[    0.299430] vgaarb: bridge control possible 0000:01:00.0
[    0.299431] vgaarb: no bridge control possible 0000:00:02.0
[    0.346403] DMAR: No ATSR found
[    0.346422] IOMMU 0 0xfed90000: using Queued invalidation
[    0.346423] IOMMU 1 0xfed91000: using Queued invalidation
[    0.346424] IOMMU: Setting RMRR:
[    0.346433] IOMMU: Setting identity map for device 0000:00:02.0 [0xcb800000 - 0xcf9fffff]
[    0.346726] IOMMU: Setting identity map for device 0000:00:14.0 [0xca48e000 - 0xca49cfff]
[    0.346741] IOMMU: Setting identity map for device 0000:00:1a.0 [0xca48e000 - 0xca49cfff]
[    0.346753] IOMMU: Setting identity map for device 0000:00:1d.0 [0xca48e000 - 0xca49cfff]
[    0.346761] IOMMU: Prepare 0-16MiB unity mapping for LPC
[    0.346767] IOMMU: Setting identity map for device 0000:00:1f.0 [0x0 - 0xffffff]
[    0.353553] fb0: EFI VGA frame buffer device
[    1.047639] systemd[1]: Starting Arbitrary Executable File Formats File System Automount Point.
[    1.047806] systemd[1]: Set up automount Arbitrary Executable File Formats File System Automount Point.
[    1.222325] [drm] Initialized drm 1.1.0 20060810
[    1.244809] [drm] Memory usable by graphics device = 2048M
[    1.244811] [drm] VT-d active for gfx access
[    1.244813] [drm] Disabling PPGTT because VT-d is on
[    1.244814] [drm] Replacing VGA console driver
[    1.244818] fb: switching to inteldrmfb from EFI VGA
[    1.262227] [drm] Supports vblank timestamp caching Rev 2 (21.10.2013).
[    1.262228] [drm] Driver supports precise vblank timestamp query.
[    1.262270] [drm] DMAR active, disabling use of stolen memory
[    1.265141] [drm] Wrong MCH_SSKPD value: 0x16040307
[    1.265143] [drm] This can cause pipe underruns and display issues.
[    1.265143] [drm] Please upgrade your BIOS to fix this.
[    1.329323] fbcon: inteldrmfb (fb0) is primary device
[    1.370543] i915 0000:00:02.0: fb0: inteldrmfb frame buffer device
[    1.392060] vgaarb: device changed decodes: PCI:0000:00:02.0,olddecodes=io+mem,decodes=io:owns=io+mem
[    1.414195] [drm] Initialized i915 1.6.0 20080730 for 0000:00:02.0 on minor 0
[    2.344673] [drm] Enabling RC6 states: RC6 on, RC6p off, RC6pp off
[   15.312138] [drm] stuck on render ring
[   15.312526] [drm] GPU HANG: ecode 0:0x85fffffd, in systemd-logind [309], reason: Ring hung, action: reset
[   15.312528] [drm] GPU hangs can indicate a bug anywhere in the entire gfx stack, including userspace.
[   15.312530] [drm] Please file a _new_ bug report on bugs.freedesktop.org against DRI -> DRM/Intel
[   15.312531] [drm] drm/i915 developers can then reassign to the right component if it's not a kernel issue.
[   15.312532] [drm] The gpu crash dump is required to analyze gpu hangs, so please always attach it.
[   15.312533] [drm] GPU crash dump saved to /sys/class/drm/card0/error
[   17.310369] [drm] Enabling RC6 states: RC6 on, RC6p off, RC6pp off
[   36.970235] VFIO - User Level meta-driver version: 0.3
[   64.491380] vfio-pci 0000:01:00.0: enabling device (0000 -> 0003)
[   64.491522] vfio_ecap_init: 0000:01:00.0 hiding ecap 0x1e@0x258
[   64.491534] vfio_ecap_init: 0000:01:00.0 hiding ecap 0x19@0x900
[   64.494601] vfio-pci 0000:01:00.1: enabling device (0000 -> 0002)

gdb core dump inspection

Program terminated with signal SIGSEGV, Segmentation fault.
#0  0x00007ffff6903fe0 in g_str_hash () from /usr/lib/libglib-2.0.so.0
(gdb) backtrace
#0  0x00007ffff6903fe0 in g_str_hash () from /usr/lib/libglib-2.0.so.0
#1  0x00007ffff6903569 in g_hash_table_lookup () from /usr/lib/libglib-2.0.so.0
#2  0x00000000006b4e94 in type_table_lookup (name=0x8 <error: Cannot access memory at address 0x8>) at qom/object.c:92
#3  0x00000000006b5112 in type_get_by_name (name=0x8 <error: Cannot access memory at address 0x8>) at qom/object.c:158
#4  0x00000000006b514a in type_get_parent (type=0x1344ff0) at qom/object.c:164
#5  0x00000000006b5292 in type_is_ancestor (type=0x1344ff0, target_type=0x10f30c0) at qom/object.c:212
#6  0x00000000006b5ec8 in object_class_dynamic_cast (class=0x7ffff22fd128 <main_arena+1192>, typename=0x7c88c3 "cpu") at qom/object.c:530
#7  0x00000000006b5c1b in object_dynamic_cast (obj=0x11ec140, typename=0x7c88c3 "cpu") at qom/object.c:443
#8  0x00000000004b1890 in acpi_add_cpu_info (o=0x11ec140, opaque=0x7fffebcbf950) at /home/marcel/src/qemu/hw/i386/acpi-build.c:133
#9  0x00000000006b6280 in object_child_foreach (obj=0x11ed260, fn=0x4b1867 <acpi_add_cpu_info>, opaque=0x7fffebcbf950) at qom/object.c:676
#10 0x00000000004b18fa in acpi_add_cpu_info (o=0x11ed260, opaque=0x7fffebcbf950) at /home/marcel/src/qemu/hw/i386/acpi-build.c:140
#11 0x00000000006b6280 in object_child_foreach (obj=0x112ed60, fn=0x4b1867 <acpi_add_cpu_info>, opaque=0x7fffebcbf950) at qom/object.c:676
#12 0x00000000004b18fa in acpi_add_cpu_info (o=0x112ed60, opaque=0x7fffebcbf950) at /home/marcel/src/qemu/hw/i386/acpi-build.c:140
#13 0x00000000006b6280 in object_child_foreach (obj=0x1117270, fn=0x4b1867 <acpi_add_cpu_info>, opaque=0x7fffebcbf950) at qom/object.c:676
#14 0x00000000004b18fa in acpi_add_cpu_info (o=0x1117270, opaque=0x7fffebcbf950) at /home/marcel/src/qemu/hw/i386/acpi-build.c:140
#15 0x00000000006b6280 in object_child_foreach (obj=0x1117f70, fn=0x4b1867 <acpi_add_cpu_info>, opaque=0x7fffebcbf950) at qom/object.c:676
#16 0x00000000004b1941 in acpi_get_cpu_info (cpu=0x7fffebcbf950) at /home/marcel/src/qemu/hw/i386/acpi-build.c:149
#17 0x00000000004b5054 in acpi_build (guest_info=0x1122470, tables=0x7fffebcbf9f0) at /home/marcel/src/qemu/hw/i386/acpi-build.c:1486
#18 0x00000000004b55e8 in acpi_build_update (build_opaque=0x13021b0, offset=0) at /home/marcel/src/qemu/hw/i386/acpi-build.c:1623
#19 0x000000000062f2f2 in fw_cfg_read (s=0x117c720) at hw/nvram/fw_cfg.c:255
#20 0x000000000062f404 in fw_cfg_comb_read (opaque=0x117c720, addr=1, size=1) at hw/nvram/fw_cfg.c:291
#21 0x000000000045e793 in memory_region_read_accessor (mr=0x117ec10, addr=1, value=0x7fffebcbfb50, size=1, shift=0, mask=255) at /home/marcel/src/qemu/memory.c:410
#22 0x000000000045e9ef in access_with_adjusted_size (addr=1, value=0x7fffebcbfb50, size=1, access_size_min=1, access_size_max=4, access=0x45e741 <memory_region_read_accessor>, mr=0x117ec10)
    at /home/marcel/src/qemu/memory.c:480
#23 0x0000000000460fc2 in memory_region_dispatch_read1 (mr=0x117ec10, addr=1, size=1) at /home/marcel/src/qemu/memory.c:1097
#24 0x0000000000461084 in memory_region_dispatch_read (mr=0x117ec10, addr=1, pval=0x7fffebcbfc08, size=1) at /home/marcel/src/qemu/memory.c:1119
#25 0x0000000000463c2c in io_mem_read (mr=0x117ec10, addr=1, pval=0x7fffebcbfc08, size=1) at /home/marcel/src/qemu/memory.c:1963
#26 0x0000000000419fcc in address_space_rw (as=0xc2e180 <address_space_io>, addr=1297, buf=0x7ffff7ea4000 <error: Cannot access memory at address 0x7ffff7ea4000>, len=1, is_write=false)
    at /home/marcel/src/qemu/exec.c:2086
#27 0x000000000045bfb0 in kvm_handle_io (port=1297, data=0x7ffff7ea4000, direction=0, size=1, count=1024) at /home/marcel/src/qemu/kvm-all.c:1597
#28 0x000000000045c3ee in kvm_cpu_exec (cpu=0x11323e0) at /home/marcel/src/qemu/kvm-all.c:1734
#29 0x0000000000445947 in qemu_kvm_cpu_thread_fn (arg=0x11323e0) at /home/marcel/src/qemu/cpus.c:939
#30 0x00007ffff3da5124 in start_thread () from /usr/lib/libpthread.so.0
#31 0x00007ffff203d4bd in clone () from /usr/lib/libc.so.6
(gdb) list
2914	
2915	    QDECREF(pdict);
2916	    g_free(id);
2917	    g_free(type);
2918	    g_free(dummy);
2919	    if (err) {
2920	        qerror_report_err(err);
2921	        error_free(err);
2922	        return -1;
2923	    }

Valgrind output: http://txt.do/k882
(Stopped right after the gpu comes to live. Lots of unimportant "Conditional jump or move depends on uninitialised value(s)", scroll almost to the end for "Invalid read of size 8")

Last edited by mauorrizze (2014-08-24 02:04:46)

somearchfan

Member

Registered: 2014-08-18

Posts: 35

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Linux-mainline 3.16.1 + patches. Additional changes: tickrate set to 1000 Hz and preemption model has been set to voluntary.

http://netload.in/dateixLakD3kkGz/linux … tar.gz.htm

abdullah

Member

Registered: 2014-06-10

Posts: 34

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

http://edencomputing.com/index.php/2014 … ebian-way/

    Virtualization
    (CONFIG_VIRTUALIZATION=y)
    Virtualization > Kernel-based Virtual Machine (KVM) Support
    (CONFIG_KVM=y)
    Virtualization > KVM for <whichever processor you have>
    (CONFIG_KVM_INTEL=y and/or CONFIG_KVM_AMD=y)
    Bus Options (PCI etc) > PCI Stub Driver
    (CONFIG_PCI_STUB=y)
    Device Drivers > VFIO Non-Privileged userspace driver framework
    (CONFIG_VFIO=y)
    Device Drivers > VFIO Non-Privileged userspace driver framework > VFIO support for PCI Devices
    (CONFIG_VFIO_PCI=y)
    Device Drivers > VFIO Non-Privileged userspace driver framework > VFIO support for VGA Devices
    (CONFIG_VFIO_PCI_VGA=y)

The rest of these are optional (but recommended) performance enhancements you may compile as modules [M]:

    Virtualization > Host kernel accelerator for virtio net
    (CONFIG_VHOST_NET=m)
    Device Drivers > Virtio drivers > PCI driver for virtio devices
    (CONFIG_VIRTIO_PCI=m)
    Device Drivers > Virtio drivers > Virtio balloon driver
    (CONFIG_VIRTIO_BALLOON=m)

And finally these options will enhance performance and responsiveness of your virtual machines:

    Processor Type and Features > Preemption Model > Preemptible Kernel (Low Latency Desktop)
    (CONFIG_PREEMPT=y)
    Processor Type and Features > Timer Frequency > 1000 Hz
    (CONFIG_HZ_1000=y)

just apply the batch, then change the above.
you have your  kernel.
I'm not relly sure about downloading files from unknown sources.. ( the prv post).

My setup is now working with 2 GPUs, I got Vm1 and Vm2, each one has 6114 Gb of ram, 2 HDDs, the first one using Nvidia GTX 760, the second one uses the AMD sapphire toxic 4780 HD, everything looks great, but if you killed an VM through the host, you will destroy it, and it wont boot anymore, thats make me afride of power loss.

somearchfan

Member

Registered: 2014-08-18

Posts: 35

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Oh right forgot to tell ya'll about dat secret NSA backdoor.

bruce1337

Member

Registered: 2014-08-24

Posts: 3

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Hello there,

thanks to this pertinent resource (and some qualified outside assistance), the inappropriately monikered linux newb that is me has succeeded in passing through dedicated graphics to a VM. So far, I have relied on reading only -- but at 100+ pages, this thread has become so unwieldy that I decided de-lurk (...making it even more unwieldy ). Nice signup captcha by the way! Hope you will forgive any violations of tech etiquette I'm bound to make and let me know if I did, so that I may avoid them in the future.

For the record, I'm running

Xeon E3-1246v3
ASRock H97 Pro4, supporting both Vt-d and IOMMU (as far as I can tell...)

My issue is with virt-manager (both 0.9.5 & 1.01), and seems to be identical to Chetyre's discussed on p.26: qemu, when invoked by virt-manager, fails to access /dev/vfio/x while (seemingly?) succeeding when run from the command line. I followed all subsequent directions I could find, i.e.

/etc/libvirt/qemu.conf

user = "+0"
group = "+0"

group_device_acl = [   
+    "/dev/vfio/1", "/dev/vfio/13"
]

clear_emulator_capabilities = 0

relaxed_acs_check = 1

(/dev/vfio/13 contains 2 entries: video and audio from the GTX [edit: or rather the respective iommu_group does...])

Unfortunately, I could locate neither additional info nor a confirmation of the problem's solution further down the thread. Strangely, the problem persists when running virt-manager itself as root. To be clear: running the barebone qemu config (ioh3420 + GTX) from the command line results in Seabios output up to "no bootable device" from the GTX, so I'm assuming the principal setup to be correct -- it's just that for now, I would prefer to rely on the convenience of virt-manager offering a working setup for interfacing to the VM by spice and mouse/keyboard emulation instead of applying everything manually through qemu-argv. The last couple of days have been filled with a lot of studying, I think it's about time for some playing...

[edit: After some wrestling with qemu (or rather my ignorance of it), my odyssey has come to a a (certainly temporary) conclusion -- things are finally working! Has anyone ever figured out what exactly determines whether qemu will grab either mouse, keyboard, both, or sometimes nothing? Is it all a function of "-device qxl" (hard to believe...)?  Also, does anyone know if I can copy my concurrent .img directly to a dedicated logical volume or will the "physical partition" rules apply and demand loopback MBR? Thanks!]

[edit #2: Well, this is somewhat embarassing, but as I now know, all the troubles I've been having with qemu's command line invocation were clearly due to PEBKAC. Pro-tip: Do NOT insert comments into the multiline command of your shell script like ol' silly bruce did. Also, dd'ing an image to a dedicated LV works without a hitch, no additional steps required.]

Last edited by bruce1337 (2014-08-26 13:09:50)

somearchfan

Member

Registered: 2014-08-18

Posts: 35

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

For the record, I'm running

Xeon E3-1246v3
ASRock H97 Pro4, supporting both Vt-d and IOMMU (as far as I can tell...)

Just a quick fyi: the H97 Pro4 does support VT-D (you can disable / enable it in the bios settings) and IOMMU, so no worries there. I'm using a fairly similar board, the H97M Pro 4. The only difference is the form factor afaik.

Cheers

bruce1337

Member

Registered: 2014-08-24

Posts: 3

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

It's not the size that matters, it's how you wield it...still, mine *is* bigger! Then again, those 2 additional PCI slots will hardly see any action, ever...

Actually, it was ASRock's reputed feature support beyond Intel's chipset standards that made me buy it, so I wasn't worried too much. Still, good to have direct confirmation! (And should you ever be bored, do take the guided tour of the UEFI, it's pretty goofy, to say the least...)

Last edited by bruce1337 (2014-08-25 00:37:32)

sitonapanotis

Member

Registered: 2014-01-08

Posts: 15

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Ive been trying to get my kvm +qemu vm up and running on an arch host, eventually running windows 8.1 as a guest.

my setup is a vt-d and IOMMU compatible mobo and intel processor with a r9 270 for the host and a GTX 770 for the guest.
I am using the 3.16.1 kernel package from the ABS to which I applied the acs override patch. (I didnt change any of the options/flags, maybe I need to do that?)

running with these options

root=/dev/mapper/VolGroup00-lvroot rw initrd=/initramfs-linux-custom-acs.img intel_iommu=on

Right now im getting the devices to bind to vfio just fine but when I try to run my test script I run into errors.

here is the script im using to test.

	
    qemu-system-x86_64 -enable-kvm -M q35 -m 1024 -cpu host \
    -smp 6,sockets=1,cores=6,threads=1 \
    -bios /usr/share/qemu/bios.bin -vga none \
    -device ioh3420,bus=pcie.0,addr=1c.0,multifunction=on,port=1,chassis=1,id=root.1 \
    -device vfio-pci,host=02:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on \
    -device vfio-pci,host=02:00.1,bus=root.1,addr=00.1

and here is the output I get from running it from root

qemu-system-x86_64: -device vfio-pci,host=02:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on: vfio: error, group 1 is not viable, please ensure all devices within the iommu_group are bound to their vfio bus driver.
qemu-system-x86_64: -device vfio-pci,host=02:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on: vfio: failed to get group 1
qemu-system-x86_64: -device vfio-pci,host=02:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on: Device initialization failed.
qemu-system-x86_64: -device vfio-pci,host=02:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on: Device 'vfio-pci' could not be initialized

any ideas as to where I am going wrong?

Last edited by sitonapanotis (2014-08-25 05:18:59)

siddharta

Member

Registered: 2014-05-03

Posts: 30

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Ive been trying to get my kvm +qemu vm up and running on an arch host, eventually running windows 8.1 as a guest.

my setup is a vt-d and IOMMU compatible mobo and intel processor with a r9 270 for the host and a GTX 770 for the guest.
I am using the 3.16.1 kernel package from the ABS to which I applied the acs override patch. (I didnt change any of the options/flags, maybe I need to do that?)

running with these options

root=/dev/mapper/VolGroup00-lvroot rw initrd=/initramfs-linux-custom-acs.img intel_iommu=on

Right now im getting the devices to bind to vfio just fine but when I try to run my test script I run into errors.

here is the script im using to test.

	
    qemu-system-x86_64 -enable-kvm -M q35 -m 1024 -cpu host \
    -smp 6,sockets=1,cores=6,threads=1 \
    -bios /usr/share/qemu/bios.bin -vga none \
    -device ioh3420,bus=pcie.0,addr=1c.0,multifunction=on,port=1,chassis=1,id=root.1 \
    -device vfio-pci,host=02:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on \
    -device vfio-pci,host=02:00.1,bus=root.1,addr=00.1

and here is the output I get from running it from root

qemu-system-x86_64: -device vfio-pci,host=02:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on: vfio: error, group 1 is not viable, please ensure all devices within the iommu_group are bound to their vfio bus driver.
qemu-system-x86_64: -device vfio-pci,host=02:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on: vfio: failed to get group 1
qemu-system-x86_64: -device vfio-pci,host=02:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on: Device initialization failed.
qemu-system-x86_64: -device vfio-pci,host=02:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on: Device 'vfio-pci' could not be initialized

any ideas as to where I am going wrong?

Look at the QEMU configuration QEMU.conf, as three posts above. Set QEMU processes to run as root, do

ls /dev/vfio

and add what you find there to the group_device_acl in qemu.conf, as above (or look through this thread for other examples). Lastly, invoke qemu-system-x86_64 as root.

aw

Member

Registered: 2013-10-04

Posts: 921

Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

I applied the acs override patch. (I didnt change any of the options/flags, maybe I need to do that?)

qemu-system-x86_64: -device vfio-pci,host=02:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on: vfio: error, group 1 is not viable, please ensure all devices within the iommu_group are bound to their vfio bus driver.

any ideas as to where I am going wrong?

The error message is indicating exactly what's wrong.  There are more devices in the IOMMU group than you're passing through to the guest.  An IOMMU group cannot be split between host and guest, therefore all the device in the group must be bound to vfio-pci (or pci-stub) for the group to be viable.  There are scripts around like lsgroup for easily viewing the IOMMU group devices, or just look in /sys/kernel/iommu_group/  Applying the ACS override patch without activating it on the commandline does nothing.

BTW, the signal to noise ratio of this thread has dropped to pretty unbearable levels.  I've started the below blog as an attempt to help.  I've got a couple things in mind to post about, but I welcome suggestions.  I'm working on another detailed post describing ACS and IOMMU groups, then maybe I'll get back to some hands-on examples.  Bear with me as I build some content.  If anyone wants to contribute a post, please contact me.

http://vfio.blogspot.com/

---

http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

sitonapanotis

Member

Registered: 2014-01-08

Posts: 15

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

I applied the acs override patch. (I didnt change any of the options/flags, maybe I need to do that?)

qemu-system-x86_64: -device vfio-pci,host=02:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on: vfio: error, group 1 is not viable, please ensure all devices within the iommu_group are bound to their vfio bus driver.

any ideas as to where I am going wrong?

The error message is indicating exactly what's wrong.  There are more devices in the IOMMU group than you're passing through to the guest.  An IOMMU group cannot be split between host and guest, therefore all the device in the group must be bound to vfio-pci (or pci-stub) for the group to be viable.  There are scripts around like lsgroup for easily viewing the IOMMU group devices, or just look in /sys/kernel/iommu_group/  Applying the ACS override patch without activating it on the commandline does nothing.

BTW, the signal to noise ratio of this thread has dropped to pretty unbearable levels.  I've started the below blog as an attempt to help.  I've got a couple things in mind to post about, but I welcome suggestions.  I'm working on another detailed post describing ACS and IOMMU groups, then maybe I'll get back to some hands-on examples.  Bear with me as I build some content.  If anyone wants to contribute a post, please contact me.

http://vfio.blogspot.com/

It apppears that my processor  radeon card/radeon sound device and nvidia card/sound device are all in the same group which is what is causing problems.

How do I go about enableing the acs override?

aw

Member

Registered: 2013-10-04

Posts: 921

Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

I applied the acs override patch. (I didnt change any of the options/flags, maybe I need to do that?)

qemu-system-x86_64: -device vfio-pci,host=02:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on: vfio: error, group 1 is not viable, please ensure all devices within the iommu_group are bound to their vfio bus driver.

any ideas as to where I am going wrong?

The error message is indicating exactly what's wrong.  There are more devices in the IOMMU group than you're passing through to the guest.  An IOMMU group cannot be split between host and guest, therefore all the device in the group must be bound to vfio-pci (or pci-stub) for the group to be viable.  There are scripts around like lsgroup for easily viewing the IOMMU group devices, or just look in /sys/kernel/iommu_group/  Applying the ACS override patch without activating it on the commandline does nothing.

BTW, the signal to noise ratio of this thread has dropped to pretty unbearable levels.  I've started the below blog as an attempt to help.  I've got a couple things in mind to post about, but I welcome suggestions.  I'm working on another detailed post describing ACS and IOMMU groups, then maybe I'll get back to some hands-on examples.  Bear with me as I build some content.  If anyone wants to contribute a post, please contact me.

http://vfio.blogspot.com/

It apppears that my processor  radeon card/radeon sound device and nvidia card/sound device are all in the same group which is what is causing problems.

How do I go about enableing the acs override?

Really?  You're not even willing to google "acs override"?

---

http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

sitonapanotis

Member

Registered: 2014-01-08

Posts: 15

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

The error message is indicating exactly what's wrong.  There are more devices in the IOMMU group than you're passing through to the guest.  An IOMMU group cannot be split between host and guest, therefore all the device in the group must be bound to vfio-pci (or pci-stub) for the group to be viable.  There are scripts around like lsgroup for easily viewing the IOMMU group devices, or just look in /sys/kernel/iommu_group/  Applying the ACS override patch without activating it on the commandline does nothing.

BTW, the signal to noise ratio of this thread has dropped to pretty unbearable levels.  I've started the below blog as an attempt to help.  I've got a couple things in mind to post about, but I welcome suggestions.  I'm working on another detailed post describing ACS and IOMMU groups, then maybe I'll get back to some hands-on examples.  Bear with me as I build some content.  If anyone wants to contribute a post, please contact me.

http://vfio.blogspot.com/

It apppears that my processor  radeon card/radeon sound device and nvidia card/sound device are all in the same group which is what is causing problems.

How do I go about enableing the acs override?

Really?  You're not even willing to google "acs override"?

Im reading mailing list posts on it as we speak, I just thought this would be a good place to ask for more specific instructions. I didn't know that was inapropriate for the disscussion here. Anyway thank you for the help so far, without this thread I wouldnt even have been able to get as far as I have already.

edit:

got it working with pcie_acs_override=downstream

edit2:

qemu starts now but I dont get output from my passed through card on my second monitor.
when testing using "vga std" in place of "vga none" the vm/seabios boots fine.
I searched through the thread for others who seemed to have similar problems and saw that some of them had solved those problems but I was unable to identify how. If anyone has experienced/fixed this problem let me know.

Last edited by sitonapanotis (2014-08-26 08:48:30)

slis

Member

Registered: 2014-06-02

Posts: 127

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

you need to read op post about vga patches (not acs) and how to activate them...

Last edited by slis (2014-08-26 10:01:44)

aw

Member

Registered: 2013-10-04

Posts: 921

Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

What's your favorite over-asked question here?  http://vfio.blogspot.com/2014/08/vfiovga-faq.html

---

http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

bruce1337

Member

Registered: 2014-08-24

Posts: 3

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

As someone who diligently tried to obtain answers by reading through this thread (yes...all of it), I believe you have already caught all prime suspects -- would be great to have this linked prominently in the OP. Also, I respectfully tip my hat to you, good sir! I certainly wouldn't be in the position I am in now were it not for your dedication, both in terms of technology as well as communication, and for that, you have my sincerest thanks. I shall try to pay it forward with the talents at my disposal.

slis

Member

Registered: 2014-06-02

Posts: 127

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

3rd

aw

Member

Registered: 2013-10-04

Posts: 921

Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

I don't expect everyone is following my new blog (yet), but I wrote this up yesterday:

http://vfio.blogspot.com/2014/08/primar … t-vga.html

I'm not the first to try this, former active user on this thread DanaGoyette attempted this, but never fully debugged the issues.  Using OVMF (UEFI) for the guest we can avoid all of the issues with the VGA aspects of GPU assignment.  This means that we don't care about how broken the i915 driver is and we don't need the x-vga option for the assigned device, making it just another assigned devices as far as libvirt is concerned.

Also:

http://vfio.blogspot.com/2014/08/upstre … -2014.html

If you're a GeForce user, libirt just got support for kvm=off and QEMU will now automatically enable the primary device quirk for Nvidia GPUs independent of VGA.

---

http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

sitonapanotis

Member

Registered: 2014-01-08

Posts: 15

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

What's your favorite over-asked question here?  http://vfio.blogspot.com/2014/08/vfiovga-faq.html

This faq is SUPER useful, I was having problem 4, which I fixed now.
Thank you!

edit: Got everything set up sucessfully! Thank you so much to everyone in the thread!

Last edited by sitonapanotis (2014-08-27 03:16:22)

Wimma77

Member

Registered: 2014-08-12

Posts: 15

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

I don't expect everyone is following my new blog (yet), but I wrote this up yesterday:
...

Thanks so much for all the details.
I've just successfully got my HD7770 passed through to my Win7 VM (trick was to not pass the sound with it - good tip redger) and proved capability.
And that was on 14.04 with vanilla qemu and 3.16 kernel.
Still have to work out networking and sound, but progress is always encouraging.
I will have to consider Win8 based on your UEFI comments, but will wait for my GTX750ti first and give that a crack.
Thanks again for the help.
Will try your blog and see how it goes.Getting closer to a steam streaming server!
Cheers.

josipva

Member

Registered: 2014-07-21

Posts: 29

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Hi! Can someone help with following questions?

- How much performance can I expect from i7-4790k running host arch-linux and 2 win 8.1 virtual machines for gaming?
- How similar performance would be compared to i5 4690 running windows natively (assuming same graphics cards in those situations) ?

Last edited by josipva (2014-08-27 13:03:50)

siddharta

Member

Registered: 2014-05-03

Posts: 30

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Hi! Can someone help with following questions?

- How much performance can I expect from i7-4790k running host arch-linux and 2 win 8.1 virtual machines for gaming?
- How similar performance would be compared to i5 4690 running windows natively (assuming same graphics cards in those situations) ?

I haven't tried this myself so I can't say for certain it would work but why not. If, for example, you would pass 3 vcpu to each guest, keeping 2 vcpu for the host and pinning accordingly I would assume you would get equal or faster performance for most tasks. I don't think there's a penalty to assigning an odd number of vcpu though I could be mistaken. Even with the performance hit compared to bare metal there is a 500 MHz difference in base clock and turbo speeds between the 4790K and 4690, that should in part or fully make compensate depending on the application. You'd likely have somewhat reduced performance in applications where having a fourth core available is a benefit. However again I haven't tried this.

I have an 4790K with Win 8.1 using 4 vcpu and a 7950, if you'd like I can run a benchmark if you tell me which one. Windows performance index is 8.1 for CPU, memory and graphics.

aw

Member

Registered: 2013-10-04

Posts: 921

Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Hi! Can someone help with following questions?

- How much performance can I expect from i7-4790k running host arch-linux and 2 win 8.1 virtual machines for gaming?
- How similar performance would be compared to i5 4690 running windows natively (assuming same graphics cards in those situations) ?

I expect your clock scaling from 4GHz to 3.5GHz for the VM is probably sufficiently conservative to account for the overhead, but if you split an i7-4790k in half, you're probably looking at something more like an i3-4330 per VM.  Threading doesn't come anywhere close to doubling the throughput of a core, so running two quad-core VMs on a quad-core + threads host is overcommitting the CPU resources.

---

http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users