Arch Linux

zzz3000

Member

Registered: 2014-02-16

Posts: 12

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Video passthrought had worked for me with reboot problem when I used pci-stub metod and started VM via virt-manager.

Reboot  problem dissapeared  for me when I used vfio method and started via console, but host intel graphics becomes spoiled.

After I had installed Radeon Driver it stopped working at all, and host begun to crash.

You'll probably need to use the i915 patch

Thanks for reply.

i915 patch could resolv intel spoiled graphics or  full crash with radeon driver installed on Windows?

Where  can I get this patch? Is  it a kernel patch or what?

Also I don't see bios booting in guest's video card.

I am using Fedora 20.

Last edited by zzz3000 (2014-02-21 06:38:06)

mostlyharmless

Member

Registered: 2014-01-16

Posts: 72

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

So, I tried switching my NVIDIA K2000 and AMD HD 7970 in my ASUS RIVE with 4930K .  Previously, everything worked without problems with the NVIDIA as guest and the AMD as host, but of course I couldn't run the AMD drivers on the host, and the radeon drivers don't support 3D acceleration. So now I have the NVIDIA on the host with the HD7970 as guest.  I ave been running the same kernel/qemu/seabios; ie 3.13.2/qemu-git and seabios git downloaded 12-5-13.   I've patched the kernel with https://git.kernel.org/cgit/linux/kerne … fe8702b098 . My kernel line has vga=normal on it, but of course I still get this warning in dmesg:

fb0: EFI VGA frame buffer device
[   18.026908] hda-intel 0000:01:00.1: Handle VGA-switcheroo audio client
[   49.298091] NVRM: Your system is not currently configured to drive a VGA console
[   49.298094] NVRM: on the primary VGA device. The NVIDIA Linux graphics driver
[   49.298095] NVRM: requires the use of a text-mode VGA console. Use of other console

because I boot in efi mode.  I'm not sure this a way to boot in any other mode on this motherboard. I've tried this https://devtalk.nvidia.com/default/topi … -warning-/
but there is no framebuffer other than efifb, so i get a black screen if I add the video=efifb:off.  All that might be irrelevent.

I uninstalled the NVIDIA drivers from the guest prior to swapping.  I patched the NVIDIA driver for the VGA arbitration. Using 319.76 (or for that matter 334.16) on the host seems to work fine with functioning OpenGL. 

I removed the 03:00.1 audio passthrough on my qemu line and installed the AMD driver without CCC.  Windows boots, and seems to work OK with no errors seen in Device Manager.  However, while I can shutdown the guest, I cannot restart the guest, and in fact the host temporarily freezes up.  I tried ejecting the AMD driver prior to shutdown, didn't help and now I have a black screen on the guest too.  Here's dmesg after the crash/freeze. Seems to be a memory leak or something

[  186.932077] vfio-pci 0000:03:00.0: enabling device (0000 -> 0003)
[  186.953915] vfio_ecap_init: 0000:03:00.0 hiding ecap 0x19@0x270
[  186.953920] vfio_ecap_init: 0000:03:00.0 hiding ecap 0x1b@0x2d0
[  187.081805] vfio_cap_init: 0000:00:1d.0 hiding cap 0xa
[  189.849067] dmar: DRHD: handling fault status reg 2
[  189.849073] dmar: DMAR:[DMA Read] Request device [00:1d.0] fault addr ee000 
[  189.849073] DMAR:[fault reason 06] PTE Read access is not set
[  201.195616] vfio-pci 0000:03:00.0: irq 103 for MSI/MSI-X
[  297.885563] vfio_ecap_init: 0000:03:00.0 hiding ecap 0x19@0x270
[  297.885568] vfio_ecap_init: 0000:03:00.0 hiding ecap 0x1b@0x2d0
[  298.006473] vfio_cap_init: 0000:00:1d.0 hiding cap 0xa
[  410.323923] vfio_ecap_init: 0000:03:00.0 hiding ecap 0x19@0x270
[  410.323930] vfio_ecap_init: 0000:03:00.0 hiding ecap 0x1b@0x2d0
[  410.452817] vfio_cap_init: 0000:00:1d.0 hiding cap 0xa
[  412.892137] kvm: zapping shadow pages for mmio generation wraparound
[  774.404347] kwin invoked oom-killer: gfp_mask=0x201da, order=0, oom_score_adj=0
[  774.404351] kwin cpuset=/ mems_allowed=0
[  774.404354] CPU: 5 PID: 1539 Comm: kwin Tainted: P           O 3.13.2 #2
[  774.404355] Hardware name: System manufacturer System Product Name/RAMPAGE IV EXTREME, BIOS 9502 11/07/2013
[  774.404356]  ffff880829b4db90 ffff880816fafa60 ffffffff816338cd ffff880829b4d640
[  774.404358]  ffff880816fafac0 ffffffff81631535 0000000001320122 ffff880816fafa88
[  774.404360]  ffffffff8114daee 0000000000000286 ffffffff81c3b820 ffff8805b227e4a0
[  774.404363] Call Trace:
[  774.404369]  [<ffffffff816338cd>] dump_stack+0x45/0x56
[  774.404372]  [<ffffffff81631535>] dump_header.isra.12+0x76/0x1a7
[  774.404376]  [<ffffffff8114daee>] ? __delayacct_freepages_end+0x2e/0x30
[  774.404379]  [<ffffffff81183d2d>] oom_kill_process+0x1dd/0x340
[  774.404382]  [<ffffffff812a3835>] ? security_capable_noaudit+0x15/0x20
[  774.404384]  [<ffffffff811842a4>] out_of_memory+0x254/0x280
[  774.404387]  [<ffffffff811890c3>] __alloc_pages_nodemask+0x8b3/0x930
[  774.404389]  [<ffffffff81182450>] filemap_fault+0x1b0/0x3d0
[  774.404393]  [<ffffffff811a4449>] __do_fault+0x69/0x4e0
[  774.404394]  [<ffffffff811821cf>] ? wait_on_page_bit_killable+0x7f/0x90
[  774.404396]  [<ffffffff811a870a>] handle_mm_fault+0x30a/0xc70
[  774.404399]  [<ffffffff81095fe0>] __do_page_fault+0x170/0x500
[  774.404403]  [<ffffffff811e31fc>] ? poll_select_copy_remaining+0xdc/0x130
[  774.404405]  [<ffffffff810963ae>] do_page_fault+0xe/0x10
[  774.404408]  [<ffffffff8163c608>] page_fault+0x28/0x30
[  774.404409] Mem-Info:
[  774.404410] DMA per-cpu:
[  774.404411] CPU    0: hi:    0, btch:   1 usd:   0
[  774.404412] CPU    1: hi:    0, btch:   1 usd:   0
[  774.404413] CPU    2: hi:    0, btch:   1 usd:   0
[  774.404414] CPU    3: hi:    0, btch:   1 usd:   0
[  774.404415] CPU    4: hi:    0, btch:   1 usd:   0
[  774.404416] CPU    5: hi:    0, btch:   1 usd:   0
[  774.404417] CPU    6: hi:    0, btch:   1 usd:   0
[  774.404418] CPU    7: hi:    0, btch:   1 usd:   0
[  774.404419] CPU    8: hi:    0, btch:   1 usd:   0
[  774.404419] CPU    9: hi:    0, btch:   1 usd:   0
[  774.404420] CPU   10: hi:    0, btch:   1 usd:   0
[  774.404421] CPU   11: hi:    0, btch:   1 usd:   0
[  774.404422] DMA32 per-cpu:
[  774.404423] CPU    0: hi:  186, btch:  31 usd:   0
[  774.404424] CPU    1: hi:  186, btch:  31 usd:   0
[  774.404425] CPU    2: hi:  186, btch:  31 usd:   0
[  774.404426] CPU    3: hi:  186, btch:  31 usd:   0
[  774.404427] CPU    4: hi:  186, btch:  31 usd:   0
[  774.404428] CPU    5: hi:  186, btch:  31 usd:   0
[  774.404429] CPU    6: hi:  186, btch:  31 usd:   0
[  774.404430] CPU    7: hi:  186, btch:  31 usd:   0
[  774.404430] CPU    8: hi:  186, btch:  31 usd:   0
[  774.404431] CPU    9: hi:  186, btch:  31 usd:   0
[  774.404432] CPU   10: hi:  186, btch:  31 usd:   0
[  774.404433] CPU   11: hi:  186, btch:  31 usd:   0
[  774.404434] Normal per-cpu:
[  774.404435] CPU    0: hi:  186, btch:  31 usd:   0
[  774.404436] CPU    1: hi:  186, btch:  31 usd:   0
[  774.404436] CPU    2: hi:  186, btch:  31 usd:   0
[  774.404437] CPU    3: hi:  186, btch:  31 usd:   0
[  774.404438] CPU    4: hi:  186, btch:  31 usd:   0
[  774.404439] CPU    5: hi:  186, btch:  31 usd:   0
[  774.404440] CPU    6: hi:  186, btch:  31 usd:   0
[  774.404441] CPU    7: hi:  186, btch:  31 usd:   0
[  774.404442] CPU    8: hi:  186, btch:  31 usd:   0
[  774.404442] CPU    9: hi:  186, btch:  31 usd:   0
[  774.404443] CPU   10: hi:  186, btch:  31 usd:   0
[  774.404444] CPU   11: hi:  186, btch:  31 usd:   0
[  774.404447] active_anon:7519336 inactive_anon:582210 isolated_anon:320
[  774.404447]  active_file:15 inactive_file:83 isolated_file:0
[  774.404447]  unevictable:0 dirty:0 writeback:0 unstable:0
[  774.404447]  free:67089 slab_reclaimable:7419 slab_unreclaimable:11764
[  774.404447]  mapped:6157 shmem:2 pagetables:9275 bounce:0
[  774.404447]  free_cma:0
[  774.404452] DMA free:15864kB min:64kB low:80kB high:96kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15996kB managed:15896kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:32kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes
[  774.404453] lowmem_reserve[]: 0 2723 32114 32114
[  774.404457] DMA32 free:128936kB min:11456kB low:14320kB high:17184kB active_anon:2072576kB inactive_anon:576816kB active_file:60kB inactive_file:52kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:2804552kB managed:2794080kB mlocked:0kB dirty:0kB writeback:0kB mapped:700kB shmem:0kB slab_reclaimable:3512kB slab_unreclaimable:2280kB kernel_stack:128kB pagetables:2548kB unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:3988935 all_unreclaimable? yes
[  774.404458] lowmem_reserve[]: 0 0 29390 29390
[  774.404463] Normal free:123556kB min:123640kB low:154548kB high:185460kB active_anon:28004768kB inactive_anon:1752024kB active_file:0kB inactive_file:280kB unevictable:0kB isolated(anon):1280kB isolated(file):0kB present:30670848kB managed:30095696kB mlocked:0kB dirty:0kB writeback:0kB mapped:23928kB shmem:8kB slab_reclaimable:26164kB slab_unreclaimable:44744kB kernel_stack:3392kB pagetables:34552kB unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:45224968 all_unreclaimable? yes
[  774.404463] lowmem_reserve[]: 0 0 0 0
[  774.404465] DMA: 0*4kB 1*8kB (U) 1*16kB (U) 1*32kB (U) 3*64kB (U) 2*128kB (U) 0*256kB 0*512kB 1*1024kB (U) 1*2048kB (U) 3*4096kB (MR) = 15864kB
[  774.404473] DMA32: 270*4kB (UEM) 180*8kB (UEM) 215*16kB (UEM) 333*32kB (UEM) 221*64kB (UEM) 145*128kB (UEM) 95*256kB (UEM) 42*512kB (UEM) 33*1024kB (UEM) 0*2048kB 0*4096kB = 128936kB
[  774.404481] Normal: 509*4kB (UEM) 389*8kB (UEM) 224*16kB (UEM) 242*32kB (UEM) 129*64kB (UEM) 103*128kB (UEM) 109*256kB (UEM) 73*512kB (UEM) 20*1024kB (UEM) 0*2048kB 0*4096kB = 123676kB
[  774.404488] 575775 total pagecache pages
[  774.404490] 575675 pages in swap cache
[  774.404491] Swap cache stats: add 1679802, delete 1104127, find 30940/40375
[  774.404492] Free swap  = 30715568kB
[  774.404493] Total swap = 33554428kB
[  774.404493] 8372849 pages RAM
[  774.404494] 0 pages HighMem/MovableOnly
[  774.404495] 143788 pages reserved
[  774.404496] [ pid ]   uid  tgid total_vm      rss nr_ptes swapents oom_score_adj name
[  774.404499] [  467]     0   467     7610       15      21      298         -1000 udevd
[  774.404501] [  710]     0   710     1617        0       8       44             0 syslogd
[  774.404502] [  714]     0   714     1090        0       8       23             0 klogd
[  774.404504] [  923]     0   923     2184        0      10       67             0 dhcpcd
[  774.404505] [  993]     1   993     1630        0       8       29             0 rpc.portmap
[  774.404507] [  997]    32   997     3206        1      12       41             0 rpc.statd
[  774.404508] [ 1015]     0  1015     1612        0       9       26             0 inetd
[  774.404509] [ 1018]     0  1018     6994        0      17      108         -1000 sshd
[  774.404511] [ 1029]     0  1029    94565        0      27      105             0 automount
[  774.404512] [ 1043]     0  1043     1093        0       8       42             0 acpid
[  774.404514] [ 1056]    81  1056     5096      111      13      130             0 dbus-daemon
[  774.404515] [ 1059]     0  1059  1063594        0      91      391             0 console-kit-dae
[  774.404517] [ 1128]     0  1128    53403        0      43     1121             0 polkitd
[  774.404518] [ 1157]     0  1157    13311       32      31      186             0 cupsd
[  774.404519] [ 1159]     0  1159     3198        0      12       37             0 crond
[  774.404521] [ 1161]     0  1161     3196        0      11       35             0 atd
[  774.404522] [ 1165]     0  1165    68585       13     131      409             0 smbd
[  774.404523] [ 1167]     0  1167    49046       39      92      272             0 nmbd
[  774.404524] [ 1169]     0  1169     1878        4       9       23             0 gpm
[  774.404526] [ 1179]     0  1179    68585       27     130      395             0 smbd
[  774.404527] [ 1193]     0  1193    75778       39      86      779             0 libvirtd
[  774.404528] [ 1212]     0  1212     1657       14       8       19             0 vde_switch
[  774.404530] [ 1306]  1000  1306     5143        1      16      178             0 bash
[  774.404531] [ 1307]     0  1307     3199        1      12       35             0 agetty
[  774.404533] [ 1308]     0  1308     3199        1      12       36             0 agetty
[  774.404534] [ 1309]     0  1309     3199        1      12       38             0 agetty
[  774.404536] [ 1310]     0  1310     3199        1      12       39             0 agetty
[  774.404537] [ 1311]     0  1311     3199        1      12       39             0 agetty
[  774.404538] [ 1365]  1000  1365     5153        1      16      188             0 bash
[  774.404540] [ 1381]  1000  1381     4002        0      13       50             0 xinit
[  774.404541] [ 1382]  1000  1382    44593     1696      89     9035             0 X
[  774.404543] [ 1385]     0  1385     7609       61      21      256         -1000 udevd
[  774.404544] [ 1388]     0  1388     7609        0      21      315         -1000 udevd
[  774.404545] [ 1399]  1000  1399     2971        0      11      102             0 sh
[  774.404547] [ 1400]  1000  1400     9189        0      24       70             0 ck-launch-sessi
[  774.404548] [ 1409]  1000  1409     2978        1      11      108             0 startkde
[  774.404550] [ 1443]  1000  1443     7720        0      20       82             0 dbus-launch
[  774.404551] [ 1444]  1000  1444     5268       35      14      252             0 dbus-daemon
[  774.404552] [ 1451]  1000  1451     1050        0       6       21          -300 start_kdeinit
[  774.404554] [ 1452]  1000  1452    87774        4     151     3670          -300 kdeinit4
[  774.404555] [ 1453]  1000  1453    90179        2     132     4257             0 klauncher
[  774.404557] [ 1455]  1000  1455   431428      378     279     5744             0 kded4
[  774.404558] [ 1457]  1000  1457     5117        1      17      103             0 gam_server
[  774.404560] [ 1461]  1000  1461   111316       13     171     4401             0 kglobalaccel
[  774.404561] [ 1475]  1000  1475   114168        4     175     4488             0 kwalletd
[  774.404562] [ 1480]     0  1480    49158        0      31      765             0 upowerd
[  774.404564] [ 1508]     0  1508    47874      120      31      655             0 udisks-daemon
[  774.404565] [ 1512]     0  1512    10848        0      25       74             0 udisks-daemon
[  774.404567] [ 1533]  1000  1533     1084        0       8       22             0 kwrapper4
[  774.404568] [ 1534]  1000  1534   133118        0     180     4075             0 ksmserver
[  774.404569] [ 1536]  1000  1536   177296        0     156     1912             0 kactivitymanage
[  774.404571] [ 1539]  1000  1539   728405     7237     273     9453             0 kwin
[  774.404572] [ 1553]  1000  1553   249972       92     213     4052             0 knotify4
[  774.404573] [ 1555]  1000  1555   846145     1662     334    23202             0 plasma-desktop
[  774.404575] [ 1561]  1000  1561     3036        0      11      151             0 ksysguardd
[  774.404576] [ 1564]  1000  1564    73196        0     129     1048             0 kuiserver
[  774.404578] [ 1567]  1000  1567    38367       38      38      163             0 akonadi_control
[  774.404579] [ 1586]  1000  1586   109022        0     131     4044             0 nepomukserver
[  774.404581] [ 1590]  1000  1590   203101        8     185     2631             0 nepomukservices
[  774.404582] [ 1591]  1000  1591   212223      167     261     5854             0 krunner
[  774.404584] [ 1593]  1000  1593    76019       13     130      988             0 nepomukcontroll
[  774.404585] [ 1595]  1000  1595   158076      113     191     5208             0 kmix
[  774.404587] [ 1600]  1000  1600   103051     3267      55     8842             0 virtuoso-t
[  774.404588] [ 1601]  1000  1601   670612       14     202     8803             0 kwrite
[  774.404589] [ 1604]  1000  1604   155527        0     160     1830             0 synergy
[  774.404591] [ 1606]  1000  1606   138834      608     190     4618             0 konsole
[  774.404592] [ 1612]  1000  1612   138974      350     191     4823             0 konsole
[  774.404593] [ 1616]  1000  1616     5134        0      16      169             0 bash
[  774.404595] [ 1619]  1000  1619     5134        0      15      169             0 bash
[  774.404596] [ 1625]  1000  1625    91018        0     136     4467             0 blueman-applet
[  774.404598] [ 1626]  1000  1626   123644        4     224     5510             0 python
[  774.404599] [ 1629]  1000  1629    97769        3     147     1089             0 polkit-kde-auth
[  774.404600] [ 1632]  1000  1632    39380      144      78     3083             0 python
[  774.404602] [ 1633]  1000  1633    32376       40      64     3163             0 python
[  774.404603] [ 1634]  1000  1634    98085      112     178     1200             0 korgac
[  774.404604] [ 1638]  1000  1638   183525    10028     240    14937             0 firefox
[  774.404606] [ 1641]  1000  1641   113540      122     176     4395             0 klipper
[  774.404607] [ 1649]  1000  1649    46340        0      27      131             0 gvfsd
[  774.404609] [ 1653]  1000  1653    70911        0      40      777             0 gvfsd-fuse
[  774.404610] [ 1661]  1000  1661    12515        5      30      136             0 gconfd-2
[  774.404611] [ 1679]  1000  1679    47423        0      86      390             0 obex-data-serve
[  774.404613] [ 1720]  1000  1720   115420      151     148     1604             0 nepomukservices
[  774.404614] [ 1721]  1000  1721    97736      144     150     1348             0 nepomukservices
[  774.404615] [ 1729]     0  1729     5137       35      16      139             0 bash
[  774.404617] [ 1845]     0  1845     5137       44      15      131             0 bash
[  774.404618] [ 1851]  1000  1851    33696       56      34      166             0 synergys
[  774.404620] [ 2132]     0  2132     2976        7      11       93             0 sh
[  774.404622] [ 2133]     0  2133  2290625   488926    2188   567641             0 qemu-system-x86
[  774.404623] [ 2146]     0  2146    49046       49      91      262             0 nmbd

My current qemu line is

QEMU_SDL_SAMPLES=2048 \
qemu-system-x86_64 -enable-kvm -M q35 -m 8192 -cpu host -smp 6,sockets=1,cores=6,threads=1 \
-bios /usr/share/qemu/bios.bin -vga none\
-device ioh3420,bus=pcie.0,addr=1c.0,multifunction=on,port=1,chassis=1,id=root.1 \
-device vfio-pci,host=03:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on \
-drive file=/dev/mapper/cryptvg2-win7,id=disk,if=virtio \
-device piix4-ide,bus=pcie.0 \
-drive file=/dev/sr0,id=cddisk -device ide-cd,bus=ide.1,drive=cddisk \
-net vde,sock=/var/run/kvm0.ctl,vlan=0 -net nic,vlan=0,model=virtio \
-device vfio-pci,host=00:1d.0,bus=pcie.0 \
-device ich9-intel-hda,bus=pcie.0,addr=1b.0,id=sound0 \
-device hda-duplex,id=sound0-codec0,bus=sound0.0,cad=0 \
-drive file=virtio-win-0.1-74.iso,id=isocd -device ide-cd,bus=ide.2,drive=isocd \
-rtc base=localtime

The DMAR error regarding device [00:1d.0] has always been there (prior to the VGA swap) and refers to a passed through USB controller for a mouse for the guest, so I doubt it is relevent.

[EDIT] Hmm, one more reboot out of many seems to have made the problem go away. Still, if anyone knows what the crash means, let me know

Last edited by mostlyharmless (2014-02-22 17:27:54)

andy123

Member

Registered: 2011-11-04

Posts: 169

Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

The nvidia guys seem to have a problem with their setup: https://devtalk.nvidia.com/default/topi … 4/#4131164

We have a number on their internal bugtracker, so maybe we should try helping them with their issues, so they can reproduce it and fix it at some point.

---

i'm sorry for my poor english wirting skills…

zzz3000

Member

Registered: 2014-02-16

Posts: 12

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

After I patched  fedora kernel-3.13 with  patches from main page, all  problems have disappeared except one.
I get BSOD under Windows7 guest when I am installing  radeon driver for 7750...

log:

Feb 23 17:03:42 DeltaS kernel: [  703.287069] kvm: zapping shadow pages for mmio generation wraparound

Last edited by zzz3000 (2014-02-23 13:06:17)

DanaGoyette

Member

Registered: 2014-01-03

Posts: 46

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

I've done a good deal more experimenting with my setup (back to one GPU at the moment).  Only one variant seems at all stable.

q35, EFI: Not an option -- OVMF is hardcoded for i440fx.
q35, radeon as primary, x-vga=on, no VGA Arbiter patch: hang (and garbled host graphics) upon trying to run VGA option rom.
q35, radeon as primary, x-vga=on, i915 arbiter patches:  Bus IRQ count assertion failure every single time I try to boot the guest (except in Safe Mode).  Also, breaks host OpenGL, which is a showstopper for me.
q35, radeon as primary, no x-vga: hang upon trying to run VGA option ROM.
q35, radeon as primary, rombar=0: guest boots and can be connected to via VNC... but AMD driver doesn't load.
q35, radeon as secondary: guest boots.  AMD driver won't load, or loads but doesn't detect displays or appear as usable in CCC.

i440fx, EFI guest: Boots reliably and runs reliably... but the bus reset doesn't work!  Any guest reboot or crash requires me to suspend and resume the host!

Last edited by DanaGoyette (2014-02-24 06:42:30)

aw

Member

Registered: 2013-10-04

Posts: 921

Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

q35, radeon as primary, x-vga=on, i915 arbiter patches:  Bus IRQ count assertion failure every single time I try to boot the guest (except in Safe Mode).

A number of other people seem to have this one working and it's certainly what I would advise for most people.

---

http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

C0NPAQ

Member

Registered: 2014-02-24

Posts: 2

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Suggesting that OP puts 1-2 sentences in his post on what passthrough really does.

I had to read the whole post to count 1:1 together. I expected some functionality which would let you forward the output of graphics card A to display on graphics card B within spice or something. But all you essentially get is one graphics card plainly assigned to the VM and that is that. Meaning you need another monitor (or one with two inputs one unused), another mouse and keyboard if you don't do weird tricks and don't want to steal those from the host OS and switch back and forth. Slightly better than rebooting ... if you happen to have a perfectly redundant graphics card ... but seriously: it sounded way better when you had not thought it through.

Don't know if there are any cards that eat DVI and pipe it to an application. Like TV cards... VNC is too slow for games but the same functionality for keyboard/mouse as a VM driver would work out. Then the problem would be solved.

EDIT: Arrived in the 21th century, there are cards for 100-200 USD, 200 USD with certain linux support. Seem to be made for video recording though, cheap ones even USB, probably latency issues.

Last edited by C0NPAQ (2014-02-24 17:14:22)

SpacePirate

Member

Registered: 2013-09-16

Posts: 55

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Hi,

I`ve been using the Passthrough setup for half a year now and it works for me quite well.
The only problem I have is that there are some performance issues. Though games run nicely on a HD 7870 + 6x4GHz Cores (AMD) the loading times are extremely high.
I also experience some stuttering when a game starts, I think cause some resources are loaded then.
I have enabled bigtables, cgroups (I use the script OP posted a while ago) and  disabled nested pagetables (kvm-amd npt=0). Do you know what I could do to verify that allocation is the problem or how to improve the loading times?

aw

Member

Registered: 2013-10-04

Posts: 921

Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Suggesting that OP puts 1-2 sentences in his post on what passthrough really does.

I had to read the whole post to count 1:1 together. I expected some functionality which would let you forward the output of graphics card A to display on graphics card B within spice or something. But all you essentially get is one graphics card plainly assigned to the VM and that is that. Meaning you need another monitor (or one with two inputs one unused), another mouse and keyboard if you don't do weird tricks and don't want to steal those from the host OS and switch back and forth. Slightly better than rebooting ... if you happen to have a perfectly redundant graphics card ... but seriously: it sounded way better when you had not thought it through.

Don't know if there are any cards that eat DVI and pipe it to an application. Like TV cards... VNC is too slow for games but the same functionality for keyboard/mouse as a VM driver would work out. Then the problem would be solved.

EDIT: Arrived in the 21th century, there are cards for 100-200 USD, 200 USD with certain linux support. Seem to be made for video recording though, cheap ones even USB, probably latency issues.

It would certainly be nice if there was some way to extract the framebuffer of the card and expose it on the host, and it may be possible, but typically that's not how PCI device assignment works.  If you assign a NIC, you don't somehow expect that it obeys the routing of the host, the NIC is an isolated device with it's own data path.  That said, I can mention that the guest is just a process to the host and that framebuffer that goes out to the screen likely lives somewhere in the physical memory of the system.  If you can find it and not interfere with the guest, you could theoretically render it into a window on a host display.  I have no tips for how to do that, it's likely proprietary and requires in-depth knowledge of how the card works, but it may be possible.  Another option is to use a PCoIP card co-assigned to the guest.  These are intended to be used for low-latency, high performance remote displays.  If they have a software client, it could be run on the host.  Another project that may be interesting is the Virgil 3D GPU project.  This intends to effectively share the host GPU with a guest and allow 3D rendering onto the host display.  If non-Linux gaming is your target, drivers are likely to be an issue with that approach for some time.

---

http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

aw

Member

Registered: 2013-10-04

Posts: 921

Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Hi,

I`ve been using the Passthrough setup for half a year now and it works for me quite well.
The only problem I have is that there are some performance issues. Though games run nicely on a HD 7870 + 6x4GHz Cores (AMD) the loading times are extremely high.
I also experience some stuttering when a game starts, I think cause some resources are loaded then.
I have enabled bigtables, cgroups (I use the script OP posted a while ago) and  disabled nested pagetables (kvm-amd npt=0). Do you know what I could do to verify that allocation is the problem or how to improve the loading times?

What do you use as a backing store for you OS image?  If you're not using a raw disk/partition/logical volume with virtio drivers, that's an obvious place to start.  Beyond that, there are tools like kvm_stat, perf, and trace-cmd that can show where you're spending time, this is how we were able to determine debug register access is a significant performance bottleneck for games like Borderlands2 (fix coming soon - at least on Intel, no data for AMD yet).

---

http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

doubledr

Member

Registered: 2013-12-26

Posts: 13

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Hi,

I`ve been using the Passthrough setup for half a year now and it works for me quite well.
The only problem I have is that there are some performance issues. Though games run nicely on a HD 7870 + 6x4GHz Cores (AMD) the loading times are extremely high.
I also experience some stuttering when a game starts, I think cause some resources are loaded then.
I have enabled bigtables, cgroups (I use the script OP posted a while ago) and  disabled nested pagetables (kvm-amd npt=0). Do you know what I could do to verify that allocation is the problem or how to improve the loading times?

What do you use as a backing store for you OS image?  If you're not using a raw disk/partition/logical volume with virtio drivers, that's an obvious place to start.  Beyond that, there are tools like kvm_stat, perf, and trace-cmd that can show where you're spending time, this is how we were able to determine debug register access is a significant performance bottleneck for games like Borderlands2 (fix coming soon - at least on Intel, no data for AMD yet).

Hi, aw

Is there any tutorial about how to use these tools? I want to help this topic too.

aw

Member

Registered: 2013-10-04

Posts: 921

Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Hi, aw

Is there any tutorial about how to use these tools? I want to help this topic too.

You'll find kvm_stat in the qemu source tree, here are some instructions:

https://docs.fedoraproject.org/en-US/Fe … 20s02.html

Tracing is similarly easy, but the report generated is massive and takes some skill to analyze:

http://www.linux-kvm.org/page/Tracing

Generally you'll want to run kvm_stat first to see what the heavy hitters are, then look in the trace output to try to figure out why they're being called.

There's a bit on perf here too:

http://www.linux-kvm.org/page/Perf_events

A simple place to start, especially if you're compiling your own qemu is to uncomment #define DEBUG_VFIO from hw/misc/vfio.c, recompile and test with that binary.  You will get a ton of initial output but once the OS drivers are running it should slow down considerably.  The output will never cease entirely though, all config space access will go through qemu and any of the quirks.  So long as you don't see more than around a dozen lines per second, vfio is probably not the bottleneck.

---

http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

SpacePirate

Member

Registered: 2013-09-16

Posts: 55

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Hi,

I`ve been using the Passthrough setup for half a year now and it works for me quite well.
The only problem I have is that there are some performance issues. Though games run nicely on a HD 7870 + 6x4GHz Cores (AMD) the loading times are extremely high.
I also experience some stuttering when a game starts, I think cause some resources are loaded then.
I have enabled bigtables, cgroups (I use the script OP posted a while ago) and  disabled nested pagetables (kvm-amd npt=0). Do you know what I could do to verify that allocation is the problem or how to improve the loading times?

What do you use as a backing store for you OS image?  If you're not using a raw disk/partition/logical volume with virtio drivers, that's an obvious place to start.  Beyond that, there are tools like kvm_stat, perf, and trace-cmd that can show where you're spending time, this is how we were able to determine debug register access is a significant performance bottleneck for games like Borderlands2 (fix coming soon - at least on Intel, no data for AMD yet).

Thanks Ill try those.
I use virtio drivers on a raw lvm partition.

kaeptnb

Member

Registered: 2013-09-04

Posts: 30

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

EDIT2: Further analysis (special thanks to Paolo Bonzini) shows two issues that are hurting performance here.  First is a known issue with Windows that it pounds on the time source, whether it's ACPI PM timer or HPET.  Code just recently went upstream (kernel and qemu) that enables a Hyper-V enlightened timer that can help to reduce this.  To enable it add "hv-time" to your -cpu option, ex. -cpu Haswell,hv-time  This gives me about 20% more FPS and is likely to help performance of any game running in a Windows guest.  The second problem is that we see extensive use of the processor debug registers.  Each move to or from these registers traps into the hypvervisor and they appear to be batched into a context switch function, so we have a burst of them all at once.  I've submitted a support ticket to gearbox software to see if they have any suggestions to avoid these registers.

Hello Andrew,

thanks for this hint. I tried "-m cpu host,hv-time" - but as soon as "hv-time" is added, the Guest (Win7 64bit) behaves aweful: IF the boot succeeds, half of the Services can't get startet, but mostly i get a BSOD with STOP 7e.
It may be this is because of the current GIT-Version of qemu - not because of "hv-time" - i cant tell exactly because i can test hv-time only with current git of qemu;
Guest behaves normal with qemu 1.7.0.
Kernel is 3.14-rc4 in both cases.

Maybe "hv-time" is only possible with a fixed cpu type, not "host" ?

Greetings!

aw

Member

Registered: 2013-10-04

Posts: 921

Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

EDIT2: Further analysis (special thanks to Paolo Bonzini) shows two issues that are hurting performance here.  First is a known issue with Windows that it pounds on the time source, whether it's ACPI PM timer or HPET.  Code just recently went upstream (kernel and qemu) that enables a Hyper-V enlightened timer that can help to reduce this.  To enable it add "hv-time" to your -cpu option, ex. -cpu Haswell,hv-time  This gives me about 20% more FPS and is likely to help performance of any game running in a Windows guest.  The second problem is that we see extensive use of the processor debug registers.  Each move to or from these registers traps into the hypvervisor and they appear to be batched into a context switch function, so we have a burst of them all at once.  I've submitted a support ticket to gearbox software to see if they have any suggestions to avoid these registers.

Hello Andrew,

thanks for this hint. I tried "-m cpu host,hv-time" - but as soon as "hv-time" is added, the Guest (Win7 64bit) behaves aweful: IF the boot succeeds, half of the Services can't get startet, but mostly i get a BSOD with STOP 7e.
It may be this is because of the current GIT-Version of qemu - not because of "hv-time" - i cant tell exactly because i can test hv-time only with current git of qemu;
Guest behaves normal with qemu 1.7.0.
Kernel is 3.14-rc4 in both cases.

Maybe "hv-time" is only possible with a fixed cpu type, not "host" ?

Greetings!

I used it with -cpu Haswell,hv-time. Use -cpu ? to see available predefined models.

---

http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

kaeptnb

Member

Registered: 2013-09-04

Posts: 30

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

I used it with -cpu Haswell,hv-time. Use -cpu ? to see available predefined models.

i tried it with "Nehalem" (only have a i7 920) and the new git-qemu - but no luck. it seems the current GIT snapshot of qemu messes up the whole virtual machine here: passed through GPU is there, but the device manager says the driver cant be loaded

Running with qemu 1.7.0 works perfect here, even with -cpu Nehalem - so it must be the current state of qemu GIT.

aw

Member

Registered: 2013-10-04

Posts: 921

Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

I used it with -cpu Haswell,hv-time. Use -cpu ? to see available predefined models.

i tried it with "Nehalem" (only have a i7 920) and the new git-qemu - but no luck. it seems the current GIT snapshot of qemu messes up the whole virtual machine here: passed through GPU is there, but the device manager says the driver cant be loaded

Running with qemu 1.7.0 works perfect here, even with -cpu Nehalem - so it must be the current state of qemu GIT.

I just refreshed my git tree, head is aa0d1f448871314bfc535da97eb003fe7766d4c2, I add this this patch because otherwise multifunction devices are broken and it works fine.

---

http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

neona

Member

Registered: 2014-02-25

Posts: 6

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

I've got things mostly working (windows boots, can play games, performs decently, though I'll probably tweak it more), though I'm having a few issues.

First, my commandline and config:

qemu-system-x86_64 -enable-kvm -M q35 -m 8192 -cpu host -vga none \
-smp 6,sockets=1,cores=6,threads=1 \
-bios /usr/share/qemu/bios.bin  \
-device ioh3420,bus=pcie.0,addr=1c.0,multifunction=on,port=1,chassis=1,id=root.1 \
-device vfio-pci,host=02:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on,romfile=Gigabyte.GTX550Ti.1024.110308.rom \
-device vfio-pci,host=02:00.1,bus=root.1,addr=00.1 \
-drive file=/dev/sda,if=virtio,format=raw \
-usb -device usb-host,hostbus=3,hostaddr=7 \
-device usb-host,hostbus=6,hostaddr=2 \
-device usb-host,hostbus=6,hostaddr=3 \
-device ich9-intel-hda,bus=pcie.0,addr=1b.0,id=sound0 \
-device hda-duplex,id=sound0-codec0,bus=sound0.0,cad=0 \
-device virtio-balloon  \
-net nic,model=virtio -net user

using the kernel, qemu, and seabios from OP, along with patched nvidia driver on host. Have all the relevant virtio drivers installed and working on guest, though it seems that the host kernel may need to be rebuilt again for me to actually enable balloon.

System specs:

FX-8350
GeForce 9500 GT (host)
GTX 550 Ti (guest)
Windows 8.1 - guest os
16gb of memory, currently allocating 8gb to guest
ASRock 970 EXTREME3

Anyway, onward to problems:

For some reason, qemu only claims to support oss, wav, and spice output, so sound won't work.  I double checked the PKGBUILD for qemu, and it's clearly being compiled with pulse and alsa support (or should be, anyway), so I have no clue what's causing this, and some googling around hasn't found anything.  I've tried various ways of wrapping oss to pulse/alsa, and none of that has worked either.  This is rather inconvienient, but at least if I can get networking to behave then I think it's possible to have windows output to pulse if nothing else works.  I might also get a usb soundcard, but I'd rather avoid spending money if I can.

The other major issue i'm having is trouble getting networking working in any configuration other than the default user mode (as it is in my commandline).  All attempts to use tap/tun result in various "device busy" or "device can't be accessed" messages.  This wouldn't be a big deal, but it's clear that tap/tun should provide better performance, and also I need the guest to be able to talk to the host so I can use synergy, so it's a bit of an issue.  Right now I'm just passing the host input devices to the guest, so I lose control of the host when I have the vm running (with ssh just in case, of course).  It'd also be nice for ICMP to work. Does anyone have experience with tap/tun?  I haven't been able to find any particularly clear guides, though I've tried a variety of things.

Oh, and does anyone know if use of hugetlbfs is mutually exclusive with proper use of balloon memory? It seems likely.  Also, how much of/what sort of an improvement does hugetlbfs provide, anyway?  Does it just reduce memory/tlb thrashing - can this reduce stutter in games?

Past that, everything is working great though!  Bit of stuttering in some games, but could be way worse.

I appreciate all the research you guys have put into getting this working.

Flyser

Member

Registered: 2013-12-19

Posts: 29

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Is there any quirk list fpr vfio-vga devices in the the kernel or qemu? I ask because a friend of mine has problems getting his Geforce 750 Ti to work with vga-passthrough, while his old G210 (or 8400GS, don't remember) worked.
The relevant dmesg errors seem to be:

[  142.835457] vfio_ecap_init: 0000:0a:00.0 hiding ecap 0x1e@0x258
[  142.835472] vfio_ecap_init: 0000:0a:00.0 hiding ecap 0x19@0x900

Best regards,
Flyser

aw

Member

Registered: 2013-10-04

Posts: 921

Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Is there any quirk list fpr vfio-vga devices in the the kernel or qemu? I ask because a friend of mine has problems getting his Geforce 750 Ti to work with vga-passthrough, while his old G210 (or 8400GS, don't remember) worked.
The relevant dmesg errors seem to be:

[  142.835457] vfio_ecap_init: 0000:0a:00.0 hiding ecap 0x1e@0x258
[  142.835472] vfio_ecap_init: 0000:0a:00.0 hiding ecap 0x19@0x900

These probably aren't relevant.  Extended capability 0x19 is a secondary PCIe capability, 0x1e is L1 PM substates (ie. power management states).  These messages simply mean that we're hiding the capability from the guest, either because it's not necessary, it's inappropriate for a guest, or nobody has added it yet.  It's very unlikely that these are blocking anything.

---

http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

deniv

Member

Registered: 2013-10-16

Posts: 27

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

[I used it with -cpu Haswell,hv-time. Use -cpu ? to see available predefined models.

Which is the correct one: hv-time or hv_time? libvirt uses hv_time [1]. The long supported hv_relaxed, hv_vapic, hv_spinlocks all use undeline characters [2].

1. http://libvirt.org/git/?p=libvirt.git;a … d0;hb=HEAD
2. http://qemu-project.org/ChangeLog/1.1#x86

deniv

Member

Registered: 2013-10-16

Posts: 27

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

The other major issue i'm having is trouble getting networking working in any configuration other than the default user mode (as it is in my commandline).  All attempts to use tap/tun result in various "device busy" or "device can't be accessed" messages.  This wouldn't be a big deal, but it's clear that tap/tun should provide better performance, and also I need the guest to be able to talk to the host so I can use synergy, so it's a bit of an issue.  Right now I'm just passing the host input devices to the guest, so I lose control of the host when I have the vm running (with ssh just in case, of course).  It'd also be nice for ICMP to work. Does anyone have experience with tap/tun?  I haven't been able to find any particularly clear guides, though I've tried a variety of things.

Do you have tun/tap enabled in kernel? CONFIG_TUN that is. The module name "tun."
Command line should look like:

-device virtio-net,netdev=t10,mac=52:54:00:00:00:00 \
-netdev tap,id=t10,vhost=on,ifname=tap10,script=no,downscript=no

I have a tun setup. When I asked people on #kvm they said the fastest setup for inter-vm communication is tun + bridges (macvtap is a tad slower). Basicly it creates just another net interface (e.g. ifname=tap10) like ethernet or wireless. Once you get the new tap interface you have to configure it like any other net interface. Setup a bridge or NAT. For example, right after the qemu command line I have:

ifconfig tap10 up
brctl addif br10 tap10

That bridge connects the vm to dhcp/router.

Oh, and does anyone know if use of hugetlbfs is mutually exclusive with proper use of balloon memory? It seems likely.  Also, how much of/what sort of an improvement does hugetlbfs provide, anyway?  Does it just reduce memory/tlb thrashing - can this reduce stutter in games?

AFAIK, yes. You can't balloon hugetlbfs backed memory. As for improvement, for me hugetlbfs gives about 2 points in win7 benchmark. From 5.5 to about 7.3-7.4.

Bit of stuttering in some games, but could be way worse.

You can try adding hv_relaxed,hv_vapic,hv_spinlocks=0x1000 to -cpu options (there is also hv_time, try it if you're running kernel 3.14, qemu-git). Might give you a couple more percents. All these options enable kvm-supported Hyper-V enlightenments which make windows behave better in virtual environment.

aw

Member

Registered: 2013-10-04

Posts: 921

Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

[I used it with -cpu Haswell,hv-time. Use -cpu ? to see available predefined models.

Which is the correct one: hv-time or hv_time? libvirt uses hv_time [1]. The long supported hv_relaxed, hv_vapic, hv_spinlocks all use undeline characters [2].

1. http://libvirt.org/git/?p=libvirt.git;a … d0;hb=HEAD
2. http://qemu-project.org/ChangeLog/1.1#x86

http://git.qemu.org/?p=qemu.git;a=commi … 9fd1933fbc

hv-time, but the others are also '-', so it's likely that either is accepted.

---

http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

redger

Member

Registered: 2014-02-26

Posts: 15

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

hi All,
firstly thanks for all the good information in this guide

I've been working through the instructions (as a new Arch user ie. n00b) and have hit a snag. I have complied the new kernel, qmeu and seabios as supplied by NBHS (thanks and compiled  pci-stub in and set the VFIO flags.
When running the first test of qemu I see this message "qemu-system-x86_64: hw/pci/pcie.c:240: pcie_cap_slot_hotplug_common: Assertion `((pci_dev->devfn) & 0x07) == 0' failed."

After a bit of hunting around I think the code responsible can be found in this post http://lists.gnu.org/archive/html/qemu- … 01183.html ... which seems to continue some code going back at least to 2010 .... which prevents hotpluging of devices with a non-zero "function" ie. 01:00.0 is ok, but 01:00.1 fails
BUT
That seems unlikely since the example quoted  includes just such a device and apparently works ok.

Some other background -
Asrock z87 extreme6 motherboard with Intel 4670 CPU and 16GB of RAM
2 add-in cards (both PCIe) AMD HD6950 GPU  and Renesas USB3 card
Arch installed as of 25-Feb-14 ie. current with sofwtare downloaded from first page on the same day

Can anybody point me in the right direction ? I'd be grateful for any advice
Thanks

PS this motherboard works fine in Opensuse KVM using virt-manager (easy setup) with GPU card as secondary ... but latency is high and frame rate low, hence trying VFIO. Furthermore the Renesas USB card passes through nicely
The motherboard also works with XEN (with XM toolset using Powerhouse's instructions for Mint) but doesn't suit me because I can't get AMD's Catalyst Control Centre to install to support the triple screen (eyefinity) facility AND there's a problem with the Renesas USB card not resetting correctly which causes untold grief. The XL toolset allows the VM to start but there's a Windows device error on the GPU which prevents its use.

qemu command and error message

qemu-system-x86_64 -enable-kvm -M q35 -m 2048 -cpu Haswell,hv-time \
> -smp 2,sockets=1,cores=2,threads=1 \
> -bios /usr/share/qemu/bios.bin -vga none \
> -device ioh3420,bus=pcie.0,addr=1c.0,multifunction=on,port=1,chassis=1,id=root.1 \
> -device vfio-pci,host=01:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on \
> -device vfio-pci,host=01:00.1,bus=root.1,addr=00.1
qemu-system-x86_64: hw/pci/pcie.c:240: pcie_cap_slot_hotplug_common: Assertion `((pci_dev->devfn) & 0x07) == 0' failed.

lspci -nn

00:00.0 Host bridge [0600]: Intel Corporation 4th Gen Core Processor DRAM Controller [8086:0c00] (rev 06)
00:01.0 PCI bridge [0604]: Intel Corporation Xeon E3-1200 v3/4th Gen Core Processor PCI Express x16 Controller [8086:0c01] (rev 06)
00:01.2 PCI bridge [0604]: Intel Corporation Xeon E3-1200 v3/4th Gen Core Processor PCI Express x4 Controller [8086:0c09] (rev 06)
00:02.0 VGA compatible controller [0300]: Intel Corporation Xeon E3-1200 v3/4th Gen Core Processor Integrated Graphics Controller [8086:0412] (rev 06)
00:03.0 Audio device [0403]: Intel Corporation Xeon E3-1200 v3/4th Gen Core Processor HD Audio Controller [8086:0c0c] (rev 06)
00:14.0 USB controller [0c03]: Intel Corporation 8 Series/C220 Series Chipset Family USB xHCI [8086:8c31] (rev 05)
00:16.0 Communication controller [0780]: Intel Corporation 8 Series/C220 Series Chipset Family MEI Controller #1 [8086:8c3a] (rev 04)
00:19.0 Ethernet controller [0200]: Intel Corporation Ethernet Connection I217-V [8086:153b] (rev 05)
00:1a.0 USB controller [0c03]: Intel Corporation 8 Series/C220 Series Chipset Family USB EHCI #2 [8086:8c2d] (rev 05)
00:1b.0 Audio device [0403]: Intel Corporation 8 Series/C220 Series Chipset High Definition Audio Controller [8086:8c20] (rev 05)
00:1c.0 PCI bridge [0604]: Intel Corporation 8 Series/C220 Series Chipset Family PCI Express Root Port #1 [8086:8c10] (rev d5)
00:1c.2 PCI bridge [0604]: Intel Corporation 8 Series/C220 Series Chipset Family PCI Express Root Port #3 [8086:8c14] (rev d5)
00:1c.3 PCI bridge [0604]: Intel Corporation 8 Series/C220 Series Chipset Family PCI Express Root Port #4 [8086:8c16] (rev d5)
00:1c.4 PCI bridge [0604]: Intel Corporation 8 Series/C220 Series Chipset Family PCI Express Root Port #5 [8086:8c18] (rev d5)
00:1d.0 USB controller [0c03]: Intel Corporation 8 Series/C220 Series Chipset Family USB EHCI #1 [8086:8c26] (rev 05)
00:1f.0 ISA bridge [0601]: Intel Corporation Z87 Express LPC Controller [8086:8c44] (rev 05)
00:1f.2 SATA controller [0106]: Intel Corporation 8 Series/C220 Series Chipset Family 6-port SATA Controller 1 [AHCI mode] [8086:8c02] (rev 05)
00:1f.3 SMBus [0c05]: Intel Corporation 8 Series/C220 Series Chipset Family SMBus Controller [8086:8c22] (rev 05)
01:00.0 VGA compatible controller [0300]: Advanced Micro Devices, Inc. [AMD/ATI] Cayman PRO [Radeon HD 6950] [1002:6719]
01:00.1 Audio device [0403]: Advanced Micro Devices, Inc. [AMD/ATI] Cayman/Antilles HDMI Audio [Radeon HD 6900 Series] [1002:aa80]
02:00.0 USB controller [0c03]: Renesas Technology Corp. uPD720201 USB 3.0 Host Controller [1912:0014] (rev 03)
04:00.0 SATA controller [0106]: ASMedia Technology Inc. ASM1062 Serial ATA Controller [1b21:0612] (rev 01)
05:00.0 Ethernet controller [0200]: Intel Corporation I211 Gigabit Network Connection [8086:1539] (rev 03)
06:00.0 SATA controller [0106]: ASMedia Technology Inc. ASM1062 Serial ATA Controller [1b21:0612] (rev 01)

dmesg |grep pci-stub

    0.000000] Command line: BOOT_IMAGE=/vmlinuz-linux-mainline root=UUID=714ebc3d-5848-4d11-9329-39c5067a8917 rw intel_iommu=on pci-stub.ids=8086:0c01,8086:0c09,1002:6719,1002:aa80,1912:0014,8086:8c10,8086:8c14,8086:8c16,8086:8c18,1b21:0612,8086:1539 quiet
[    0.000000] Kernel command line: BOOT_IMAGE=/vmlinuz-linux-mainline root=UUID=714ebc3d-5848-4d11-9329-39c5067a8917 rw intel_iommu=on pci-stub.ids=8086:0c01,8086:0c09,1002:6719,1002:aa80,1912:0014,8086:8c10,8086:8c14,8086:8c16,8086:8c18,1b21:0612,8086:1539 quiet
[    0.332791] pci-stub: add 8086:0C01 sub=FFFFFFFF:FFFFFFFF cls=00000000/00000000
[    0.332794] pci-stub: add 8086:0C09 sub=FFFFFFFF:FFFFFFFF cls=00000000/00000000
[    0.332796] pci-stub: add 1002:6719 sub=FFFFFFFF:FFFFFFFF cls=00000000/00000000
[    0.332800] pci-stub 0000:01:00.0: claimed by stub
[    0.332801] pci-stub: add 1002:AA80 sub=FFFFFFFF:FFFFFFFF cls=00000000/00000000
[    0.332804] pci-stub 0000:01:00.1: claimed by stub
[    0.332806] pci-stub: add 1912:0014 sub=FFFFFFFF:FFFFFFFF cls=00000000/00000000
[    0.332809] pci-stub 0000:02:00.0: claimed by stub
[    0.332810] pci-stub: add 8086:8C10 sub=FFFFFFFF:FFFFFFFF cls=00000000/00000000
[    0.332813] pci-stub: add 8086:8C14 sub=FFFFFFFF:FFFFFFFF cls=00000000/00000000
[    0.332816] pci-stub: add 8086:8C16 sub=FFFFFFFF:FFFFFFFF cls=00000000/00000000
[    0.332819] pci-stub: add 8086:8C18 sub=FFFFFFFF:FFFFFFFF cls=00000000/00000000
[    0.332822] pci-stub: add 1B21:0612 sub=FFFFFFFF:FFFFFFFF cls=00000000/00000000
[    0.332826] pci-stub 0000:04:00.0: claimed by stub
[    0.332828] pci-stub 0000:06:00.0: claimed by stub
[    0.332829] pci-stub: add 8086:1539 sub=FFFFFFFF:FFFFFFFF cls=00000000/00000000
[    0.332833] pci-stub 0000:05:00.0: claimed by stub

dmesg |grep -e IOMMU -e DMAR

[    0.000000] ACPI: DMAR 00000000bc173bd8 0000B8 (v01 INTEL      HSW  00000001 INTL 00000001)
[    0.000000] Intel-IOMMU: enabled
[    0.022553] dmar: IOMMU 0: reg_base_addr fed90000 ver 1:0 cap c0000020660462 ecap f0101a
[    0.022557] dmar: IOMMU 1: reg_base_addr fed91000 ver 1:0 cap d2008020660462 ecap f010da                  
[    0.022623] IOAPIC id 2 under DRHD base  0xfed91000 IOMMU 1                                               
[    0.319935] DMAR: No ATSR found
[    0.319952] IOMMU 0 0xfed90000: using Queued invalidation
[    0.319952] IOMMU 1 0xfed91000: using Queued invalidation
[    0.319953] IOMMU: Setting RMRR:
[    0.319961] IOMMU: Setting identity map for device 0000:00:02.0 [0xbf800000 - 0xcf9fffff]
[    0.321223] IOMMU: Setting identity map for device 0000:00:1d.0 [0xbc027000 - 0xbc033fff]
[    0.321244] IOMMU: Setting identity map for device 0000:00:1a.0 [0xbc027000 - 0xbc033fff]
[    0.321261] IOMMU: Setting identity map for device 0000:00:14.0 [0xbc027000 - 0xbc033fff]
[    0.321273] IOMMU: Prepare 0-16MiB unity mapping for LPC
[    0.321279] IOMMU: Setting identity map for device 0000:00:1f.0 [0x0 - 0xffffff]

IOMMU groups

(for i in /sys/kernel/iommu_groups/*;do echo "IOMMU: $i";for dev in $i/devices/*;do lspci -s "${dev##*/}"; done; echo;done)

IOMMU: /sys/kernel/iommu_groups/0
00:00.0 Host bridge: Intel Corporation 4th Gen Core Processor DRAM Controller (rev 06)

IOMMU: /sys/kernel/iommu_groups/1
00:01.0 PCI bridge: Intel Corporation Xeon E3-1200 v3/4th Gen Core Processor PCI Express x16 Controller (rev 06)
00:01.2 PCI bridge: Intel Corporation Xeon E3-1200 v3/4th Gen Core Processor PCI Express x4 Controller (rev 06)
01:00.0 VGA compatible controller: Advanced Micro Devices, Inc. [AMD/ATI] Cayman PRO [Radeon HD 6950]
01:00.1 Audio device: Advanced Micro Devices, Inc. [AMD/ATI] Cayman/Antilles HDMI Audio [Radeon HD 6900 Series]
02:00.0 USB controller: Renesas Technology Corp. uPD720201 USB 3.0 Host Controller (rev 03)

IOMMU: /sys/kernel/iommu_groups/10
00:1d.0 USB controller: Intel Corporation 8 Series/C220 Series Chipset Family USB EHCI #1 (rev 05)

IOMMU: /sys/kernel/iommu_groups/11
00:1f.0 ISA bridge: Intel Corporation Z87 Express LPC Controller (rev 05)
00:1f.2 SATA controller: Intel Corporation 8 Series/C220 Series Chipset Family 6-port SATA Controller 1 [AHCI mode] (rev 05)
00:1f.3 SMBus: Intel Corporation 8 Series/C220 Series Chipset Family SMBus Controller (rev 05)

IOMMU: /sys/kernel/iommu_groups/2
00:02.0 VGA compatible controller: Intel Corporation Xeon E3-1200 v3/4th Gen Core Processor Integrated Graphics Controller (rev 06)

IOMMU: /sys/kernel/iommu_groups/3
00:03.0 Audio device: Intel Corporation Xeon E3-1200 v3/4th Gen Core Processor HD Audio Controller (rev 06)

IOMMU: /sys/kernel/iommu_groups/4
00:14.0 USB controller: Intel Corporation 8 Series/C220 Series Chipset Family USB xHCI (rev 05)

IOMMU: /sys/kernel/iommu_groups/5
00:16.0 Communication controller: Intel Corporation 8 Series/C220 Series Chipset Family MEI Controller #1 (rev 04)

IOMMU: /sys/kernel/iommu_groups/6
00:19.0 Ethernet controller: Intel Corporation Ethernet Connection I217-V (rev 05)

IOMMU: /sys/kernel/iommu_groups/7
00:1a.0 USB controller: Intel Corporation 8 Series/C220 Series Chipset Family USB EHCI #2 (rev 05)

IOMMU: /sys/kernel/iommu_groups/8
00:1b.0 Audio device: Intel Corporation 8 Series/C220 Series Chipset High Definition Audio Controller (rev 05)

IOMMU: /sys/kernel/iommu_groups/9
00:1c.0 PCI bridge: Intel Corporation 8 Series/C220 Series Chipset Family PCI Express Root Port #1 (rev d5)
00:1c.2 PCI bridge: Intel Corporation 8 Series/C220 Series Chipset Family PCI Express Root Port #3 (rev d5)
00:1c.3 PCI bridge: Intel Corporation 8 Series/C220 Series Chipset Family PCI Express Root Port #4 (rev d5)
00:1c.4 PCI bridge: Intel Corporation 8 Series/C220 Series Chipset Family PCI Express Root Port #5 (rev d5)
04:00.0 SATA controller: ASMedia Technology Inc. ASM1062 Serial ATA Controller (rev 01)
05:00.0 Ethernet controller: Intel Corporation I211 Gigabit Network Connection (rev 03)
06:00.0 SATA controller: ASMedia Technology Inc. ASM1062 Serial ATA Controller (rev 01)

contents of vfio-pci.cfg

DEVICES="0000:01:00.0 0000:01:00.1 0000:02:00.0 0000:04:00.0 0000:05:00.0 0000:06:00.0"

ie. the GPU, Renesas USB card, Asmedia Sata controllers and 2nd Intel NIC will be passed to the guest (this works nicely on OpenSuse via Virt-Manager ... hopefully even better with VFIO)

kaeptnb

Member

Registered: 2013-09-04

Posts: 30

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Can anybody point me in the right direction ? I'd be grateful for any advice
Thanks

Try it with qemu-1.7.0 stable. and then without "hv-time" because 1.7.0 doesn't know about hv-time yet.

aw posted a patch (Post #1217) which should eliminate your error - it did here; BUT my guest produces BSOD with current qemu git even with this patch - but works fine with qemu 1.7.0...

Last edited by kaeptnb (2014-02-26 15:26:56)