Arch Linux

Owen77

Member

Registered: 2013-12-12

Posts: 7

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Congratulation, it works.  When the Nvidia driver is used, the emulated VGA is only used for text mode and the GPU is used for graphical mode.  They're mutually exclusive, so it's expected that the emulated VGA "freezes" when X starts.  In your first case, apps can't find any display device, probably because the graphical desktop is sitting at a login prompt.  In the second case, you've started an X session, therefore there is a display to connect to.  This is the way that GRID cards work and Quadro as well if you ignore that it has a physical display connector.  You need to setup remote access to the desktop first.  You can run a vnc server in the guest or use some other remote'ing solution.  If you were expecting X on the emulated VGA with rendering in the GPU, that doesn't happen.

Great! Thanks, I don't know why I was expecting the qemu vnc server to switch over, but it makes sense that it's fixed to the emulated vga framebuffer. Looking into vnc installation in the guest now.

syllopsium

Member

Registered: 2013-12-18

Posts: 4

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

VGA passthrough is working well for me with a Radeon R9 270X (i.e. a rebranded 7870 GHz Edition), however after installing the latest Catalyst driver version 13.11 beta 9.5 on Windows 7 SP1 I am having trouble booting Windows.

I see 'Starting Windows...' with the windows splash screen animation but then the screen goes black and after a bit the monitor loses signal. In Safe mode, the event log shows error 0x00000116 - MSDN says this: "The VIDEO_TDR_ ERROR bug check has a value of 0x00000116. This indicates that an attempt to reset the display driver and recover from a timeout failed."

I have tried so moving the HDMI audio device (ID 0000:01:00.1) to pcie.0 instead of root.1 as well as eliminating passthrough of the HDMI audio device entirely, neither of which helped. I also tried manually specifying the ROM (rombar=0 romfile=...) from a ROM bin I dumped using GPU-Z. I dumped it from within the VM, which I read may not work but it didn't throw any errors and GPU-Z properly detected the card so I think it dumped the VBIOS correctly. Has anyone encountered something similar, or has any ideas on what I could do?
[..] snip
VFIO passthrough was normally prior to installing the Catalyst drivers, so my guess is that pass-through is working correctly and the bug is in the machine config or the Windows driver.

I have a very similar problem here, but suspect the issue is Windows and Qemu/KVM's virtualised hardware presented, not the passthrough capability.

Alex, is there any way to debug this, or are we at the mercy of AMD/Microsoft/firing up a Windows kernel debugger? I've fiddled around with this a lot and got nowhere. There's nothing in dmesg.

I'm able to successfully pass a 6950 through to a Ubuntu VM using both the radeon and fglrx drivers on an S3210SHLC board (basically a server version of the X38 chipset, using a Core2Quad). That's using the southbridge x4 PCIe 1.x slot (the only one that can run graphics at a sensible rate, due to the enforced chipset limitation of x1 for graphics cards in other slots).

For Windows it always provides a driver timeout under both Windows 7 and 8 x64, whether catalyst's control panel is installed or not (before Catalyst, it works fine unaccelerated). In some cases it seems enabling vmx in the virtual CPU causes problems, but this appears to be less of an issue of late. Using 3.12.0 rc4 and qemu 1.7.5.0

I do have to specify the GPU BIOS when booting the VM - it doesn't work otherwise.

I'm also much less importantly having issues with passing through the built in Matrox G200e on the board - that's a case where it works in Xen and VMWare, but not in KVM.

Startup, if you're bothered :

qemu-system-x86_64 -enable-kvm -M q35 -m 3184 -cpu qemu64,+ssse3,+cx16,+xtpr,+acpi,+ss,+ht,+tm,+pbe,+dtes64,+monitor,+ds_cpl,+est,+tm2,+pdcm,+lahf_lm, -smp 2,sockets=1,cores=2,threads=1 -bios /usr/share/qemu/bios.bin -vga none  -device ioh3420,bus=pcie.0,addr=1c.0,multifunction=on,port=1,chassis=1,id=root.1  -device vfio-pci,host=03:00.0,bus=root.1,addr=00.0,multifunction=on,rombar=0,romfile=/home/peter/6950bios.rom,x-vga=on -device vfio-pci,host=03:00.1,bus=root.1,addr=00.1   -device ahci,bus=pcie.0,id=ahci  -drive file=/home/peter/debian.img,if=virtio  -drive file=/home/peter/Downloads/ubuntu-13.04-desktop-amd64.iso,id=isocd -device ide-cd,bus=ahci.1,drive=isocd  -drive file=/home/peter/Downloads/virtio-win-0.1-65.iso,id=driviso -device ide-cd,bus=ahci.2,drive=driviso -usb   -boot order=cd -usb -usbdevice host:03f0:0024 -usbdevice host:046d:c051

dmesg | grep KVM

[18766.868390] kvm [3596]: vcpu0 disabled perfctr wrmsr: 0xc1 data 0xffff
[18767.310411] kvm: zapping shadow pages for mmio generation wraparound

dmesg | grep VFIO

[    0.000000] Kernel command line: BOOT_IMAGE=/vmlinuz-3.12.0-rc4 root=/dev/mapper/vmtest--vg-root ro quiet modprobe.blacklist=radeon pci-stub.ids=1002:6719,1002:aa80 intel_iommu=on kvm-intel.emulate_invalid_guest_state=1 vfio_iommu_type1.allow_unsafe_interrupts=1 pcie_acs_override=downstream console=tty0 console=ttyS0,115200n8
[    2.959621] VFIO - User Level meta-driver version: 0.3
[18723.504293] vfio-pci 0000:03:00.0: enabling device (0140 -> 0143)
[18771.524071] vfio-pci 0000:03:00.0: irq 48 for MSI/MSI-X
[18776.940059] vfio-pci 0000:03:00.1: irq 49 for MSI/MSI-X

dmesg | grep iommu

[    0.000000] Warning: PCIe ACS overrides enabled; This may allow non-IOMMU protected peer-to-peer DMA
[    0.000000] Kernel command line: BOOT_IMAGE=/vmlinuz-3.12.0-rc4 root=/dev/mapper/vmtest--vg-root ro quiet modprobe.blacklist=radeon pci-stub.ids=1002:6719,1002:aa80 intel_iommu=on kvm-intel.emulate_invalid_guest_state=1 vfio_iommu_type1.allow_unsafe_interrupts=1 pcie_acs_override=downstream console=tty0 console=ttyS0,115200n8
[    0.000000] Intel-IOMMU: enabled
[    0.020122] dmar: IOMMU 0: reg_base_addr feb03000 ver 1:0 cap c9008020230272 ecap 1000
[    2.886000] IOMMU 0 0xfeb03000: using Register based invalidation
[    2.886004] IOMMU: Setting RMRR:
[    2.886022] IOMMU: Setting identity map for device 0000:00:1a.7 [0x9fcfc000 - 0x9fcfefff]
[    2.886046] IOMMU: Setting identity map for device 0000:00:1d.7 [0x9fcff000 - 0x9fd01fff]
[    2.886066] IOMMU: Setting identity map for device 0000:00:1d.0 [0x9fd32000 - 0x9fd34fff]
[    2.886084] IOMMU: Setting identity map for device 0000:00:1d.1 [0x9fd32000 - 0x9fd34fff]
[    2.886103] IOMMU: Setting identity map for device 0000:00:1d.2 [0x9fd32000 - 0x9fd34fff]
[    2.886121] IOMMU: Setting identity map for device 0000:00:1a.0 [0x9fd32000 - 0x9fd34fff]
[    2.886140] IOMMU: Setting identity map for device 0000:00:1a.1 [0x9fd32000 - 0x9fd34fff]
[    2.886150] IOMMU: Setting identity map for device 0000:00:1d.7 [0x9fd32000 - 0x9fd34fff]
[    2.886153] IOMMU: Setting identity map for device 0000:00:1a.7 [0x9fd32000 - 0x9fd34fff]
[    2.886156] IOMMU: Setting identity map for device 0000:00:1a.0 [0x9fd02000 - 0x9fd07fff]
[    2.886159] IOMMU: Setting identity map for device 0000:00:1a.1 [0x9fd02000 - 0x9fd07fff]
[    2.886162] IOMMU: Setting identity map for device 0000:00:1d.0 [0x9fd08000 - 0x9fd10fff]
[    2.886165] IOMMU: Setting identity map for device 0000:00:1d.1 [0x9fd08000 - 0x9fd10fff]
[    2.886168] IOMMU: Setting identity map for device 0000:00:1d.2 [0x9fd08000 - 0x9fd10fff]
[    2.886172] IOMMU: Prepare 0-16MiB unity mapping for LPC
[    2.886182] IOMMU: Setting identity map for device 0000:00:1f.0 [0x0 - 0xffffff]

Yes, it is using Debian in this case. I started with Arch and it does exactly the same thing (Ubuntu ok, Windows not).

Any help appreciated!

tuoni

Member

Registered: 2013-12-19

Posts: 7

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Hi guys, I just registered to thank you all for your work - because of this thread, I have finally got KVM/QEMU VGA passthrough working on my GT660Ti.

My setup is i7-4770 / ASRock Z87 Extreme 4 / Nvidia GT660Ti, host OS is Debian x64, guest OS is Windows 7 x64, keyboard/mouse sharing with Synergy.

I had bought the processor/mobo combo specifically to try and do VGA passthrough and had been really struggling first with Xen and latterly with KVM until I found this thread - the information in the OP and the patches it links to have been invaluable.

Thanks once again, all.

Jesse2004

Member

Registered: 2012-06-25

Posts: 9

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Wow, really a GREAT thread!!!

It seems that the current generation (haswell) of Intel mobile processors supports vt-d in both cpu (i.e. i7-4700HQ) and chipset (i.e. HM87). Has anyone tried vga-passthrough on a laptop? Looking forward to seeing any successful story.

firewing1

Member

Registered: 2013-12-17

Posts: 9

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Regarding my initial troubles with Code 10 and BSOD 0x116: I posted here to Fedora's virt mailing list and Alex Williamson suggested I try with Kernel 3.13 release candidates as well as the Haswell i915 VGA arbritration patches. After compiling qemu.git and kernel 3.13 from Fedora rawhide with the VGA arbitration patches, I'm happy my setup is now working flawlessly and I don't get the BSOD anymore. I didn't even have to reinstall Windows!

Last edited by firewing1 (2013-12-21 08:52:50)

TripleSpeeder

Member

Registered: 2011-05-02

Posts: 47

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Regarding my initial troubles with Code 10 and BSOD 0x116: I posted here to Fedora's virt mailing list and Alex Williamson suggested I try with Kernel 3.13 release candidates as well as the Haswell i915 VGA arbritration patches. After compiling qemu.git and kernel 3.13 from Fedora rawhide with the VGA arbitration patches, I'm happy my setup is now working flawlessly and I don't get the BSOD anymore. I didn't even have to reinstall Windows!

This is good news, as I am seeing exactly the same problem on my system now. Could you provide sources to the patches you applied so I can also give it a try? Thanks!

Val532

Member

Registered: 2013-11-13

Posts: 35

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Regarding my initial troubles with Code 10 and BSOD 0x116: I posted here to Fedora's virt mailing list and Alex Williamson suggested I try with Kernel 3.13 release candidates as well as the Haswell i915 VGA arbritration patches. After compiling qemu.git and kernel 3.13 from Fedora rawhide with the VGA arbitration patches, I'm happy my setup is now working flawlessly and I don't get the BSOD anymore. I didn't even have to reinstall Windows!

Hello,

Can you provide your patch please.

Thanks.

revasm

Member

Registered: 2013-12-22

Posts: 2

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

i915_313rc4.patch

Applies cleanly, but 100% untested. YMMV.

The patch was generated by reverting to Linus's commit:
   cd4edf7 Merge branch 'drm-fixes' of git://people.freedesktop.org/~airlied/linux

Followed by undoing the below changes and merging onto rc4.
   e1264eb Revert "drm/i915: Delay disabling of VGA memory until vgacon->fbcon handoff is done"
   ebff5fa9 Revert "i915: Update VGA arbiter support for newer devices"

Alex wrote that these changes are necessary, but they seem intact.
   5c0f6ee vgaarb: Fix VGA decodes changes
   f22d776 vgaarb: Don't disable resources that are not owned

Further reading:

1. https://lkml.org/lkml/2013/8/15/458

2. https://lkml.org/lkml/2013/8/28/362

3. http://cgit.freedesktop.org/~danvet/drm … ba24389bee

firewing1

Member

Registered: 2013-12-17

Posts: 9

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

This is good news, as I am seeing exactly the same problem on my system now. Could you provide sources to the patches you applied so I can also give it a try? Thanks!

Can you provide your patch please.

The poster above covers it pretty well - apply the VGA arbitration patch to 3.13rc and you should be good.

A.W. recommended using kernel 3.13rc + qemu.git OR kernel 3.12.5 + qemu 1.7.0 with the following commits patched in: https://lists.fedoraproject.org/piperma … 03916.html

I have tried both 3.13rc4 and 3.12.5, and as A.W. said either work perfectly after applying the patches. I believe the key is the 915 VGA arbiter patch, and NoSnoop if your device is listing NoSnoop+ in "lspci -vv".

For those interested in replicating my configuration exactly, I also used this kernel patch to fix an IOMMU-related traceback present in dmesg on boot: Quirk non-compliant PCIe-to-PCI bridges - fixes backtrace in dmesg on certain systems with broken PCIe-to-PCI bridges - http://lists.linux-foundation.org/piper … 05668.html

Last edited by firewing1 (2013-12-23 05:46:08)

Val532

Member

Registered: 2013-11-13

Posts: 35

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Ok thanks.

But i have one problem, with kernel 3.13-rc4 (with i915-patch and acs patch) and qemu from git, my VM (Windows 7) refuse to boot ! But with qemu 1.7 it work.

And my VGA card, an 7770HD have NoSnoop+ so i patch qemu with NoSnoop patch but the error 0x00000116 is always here !

It's crazy, because in the past i managed to work, but i don't understand what is the difference with my setup now !

firewing1

Member

Registered: 2013-12-17

Posts: 9

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

And my VGA card, an 7770HD have NoSnoop+ so i patch qemu with NoSnoop patch but the error 0x00000116 is always here !

It's crazy, because in the past i managed to work, but i don't understand what is the difference with my setup now !

Are you passing an HDMI audio device along with your GPU? If so, try booting after changing "bus=root.1,addr=0x01" for "bus=pcie.0" on the HDMI PCI device or alternatively remove it altogether. In my case (as other Radeon users have suggested in this thread), I had to move my HDMI device to pcie.0 in order to overcome BSOD 0x116, even with all patches applied. Prior to the patches, I would get 0x116 regardless of the status of the HDMI PCI device.

aw

Member

Registered: 2013-10-04

Posts: 921

Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

A.W. recommended using kernel 3.13rc + qemu.git OR kernel 3.12.5 + qemu 1.7.0 with the following commits patched in: https://lists.fedoraproject.org/piperma … 03916.html

I have tried both 3.13rc4 and 3.12.5, and as A.W. said either work perfectly after applying the patches. I believe the key is the 915 VGA arbiter patch, and NoSnoop if your device is listing NoSnoop+ in "lspci -vv".

The commits in the above url supersede the NoSnoop patch.  The NoSnoop patch should no longer be necessary.

---

http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Val532

Member

Registered: 2013-11-13

Posts: 35

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Are you passing an HDMI audio device along with your GPU? If so, try booting after changing "bus=root.1,addr=0x01" for "bus=pcie.0" on the HDMI PCI device or alternatively remove it altogether. In my case (as other Radeon users have suggested in this thread), I had to move my HDMI device to pcie.0 in order to overcome BSOD 0x116, even with all patches applied. Prior to the patches, I would get 0x116 regardless of the status of the HDMI PCI device.

I tested all possibility and i do not understand ! Is it possible that an external library causes the problem ?

I will do more test during the next hours.

Thanks.

And

The commits in the above url supersede the NoSnoop patch.  The NoSnoop patch should no longer be necessary.

So you're saying with Kernel 3.13-rc4 and qemu git is no need to use NoSnoop patch, but acs patch and i915 patch is needed ?

Last edited by Val532 (2013-12-23 15:05:16)

aw

Member

Registered: 2013-10-04

Posts: 921

Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

The commits in the above url supersede the NoSnoop patch.  The NoSnoop patch should no longer be necessary.

So you say with Kernel 3.13-rc4 and qemu git is no need to use NoSnoop patch, but acs patch and i915 patch is needed ?

The ACS patch should be used with caution.  It's purpose is to allow the user to override and supplement the PCI ACS information provided by the hardware.  If used improperly this can indicate that devices are isolated from each other when they actually are not.  Lack of real isolation can mean DMA transactions can go to other devices rather than be translated through the IOMMU.  This can create very difficult to debug problems.  There's a reason why this patch was rejected upstream.

The i915 patches are necessary if you have Intel graphics and it will cause DRI to be disabled on the host X session using the Intel device.  AFAIK, these patches are correct other than inducing poor behavior in other layers.

---

http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

techdude300

Member

Registered: 2012-04-09

Posts: 4

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Thanks for this guide! I've been wanting to do something like this for a long time, was about to go the Xen route but this seemed a little easier for a noob like me. Mostly everything works, though I'm still having issues with getting it to play nice with pulseaudio (I may go get a cheap usb soundcard and just pass that through if all else fails).

One strange thing I've noticed is that when I start a VM and then shut it down, either using system_powerdown qemu command or shutting it down from within the Windows VM (Win 8) it shuts down cleanly, but doesn't free up the memory it allocated to the VM. This means that after running and stopping a VM, I have about 3GB of memory my system is telling me is still in use. I can't find any process that's using all this memory. There's no lingering qemu processes, and top/htop doesn't show anything using up all that memory (even run as root). There also aren't any huge files in /tmp that might cause this, but free -h and similar tools insist that 3GB is being used and it certainly acts like it, since if I try to run the command to start the VM again it just steals another 3GB, and trying it a 3rd time pushes my poor 8GB machine to start swapping like a maniac, usually so bad I just have to hard reset the host, which I hate to do. Is this normal for qemu, and I'm just not understanding something right? Even if I could allocate that 3GB on boot and have it be re-used by subsequent VM launches would be nice. The command I'm using (as root):

QEMU_PA_SAMPLES=128 QEMU_AUDIO_DRV=pa qemu-system-x86_64 -enable-kvm -M q35 -m 3072 \
-cpu host -smp 6,sockets=1,cores=6,threads=1 -bios /usr/share/qemu/bios.bin -vga none \
-device ioh3420,bus=pcie.0,addr=1c.0,multifunction=on,port=1,chassis=1,id=root.1 \
-device vfio-pci,host=04:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on \
-device vfio-pci,host=04:00.1,bus=root.1,addr=00.1 \
 -nographic \
-device ahci,bus=pcie.0,id=ahci \
-drive file=/home/william/VirtualBox\ VMs/Windows\ 8/Windows\ 8.vdi,id=disk,format=vdi,cache=none \
-device ide-hd,bus=ahci.0,drive=disk \
-device ich9-intel-hda,bus=pcie.0,addr=1b.0,id=sound0 \
-device hda-duplex,id=sound0-codec0,bus=sound0.0,cad=0 \
-monitor stdio \
-device vfio-pci,host=00:12.0,bus=pcie.0 \
-device vfio-pci,host=00:12.2,bus=pcie.0

Where 04:00.* is the video card (radeon 4850), and 00:12.* is a usb controller. Guest VM is Windows 8 x64, host is running kernel 3.13.0-rc3.

(EDIT: spaced things out better in the above command for readbility)

Last edited by techdude300 (2013-12-23 16:52:23)

firewing1

Member

Registered: 2013-12-17

Posts: 9

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

I tested all possibility and i do not understand ! Is it possible that an external library causes the problem ?

In my case it was definitely kernel + QEMU that was the cause. As soon as I upgraded to kernel-3.13rc and QEMU.git, the very same virtual machine image started working without a BSOD.

So you're saying with Kernel 3.13-rc4 and qemu git is no need to use NoSnoop patch, but acs patch and i915 patch is needed ?

The i915 patches are needed in either case, NoSnoop is only needed if your device is showing "NoSnoop+" when you check "lspci -vv". I didn't use the ACS patch, but if you need to refer to A.W.'s post above.

One strange thing I've noticed is that when I start a VM and then shut it down, either using system_powerdown qemu command or shutting it down from within the Windows VM (Win 8) it shuts down cleanly, but doesn't free up the memory it allocated to the VM

What's the output of "free -m" after closing the VM? My guess is that the memory is just being cached in case you re-start the VM again, and will be freed automatically as the system demands it. If you see a large amount of memory in the "cached" column from the command, then this is the case.

Last edited by firewing1 (2013-12-23 18:15:07)

techdude300

Member

Registered: 2012-04-09

Posts: 4

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

One strange thing I've noticed is that when I start a VM and then shut it down, either using system_powerdown qemu command or shutting it down from within the Windows VM (Win 8) it shuts down cleanly, but doesn't free up the memory it allocated to the VM

What's the output of "free -m" after closing the VM? My guess is that the memory is just being cached in case you re-start the VM again, and will be freed automatically as the system demands it. If you see a large amount of memory in the "cached" column from the command, then this is the case.

I don't think it's cached, because the system doesn't seem to be freeing it when it's needed, because when I stop and start the VM three times (3gb vm, 8gb mem on host) it starts swapping very heavily. Here's free -m before a run:

             total       used       free     shared    buffers     cached
Mem:          7953       1666       6287         32         56        564
-/+ buffers/cache:       1046       6907
Swap:         8187          0       8187

While qemu is running:

             total       used       free     shared    buffers     cached
Mem:          7953       4837       3116         33         56        577
-/+ buffers/cache:       4202       3751
Swap:         8187          0       8187

And after a clean shutdown (shutdown via guest), qemu not running:

             total       used       free     shared    buffers     cached
Mem:          7953       4787       3166         28         57        579
-/+ buffers/cache:       4150       3803
Swap:         8187          0       8187

I'm not very knowledgable on Linux memory management, but if the memory doesn't appear to be used by a process, or any tmpfs using RAM, could that mean that the kernel is allocating all that memory and not releasing it? Could this be a kernel bug? Some things I've noticed after further testing that the issue is not affected by choice of guest OS (tested Ubuntu and Windows 8), and memory is successfully returned if I don't pass any devices through (both Ubuntu and Windows). However, when I pass either the USB controller, VGA card, or both to the VM (both Ubuntu and Win 8) memory is not returned on qemu exiting. This makes me think it's some kind of bug in either the kernel, qemu, or my motherboard's IOMMU stuff. Motherboard is Gigabyte 531020 GA-990FXA-UD3, just bought it a few days ago. I've been having issues with USB3 while UEFI booting, and I remember there being something about vga passthrough on Xen not working right when the host was UEFI booted, so I wouldn't be surprised if that's messing something up. I'll set up grub and do a normal bios boot and see if it changes anything.

Val532

Member

Registered: 2013-11-13

Posts: 35

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

The ACS patch should be used with caution.  It's purpose is to allow the user to override and supplement the PCI ACS information provided by the hardware.  If used improperly this can indicate that devices are isolated from each other when they actually are not.  Lack of real isolation can mean DMA transactions can go to other devices rather than be translated through the IOMMU.  This can create very difficult to debug problems.  There's a reason why this patch was rejected upstream.

The i915 patches are necessary if you have Intel graphics and it will cause DRI to be disabled on the host X session using the Intel device.  AFAIK, these patches are correct other than inducing poor behavior in other layers.

Ok so with a 3.13-rc4 without patch (no i915 or acs) my VM do not work, but i do not see any BSOD just reboot so it's maybe an other problem in my machine. Also i see a new problem in dmesg durring the VM work  :

vfio_pin_pages: RLIMIT_MEMLOCK (65536) exceeded

and at the stop of the VM :

[  663.077661] WARNING: CPU: 1 PID: 532 at drivers/gpu/drm/i915/intel_display.c:6309 hsw_enable_pc8_work+0x6c9/0x6f0 [i915]()
[  663.077662] Power well on
[  663.077663] Modules linked in: vfio_iommu_type1(F) vfio_pci(F) vfio(F) ip6table_filter(F) ip6_tables(F) iptable_filter(F) ip_tables(F) ebtable_nat(F) ebtables(F) x_tables(F) bnep(F) bridge(F) rfcomm(F) stp(F) bluetooth(F) llc(F) parport_pc(F) ppdev(F) x86_pkg_temp_thermal(F) intel_powerclamp(F) coretemp(F) kvm_intel(F) kvm(F) crct10dif_pclmul(F) crc32_pclmul(F) ghash_clmulni_intel(F) aesni_intel(F) aes_x86_64(F) lrw(F) gf128mul(F) glue_helper(F) ablk_helper(F) cryptd(F) mxm_wmi(F) snd_hda_codec_hdmi(F) snd_hda_codec_realtek(F) snd_hda_intel(F) snd_hda_codec(F) snd_hwdep(F) snd_pcm(F) snd_page_alloc(F) i915(F) snd_seq_midi(F) snd_seq_midi_event(F) snd_rawmidi(F) snd_seq(F) snd_seq_device(F) snd_timer(F) psmouse(F) mei_me(F) drm_kms_helper(F) drm(F) mei(F) microcode(F) snd(F) soundcore(F) serio_raw(F) video(F) wmi(F) lpc_ich(F) mac_hid(F) lp(F) parport(F) hid_generic(F) usbhid(F) hid(F) igb(F) i2c_algo_bit(F) dca(F) ahci(F) libahci(F) e1000e(F) ptp(F) pps_core(F)
[  663.077693] CPU: 1 PID: 532 Comm: kworker/1:2 Tainted: GF       W    3.13.0-rc4-no-patch #1
[  663.077694] Hardware name: To Be Filled By O.E.M. To Be Filled By O.E.M./Z87 Extreme6, BIOS P2.10 07/03/2013
[  663.077702] Workqueue: events hsw_enable_pc8_work [i915]
[  663.077703]  0000000000000009 ffff88060a671d58 ffffffff81714c09 ffff88060a671da0
[  663.077706]  ffff88060a671d90 ffffffff810643ad ffff88060bf83bc8 ffff88060bf80000
[  663.077707]  ffff88060d116350 ffff88060d116358 0000000000000040 ffff88060a671df0
[  663.077710] Call Trace:
[  663.077715]  [<ffffffff81714c09>] dump_stack+0x45/0x56
[  663.077718]  [<ffffffff810643ad>] warn_slowpath_common+0x7d/0xa0
[  663.077720]  [<ffffffff8106441c>] warn_slowpath_fmt+0x4c/0x50
[  663.077728]  [<ffffffffa026d0e9>] hsw_enable_pc8_work+0x6c9/0x6f0 [i915]
[  663.077730]  [<ffffffff810800f2>] process_one_work+0x182/0x450
[  663.077732]  [<ffffffff81080eb1>] worker_thread+0x121/0x410
[  663.077734]  [<ffffffff81080d90>] ? rescuer_thread+0x3e0/0x3e0
[  663.077737]  [<ffffffff81087b52>] kthread+0xd2/0xf0
[  663.077739]  [<ffffffff81087a80>] ? kthread_create_on_node+0x190/0x190
[  663.077742]  [<ffffffff8172577c>] ret_from_fork+0x7c/0xb0
[  663.077744]  [<ffffffff81087a80>] ? kthread_create_on_node+0x190/0x190
[  663.077745] ---[ end trace 47811739132dc27e ]---
[  726.106603] ------------[ cut here ]------------
[  726.106623] WARNING: CPU: 1 PID: 532 at drivers/gpu/drm/i915/intel_display.c:6309 hsw_enable_pc8_work+0x6c9/0x6f0 [i915]()
[  726.106624] Power well on
[  726.106625] Modules linked in: vfio_iommu_type1(F) vfio_pci(F) vfio(F) ip6table_filter(F) ip6_tables(F) iptable_filter(F) ip_tables(F) ebtable_nat(F) ebtables(F) x_tables(F) bnep(F) bridge(F) rfcomm(F) stp(F) bluetooth(F) llc(F) parport_pc(F) ppdev(F) x86_pkg_temp_thermal(F) intel_powerclamp(F) coretemp(F) kvm_intel(F) kvm(F) crct10dif_pclmul(F) crc32_pclmul(F) ghash_clmulni_intel(F) aesni_intel(F) aes_x86_64(F) lrw(F) gf128mul(F) glue_helper(F) ablk_helper(F) cryptd(F) mxm_wmi(F) snd_hda_codec_hdmi(F) snd_hda_codec_realtek(F) snd_hda_intel(F) snd_hda_codec(F) snd_hwdep(F) snd_pcm(F) snd_page_alloc(F) i915(F) snd_seq_midi(F) snd_seq_midi_event(F) snd_rawmidi(F) snd_seq(F) snd_seq_device(F) snd_timer(F) psmouse(F) mei_me(F) drm_kms_helper(F) drm(F) mei(F) microcode(F) snd(F) soundcore(F) serio_raw(F) video(F) wmi(F) lpc_ich(F) mac_hid(F) lp(F) parport(F) hid_generic(F) usbhid(F) hid(F) igb(F) i2c_algo_bit(F) dca(F) ahci(F) libahci(F) e1000e(F) ptp(F) pps_core(F)
[  726.106654] CPU: 1 PID: 532 Comm: kworker/1:2 Tainted: GF       W    3.13.0-rc4-no-patch #1
[  726.106655] Hardware name: To Be Filled By O.E.M. To Be Filled By O.E.M./Z87 Extreme6, BIOS P2.10 07/03/2013
[  726.106663] Workqueue: events hsw_enable_pc8_work [i915]
[  726.106664]  0000000000000009 ffff88060a671d58 ffffffff81714c09 ffff88060a671da0
[  726.106667]  ffff88060a671d90 ffffffff810643ad ffff88060bf83bc8 ffff88060bf80000
[  726.106669]  ffff88060d116350 ffff88060d116358 0000000000000040 ffff88060a671df0
[  726.106671] Call Trace:
[  726.106676]  [<ffffffff81714c09>] dump_stack+0x45/0x56
[  726.106679]  [<ffffffff810643ad>] warn_slowpath_common+0x7d/0xa0
[  726.106681]  [<ffffffff8106441c>] warn_slowpath_fmt+0x4c/0x50
[  726.106690]  [<ffffffffa026d0e9>] hsw_enable_pc8_work+0x6c9/0x6f0 [i915]
[  726.106693]  [<ffffffff810800f2>] process_one_work+0x182/0x450
[  726.106694]  [<ffffffff81080eb1>] worker_thread+0x121/0x410
[  726.106696]  [<ffffffff81080d90>] ? rescuer_thread+0x3e0/0x3e0
[  726.106699]  [<ffffffff81087b52>] kthread+0xd2/0xf0
[  726.106701]  [<ffffffff81087a80>] ? kthread_create_on_node+0x190/0x190
[  726.106703]  [<ffffffff8172577c>] ret_from_fork+0x7c/0xb0
[  726.106706]  [<ffffffff81087a80>] ? kthread_create_on_node+0x190/0x190
[  726.106707] ---[ end trace 47811739132dc27f ]---
[  741.101725] ------------[ cut here ]------------
[  741.101745] WARNING: CPU: 0 PID: 46 at drivers/gpu/drm/i915/intel_display.c:6309 hsw_enable_pc8_work+0x6c9/0x6f0 [i915]()
[  741.101746] Power well on
[  741.101747] Modules linked in: vfio_iommu_type1(F) vfio_pci(F) vfio(F) ip6table_filter(F) ip6_tables(F) iptable_filter(F) ip_tables(F) ebtable_nat(F) ebtables(F) x_tables(F) bnep(F) bridge(F) rfcomm(F) stp(F) bluetooth(F) llc(F) parport_pc(F) ppdev(F) x86_pkg_temp_thermal(F) intel_powerclamp(F) coretemp(F) kvm_intel(F) kvm(F) crct10dif_pclmul(F) crc32_pclmul(F) ghash_clmulni_intel(F) aesni_intel(F) aes_x86_64(F) lrw(F) gf128mul(F) glue_helper(F) ablk_helper(F) cryptd(F) mxm_wmi(F) snd_hda_codec_hdmi(F) snd_hda_codec_realtek(F) snd_hda_intel(F) snd_hda_codec(F) snd_hwdep(F) snd_pcm(F) snd_page_alloc(F) i915(F) snd_seq_midi(F) snd_seq_midi_event(F) snd_rawmidi(F) snd_seq(F) snd_seq_device(F) snd_timer(F) psmouse(F) mei_me(F) drm_kms_helper(F) drm(F) mei(F) microcode(F) snd(F) soundcore(F) serio_raw(F) video(F) wmi(F) lpc_ich(F) mac_hid(F) lp(F) parport(F) hid_generic(F) usbhid(F) hid(F) igb(F) i2c_algo_bit(F) dca(F) ahci(F) libahci(F) e1000e(F) ptp(F) pps_core(F)
[  741.101776] CPU: 0 PID: 46 Comm: kworker/0:1 Tainted: GF       W    3.13.0-rc4-no-patch #1
[  741.101777] Hardware name: To Be Filled By O.E.M. To Be Filled By O.E.M./Z87 Extreme6, BIOS P2.10 07/03/2013
[  741.101785] Workqueue: events hsw_enable_pc8_work [i915]
[  741.101787]  0000000000000009 ffff88060d3efd58 ffffffff81714c09 ffff88060d3efda0
[  741.101789]  ffff88060d3efd90 ffffffff810643ad ffff88060bf83bc8 ffff88060bf80000
[  741.101791]  ffff88060d116350 ffff88060d116358 0000000000000000 ffff88060d3efdf0
[  741.101793] Call Trace:
[  741.101797]  [<ffffffff81714c09>] dump_stack+0x45/0x56
[  741.101800]  [<ffffffff810643ad>] warn_slowpath_common+0x7d/0xa0
[  741.101802]  [<ffffffff8106441c>] warn_slowpath_fmt+0x4c/0x50
[  741.101810]  [<ffffffffa026d0e9>] hsw_enable_pc8_work+0x6c9/0x6f0 [i915]
[  741.101813]  [<ffffffff810800f2>] process_one_work+0x182/0x450
[  741.101815]  [<ffffffff81080eb1>] worker_thread+0x121/0x410
[  741.101817]  [<ffffffff81080d90>] ? rescuer_thread+0x3e0/0x3e0
[  741.101820]  [<ffffffff81087b52>] kthread+0xd2/0xf0
[  741.101822]  [<ffffffff81087a80>] ? kthread_create_on_node+0x190/0x190
[  741.101825]  [<ffffffff8172577c>] ret_from_fork+0x7c/0xb0
[  741.101827]  [<ffffffff81087a80>] ? kthread_create_on_node+0x190/0x190
[  741.101828] ---[ end trace 47811739132dc280 ]---

teekay

Member

Registered: 2011-10-26

Posts: 271

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

@techdude300: try to uprade your kernel to latest -rc5. -rc3 was very flaky here, too.

revasm

Member

Registered: 2013-12-22

Posts: 2

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Code 43 on i5-2500 IGP and Nvidia 560 TI. VGA works well (1200x800). Device/bus reset seems broken: QEMU issues "Invalid ROM" error on restart.

Z68 was not supposed to have full support for an IOMMU anyway, so I'm surprised that VFIO works at all.

Linux 3.13-rc4
qemu.git 2013-12-20
i915 driver arbiter patches
qemu-vfio NoSnoop v1 patch (unpatched/patched, didn't make a difference, except to flip the NoSnoop bit)

# cat run-kvm.sh
#!/bin/sh
./pcistub-bind "8086 1c2d" "0000:00:1a.0" # USB 2.0 controller
./vfio-group 1
qemu-system-x86_64 -nodefaults \
	-enable-kvm -M q35 -m 2048 -cpu SandyBridge \
	-smp cores=2,threads=1,sockets=1 \
	-bios /usr/share/qemu/bios.bin -rtc base=localtime \
	-vga none -nographic -monitor stdio -boot menu=on \
	-device ioh3420,bus=pcie.0,addr=1c.0,multifunction=on,port=1,chassis=1,id=ich9-pcie-port-1 \
	-device vfio-pci,host=01:00.0,bus=ich9-pcie-port-1,addr=00.0,multifunction=on,x-vga=on,romfile=/export/misc/GF114.bin \
	-device vfio-pci,host=01:00.1,bus=ich9-pcie-port-1,addr=00.1 \
	-device pci-assign,host=00:1a.0,addr=1a.0,multifunction=on,id=ehci-1 \
	-device ide-cd,drive=cd1,bus=ide.1 \
	-device ide-cd,drive=cd2,bus=ide.2 \
	-drive id=cd1,media=cdrom,index=1,file=/export/iso/en_windows_7*.iso \
	-drive id=cd2,media=cdrom,index=2,file=/export/iso/virtio-win-0.1-74.iso \
	-drive if=virtio,index=0,cache=none,format=qcow2,file=/export/vm/win7sp1.qcow2 \
	-net bridge,br=br0 -net nic,model=virtio

# cat /proc/cmdline
initrd=\initramfs-linux-mainline.img root=PARTUUID="667c0274-455e-46a4-8f37-cab6d73269a6" rw acpi_osi=Linux intel_iommu=on pci-stub.ids=8086:1c2d,10de:1200,10de:0e0c

# cat /proc/version
Linux version 3.13.0-1-mainline (gcc version 4.8.2 (GCC) ) #1 SMP PREEMPT Sun Dec 22 14:50:44 PST 2013

# lsgroup | grep "Group 1" -A4
### Group 1 ###
    00:01.0 PCI bridge: Intel Corporation Xeon E3-1200/2nd Generation Core Processor Family PCI Express Root Port (rev 09)
    01:00.0 VGA compatible controller: NVIDIA Corporation GF114 [GeForce GTX 560 Ti] (rev a1)
    01:00.1 Audio device: NVIDIA Corporation GF114 HDMI Audio Controller (rev a1)
### Group 2 ###

# lspci | egrep "VGA|Audio"
00:02.0 VGA compatible controller: Intel Corporation 2nd Generation Core Processor Family Integrated Graphics Controller (rev 09)
00:1b.0 Audio device: Intel Corporation 6 Series/C200 Series Chipset Family High Definition Audio Controller (rev 05)
01:00.0 VGA compatible controller: NVIDIA Corporation GF114 [GeForce GTX 560 Ti] (rev a1)
01:00.1 Audio device: NVIDIA Corporation GF114 HDMI Audio Controller (rev a1)

# head -n1 /dev/vga_arbiter
count:2,PCI:0000:00:02.0,decodes=io,owns=none,locks=none(0:0)

# dmesg | grep -i iommu
[    0.000000] Intel-IOMMU: enabled
[    0.286043] dmar: IOMMU 0: reg_base_addr fed90000 ver 1:0 cap c0000020e60262 ecap f0101a
[    0.286048] dmar: IOMMU 1: reg_base_addr fed91000 ver 1:0 cap c9008020660262 ecap f0105a
[    0.286118] IOAPIC id 2 under DRHD base  0xfed91000 IOMMU 1
[    0.567671] IOMMU 0 0xfed90000: using Queued invalidation
[    0.567672] IOMMU 1 0xfed91000: using Queued invalidation
[    0.567673] IOMMU: Setting RMRR:
[    0.567683] IOMMU: Setting identity map for device 0000:00:02.0 [0xcb800000 - 0xcf9fffff]
[    0.567983] IOMMU: Setting identity map for device 0000:00:1d.0 [0xca8c1000 - 0xca8d2fff]
[    0.567999] IOMMU: Setting identity map for device 0000:00:1a.0 [0xca8c1000 - 0xca8d2fff]
[    0.568008] IOMMU: Prepare 0-16MiB unity mapping for LPC
[    0.568015] IOMMU: Setting identity map for device 0000:00:1f.0 [0x0 - 0xffffff]

# dmesg | tail -n12
[   70.786246] pci-stub 0000:00:1a.0: claimed by stub
[   70.831198] VFIO - User Level meta-driver version: 0.3
[   71.564129] tun: Universal TUN/TAP device driver, 1.6
[   71.564132] tun: (C) 1999-2004 Max Krasnyansky <maxk@qualcomm.com>
[   71.564688] device tap0 entered promiscuous mode
[   71.564776] br0: port 2(tap0) entered forwarding state
[   71.564781] br0: port 2(tap0) entered forwarding state
[   72.140304] vfio-pci 0000:01:00.0: enabling device (0000 -> 0003)
[   72.232772] vfio-pci 0000:01:00.1: enabling device (0000 -> 0002)
[   72.675966] pci-stub 0000:00:1a.0: kvm assign device
[   86.574242] br0: port 2(tap0) entered forwarding state
[   87.229914] kvm: zapping shadow pages for mmio generation wraparound

# dmesg | grep -i vga
[    0.482563] vgaarb: device added: PCI:0000:00:02.0,decodes=io+mem,owns=io+mem,locks=none
[    0.482566] vgaarb: device added: PCI:0000:01:00.0,decodes=io+mem,owns=none,locks=none
[    0.482567] vgaarb: loaded
[    0.482568] vgaarb: bridge control possible 0000:01:00.0
[    0.482568] vgaarb: no bridge control possible 0000:00:02.0
[    0.582479] fb0: EFI VGA frame buffer device
[    4.441728] fb: conflicting fb hw usage inteldrmfb vs EFI VGA - removing generic driver
[    4.572183] [drm] GMBUS [i915 gmbus vga] timed out, falling back to bit banging on pin 2
[    4.852049] vgaarb: device changed decodes: PCI:0000:00:02.0,olddecodes=io+mem,decodes=io:owns=io

# lspci -vvv | grep GTX -A48
01:00.0 VGA compatible controller: NVIDIA Corporation GF114 [GeForce GTX 560 Ti] (rev a1) (prog-if 00 [VGA controller])
	Subsystem: eVga.com. Corp. Device 1563
	Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR+ FastB2B- DisINTx-
	Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR+ <PERR- INTx-
	Latency: 0, Cache Line Size: 64 bytes
	Interrupt: pin A routed to IRQ 16
	Region 0: Memory at f4000000 (32-bit, non-prefetchable) [size=32M]
	Region 1: Memory at e0000000 (64-bit, prefetchable) [size=128M]
	Region 3: Memory at e8000000 (64-bit, prefetchable) [size=64M]
	Region 5: I/O ports at e000 [size=128]
	Expansion ROM at f6000000 [disabled] [size=512K]
	Capabilities: [60] Power Management version 3
		Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0-,D1-,D2-,D3hot-,D3cold-)
		Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=0 PME-
	Capabilities: [68] MSI: Enable- Count=1/1 Maskable- 64bit+
		Address: 0000000000000000  Data: 0000
	Capabilities: [78] Express (v2) Endpoint, MSI 00
		DevCap:	MaxPayload 128 bytes, PhantFunc 0, Latency L0s <4us, L1 <64us
			ExtTag+ AttnBtn- AttnInd- PwrInd- RBE+ FLReset-
		DevCtl:	Report errors: Correctable- Non-Fatal- Fatal- Unsupported-
			RlxdOrd+ ExtTag+ PhantFunc- AuxPwr- NoSnoop-
			MaxPayload 128 bytes, MaxReadReq 512 bytes
		DevSta:	CorrErr- UncorrErr+ FatalErr- UnsuppReq- AuxPwr- TransPend-
		LnkCap:	Port #0, Speed 2.5GT/s, Width x16, ASPM L0s L1, Latency L0 <256ns, L1 <4us
			ClockPM+ Surprise- LLActRep- BwNot-
		LnkCtl:	ASPM Disabled; RCB 128 bytes Disabled- Retrain- CommClk+
			ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
		LnkSta:	Speed 2.5GT/s, Width x16, TrErr- Train- SlotClk+ DLActive- BWMgmt- ABWMgmt-
		DevCap2: Completion Timeout: Not Supported, TimeoutDis+, LTR-, OBFF Not Supported
		DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis-, LTR-, OBFF Disabled
		LnkCtl2: Target Link Speed: 2.5GT/s, EnterCompliance- SpeedDis-
			 Transmit Margin: Normal Operating Range, EnterModifiedCompliance- ComplianceSOS-
			 Compliance De-emphasis: -6dB
		LnkSta2: Current De-emphasis Level: -6dB, EqualizationComplete-, EqualizationPhase1-
			 EqualizationPhase2-, EqualizationPhase3-, LinkEqualizationRequest-
	Capabilities: [b4] Vendor Specific Information: Len=14 <?>
	Capabilities: [100 v1] Virtual Channel
		Caps:	LPEVC=0 RefClk=100ns PATEntryBits=1
		Arb:	Fixed- WRR32- WRR64- WRR128-
		Ctrl:	ArbSelect=Fixed
		Status:	InProgress-
		VC0:	Caps:	PATOffset=00 MaxTimeSlots=1 RejSnoopTrans-
			Arb:	Fixed- WRR32- WRR64- WRR128- TWRR128- WRR256-
			Ctrl:	Enable+ ID=0 ArbSelect=Fixed TC/VC=ff
			Status:	NegoPending- InProgress-
	Capabilities: [128 v1] Power Budgeting <?>
	Capabilities: [600 v1] Vendor Specific Information: ID=0001 Rev=1 Len=024 <?>
	Kernel driver in use: vfio-pci
	Kernel modules: nouveau

Val532

Member

Registered: 2013-11-13

Posts: 35

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

@aw :

My first sucess, now i have a working setup, with 3.12.0 kernel and i915 patch, and qemu git + NoSnoop patch (probably v2 but i'm not sure).
Now i will try with 3.12.6 kernel.

Edit : And only with hdmi audio on pcie.0 insted of root.1

Edit 2nd : Now it work for me with 3.13-rc5 + i915 patch and qemu git without any patch. But only with hdmi audio as bus=pcie.0 and not root.1

Last edited by Val532 (2013-12-24 04:10:16)

paradexes

Member

Registered: 2013-11-30

Posts: 13

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

This thread has been great. Thanks for the all the info so far. Also I have hardware now that does work and is supported for IOMMU. However now I am running into an issue when I run the qemu commands. Here is my commands and output:

qemu-system-x86_64 -enable-kvm -M q35 -m 16024 -cpu host -smp 6,sockets=1,cores=6,threads=1 -bios /usr/share/qemu/bios.bin -vga none -device ioh3420,bus=pcie.0,addr=1c.0,multifunction=on,port=1,chassis=1,id=root.1 -device vfio-pci,host=03:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on -device vfio-pci,host=03:00.1,bus=root.1,addr=00.1

qemu-system-x86_64: -device vfio-pci,host=03:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on: vfio: error, group 17 is not viable, please ensure all devices within the iommu_group are bound to their vfio bus driver.
qemu-system-x86_64: -device vfio-pci,host=03:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on: vfio: failed to get group 17
qemu-system-x86_64: -device vfio-pci,host=03:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on: Device initialization failed.
qemu-system-x86_64: -device vfio-pci,host=03:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on: Device 'vfio-pci' could not be initialized

What am I missing here? It seems like its almost there but I am sure I am issing something here. The error seems to indicate to put this in an iommu_group which I was under the impression I did.

Edit: Specs. Im running on an TYAN S8225 board. Running 32Gb of memory with 2x 4234 Opterons, OS: Linuxmint 16 with 3.12.6 kernel. I have an nvidia 660ti as the host card and my intended card is an AMD R9 270X.

Thanks

Last edited by paradexes (2013-12-24 04:48:01)

noobman

Member

Registered: 2013-12-13

Posts: 21

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

@techdude300: i have exactly same problem with memory, pls let me/us know if you find a fix/workaround;

@paradexes: i had same output, but i saw it works prefix sudo ...  so looking into it i found this :

...The final step is to provide the user with access to the group if unprivileged operation is desired (note that /dev/vfio/vfio provides no capabilities on its own and is therefore expected to be set to mode 0666 by the system)...

# chown user:user /dev/vfio/17

Afterwards i think should work. This is like a default and needs a change, i would vote for this to be mentioned by OP because it was not obvious for me either.

BulliteShot

Member

Registered: 2013-07-28

Posts: 26

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Well i'm back with a "Crosshair V Formula" motherboard and FX CPU. Upon booting the VM, my system freezes up when the BIOS screen loads and says no bootable device. The text is also strangly spaced out wide.

Anyone know how to fix the crash or is this because the BIOS is resetting after failing to boot?

xani

Member

Registered: 2013-12-24

Posts: 6

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Hi, first of all big thanks to nbhs and Alex for your awesome job!

I managed to make this work with pci-assign back in July, but I couldn't reboot even with ejecting gpu on shutdown. I was using windows 8 and it gave BSOD on shutdown everytime and gpu wouldn't reset.
Now I want to do this using vfio. I managed to get it working with acs override and i915 patches for rc3. I'm using qemu-git, seabios-git and linux-mainline provided by nbhs at first page.

My setup:
cpu: i7-4770
mobo: asrock z87 extreme6
gpu: radeon hd7870 GHz edition

Anyway, I had the same problem as noobman and techdude300, memory wouldn't free after closing qemu.
I bisected problem to commit

0c44c2d0f459cd7e275242b72f500137c4fa834d

I also reverted this one, I think it would'nt build without reverting this but now I'm not sure.

b021fe3e25094fbec22d0eff846d2adeee1b9736

Now it does free memory, but I cannot reboot anyway because seabios is hanging at '_' and after a while it shows that it cannot find hard drive to boot from.
Host reboot is necessary to fire up vm again.

My config:

$ cat /proc/cmdline
BOOT_IMAGE=/vmlinuz-linux-mainline root=UUID=a63d1c94-4682-431c-9788-0cccd29ee598 rw intel_iommu=on pci-stub.ids=1002:6818,1002:aab0,8086:0
c0c,8086:153b pcie_acs_override=downstream

qemu-system-x86_64 -M q35 \
-m 8196 -enable-kvm -cpu host -monitor stdio \
-smp 4,sockets=1,cores=4,threads=1 \
-rtc clock=host,base=localtime \
-vga none -net none \
-parallel none -serial none \
-usb -usbdevice host:05ac:0221 -usbdevice host:1532:0037 \
-device ahci,bus=pcie.0,id=ahci \
-device ioh3420,bus=pcie.0,addr=1c.0,multifunction=on,port=1,chassis=1,id=root.1 \
-device vfio-pci,host=01:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on \
-device vfio-pci,host=01:00.1,bus=pcie.0 \
-device vfio-pci,host=00:19.0,bus=pcie.0 \
-device vfio-pci,host=05:00.0,bus=pcie.0 

I use nbhs script from first page to bind devices.

Here's my directory from which you can test that memory leak issues.
linux-mainline.tar.gz

Provide me with feedback if you have the same problem with seabios after reverting those patches.

EDIT:
Further investigation showed that this problem can lie within gcc itself. Here's link that can give you explanation. I will report on my findings asap.

EDIT2:
OK, so after disabling asm goto with this little patch, it works just like reversing commits. But this bug should be fixed by commit

88f182dd77

Maybe it's another thing then. I still have few options to try.

Last edited by xani (2013-12-25 14:49:06)