Arch Linux

aw

Member

Registered: 2013-10-04

Posts: 921

Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

@aw: Your VM startup script uses virsh nodedev-detach pci_0000_01_00_0 and virsh nodedev-reattach pci_0000_01_00_0 to detach and later reattach the graphics card. Does the reattach command reset the graphics card

no

and/or make it available to the host?

yes

Are these virsh commands in lieu of binding the card to pci-stub?

It's in place of binding and unbinding the device to vfio-pci

man virsh:

       nodedev-detach nodedev [--driver backend_driver]
           Detach nodedev from the host, so that it can safely be used by
           guests via <hostdev> passthrough.  This is reversed with nodedev-
           reattach, and is done automatically for managed devices.  For
           compatibility purposes, this command can also be spelled nodedev-
           dettach.

           Different backend drivers expect the device to be bound to
           different dummy devices. For example, QEMU's "kvm" backend driver
           (the default) expects the device to be bound to pci-stub, but its
           "vfio" backend driver expects the device to be bound to vfio-pci.
           The --driver parameter can be used to specify the desired backend
           driver.

virsh nodedev-dumpxml will also list the iommu group dependencies

Now to the qemu / kvm command options: Aside from running --help and getting a shortlist of options, is there any source of information on the many command line options that are used?

Use the source.  It's also useful to use virt-manager then look in the libvirt log files to extract the options it used for a given feature.

---

http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

aw

Member

Registered: 2013-10-04

Posts: 921

Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

You can also use -device ? to get a list of devices and -device foo,? to get a list of options for device foo.  '?' works to get options lists for various other commands too.  So start with --help and drill down with other options.  There's also a fairly extensive man page for qemu.

---

http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

manjack

Member

Registered: 2014-05-11

Posts: 9

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Are those the first errors?  Can you capture the entire output and post it to pastebin and provide us a link?
How are you compiling it?  Are you using makepkg, or are you using something like  config and make install ?

Sorry, first time posting code here. i have already fixed the post.

Yes, I'm using makepkg to compile it. The log is here

Thanks!

Last edited by manjack (2014-05-15 11:57:05)

ewaller

Administrator

From: Pasadena, CA

Registered: 2009-07-13

Posts: 20,214

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Sorry, first time posting code here. i have already fixed the post.

No worries.  And your readers appreciate it.

Yes, I'm using makepkg to compile it. The log is here

There is a mismatch somewhere.  The there is a macro definition in an included file that does not match the way it is being called.
Which qemu is it that you are trying to build?  I looked in the AUR and there are several, and of course, there is the one in the ABS.
Also, is your system fully up to date?  Have you run pacman -Syu to ensure you've the latest version of all packagers?

---

Nothing is too wonderful to be true, if it be consistent with the laws of nature -- Michael Faraday
Sometimes it is the people no one can imagine anything of who do the things no one can imagine. -- Alan Turing
---
How to Ask Questions the Smart Way

powerhouse

Member

Registered: 2013-05-22

Posts: 15

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

You can also use -device ? to get a list of devices and -device foo,? to get a list of options for device foo.  '?' works to get options lists for various other commands too.  So start with --help and drill down with other options.  There's also a fairly extensive man page for qemu.

Thanks! And also for explaining virsh nodedev-detach... etc. I had used virsh before, but only for USB hosts. Under Xen I didn't have luck reattaching my graphics card to the host and using it (IIRC it would crash the host). Is that possible with kvm and a little luck?

A second scenario would be to pass the graphics card to another guest. Is this possible?

aw

Member

Registered: 2013-10-04

Posts: 921

Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

You can also use -device ? to get a list of devices and -device foo,? to get a list of options for device foo.  '?' works to get options lists for various other commands too.  So start with --help and drill down with other options.  There's also a fairly extensive man page for qemu.

Thanks! And also for explaining virsh nodedev-detach... etc. I had used virsh before, but only for USB hosts. Under Xen I didn't have luck reattaching my graphics card to the host and using it (IIRC it would crash the host). Is that possible with kvm and a little luck?

Unfortunately the Linux graphics drivers behave very poorly when you attempt to unbind devices from them.  I haven't looked at fixing any of them yet, but I did run across this video recently that seems to show a GPU being dynamically released from Xorg, booting Windows, then being re-attached to Xorg - http://163216.genaker.web.hosting-test. … sthfI.html  Sounds like some scripts to do some xrandr magic before and after using the vm.  blu3bird84, are you around to share your scripts?

A second scenario would be to pass the graphics card to another guest. Is this possible?

If you mean sequentially, then yes, of course that works (assuming we can reset the device when qemu starts up).  There's no concurrent access support.

I'll also note for this thread that I tried a GeForce 8400GS and found that there was absolutely no way to reset the card.  A secondary bus reset doesn't phase it, the mode and framebuffer remain intact.  Windows guests still manage to work well with it, but not so much Linux.  The card is too old to spend much effort on it, but fyi, folks using vintage cards may run into such things.

---

http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Destroy

Member

Registered: 2014-05-05

Posts: 10

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Was it upgrading to qemu 2 that did the trick with booting the VM?

Yes. Qemu2, kernel 3.14.2 (with a patch for a problem with chipset SATA Marvell)

About using only one VCPU: I noticed that you don't have any

smp sockets=...,cores=...

I have added

-smp sockets=1,cores=2 -cpu host,hv-time 

and I getting a better response CPU Mark from 1633 to 2950, it is a 80%

AKSN74

Member

From: Taichung, Taiwan

Registered: 2014-02-18

Posts: 52

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Finally, in today, I success to build 2 VMs with 2 x GTX480 on another PC.
Before that, I can't boot up 2nd VM when 1st VM is working.

And here is another PC's spec:
CPU: Intel Core i7 4820K
MB: MSI X79A-GD45 Plus
RAM: DDR3-1600 4G x 8
GPU(Host): AMD Radeon HD5450
GPU(Guest 1): NVIDIA GeForce GTX480
GPU(Guest 2): Same as Guest 1
Host OS: Ubuntu Server 12.04-3 (Kernel: Linux 3.14.0-rc4 x86_64)
Guest OS: Ubuntu Server 12.04-4 (Both)

Few days ago I found that I doing a wrong installation steps before, that's why I can't passthrough my USB keyboard into my VM.
So, I decide to delete everything and restart at begin.

Here is my new steps:

1. Download VFIO kernel and qemu from aw's github (qemu-vfio & linux-vfio)

2. Install qemu-vfio first, note that if you want to use usb device in VM, you need to set configure as

./configure --prefix=/usr --enable-libusb

3. Then install linux-vfio, before reboot, set /etc/default/grub as

GRUB_CMDLINE_LINUX_DEFAULT="quiet splash intel_iommu=on pci-stub.ids=10de:06c0,10de:0be5 vfio_iommu_type1.allow_unsafe_interrupts=1"

and update grub, then reboot.

4. After reboot, make vfio-bind.sh file at #1 post then save to /usr/bin

5. In #1 post, nbhs wrote Systemd Service so can make vfio-bind to bind GPU into VFIO automatically when boot, but Ubuntu 12.04 still using Upstart, so I have no idea but skip it.

6. So I make sh file so it can bind into VFIO when start VM:

#!/bin/bash

#1st VM settings
vfio-bind 0000:03:00.0 0000:03:00.1

qemu-system-x86_64 -enable-kvm -M q35 -m 4096 -cpu host \
-smp 3,sockets=1,cores=3,threads=1 \
-bios /usr/share/qemu/bios.bin -vga none -nographic -localtime \
-device ioh3420,bus=pcie.0,addr=1c.0,multifunction=on,port=1,chassis=1,id=root.1 \
-device vfio-pci,host=03:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on \
-device vfio-pci,host=03:00.1,bus=root.1,addr=00.1 \
-device ahci,bus=pcie.0,id=ahci \
-usb -usbdevice host:03f0:0024 \
-drive file=/home/eddieak/uc1.img,id=disk,format=raw,if=virtio

7. 1st VM is bootup, now bootup 2nd VM, same code, but different USB HID, GPU PCI num, and image:

#!/bin/bash

#2nd VM settings
vfio-bind 0000:01:00.0 0000:01:00.1

qemu-system-x86_64 -enable-kvm -M q35 -m 4096 -cpu host \
-smp 3,sockets=1,cores=3,threads=1 \
-bios /usr/share/qemu/bios.bin -vga none -nographic -localtime \
-device ioh3420,bus=pcie.0,addr=1c.0,multifunction=on,port=1,chassis=1,id=root.1 \
-device vfio-pci,host=01:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on \
-device vfio-pci,host=01:00.1,bus=root.1,addr=00.1 \
-device ahci,bus=pcie.0,id=ahci \
-usb -usbdevice host:413c:2003 \
-drive file=/home/eddieak/uc2.img,id=disk,format=raw,if=virtio

8. Finally, this time each VM can bootup, and can boot into Ubuntu tty.
Also, type lspci can found GTX480.

Now it can do well at this time, but I got problems

1. When I reboot one of VM, it will stuck at blank screen again, need to reboot whole host so that can boot 2 VMs together again.
2. If one of the VM stuck, I can't terminate it immediately, even I tried Ctrl+C is no use. Need to reboot or hit restart button.

Does any solutions?

Last edited by AKSN74 (2014-05-16 09:16:19)

aw

Member

Registered: 2013-10-04

Posts: 921

Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Here is my new steps:

1. Download VFIO kernel and qemu from aw's github (qemu-vfio & linux-vfio)

No, there's nothing there that you need.  In your setup qemu 2.0 and kernel 3.15-rc should be perfectly fine.  Most users will also need to use the i915 VGA arbiter patches for host IGD graphics.  The only reason for 3.15-rc is because it has the bulk of the PCH root port ACS quirks.

2. Install qemu-vfio first, note that if you want to use usb device in VM, you need to set configure as

./configure --prefix=/usr --enable-libusb

3. Then install linux-vfio, before reboot, set /etc/default/grub as

GRUB_CMDLINE_LINUX_DEFAULT="quiet splash intel_iommu=on pci-stub.ids=10de:06c0,10de:0be5 vfio_iommu_type1.allow_unsafe_interrupts=1"

and update grub, then reboot.

You also likely don't need vfio_iommu_type1.allow_unsafe_interrupts=1.  Any one on an AMD system and most people on Intel don't need this option.

4. After reboot, make vfio-bind.sh file at #1 post then save to /usr/bin

5. In #1 post, nbhs wrote Systemd Service so can make vfio-bind to bind GPU into VFIO automatically when boot, but Ubuntu 12.04 still using Upstart, so I have no idea but skip it.

6. So I make sh file so it can bind into VFIO when start VM:

#!/bin/bash

#1st VM settings
vfio-bind 0000:03:00.0 0000:03:00.1

qemu-system-x86_64 -enable-kvm -M q35 -m 4096 -cpu host \
-smp 3,sockets=1,cores=3,threads=1 \
-bios /usr/share/qemu/bios.bin -vga none -nographic -localtime \
-device ioh3420,bus=pcie.0,addr=1c.0,multifunction=on,port=1,chassis=1,id=root.1 \
-device vfio-pci,host=03:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on \
-device vfio-pci,host=03:00.1,bus=root.1,addr=00.1 \
-device ahci,bus=pcie.0,id=ahci \
-usb -usbdevice host:03f0:0024 \
-drive file=/home/eddieak/uc1.img,id=disk,format=raw,if=virtio

7. 1st VM is bootup, now bootup 2nd VM, same code, but different USB HID, GPU PCI num, and image:

#!/bin/bash

#2nd VM settings
vfio-bind 0000:01:00.0 0000:01:00.1

qemu-system-x86_64 -enable-kvm -M q35 -m 4096 -cpu host \
-smp 3,sockets=1,cores=3,threads=1 \
-bios /usr/share/qemu/bios.bin -vga none -nographic -localtime \
-device ioh3420,bus=pcie.0,addr=1c.0,multifunction=on,port=1,chassis=1,id=root.1 \
-device vfio-pci,host=01:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on \
-device vfio-pci,host=01:00.1,bus=root.1,addr=00.1 \
-device ahci,bus=pcie.0,id=ahci \
-usb -usbdevice host:413c:2003 \
-drive file=/home/eddieak/uc2.img,id=disk,format=raw,if=virtio

8. Finally, this time each VM can bootup, and can boot into Ubuntu tty.
Also, type lspci can found GTX480.

Now it can do well at this time, but I got problems

1. When I reboot one of VM, it will stuck at blank screen again, need to reboot whole host so that can boot 2 VMs together again.
2. If one of the VM stuck, I can't terminate it immediately, even I tried Ctrl+C is no use. Need to reboot or hit restart button.

Does any solutions?

Could you remove the comment around #define DEBUG_VFIO in hw/misc/vfio.c in your qemu tree, rebuild and install and report the last few lines you see before it hangs?

---

http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

manjack

Member

Registered: 2014-05-11

Posts: 9

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Sorry, first time posting code here. i have already fixed the post.

No worries.  And your readers appreciate it.

Yes, I'm using makepkg to compile it. The log is here

There is a mismatch somewhere.  The there is a macro definition in an included file that does not match the way it is being called.
Which qemu is it that you are trying to build?  I looked in the AUR and there are several, and of course, there is the one in the ABS.
Also, is your system fully up to date?  Have you run pacman -Syu to ensure you've the latest version of all packagers?

Im trying to build the one on this post. I have tried to build again after i run pacman -Syu but i got the same error.

AKSN74

Member

From: Taichung, Taiwan

Registered: 2014-02-18

Posts: 52

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Very thanks for your reply, I'm really appreciate. Because I'm still a beginner, so there is some steps I don't know how to do.

No, there's nothing there that you need.  In your setup qemu 2.0 and kernel 3.15-rc should be perfectly fine.  Most users will also need to use the i915 VGA arbiter patches for host IGD graphics.  The only reason for 3.15-rc is because it has the bulk of the PCH root port ACS quirks.

Do I need to uninstall previous QEMU and kernel? If so, how do I uninstall it? Because I tried "make uninstall" is no use, and type "make clean" will delete whole install data.

EDIT: I successfully uninstall the kernel and installing a new kernel now.

Could you remove the comment around #define DEBUG_VFIO in hw/misc/vfio.c in your qemu tree, rebuild and install and report the last few lines you see before it hangs?

I use vi to open this file and found what you said

/* #define DEBUG_VFIO */
#ifdef DEBUG_VFIO
#define DPRINTF(fmt, ...) \
    do { fprintf(stderr, "vfio: " fmt, ## __VA_ARGS__); } while (0)
#else
#define DPRINTF(fmt, ...) \
    do { } while (0)
#endif

So I just need to remove these code and compile again?

Last edited by AKSN74 (2014-05-17 01:58:52)

aw

Member

Registered: 2013-10-04

Posts: 921

Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Very thanks for your reply, I'm really appreciate. Because I'm still a beginner, so there is some steps I don't know how to do.

No, there's nothing there that you need.  In your setup qemu 2.0 and kernel 3.15-rc should be perfectly fine.  Most users will also need to use the i915 VGA arbiter patches for host IGD graphics.  The only reason for 3.15-rc is because it has the bulk of the PCH root port ACS quirks.

Do I need to uninstall previous QEMU and kernel? If so, how do I uninstall it? Because I tried "make uninstall" is no use, and type "make clean" will delete whole install data.

EDIT: I successfully uninstall the kernel and installing a new kernel now.

Could you remove the comment around #define DEBUG_VFIO in hw/misc/vfio.c in your qemu tree, rebuild and install and report the last few lines you see before it hangs?

I use vi to open this file and found what you said

/* #define DEBUG_VFIO */
#ifdef DEBUG_VFIO
#define DPRINTF(fmt, ...) \
    do { fprintf(stderr, "vfio: " fmt, ## __VA_ARGS__); } while (0)
#else
#define DPRINTF(fmt, ...) \
    do { } while (0)
#endif

So I just need to remove these code and compile again?

Just change:

/* #define DEBUG_VFIO */

to

#define DEBUG_VFIO

Recompile, install, and report the last few lines after it hangs, hopefully it gives us something to investigate next.  You can change it back and recompile/install to remove the debug output.

---

http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

AKSN74

Member

From: Taichung, Taiwan

Registered: 2014-02-18

Posts: 52

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Just change:

/* #define DEBUG_VFIO */

to

#define DEBUG_VFIO

Recompile, install, and report the last few lines after it hangs, hopefully it gives us something to investigate next.  You can change it back and recompile/install to remove the debug output.

OK, I'll try in these days and report immediately after the result.

Oh, I forgot to ask, this will show on dmesg or tty?

Last edited by AKSN74 (2014-05-17 08:11:31)

manjack

Member

Registered: 2014-05-11

Posts: 9

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

does the qemu on this post have problem?

aw

Member

Registered: 2013-10-04

Posts: 921

Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

does the qemu on this post have problem?

Do you really think this is a productive question?

---

http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

AKSN74

Member

From: Taichung, Taiwan

Registered: 2014-02-18

Posts: 52

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Just change:

/* #define DEBUG_VFIO */

to

#define DEBUG_VFIO

Recompile, install, and report the last few lines after it hangs, hopefully it gives us something to investigate next.  You can change it back and recompile/install to remove the debug output.

Hello, I'm testing right now.

when I reboot one of VM, it hangs, and vfio debug message unstoppable and repeatedly printing this

vfio: vfio_vga_read (0x3cc, 1) = 0x1
vfio: vfio_vga_read (0x3d4, 1) = 0x0
vfio: vfio_vga_write (0x3d4, 0x0,1)
vfio: vfio_vga_write (0x3d4, 0xa8, 1)
vfio: vfio_vga_write (0x3d4, 0x80a8, 2)
vfio: vfio_bar_write (0000:01:00.0:BAR5+0x8, 0x61000c, 4) ← This may happen in some times

Because its too fast. So I use camera with high-speed shutter.
That binary code not only 0x3d4, but also 0x3cc 0x3d5, and recursing.
So I realize that it hangs like it stuck at these binary code.

EDIT: Disable nouveau still useless. And I still observe to message, so code may update.

EDIT2: After the message runs about 18 minutes the message only shows:

vfio: vfio_vga_read (0xa8044, 2) = 0x0

But after about 3 minutes, the message back to begining code again

Last edited by AKSN74 (2014-05-19 03:59:08)

Namelles_One

Member

Registered: 2014-05-18

Posts: 8

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Hi, guys!

Nowdays, I also get problems with vga passthrough
Maybe, it is simple and you can help me?

I have Gentoo Linux, GeForce 7900GS with nvidia proprietary driver on host and Radeon HD6950 passthrough to guest.
On guest I tried to install Windows 7 SP1.

So, the main problem is very common - when I installed ATI driver on Windows (so, I have installed only video driver, not all CCC) - after reboot I got BSOD with atikmpag.sys

Sad, but true

Some more info about my system:

Linux StaticEvil 3.12.13-gentoo #8 SMP Sun May 18 23:54:05 MSK 2014 x86_64 Intel(R) Core(TM) i5-4440 CPU @ 3.10GHz GenuineIntel GNU/Linux

dmesg | grep IOMMU

[    0.017465] dmar: IOMMU 0: reg_base_addr fed90000 ver 1:0 cap c0000020660462 ecap f0101a
[    0.017888] dmar: IOMMU 1: reg_base_addr fed91000 ver 1:0 cap d2008020660462 ecap f010da
[    0.018537] IOAPIC id 2 under DRHD base  0xfed91000 IOMMU 1
[    0.204011] IOMMU 0 0xfed90000: using Queued invalidation
[    0.204168] IOMMU 1 0xfed91000: using Queued invalidation
[    0.204325] IOMMU: Setting RMRR:
[    0.204487] IOMMU: Setting identity map for device 0000:00:02.0 [0xab800000 - 0xaf9fffff]
[    0.205117] IOMMU: Setting identity map for device 0000:00:1d.0 [0xa837a000 - 0xa8386fff]
[    0.205407] IOMMU: Setting identity map for device 0000:00:1a.0 [0xa837a000 - 0xa8386fff]
[    0.205694] IOMMU: Setting identity map for device 0000:00:14.0 [0xa837a000 - 0xa8386fff]
[    0.205988] IOMMU: Prepare 0-16MiB unity mapping for LPC
[    0.206150] IOMMU: Setting identity map for device 0000:00:1f.0 [0x0 - 0xffffff]

cat /proc/cmdline

BOOT_IMAGE=/vmlinuz-3.12.13-gentoo root=/dev/sda3 ro net.ifnames=0 modprobe.blacklist=fglrx \
pci-stub.ids=1002:6719,1002:aa80 vfio_iommu_type1.allow_unsafe_interrupts=1

lsmod

Module                  Size  Used by
***
kvm_intel             126738  0
kvm                   369668  1 kvm_intel
nvidia              11271958  30
***

sudo lspci -nn

***
01:00.0 VGA compatible controller [0300]: Advanced Micro Devices, Inc. [AMD/ATI] Cayman PRO [Radeon HD 6950] [1002:6719]
01:00.1 Audio device [0403]: Advanced Micro Devices, Inc. [AMD/ATI] Cayman/Antilles HDMI Audio [Radeon HD 6900 Series] [1002:aa80]
***

qemu-system-x86_64 --version

QEMU emulator version 2.0.0, Copyright (c) 2003-2008 Fabrice Bellard

cat /etc/modprobe.d/kvm.conf

options kvm ignore_msrs=1
options kvm allow_unsafe_assigned_interrupts=1

What can I forget? So, the most strange is, that once it boot good, without BSOD, but after reboot I get BSOD again and again. Strange and sad.

P.S. Before starting qemu I do:

sudo vfio-bind 0000:01:00.0 0000:01:00.1

then

pkexec qemu-system-x86_64 -enable-kvm -M q35 -m 4096 -cpu host -smp 6,sockets=1,cores=2,threads=1 -bios /usr/share/qemu/bios.bin -vga none -device vfio-pci,host=01:00.0,x-vga=on -drive file=/var/lib/libvirt/images/Windows_7.img,id=disk,format=raw

zzz3000

Member

Registered: 2014-02-16

Posts: 12

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Hello.
Is it possible to passthrough radeon card and use another radeon in host?
How don't blacklist radeon driver and get it works.

Looks like I just have to use psi-stub for guest card and don't blacklist radeon driver.

Last edited by zzz3000 (2014-05-19 07:19:21)

AKSN74

Member

From: Taichung, Taiwan

Registered: 2014-02-18

Posts: 52

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Hi, guys!

Nowdays, I also get problems with vga passthrough
Maybe, it is simple and you can help me?

I have Gentoo Linux, GeForce 7900GS with nvidia proprietary driver on host and Radeon HD6950 passthrough to guest.
On guest I tried to install Windows 7 SP1.

So, the main problem is very common - when I installed ATI driver on Windows (so, I have installed only video driver, not all CCC) - after reboot I got BSOD with atikmpag.sys

Sad, but true

Some more info about my system:

Linux StaticEvil 3.12.13-gentoo #8 SMP Sun May 18 23:54:05 MSK 2014 x86_64 Intel(R) Core(TM) i5-4440 CPU @ 3.10GHz GenuineIntel GNU/Linux

dmesg | grep IOMMU

[    0.017465] dmar: IOMMU 0: reg_base_addr fed90000 ver 1:0 cap c0000020660462 ecap f0101a
[    0.017888] dmar: IOMMU 1: reg_base_addr fed91000 ver 1:0 cap d2008020660462 ecap f010da
[    0.018537] IOAPIC id 2 under DRHD base  0xfed91000 IOMMU 1
[    0.204011] IOMMU 0 0xfed90000: using Queued invalidation
[    0.204168] IOMMU 1 0xfed91000: using Queued invalidation
[    0.204325] IOMMU: Setting RMRR:
[    0.204487] IOMMU: Setting identity map for device 0000:00:02.0 [0xab800000 - 0xaf9fffff]
[    0.205117] IOMMU: Setting identity map for device 0000:00:1d.0 [0xa837a000 - 0xa8386fff]
[    0.205407] IOMMU: Setting identity map for device 0000:00:1a.0 [0xa837a000 - 0xa8386fff]
[    0.205694] IOMMU: Setting identity map for device 0000:00:14.0 [0xa837a000 - 0xa8386fff]
[    0.205988] IOMMU: Prepare 0-16MiB unity mapping for LPC
[    0.206150] IOMMU: Setting identity map for device 0000:00:1f.0 [0x0 - 0xffffff]

cat /proc/cmdline

BOOT_IMAGE=/vmlinuz-3.12.13-gentoo root=/dev/sda3 ro net.ifnames=0 modprobe.blacklist=fglrx \
pci-stub.ids=1002:6719,1002:aa80 vfio_iommu_type1.allow_unsafe_interrupts=1

lsmod

Module                  Size  Used by
***
kvm_intel             126738  0
kvm                   369668  1 kvm_intel
nvidia              11271958  30
***

sudo lspci -nn

***
01:00.0 VGA compatible controller [0300]: Advanced Micro Devices, Inc. [AMD/ATI] Cayman PRO [Radeon HD 6950] [1002:6719]
01:00.1 Audio device [0403]: Advanced Micro Devices, Inc. [AMD/ATI] Cayman/Antilles HDMI Audio [Radeon HD 6900 Series] [1002:aa80]
***

qemu-system-x86_64 --version

QEMU emulator version 2.0.0, Copyright (c) 2003-2008 Fabrice Bellard

cat /etc/modprobe.d/kvm.conf

options kvm ignore_msrs=1
options kvm allow_unsafe_assigned_interrupts=1

What can I forget? So, the most strange is, that once it boot good, without BSOD, but after reboot I get BSOD again and again. Strange and sad.

P.S. Before starting qemu I do:

sudo vfio-bind 0000:01:00.0 0000:01:00.1

then

pkexec qemu-system-x86_64 -enable-kvm -M q35 -m 4096 -cpu host -smp 6,sockets=1,cores=2,threads=1 -bios /usr/share/qemu/bios.bin -vga none -device vfio-pci,host=01:00.0,x-vga=on -drive file=/var/lib/libvirt/images/Windows_7.img,id=disk,format=raw

Not only AMD GPU need to mount into VM, but also the AMD Audio device need to mount together.

gneville

Member

Registered: 2014-04-02

Posts: 27

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Hi Namelles_One

I managed to fix the bsods on reboots by following the instructions this guy had in his blog http://blog.ktz.me/?p=219 it works perfectly for me now.

What can I forget? So, the most strange is, that once it boot good, without BSOD, but after reboot I get BSOD again and again. Strange and sad.

Namelles_One

Member

Registered: 2014-05-18

Posts: 8

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Not only AMD GPU need to mount into VM, but also the AMD Audio device need to mount together.

So, I tried with Audio too - BSOD anyway, sorry.

I managed to fix the bsods on reboots by following the instructions this guy had in his blog http://blog.ktz.me/?p=219 it works perfectly for me now.

So, you did not with qemu-kvm, but with xen?

gneville

Member

Registered: 2014-04-02

Posts: 27

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

It works with qemu as well as Xen.

So, you did not with qemu-kvm, but with xen?

Namelles_One

Member

Registered: 2014-05-18

Posts: 8

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

It works with qemu as well as Xen.

So, you did not with qemu-kvm, but with xen?

It's all about rebooting, I see. But I get BSOD also when I boot qemu firstly, after rebooting host.

Or, even when I reboot guest with ATI drivers at least once - I got corrupted state and then can't boot it, before driver removing?

zzz3000

Member

Registered: 2014-02-16

Posts: 12

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

I have the same problem with HD7750.
The problem with windows radeon driver dissapeared only after I had installed  fedora rawhide, so I am waiting fedora 21.

Namelles_One

Member

Registered: 2014-05-18

Posts: 8

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

I have the same problem with HD7750.
The problem with windows radeon driver dissapeared only after I had installed  fedora rawhide, so I am waiting fedora 21.

Thank you! "Rawhide" - it is a keyword ^^

I've update to 3.14.4 kernel and have stable Windows with driver!

So, let's rock!