Arch Linux

aw

Member

Registered: 2013-10-04

Posts: 921

Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

I have the same problem with HD7750.
The problem with windows radeon driver dissapeared only after I had installed  fedora rawhide, so I am waiting fedora 21.

Thank you! "Rawhide" - it is a keyword ^^

If you don't want to go bleeding edge for the whole distro, you can use the Fedora virt-preview repo: http://fedoraproject.org/wiki/Virtualiz … Repository

It effectively gives you "rawhide" virt bits while keeping the rest on the latest stable version.

---

http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Namelles_One

Member

Registered: 2014-05-18

Posts: 8

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

I have the same problem with HD7750.
The problem with windows radeon driver dissapeared only after I had installed  fedora rawhide, so I am waiting fedora 21.

Thank you! "Rawhide" - it is a keyword ^^

If you don't want to go bleeding edge for the whole distro, you can use the Fedora virt-preview repo: http://fedoraproject.org/wiki/Virtualiz … Repository

It effectively gives you "rawhide" virt bits while keeping the rest on the latest stable version.

I use Gentoo. "Rawhide" - gives me an idea about kernel version.

AKSN74

Member

From: Taichung, Taiwan

Registered: 2014-02-18

Posts: 52

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Also, I got another problem yesterday.

Because my target is build VM with GTX480, and want to run CUDA with GTX480 in VM.

As guest OS is Ubuntu 12.04-4, first I install NVIDIA Driver without a problem.
Then I install CUDA SDK 6.0, and still no problem. And it can compile the sample application.

But when I tried to run deviceQueryDrv, it got error.

CUDA_ERROR_INVALID_DEVICE (device specified is not a valid CUDA device)

So I downgrade my SDK version to 5.5, because GTX480 is old GPU, some functions in 6.0 may not support.
When I installed 5.5, and compile ver 5.5 samples, still got a same error while I ran deviceQueryDrv.

But before that, I can run CUDA while guest OS is Windows 7. And it can get matrix multiply result.
Maybe it's driver and SDK compatible problem (if it is VFIO problem, graphic driver will got issue first. Neither NVIDIA offical driver nor Nouveau.)

Last edited by AKSN74 (2014-05-20 01:58:52)

Destroy

Member

Registered: 2014-05-05

Posts: 10

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

This post is very funny and exciting!!! Thanks again to all, specially to nbhs and AW

Now I'm trying to pass my SATA port Marvell at 08:00.0
lspci -tv

-[0000:00]-+-00.0  Intel Corporation Xeon E3-1200 v2/3rd Gen Core processor DRAM Controller
           +-01.0-[01]----00.0  NVIDIA Corporation G73 [GeForce 7600 GT]
           +-02.0  Intel Corporation Xeon E3-1200 v2/3rd Gen Core processor Graphics Controller
           +-14.0  Intel Corporation 7 Series/C210 Series Chipset Family USB xHCI Host Controller
           +-16.0  Intel Corporation 7 Series/C210 Series Chipset Family MEI Controller #1
           +-19.0  Intel Corporation 82579V Gigabit Network Connection
           +-1a.0  Intel Corporation 7 Series/C210 Series Chipset Family USB Enhanced Host Controller #2
           +-1b.0  Intel Corporation 7 Series/C210 Series Chipset Family High Definition Audio Controller
           +-1c.0-[02]--
           +-1c.1-[03]----00.0  Marvell Technology Group Ltd. 88SE9172 SATA 6Gb/s Controller
           +-1c.3-[04]----00.0  Atheros Communications Inc. AR9462 Wireless Network Adapter
           +-1c.5-[05-06]----00.0-[06]----01.0  VIA Technologies, Inc. VT6306/7/8 [Fire II(M)] IEEE 1394 OHCI Controller
           +-1c.6-[07]----00.0  Qualcomm Atheros AR8151 v2.0 Gigabit Ethernet
           +-1c.7-[08]----00.0  Marvell Technology Group Ltd. 88SE9172 SATA 6Gb/s Controller
           +-1d.0  Intel Corporation 7 Series/C210 Series Chipset Family USB Enhanced Host Controller #1
           +-1f.0  Intel Corporation Z77 Express Chipset LPC Controller
           +-1f.2  Intel Corporation 7 Series/C210 Series Chipset Family 6-port SATA Controller [AHCI mode]
           \-1f.3  Intel Corporation 7 Series/C210 Series Chipset Family SMBus Controller

but it's in a group

lsgroup

### Group 8 ###
    00:1c.0 PCI bridge: Intel Corporation 7 Series/C210 Series Chipset Family PCI Express Root Port 1 (rev c4)
    00:1c.1 PCI bridge: Intel Corporation 7 Series/C210 Series Chipset Family PCI Express Root Port 2 (rev c4)
    00:1c.3 PCI bridge: Intel Corporation 7 Series/C210 Series Chipset Family PCI Express Root Port 4 (rev c4)
    00:1c.5 PCI bridge: Intel Corporation 82801 PCI Bridge (rev c4)
    00:1c.6 PCI bridge: Intel Corporation 7 Series/C210 Series Chipset Family PCI Express Root Port 7 (rev c4)
    00:1c.7 PCI bridge: Intel Corporation 7 Series/C210 Series Chipset Family PCI Express Root Port 8 (rev c4)
    03:00.0 SATA controller: Marvell Technology Group Ltd. 88SE9172 SATA 6Gb/s Controller (rev 11)
    04:00.0 Network controller: Atheros Communications Inc. AR9462 Wireless Network Adapter (rev 01)
    05:00.0 PCI bridge: Intel Corporation 82801 PCI Bridge (rev 30)
    06:01.0 FireWire (IEEE 1394): VIA Technologies, Inc. VT6306/7/8 [Fire II(M)] IEEE 1394 OHCI Controller (rev c0)
    07:00.0 Ethernet controller: Qualcomm Atheros AR8151 v2.0 Gigabit Ethernet (rev c0)
    08:00.0 SATA controller: Marvell Technology Group Ltd. 88SE9172 SATA 6Gb/s Controller (rev 11)

I have been reading and it says  I must apply the path to override ACS but that is from march 2013 (https://lkml.org/lkml/2013/5/30/513) , also I found the v2 from February 2014 (https://lkml.org/lkml/2014/2/3/487) but it does haven the kernel parameters.

the questions are:
are for the same?
then I must apply the v2?
it's included in kernel 3.15?

aw

Member

Registered: 2013-10-04

Posts: 921

Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

I have been reading and it says  I must apply the path to override ACS but that is from march 2013 (https://lkml.org/lkml/2013/5/30/513) , also I found the v2 from February 2014 (https://lkml.org/lkml/2014/2/3/487) but it does haven the kernel parameters.

the questions are:
are for the same?
then I must apply the v2?
it's included in kernel 3.15?

Upgrade to 3.15 and you won't need the ACS override patch.  That doesn't mean that the Marvell card is going to work, because it's terribly broken (remember those DMA patches you've applied?)

---

http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

AKSN74

Member

From: Taichung, Taiwan

Registered: 2014-02-18

Posts: 52

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Upgrade to 3.15 and you won't need the ACS override patch.  That doesn't mean that the Marvell card is going to work, because it's terribly broken (remember those DMA patches you've applied?)

Hi, aw.

I found that not only VM hangs when reboot with 2 VMs working together, but also only 1 VM working.

And I tried to shutdown, not reboot, it can back to tty successfully.
But when I try to start again with same command, it got another error message.

qemu-system-x86_64: vfio-pci: Cannot read device rom at 0000:03:00.0
Device option ROM contents are probably invalid (check dmesg).
Skip option ROM probe with rombar=0, or load from file with romfile=

It's may a reason why it hangs when reboot.
But I found that when  I reboot VM while guest OS is Windows 7, it can reboot successfully some times.
So it is very strange for this problem.......

I'll try to use 3.14.4 kernel and see it still a same problem or not.

Last edited by AKSN74 (2014-05-20 13:35:14)

mbroemme

Member

From: Cologne

Registered: 2014-02-05

Posts: 40

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Hello.
Is it possible to passthrough radeon card and use another radeon in host?
How don't blacklist radeon driver and get it works.

Looks like I just have to use psi-stub for guest card and don't blacklist radeon driver.

If you are on Arch Linux you can make it even more simple. I use a 7870 for my Linux system and a R9 290X for my VM. In Arch Linux just do the following:

#1 Create file '/etc/modules-load.d/vfio-pci.conf' with content:
vfio-pci

#2 Create file '/etc/modprobe.d/vfio-pci.conf' with content:
install vfio-pci /usr/bin/modprobe --ignore-install vfio-pci ; echo VENDOR DEVICE > /sys/bus/pci/drivers/vfio-pci/new_id

Just replace VENDOR and DEVICE with your appropriate IDs from 'lspci -n' output. Also this can be extended with any other device, if they are bound already (most probably USB devices) you can unbind them with the modprobe configuration.

Chrissi

Member

Registered: 2014-05-20

Posts: 5

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Hey guys!
I followed the instruction step by step, but when i try to start the vm with

qemu-system-x86_64 -enable-kvm -M q35 -m 1024 -cpu host \
-smp 6,sockets=1,cores=6,threads=1 \
-bios /usr/share/qemu/bios.bin -vga none \
-device ioh3420,bus=pcie.0,addr=1c.0,multifunction=on,port=1,chassis=1,id=root.$
-device vfio-pci,host=01:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on \
-device vfio-pci,host=01:00.1,bus=root.1,addr=00.1

i get the following error:

qemu-system-x86_64: -device vfio-pci,host=01:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on: vfio: error no iommu_group for device
qemu-system-x86_64: -device vfio-pci,host=01:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on: Device initialization failed.
qemu-system-x86_64: -device vfio-pci,host=01:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on: Device 'vfio-pci' could not be initialized

The interesting thing is I got the same error output on a ubuntu based machine, so I guess its something hardware related or something I just don't get.

dmesg | grep -e DMAR -e IOMMU

[    0.000000] ACPI: DMAR 000000007c1789f8 0000B8 (v01 INTEL      HSW  00000001 INTL 00000001)
[    0.016013] dmar: IOMMU 0: reg_base_addr fed90000 ver 1:0 cap c0000020660462 ecap f0101a
[    0.016017] dmar: IOMMU 1: reg_base_addr fed91000 ver 1:0 cap d2008020660462 ecap f010da
[    0.016082] IOAPIC id 2 under DRHD base  0xfed91000 IOMMU 1

dmesg | grep vfio

[    2.966707] vfio-pci: probe of 0000:01:00.0 failed with error -22 ... for every card

full dmesg on pastebin

my system:
i7 4770
asrock extreme 6 (vt-d in bios enabled)
2x 290x / 1x290 (my good old coin mining cards)
grub boots with  intel_iommu=on

i am using qemu 2.0.0.3 (installed via pacman) and the mainline kernel with patches by OP.
Since I want to pass all my cards, I just blacklisted "radeon" in modprobe.d.

would be great if someone could give me a little advice how i may proceed with this. Thanks in advance !

aw

Member

Registered: 2013-10-04

Posts: 921

Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

grub boots with  intel_iommu=on

Your pastebin says otherwise

[    0.000000] Command line: BOOT_IMAGE=/boot/vmlinuz-linux root=UUID=e60e5956-3c30-4b5d-a86b-5272a29bc246 rw quiet

---

http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Chrissi

Member

Registered: 2014-05-20

Posts: 5

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

grub boots with  intel_iommu=on

Your pastebin says otherwise

[    0.000000] Command line: BOOT_IMAGE=/boot/vmlinuz-linux root=UUID=e60e5956-3c30-4b5d-a86b-5272a29bc246 rw quiet

sometimes there is a very easy way but we just don't see it. thank you.

AKSN74

Member

From: Taichung, Taiwan

Registered: 2014-02-18

Posts: 52

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Also, I got another problem yesterday.

Because my target is build VM with GTX480, and want to run CUDA with GTX480 in VM.

As guest OS is Ubuntu 12.04-4, first I install NVIDIA Driver without a problem.
Then I install CUDA SDK 6.0, and still no problem. And it can compile the sample application.

But when I tried to run deviceQueryDrv, it got error.

CUDA_ERROR_INVALID_DEVICE (device specified is not a valid CUDA device)

So I downgrade my SDK version to 5.5, because GTX480 is old GPU, some functions in 6.0 may not support.
When I installed 5.5, and compile ver 5.5 samples, still got a same error while I ran deviceQueryDrv.

But before that, I can run CUDA while guest OS is Windows 7. And it can get matrix multiply result.
Maybe it's driver and SDK compatible problem (if it is VFIO problem, graphic driver will got issue first. Neither NVIDIA offical driver nor Nouveau.)

Also I found that if I using Ubuntu 12.04 as guest OS, it always came out these message:

[81.114285] kvm [1993]: vcpu0 unhandled wrmsr: 0xxxx data 0

And here is my dmesg.
dmesg download

Hope it can solve reboot hangs when I using Ubuntu as guest OS.

Slabity

Member

Registered: 2013-12-29

Posts: 40

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Hey all, so I'm having some trouble. I feel like I'm close to finishing this, but the qemu test does not display video though my passthrough'd card. And in my Windows guest OS, it disables the card because it's "not functioning correctly":

https://i.imgur.com/inKbcua.png

Here's my dmesg | grep pci-stub:

[    0.000000] Command line: root=/dev/sda2 rw initrd=/initramfs-linux-mainline.img pcie_acs_override=downstream intel_iommu=on pci-stub.ids=10de:1004,10de:0e1a
[    0.000000] Kernel command line: root=/dev/sda2 rw initrd=/initramfs-linux-mainline.img pcie_acs_override=downstream intel_iommu=on pci-stub.ids=10de:1004,10de:0e1a
[    0.438320] pci-stub: add 10DE:1004 sub=FFFFFFFF:FFFFFFFF cls=00000000/00000000
[    0.438326] pci-stub 0000:02:00.0: claimed by stub
[    0.438329] pci-stub: add 10DE:0E1A sub=FFFFFFFF:FFFFFFFF cls=00000000/00000000
[    0.438333] pci-stub 0000:02:00.1: claimed by stub

Anyone got any ideas?

Last edited by Slabity (2014-05-22 06:12:08)

AKSN74

Member

From: Taichung, Taiwan

Registered: 2014-02-18

Posts: 52

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Hey all, so I'm having some trouble. I feel like I'm close to finishing this, but the qemu test does not display video though my passthrough'd card. And in my Windows guest OS, it disables the card because it's "not functioning correctly":

https://i.imgur.com/inKbcua.png

Here's my dmesg | grep pci-stub:

[    0.000000] Command line: root=/dev/sda2 rw initrd=/initramfs-linux-mainline.img pcie_acs_override=downstream intel_iommu=on pci-stub.ids=10de:1004,10de:0e1a
[    0.000000] Kernel command line: root=/dev/sda2 rw initrd=/initramfs-linux-mainline.img pcie_acs_override=downstream intel_iommu=on pci-stub.ids=10de:1004,10de:0e1a
[    0.438320] pci-stub: add 10DE:1004 sub=FFFFFFFF:FFFFFFFF cls=00000000/00000000
[    0.438326] pci-stub 0000:02:00.0: claimed by stub
[    0.438329] pci-stub: add 10DE:0E1A sub=FFFFFFFF:FFFFFFFF cls=00000000/00000000
[    0.438333] pci-stub 0000:02:00.1: claimed by stub

Anyone got any ideas?

Looks like you're using virtual display right?
You need to disable virtual display, and plug a cable on your graphic card that you mounted.
Also, can you show us your launch command?

Last edited by AKSN74 (2014-05-21 11:27:38)

Slabity

Member

Registered: 2013-12-29

Posts: 40

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Looks like you're using virtual display right?
You need to disable virtual display, and plug a cable on your graphic card that you mounted.
Also, can you show us your launch command?

Well, if I run the example qemu script:

qemu-system-x86_64 -enable-kvm -M q35 -m 1024 -cpu host \
-smp 6,sockets=1,cores=6,threads=1 \
-bios /usr/share/qemu/bios.bin -vga none \
-device ioh3420,bus=pcie.0,addr=1c.0,multifunction=on,port=1,chassis=1,id=root.1 \
-device vfio-pci,host=02:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on \
-device vfio-pci,host=02:00.1,bus=root.1,addr=00.1

In the above case, a black window appears with nothing in it. Nothing appears on my monitor attached through the passthrough'd card.

I am running the Windows VM through Virt-manager. Here's my configuration file:

http://pastebin.com/zJ4CbqZc

I try to remove the <video> section, but it seems to add itself back each time I run virt-manager.

EDIT: It seems the qemu script creates a black window, but if I type anything or resize the window, the window displays a console with the following:

compat_monitor0 console
QEMU 2.0.0 monitor - type 'help' for more information'
(qemu)

How is that possible if the script disables vga?

Last edited by Slabity (2014-05-21 16:32:15)

nbhs

Member

From: Montevideo, Uruguay

Registered: 2013-05-02

Posts: 402

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Looks like you're using virtual display right?
You need to disable virtual display, and plug a cable on your graphic card that you mounted.
Also, can you show us your launch command?

Well, if I run the example qemu script:

qemu-system-x86_64 -enable-kvm -M q35 -m 1024 -cpu host \
-smp 6,sockets=1,cores=6,threads=1 \
-bios /usr/share/qemu/bios.bin -vga none \
-device ioh3420,bus=pcie.0,addr=1c.0,multifunction=on,port=1,chassis=1,id=root.1 \
-device vfio-pci,host=02:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on \
-device vfio-pci,host=02:00.1,bus=root.1,addr=00.1

In the above case, a black window appears with nothing in it. Nothing appears on my monitor attached through the passthrough'd card.

I am running the Windows VM through Virt-manager. Here's my configuration file:

http://pastebin.com/zJ4CbqZc

I try to remove the <video> section, but it seems to add itself back each time I run virt-manager.

EDIT: It seems the qemu script creates a black window, but if I type anything or resize the window, the window displays a console with the following:

compat_monitor0 console
QEMU 2.0.0 monitor - type 'help' for more information'
(qemu)

How is that possible if the script disables vga?

Are you using the i915 vga arbiter patch?

Slabity

Member

Registered: 2013-12-29

Posts: 40

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Are you using the i915 vga arbiter patch?

I believe so. I'm using the linux-mainline from the first post. The makepkg command automatically patches it, right? Then I just used pacman -U to install the kernel and change my boot command to use that kernel.

thehighhat

Member

Registered: 2014-05-12

Posts: 5

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Upgrade to 3.15 and you won't need the ACS override patch.  That doesn't mean that the Marvell card is going to work, because it's terribly broken (remember those DMA patches you've applied?)

Hi, aw.

I found that not only VM hangs when reboot with 2 VMs working together, but also only 1 VM working.

And I tried to shutdown, not reboot, it can back to tty successfully.
But when I try to start again with same command, it got another error message.

qemu-system-x86_64: vfio-pci: Cannot read device rom at 0000:03:00.0
Device option ROM contents are probably invalid (check dmesg).
Skip option ROM probe with rombar=0, or load from file with romfile=

It's may a reason why it hangs when reboot.
But I found that when  I reboot VM while guest OS is Windows 7, it can reboot successfully some times.
So it is very strange for this problem.......

I'll try to use 3.14.4 kernel and see it still a same problem or not.

Same problem here.  I cannot reboot or reissue the qemu command without this error.  Do I need to undo the vfio-bind stuffs?

esmth

Member

Registered: 2014-05-22

Posts: 4

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Hey guys, first post here. I'm having some trouble trying to get this to work correctly. My hardware supports IOMMU and it is enabled in the BIOS, and the IOMMU kernel driver loads. My system specs are here: AMD A10-7850k, Gigabyte F2A88X-D3H. I'm using the integrated graphics as the host, and i'm passing though a reference R9-290x. I also have older GPUs i've tested that cause the same issue.  The problem is that when I run the command to start the VM, the colors on the host's monitor become distorted, and the 2nd monitor connected to the passed though GPU turns on, and sometimes gets passed the seabios screen and onto the  OS and then freezes the host and guest, sometimes it doesn't, and the both the guest and host freeze. When the PC freezes, it seems like everything connected to the PCI bus is down, but the host is still running (the terminal cursor on the host continues to blink), while the guest also seems to be frozen.

I use this command to start the guest:

sudo su
bin/vfio-bind 0000:01:00:0 0000:01:00.1
./qemu-system-x86_64 -enable-kvm -M q35 -m 1024 -cpu host -smp 6,sockets=1,cores=6,threads=1 -bios /home/esmth/src/seabios/out/bios.bin -vga std -device ioh3420,bus=pcie.0,addr=1c.0,multifunction=on,port=1,chassis=1,id=root.1 -device vfio-pci,host=01:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on -device vfio-pci,host=01:00.1,bus=root.1,addr=00.1 -drive file=/home/esmth/image.img,id=disk,format=raw -device ide-hd,bus=ide.0,drive=disk -drive file=/home/esmth/tmp/windows.iso,id=isocd -device ide-cd,bus=ide.1,drive=isocd -usb -usbdevice host:046d:c52b

if anyone can help, it'd be appreciated, thanks

edit: i git clone'd both seabios and qemu and compiled them , and am running kernel 3.15rc5 with the vfio and iommu options enabled

Last edited by esmth (2014-05-22 12:59:21)

Destroy

Member

Registered: 2014-05-05

Posts: 10

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Upgrade to 3.15 and you won't need the ACS override patch.  That doesn't mean that the Marvell card is going to work, because it's terribly broken (remember those DMA patches you've applied?)

1.- I've compiled 3.15-rc5 but after boot, it gave me error with the Marvell card as you say.
2.- I've tried to applied the same patch for the Marvell card but it gave me one error
cat intel-iommu.c.rej

--- drivers/iommu/intel-iommu.c	
+++ drivers/iommu/intel-iommu.c	
@@ -3878,6 +3990,9 @@
 			iommu_disable_dev_iotlb(info);
 			iommu_detach_dev(iommu, info->bus, info->devfn);
 			iommu_detach_dependent_devices(iommu, pdev);
+			quirk_unmap_multi_requesters(pdev,
+						pci_multi_requesters(pdev));
+			quirk_unmap_requester_id(pdev);
 			free_devinfo_mem(info);
 
 			spin_lock_irqsave(&device_domain_lock, flags);

so I manually edit the file and added the lines closes to line 4197 and compiled it but I had an error

  CC      drivers/iommu/intel-iommu.o
drivers/iommu/intel-iommu.c: In function 'quirk_unmap_multi_requesters':
drivers/iommu/intel-iommu.c:1850:7: warning: passing argument 1 of 'device_to_iommu' makes pointer from integer without a cast [enabled by default]
       pdev->bus->number, pdev->devfn);
       ^
drivers/iommu/intel-iommu.c:667:28: note: expected 'struct device *' but argument is of type 'int'
 static struct intel_iommu *device_to_iommu(struct device *dev, u8 *bus, u8 *devfn)
                            ^
drivers/iommu/intel-iommu.c:1850:7: warning: passing argument 2 of 'device_to_iommu' makes pointer from integer without a cast [enabled by default]
       pdev->bus->number, pdev->devfn);
       ^
drivers/iommu/intel-iommu.c:667:28: note: expected 'u8 *' but argument is of type 'unsigned char'
 static struct intel_iommu *device_to_iommu(struct device *dev, u8 *bus, u8 *devfn)
                            ^
drivers/iommu/intel-iommu.c:1850:7: warning: passing argument 3 of 'device_to_iommu' makes pointer from integer without a cast [enabled by default]
       pdev->bus->number, pdev->devfn);
       ^
drivers/iommu/intel-iommu.c:667:28: note: expected 'u8 *' but argument is of type 'unsigned int'
 static struct intel_iommu *device_to_iommu(struct device *dev, u8 *bus, u8 *devfn)
                            ^
drivers/iommu/intel-iommu.c: In function 'quirk_map_multi_requester_ids':
drivers/iommu/intel-iommu.c:1888:6: warning: passing argument 2 of 'domain_context_mapping_one' makes pointer from integer without a cast [enabled by default]
      translation);
      ^
drivers/iommu/intel-iommu.c:1718:12: note: expected 'struct intel_iommu *' but argument is of type 'int'
 static int domain_context_mapping_one(struct dmar_domain *domain,
            ^
drivers/iommu/intel-iommu.c: In function 'quirk_unmap_requester_id':
drivers/iommu/intel-iommu.c:1907:7: warning: passing argument 1 of 'device_to_iommu' makes pointer from integer without a cast [enabled by default]
       pdev->bus->number, pdev->devfn);
       ^
drivers/iommu/intel-iommu.c:667:28: note: expected 'struct device *' but argument is of type 'int'
 static struct intel_iommu *device_to_iommu(struct device *dev, u8 *bus, u8 *devfn)
                            ^
drivers/iommu/intel-iommu.c:1907:7: warning: passing argument 2 of 'device_to_iommu' makes pointer from integer without a cast [enabled by default]
       pdev->bus->number, pdev->devfn);
       ^
drivers/iommu/intel-iommu.c:667:28: note: expected 'u8 *' but argument is of type 'unsigned char'
 static struct intel_iommu *device_to_iommu(struct device *dev, u8 *bus, u8 *devfn)
                            ^
drivers/iommu/intel-iommu.c:1907:7: warning: passing argument 3 of 'device_to_iommu' makes pointer from integer without a cast [enabled by default]
       pdev->bus->number, pdev->devfn);
       ^
drivers/iommu/intel-iommu.c:667:28: note: expected 'u8 *' but argument is of type 'unsigned int'
 static struct intel_iommu *device_to_iommu(struct device *dev, u8 *bus, u8 *devfn)
                            ^
drivers/iommu/intel-iommu.c: In function 'quirk_map_requester_id':
drivers/iommu/intel-iommu.c:1933:4: warning: passing argument 2 of 'domain_context_mapping_one' makes pointer from integer without a cast [enabled by default]
    translation);
    ^
drivers/iommu/intel-iommu.c:1718:12: note: expected 'struct intel_iommu *' but argument is of type 'int'
 static int domain_context_mapping_one(struct dmar_domain *domain,
            ^
drivers/iommu/intel-iommu.c: In function 'domain_remove_one_dev_info':
drivers/iommu/intel-iommu.c:4197:33: error: 'pdev' undeclared (first use in this function)
    quirk_unmap_multi_requesters(pdev,
                                 ^
drivers/iommu/intel-iommu.c:4197:33: note: each undeclared identifier is reported only once for each function it appears in
make[2]: *** [drivers/iommu/intel-iommu.o] Error 1
make[1]: *** [drivers/iommu] Error 2
make: *** [drivers] Error 2

So I'll wait for more of your magic AW
Thanks again for your harder work

Slabity

Member

Registered: 2013-12-29

Posts: 40

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

So I've completely reset my system from scratch to see if I could fix my issue. I still do not get any output on my passthrough'd VGA card.

Here is the qemu command I'm running:

qemu-system-x86_64 -enable-kvm -M q35 -m 1024 -cpu host \
-smp 6,sockets=1,cores=6,threads=1 \
-bios /usr/share/qemu/bios.bin -vga none \
-device ioh3420,bus=pcie.0,addr=1c.0,multifunction=on,port=1,chassis=1,id=root.1 \
-device vfio-pci,host=02:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on \
-device vfio-pci,host=02:00.1,bus=root.1,addr=00.1

The result is a blank window with nothing being displayed on the monitor.

My host is running off of a Radeon HD 6870 with open-source drivers. My passthrough card is a Geforce 780:

$ lspci -v

02:00.0 VGA compatible controller: NVIDIA Corporation GK110 [GeForce GTX 780] (rev a1) (prog-if 00 [VGA controller])
	Subsystem: eVga.com. Corp. Device 2784
	Flags: fast devsel, IRQ 17
	Memory at ee000000 (32-bit, non-prefetchable) [size=16M]
	Memory at e0000000 (64-bit, prefetchable) [size=128M]
	Memory at e8000000 (64-bit, prefetchable) [size=32M]
	I/O ports at d000 [size=128]
	Expansion ROM at ef000000 [disabled] [size=512K]
	Capabilities: <access denied>
	Kernel driver in use: vfio-pci
	Kernel modules: nouveau

02:00.1 Audio device: NVIDIA Corporation GK110 HDMI Audio (rev a1)
	Subsystem: eVga.com. Corp. Device 2784
	Flags: fast devsel, IRQ 18
	Memory at ef080000 (32-bit, non-prefetchable) [size=16K]
	Capabilities: <access denied>
	Kernel driver in use: vfio-pci
	Kernel modules: snd_hda_intel

$ lspci -n

01:00.0 0300: 1002:6738
01:00.1 0403: 1002:aa88
02:00.0 0300: 10de:1004 (rev a1)
02:00.1 0403: 10de:0e1a (rev a1)

My kernel's command line is:

root=/dev/sda2 rw initrd=/initramfs-linux-mainline.img pci-stub.ids=10de:1004,10de:0e1a intel_iommu=on pcie_acs_override=downstream

I see some other people had similar issues but didn't report a solution. Is there something fundamental that I'm missing here?

Last edited by Slabity (2014-05-22 06:18:33)

mbroemme

Member

From: Cologne

Registered: 2014-02-05

Posts: 40

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

So I've completely reset my system from scratch to see if I could fix my issue. I still do not get any output on my passthrough'd VGA card.

Here is the qemu command I'm running:

qemu-system-x86_64 -enable-kvm -M q35 -m 1024 -cpu host \
-smp 6,sockets=1,cores=6,threads=1 \
-bios /usr/share/qemu/bios.bin -vga none \
-device ioh3420,bus=pcie.0,addr=1c.0,multifunction=on,port=1,chassis=1,id=root.1 \
-device vfio-pci,host=02:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on \
-device vfio-pci,host=02:00.1,bus=root.1,addr=00.1

The result is a blank window with nothing being displayed on the monitor.

My host is running off of a Radeon HD 6870 with open-source drivers. My passthrough card is a Geforce 780:

I had similar troubles in the past with my AMD R9 290X inside guest and what helped me to get it fairly stable (at least for the first boot of VM as AMD Radeon cards are known to not support proper reset) was to boot my host with VGA enabled on card used for passthrough. So I changed in BIOS that my R9 290X will show the host BIOS post, boot the system and after binding this card to VFIO during boot it loads radeon driver for my second card (HD 7870) which works now as my Linux primary card. After this change my VM always allows me to use x-vga=on and shows SeaBIOS screen.

If I boot my system with second HD 7870 enabled as primary device in BIOS I was not able to use x-vga=on ever for the R9 290X.

mbroemme

Member

From: Cologne

Registered: 2014-02-05

Posts: 40

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

sudo su
bin/vfio-bind 0000:01:00:0 0000:01:00.1
./qemu-system-x86_64 -enable-kvm -M q35 -m 1024 -cpu host -smp 6,sockets=1,cores=6,threads=1 -bios /home/esmth/src/seabios/out/bios.bin -vga std -device ioh3420,bus=pcie.0,addr=1c.0,multifunction=on,port=1,chassis=1,id=root.1 -device vfio-pci,host=01:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on -device vfio-pci,host=01:00.1,bus=root.1,addr=00.1 -drive file=/home/esmth/image.img,id=disk,format=raw -device ide-hd,bus=ide.0,drive=disk -drive file=/home/esmth/tmp/Windows_8.1_Pro_X64_Activated_Final/Windows_8.1_Pro_X64_Activated.iso,id=isocd -device ide-cd,bus=ide.1,drive=isocd -usb -usbdevice host:046d:c52b

if anyone can help, it'd be appreciated, thanks

Could you try to use it with

-vga none

Namelles_One

Member

Registered: 2014-05-18

Posts: 8

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

So, now please help me another way)

With VGA - all is good, but audio  - not(

In host Linus (I use ALSA) - all is good and best quality, but in Windows guest - quality is ugly, sound is like a frog - "He-quauq-llo!" 0_o

Where is problem may be?

P.S. I use

QEMU_ALSA_DAC_BUFFER_SIZE=512 QEMU_ALSA_DAC_PERIOD_SIZE=170 QEMU_AUDIO_DRV=alsa

and

-device ich9-intel-hda,bus=pcie.0,addr=1b.0,id=sound0 -device hda-duplex,id=sound0-codec0,bus=sound0.0,cad=0 

IyaFR

Member

Registered: 2014-05-22

Posts: 1

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

It's possible to do this vga-passthrough with 1 gpu. I boot in multi-user.target and  i have modify my grub for enable the serial mode.
in grub.cfg:
terminal_input serial
terminal_output serial

And in the menuentry:
linux    .... quiet  APPEND console=ttyS0

After that, i run my vm with a ssh access.

siddharta

Member

Registered: 2014-05-03

Posts: 30

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Hi,

I've had partial success so far with passing through an AMD 6450 on an Asrock H61 and i5-2400. Partial success meaning I can get it to work invoking qemu directly but I can't get it working when using libvirt. There is no other discrete graphics present, the integrated graphics is for the host.

The kernel is 3.14 recompiled with vfio flags set and the i915 VGA arbiter patch applied, qemu version 2.0.0 and seabios version 1.7.4

lsgroups.sh shows

### Group 1 ###
    00:01.0 PCI bridge: Intel Corporation Xeon E3-1200/2nd Generation Core Processor Family PCI Express Root Port (rev 09)
    01:00.0 VGA compatible controller: Advanced Micro Devices, Inc. [AMD/ATI] Caicos [Radeon HD 6450/7450/8450]
    01:00.1 Audio device: Advanced Micro Devices, Inc. [AMD/ATI] Caicos HDMI Audio [Radeon HD 6400 Series]

   
hwinfo --gfxcard shows the 6450 (which is not the primary card) bound to vfio-pci (done through vfio-bind.sh 0000:01:00.0 0000:01:00.1)

20: PCI 100.0: 0300 VGA compatible controller (VGA)
  SysFS ID: /devices/pci0000:00/0000:00:01.0/0000:01:00.0
  SysFS BusID: 0000:01:00.0
  Hardware Class: graphics card
  Model: "ATI VGA compatible controller"
  Vendor: pci 0x1002 "ATI Technologies Inc"
  Device: pci 0x6779
  SubVendor: pci 0x1043 "ASUSTeK Computer Inc."
  SubDevice: pci 0x047b
  Driver: "vfio-pci"
  Driver Modules: "vfio_pci"
  Memory Range: 0xe0000000-0xefffffff (ro,non-prefetchable,disabled)
  Memory Range: 0xf7c20000-0xf7c3ffff (rw,non-prefetchable,disabled)
  I/O Ports: 0xe000-0xefff (rw,disabled)
  Memory Range: 0xf7c00000-0xf7c1ffff (ro,non-prefetchable,disabled)
  IRQ: 11 (no events)
  Module Alias: "pci:v00001002d00006779sv00001043sd0000047Bbc03sc00i00"
  Driver Info #0:
    Driver Status: radeon is not active
    Driver Activation Cmd: "modprobe radeon"
  Config Status: cfg=no, avail=yes, need=no, active=unknown
  Attached to: #9 (PCI bridge)

Primary display adapter: #10

ls -l /sys/bus/pci/drivers/vfio-pci/ looks well

total 0
lrwxrwxrwx 1 root root    0 May 22 11:14 0000:01:00.0 -> ../../../../devices/pci0000:00/0000:00:01.0/0000:01:00.0
lrwxrwxrwx 1 root root    0 May 22 11:14 0000:01:00.1 -> ../../../../devices/pci0000:00/0000:00:01.0/0000:01:00.1
--w------- 1 root root 4096 May 22 11:14 bind
lrwxrwxrwx 1 root root    0 May 22 11:14 module -> ../../../../module/vfio_pci
--w------- 1 root root 4096 May 22 11:14 new_id
--w------- 1 root root 4096 May 22 11:14 remove_id
--w------- 1 root root 4096 May 22 11:14 uevent
--w------- 1 root root 4096 May 22 11:14 unbind

 
Invoking qemu as

qemu-system-x86_64 -enable-kvm -M q35 -m 512 \
    -smp 1,sockets=1,cores=1,threads=1 \
    -bios /usr/share/qemu/bios.bin -vga none \
    -device ioh3420,bus=pcie.0,addr=1c.0,multifunction=on,port=1,chassis=1,id=root.1 \
    -device vfio-pci,host=01:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on \
    -device vfio-pci,host=01:00.1,bus=root.1,addr=00.1 

works (showing seabios output on the display attached to the 6450) with dmesg | grep -e vfio showing

[  526.686460] vfio-pci 0000:01:00.0: enabling device (0000 -> 0003)

but! When I attempt to start a libvirt managed guest defined from the following xml (note that here I'm not including the HDMI audio device)

<domain type='kvm' xmlns:qemu='http://libvirt.org/schemas/domain/qemu/1.0'>
  <name>test</name>
  <uuid>c4658435-dfb4-4c8d-afc2-f9dfad2e10f0</uuid>
  <memory unit='KiB'>524288</memory>
  <currentMemory unit='KiB'>524288</currentMemory>
  <vcpu placement='static'>1</vcpu>
  <os>
    <type arch='x86_64' machine='q35'>hvm</type>
    <loader>/usr/share/qemu/bios.bin</loader>
  </os>
  <features>
    <acpi/>
  </features>
  <cpu>
    <topology sockets='1' cores='1' threads='1'/>
  </cpu>
  <clock offset='utc'/>
  <on_poweroff>destroy</on_poweroff>
  <on_reboot>restart</on_reboot>
  <on_crash>destroy</on_crash>
  <devices>
    <emulator>/usr/bin/qemu-system-x86_64</emulator>
    <controller type='sata' index='0'/>
    <controller type='pci' index='0' model='pcie-root'/>
    <controller type='pci' index='1' model='dmi-to-pci-bridge'/>
    <controller type='pci' index='2' model='pci-bridge'/>
    <input type='mouse' bus='ps2'/>
    <input type='keyboard' bus='ps2'/>
    <memballoon model='virtio'>
      <address type='pci' domain='0x0000' bus='0x02' slot='0x05' function='0x0'/>
    </memballoon>
  </devices>
  <qemu:commandline>
    <qemu:arg value='-device'/>
    <qemu:arg value='ioh3420,bus=pcie.0,addr=1c.0,multifunction=on,port=1,chassis=1,id=root.1'/>
    <qemu:arg value='-device'/>
    <qemu:arg value='vfio-pci,host=01:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on'/>
  </qemu:commandline>
</domain>

the error output is

error: Failed to start domain test
error: internal error: process exited while connecting to monitor: qemu-system-x86_64: -device vfio-pci,host=01:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on: vfio_dma_map(0x7fce2c9e4b00, 0x0, 0x20000000, 0x7fcdf0000000) = -12 (Cannot allocate memory)
qemu-system-x86_64: -device vfio-pci,host=01:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on: vfio: memory listener initialization failed for container
qemu-system-x86_64: -device vfio-pci,host=01:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on: vfio: failed to setup container for group 1
qemu-system-x86_64: -device vfio-pci,host=01:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on: vfio: failed to get group 1
qemu-system-x86_64: -device vfio-pci,host=01:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on: Device initialization failed.
qemu-system-x86_64: -device vfio-pci,host=01:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on: Device 'vfio-pci' could not be initialized

dmesg | grep -e vfio then shows

[302278.633004] vfio_pin_pages: RLIMIT_MEMLOCK (65536) exceeded
[302278.633015] vfio_pin_pages: RLIMIT_MEMLOCK (65536) exceeded

VT-d is enabled as per dmesg | grep -e DMAR -e IOMMU output

[    0.000000] ACPI: DMAR 00000000be148e78 0000B0 (v01 INTEL      SNB  00000001 INTL 00000001)
[    0.000000] Intel-IOMMU: enabled
[    0.018580] dmar: IOMMU 0: reg_base_addr fed90000 ver 1:0 cap c0000020e60262 ecap f0101a
[    0.018585] dmar: IOMMU 1: reg_base_addr fed91000 ver 1:0 cap c9008020660262 ecap f0105a
[    0.018657] IOAPIC id 2 under DRHD base  0xfed91000 IOMMU 1
[    0.485319] DMAR: No ATSR found
[    0.485340] IOMMU 0 0xfed90000: using Queued invalidation
[    0.485341] IOMMU 1 0xfed91000: using Queued invalidation
[    0.485342] IOMMU: Setting RMRR:
[    0.485352] IOMMU: Setting identity map for device 0000:00:02.0 [0xbf800000 - 0xcf9fffff]
[    0.486624] IOMMU: Setting identity map for device 0000:00:1d.0 [0xbdf8d000 - 0xbdfb9fff]
[    0.486642] IOMMU: Setting identity map for device 0000:00:1a.0 [0xbdf8d000 - 0xbdfb9fff]
[    0.486652] IOMMU: Prepare 0-16MiB unity mapping for LPC
[    0.486659] IOMMU: Setting identity map for device 0000:00:1f.0 [0x0 - 0xffffff]
[    1.157895] [drm] DMAR active, disabling use of stolen memory

Permissions for /dev/vfio/ are

crw-rw-rw- 1 root root 251,   0 May 18 22:48 1
crw-rw-rw- 1 root root  10, 196 May 18 22:45 vfio

qemu is running its processes as root and the acl in /etc/libvirt/qemu.conf is

cgroup_device_acl = [
    "/dev/null", "/dev/full", "/dev/zero",
    "/dev/random", "/dev/urandom",
    "/dev/ptmx", "/dev/kvm", "/dev/kqemu",
    "/dev/rtc","/dev/hpet", "/dev/vfio/vfio",
    "/dev/vfio/1"
]

I'm researched this but haven't found a way forward. Any pointers are very much appreciated. Thanks to the Arch community (in particular you who have provided mountains of insight here!) for all the work done so far, this has all been highly educational!