You are not logged in.

#126 2013-05-30 18:54:58

nbhs
Member
From: Montevideo, Uruguay
Registered: 2013-05-02
Posts: 402

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Diggo wrote:
nbhs wrote:

The redhat dev recommends using q35 chipset and the gpu behind a pcie root port, i dont use hdmi, so i have no experience with it, i havent experienced stutter in flash but i did experience some in games, i solved this by disabling nested page tables and tweaking the kernel config (see my second post on first page).

Xen 3dmark vs kvm is about the same with the above tweaks.

Okay, i´ll try out Q35 when i´ll have more time. I was going to try the NPT Flag when i first found this thread and today i had some spare time to test ist. Just to give an Impression how much this did improve my results (maybe for some other thread readers):

Unigine Heaven Benchmark 4.0:
Settings were Direct3D11, 1920x1080 fullscreen, Quality low, Tesselation disabled:

1. Run (with running, but stuttering config)
FPS: 9.5
Core: 239
Min FPS: 4.9
Max FPS: 16.5
Impression: Overall Stuttering, stuttering in sound too

2. Run, now NPT disabled:
FPS: 39.2
Score: 988
Min FPS: 7.8
Max FPS: 83.0
Impression: Seems smooth, but sometimes frames seem to get into a jam: then it looks like someone is playing a movie in fast forward for some microseconds.

3. Run, NPT disabled and now Governor set to performance:
FPS: 62.8
Score: 1582
Min FPS: 8.7
Max FPS: 103.8
Impression: Really smooth, good FPS smile

Still running on stock kernel and so on, but still i am impressed by these improvements. Thanks for the good advices with nested page tables and the governor smile
Sadly flash is still not running good. Looks better, but still stuttering. At least the sound seems to play continously.

Try using my kernel config and disabling dynticks on your bootloader (nohz=off)  + q35 and gpu behind pcie root port, i got a massive boost using that.

Offline

#127 2013-06-01 00:46:27

apoapo
Member
Registered: 2013-05-16
Posts: 15

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

nbhs wrote:

Hi. tested the patches.
The warning output with the trace is gone, but I have not seen any changed behavior wink.
Edit: But I have only tested with intel GPU as host.

Did also a benchmark test with pci-assign and no changes to my kvm-start:

qemu-system-x86_64 --enable-kvm -m 6G -cpu host \
-smp 4,sockets=1,cores=4,threads=1 \
-device pci-assign,host=01:00.0 \
-device pci-assign,host=01:00.1 \
-drive file=/var/lib/libvirt/images/win-test.img \
-hdb /dev/sda3

Unigine Heaven Benchmark 4.0. settings like diggo:
Settings were Direct3D11, 1920x1080 fullscreen, Quality low, Tesselation disabled:
Hardware is a AMD HD 7950

FPS: 117,5
Score: 2959
Min FPS: 9,2
Max FPS: 194,3

Last edited by apoapo (2013-06-01 01:10:18)

Offline

#128 2013-06-01 02:11:19

nbhs
Member
From: Montevideo, Uruguay
Registered: 2013-05-02
Posts: 402

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

apoapo wrote:
nbhs wrote:

Hi. tested the patches.
The warning output with the trace is gone, but I have not seen any changed behavior wink.
Edit: But I have only tested with intel GPU as host.

Did also a benchmark test with pci-assign and no changes to my kvm-start:

qemu-system-x86_64 --enable-kvm -m 6G -cpu host \
-smp 4,sockets=1,cores=4,threads=1 \
-device pci-assign,host=01:00.0 \
-device pci-assign,host=01:00.1 \
-drive file=/var/lib/libvirt/images/win-test.img \
-hdb /dev/sda3

Unigine Heaven Benchmark 4.0. settings like diggo:
Settings were Direct3D11, 1920x1080 fullscreen, Quality low, Tesselation disabled:
Hardware is a AMD HD 7950

FPS: 117,5
Score: 2959
Min FPS: 9,2
Max FPS: 194,3

Too bad it didnt work sad im gonna try unigine and see what i get.

Offline

#129 2013-06-01 22:50:16

nbhs
Member
From: Montevideo, Uruguay
Registered: 2013-05-02
Posts: 402

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Well i ran the benchmark

Unigine 4.0 on radeon 6950

resolution 1980x1080

Low settings & tessellation off: ~1680
Extreme settings & tessellation normal: ~660

Last edited by nbhs (2013-06-01 22:51:57)

Offline

#130 2013-06-02 10:00:31

powerhouse
Member
Registered: 2013-05-22
Posts: 15

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

apocolypse600 wrote:

Firstly, cheers for the tutorial. That must have taken awhile to write, and the effort is appreciated.

I've been looking into setting up a VGA passthrough setup under Xen for my next build for awhile now. I'm trying to work out what the advantages of this setup are over a setup under Xen. From what I've seen thrown around in different places, the main advantages are:

  • a less hacky setup for the passthrough, requiring less patches (once all this stuff gets released)

  • better support for NVIDIA cards? I've seen a few people say this, but nothing official to confirm it.

  • KVM receives more support than Xen, mainly due to Red Hat

  • Xen has a (relatively small) impact on the speed of the host, whereas KVM has no impact

  • not having to mess around with xl vs xm under Xen

Am I missing anything? I've seen a lot of people getting excited about using KVM over Xen for VGA passthrough, but not many people saying why they are so excited.

I'm doing VGA passthrough on Xen 4.1 (Linux Mint 14 / Ubuntu 12.10) and here some observations:

  • Works well with a number of AMD (ATI) as well as some Nvidia Quadro video cards doing secondary passthrough - no patches, no nothing!

  • I've been using standard Xen packages from the Ubuntu / Linux Mint repos

  • Xen doesn't play well with Nvidia cards under dom0 - only Nouveau driver, no proprietary drivers (there may be a work-around, but I haven't found/tested it yet)

  • The latest Xen versions in the Ubuntu 12.10 and 13 repos break PCI passthrough - "force version" provides a work-around

  • Except for some expensive Quadro etc. cards, VGA passthrough using Nvidia cards usually require patches

  • I've seen reports that AMD CPUs may have issues with CPU power management/frequency settings under Xen - this may already be solved

  • Xen and specifically VGA passthrough under Xen is better documented (KVM is quite new to VGA passthrough)

  • Regarding performance, I wouldn't pay attention to the Phoronix benchmarks (they are unclear at best and inaccurate at worst) - I believe the OP gave some comparisons between KVM and Xen. My experiences with Xen are very good with regard to performance.

  • I'm using xm and will wait until xl provides the features I'm using now (specifically pci reattach)

To sum it up, with Xen you can get a working system with VGA passthrough using standard (unpatched) packages from a number of distros, as long as you got the right hardware. Using KVM, you will have to patch and built your own system, often using experimental code. The benefit of KVM is that it supports more Nvidia cards and that it might be less tricky with video cards/drivers in general.

By the way, this (nbhs) how-to is probably the most useful guide I found on KVM VGA passthrough so far.

Offline

#131 2013-06-04 04:05:38

nbhs
Member
From: Montevideo, Uruguay
Registered: 2013-05-02
Posts: 402

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

powerhouse wrote:

To sum it up, with Xen you can get a working system with VGA passthrough using standard (unpatched) packages from a number of distros, as long as you got the right hardware. Using KVM, you will have to patch and built your own system, often using experimental code. The benefit of KVM is that it supports more Nvidia cards and that it might be less tricky with video cards/drivers in general.

By the way, this (nbhs) how-to is probably the most useful guide I found on KVM VGA passthrough so far.

Actually vanilla kvm/qemu has supported, for a long time, secondary vga passthrough of quadro and radeon cards, the only patching needed atm ( for primary passthrough using vfio ) is if you're using radeon cards to allow the card to properly reset, XEN also has this problem but unfortunately the only solution is to eject the card before rebooting/shutting down the guest, which can also be applied to kvm/qemu without patching

Last edited by nbhs (2013-06-04 04:12:30)

Offline

#132 2013-06-05 02:39:17

powerhouse
Member
Registered: 2013-05-22
Posts: 15

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

nbhs wrote:
powerhouse wrote:

To sum it up, with Xen you can get a working system with VGA passthrough using standard (unpatched) packages from a number of distros, as long as you got the right hardware. Using KVM, you will have to patch and built your own system, often using experimental code. The benefit of KVM is that it supports more Nvidia cards and that it might be less tricky with video cards/drivers in general.

By the way, this (nbhs) how-to is probably the most useful guide I found on KVM VGA passthrough so far.

Actually vanilla kvm/qemu has supported, for a long time, secondary vga passthrough of quadro and radeon cards, the only patching needed atm ( for primary passthrough using vfio ) is if you're using radeon cards to allow the card to properly reset, XEN also has this problem but unfortunately the only solution is to eject the card before rebooting/shutting down the guest, which can also be applied to kvm/qemu without patching

Thanks! I'm learning every day.

Offline

#133 2013-06-08 06:12:44

dalingrin
Member
Registered: 2009-03-18
Posts: 128

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

nbhs wrote:
powerhouse wrote:

To sum it up, with Xen you can get a working system with VGA passthrough using standard (unpatched) packages from a number of distros, as long as you got the right hardware. Using KVM, you will have to patch and built your own system, often using experimental code. The benefit of KVM is that it supports more Nvidia cards and that it might be less tricky with video cards/drivers in general.

By the way, this (nbhs) how-to is probably the most useful guide I found on KVM VGA passthrough so far.

Actually vanilla kvm/qemu has supported, for a long time, secondary vga passthrough of quadro and radeon cards, the only patching needed atm ( for primary passthrough using vfio ) is if you're using radeon cards to allow the card to properly reset, XEN also has this problem but unfortunately the only solution is to eject the card before rebooting/shutting down the guest, which can also be applied to kvm/qemu without patching

I can confirm that KVM without vfio has VGA reset issues with Radeon on cards. However, Xen ~4.2 does not have issues with reset on my Radeon 7970. I used Xen VGA pass through for the better part of a year as my daily driver for games without issue.

Offline

#134 2013-06-08 07:56:40

nbhs
Member
From: Montevideo, Uruguay
Registered: 2013-05-02
Posts: 402

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

dalingrin wrote:
nbhs wrote:
powerhouse wrote:

To sum it up, with Xen you can get a working system with VGA passthrough using standard (unpatched) packages from a number of distros, as long as you got the right hardware. Using KVM, you will have to patch and built your own system, often using experimental code. The benefit of KVM is that it supports more Nvidia cards and that it might be less tricky with video cards/drivers in general.

By the way, this (nbhs) how-to is probably the most useful guide I found on KVM VGA passthrough so far.

Actually vanilla kvm/qemu has supported, for a long time, secondary vga passthrough of quadro and radeon cards, the only patching needed atm ( for primary passthrough using vfio ) is if you're using radeon cards to allow the card to properly reset, XEN also has this problem but unfortunately the only solution is to eject the card before rebooting/shutting down the guest, which can also be applied to kvm/qemu without patching

I can confirm that KVM without vfio has VGA reset issues with Radeon on cards. However, Xen ~4.2 does not have issues with reset on my Radeon 7970. I used Xen VGA pass through for the better part of a year as my daily driver for games without issue.

Using the xl or the xm toolkit?

I didnt have this issue on xen using the xm toolkit but i experienced performance degradation using xl

Last edited by nbhs (2013-06-08 08:06:27)

Offline

#135 2013-06-08 21:11:43

mastersplinter777
Member
Registered: 2012-10-09
Posts: 8

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Hi! I'm also interested in passing thru my VGA to VM. But I get an error 12 (Not enough system resources) and can't solve this problem. My HW is: ASRock Z77 Extreme 6 (BIOS 2.70), Intel Core i5 3470, GPUS: Intel HD 2500 (host), AMD Sapphire Radeon 6750 (68bf), ATI Radeon 4850 (9442).

I use kernel version 3.9.4 with applied kvm polarity patch (to properly run OSX) and

 
CONFIG_VFIO_IOMMU_TYPE1=m
CONFIG_VFIO=m
CONFIG_VFIO_PCI=m
CONFIG_VFIO_PCI_VGA=y

(i tried to do everything exactly as written in this article with the same result)

GRUB_CMDLINE_LINUX_DEFAULT="quiet ipv6.disable=1 intel_iommu=on iommu=group_mf pci-stub.ids=1002:68bf,1002:9442 vfio_iommu_type1.allow_unsafe_interrupts=1 kvm_intel.emulate_invalid_guest_state=0"

mkinitcpio.conf MODULES:

MODULES="pci-stub kvm kvm-intel i915 tun bridge fuse"

QEMU Win7 conf:

sudo qemu-system-x86_64 --enable-kvm -M q35 -m 2048 -cpu host \
-boot menu=on \
-uuid 91848507-02ce-7d89-1ed4-40039265bbf9 \
-smp 2,sockets=1,cores=2,threads=1 \
-vga qxl \
-spice port=5902,disable-ticketing \
-device virtio-serial \
-chardev spicevmc,id=vdagent,name=vdagent \
-device virtserialport,chardev=vdagent,name=com.redhat.spice.0 \
-readconfig /etc/qemu/ich9-ehci-uhci.cfg \
-chardev spicevmc,name=usbredir,id=usbredirchardev1 \
-device usb-redir,chardev=usbredirchardev1,id=usbredirdev1 \
-chardev spicevmc,name=usbredir,id=usbredirchardev2 \
-device usb-redir,chardev=usbredirchardev2,id=usbredirdev2 \
-chardev spicevmc,name=usbredir,id=usbredirchardev3 \
-device usb-redir,chardev=usbredirchardev3,id=usbredirdev3 \
-device ich9-intel-hda,bus=pcie.0,addr=1b.0,id=sound0 \
-device hda-duplex,id=sound0-codec0,bus=sound0.0,cad=0 \
-net nic,macaddr=00:00:00:0a:5b:2c,model=virtio \
-net user \
-device ahci,bus=pcie.0,id=ahci \
-device ioh3420,bus=pcie.0,addr=1c.0,multifunction=on,port=1,chassis=1,id=root.1 \
-device vfio-pci,host=01:00.0,bus=root.1,addr=01.0,multifunction=on,x-vga=on \
-drive file='/home/splinter/Qemu/Win7/win7.img',if=virtio

(also I tried with -bios "/path/to/patched-bios/" and -vga none options - got "VM has no graphics card" message without any output to the secondary display)

dmesg | grep IOMMU
[    0.000000] Intel-IOMMU: enabled
[    0.018497] dmar: IOMMU 0: reg_base_addr fed90000 ver 1:0 cap c0000020e60262 ecap f0101a
[    0.018502] dmar: IOMMU 1: reg_base_addr fed91000 ver 1:0 cap c9008020660262 ecap f0105a
[    0.018572] IOAPIC id 2 under DRHD base  0xfed91000 IOMMU 1
[    0.326767] IOMMU 0 0xfed90000: using Queued invalidation
[    0.326768] IOMMU 1 0xfed91000: using Queued invalidation
[    0.326770] IOMMU: Setting RMRR:
[    0.326779] IOMMU: Setting identity map for device 0000:00:02.0 [0xaf800000 - 0xbf9fffff]
[    0.327898] IOMMU: Setting identity map for device 0000:00:1d.0 [0xad6f5000 - 0xad721fff]
[    0.327915] IOMMU: Setting identity map for device 0000:00:1a.0 [0xad6f5000 - 0xad721fff]
[    0.327929] IOMMU: Setting identity map for device 0000:00:14.0 [0xad6f5000 - 0xad721fff]
[    0.327938] IOMMU: Prepare 0-16MiB unity mapping for LPC
[    0.327944] IOMMU: Setting identity map for device 0000:00:1f.0 [0x0 - 0xffffff]

dmesg | grep vfio
[   16.812467] vfio-pci 0000:01:00.0: enabling device (0000 -> 0003)

dmesg | grep DMAR
[    0.000000] ACPI: DMAR 00000000adafee70 000B8 (v01 INTEL      SNB  00000001 INTL 00000001)
[    0.326743] DMAR: No ATSR found
[    0.765236] dmar: DMAR:[DMA Read] Request device [0c:00.0] fault addr fffff000 
DMAR:[fault reason 02] Present bit in context entry is clear

Thx for your help! smile

Last edited by mastersplinter777 (2013-06-08 21:47:11)

Offline

#136 2013-06-09 05:03:42

nbhs
Member
From: Montevideo, Uruguay
Registered: 2013-05-02
Posts: 402

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

mastersplinter777 wrote:

Hi! I'm also interested in passing thru my VGA to VM. But I get an error 12 (Not enough system resources) and can't solve this problem. My HW is: ASRock Z77 Extreme 6 (BIOS 2.70), Intel Core i5 3470, GPUS: Intel HD 2500 (host), AMD Sapphire Radeon 6750 (68bf), ATI Radeon 4850 (9442).

I use kernel version 3.9.4 with applied kvm polarity patch (to properly run OSX) and

 
CONFIG_VFIO_IOMMU_TYPE1=m
CONFIG_VFIO=m
CONFIG_VFIO_PCI=m
CONFIG_VFIO_PCI_VGA=y

(i tried to do everything exactly as written in this article with the same result)

GRUB_CMDLINE_LINUX_DEFAULT="quiet ipv6.disable=1 intel_iommu=on iommu=group_mf pci-stub.ids=1002:68bf,1002:9442 vfio_iommu_type1.allow_unsafe_interrupts=1 kvm_intel.emulate_invalid_guest_state=0"

mkinitcpio.conf MODULES:

MODULES="pci-stub kvm kvm-intel i915 tun bridge fuse"

QEMU Win7 conf:

sudo qemu-system-x86_64 --enable-kvm -M q35 -m 2048 -cpu host \
-boot menu=on \
-uuid 91848507-02ce-7d89-1ed4-40039265bbf9 \
-smp 2,sockets=1,cores=2,threads=1 \
-vga qxl \
-spice port=5902,disable-ticketing \
-device virtio-serial \
-chardev spicevmc,id=vdagent,name=vdagent \
-device virtserialport,chardev=vdagent,name=com.redhat.spice.0 \
-readconfig /etc/qemu/ich9-ehci-uhci.cfg \
-chardev spicevmc,name=usbredir,id=usbredirchardev1 \
-device usb-redir,chardev=usbredirchardev1,id=usbredirdev1 \
-chardev spicevmc,name=usbredir,id=usbredirchardev2 \
-device usb-redir,chardev=usbredirchardev2,id=usbredirdev2 \
-chardev spicevmc,name=usbredir,id=usbredirchardev3 \
-device usb-redir,chardev=usbredirchardev3,id=usbredirdev3 \
-device ich9-intel-hda,bus=pcie.0,addr=1b.0,id=sound0 \
-device hda-duplex,id=sound0-codec0,bus=sound0.0,cad=0 \
-net nic,macaddr=00:00:00:0a:5b:2c,model=virtio \
-net user \
-device ahci,bus=pcie.0,id=ahci \
-device ioh3420,bus=pcie.0,addr=1c.0,multifunction=on,port=1,chassis=1,id=root.1 \
-device vfio-pci,host=01:00.0,bus=root.1,addr=01.0,multifunction=on,x-vga=on \
-drive file='/home/splinter/Qemu/Win7/win7.img',if=virtio

(also I tried with -bios "/path/to/patched-bios/" and -vga none options - got "VM has no graphics card" message without any output to the secondary display)

dmesg | grep IOMMU
[    0.000000] Intel-IOMMU: enabled
[    0.018497] dmar: IOMMU 0: reg_base_addr fed90000 ver 1:0 cap c0000020e60262 ecap f0101a
[    0.018502] dmar: IOMMU 1: reg_base_addr fed91000 ver 1:0 cap c9008020660262 ecap f0105a
[    0.018572] IOAPIC id 2 under DRHD base  0xfed91000 IOMMU 1
[    0.326767] IOMMU 0 0xfed90000: using Queued invalidation
[    0.326768] IOMMU 1 0xfed91000: using Queued invalidation
[    0.326770] IOMMU: Setting RMRR:
[    0.326779] IOMMU: Setting identity map for device 0000:00:02.0 [0xaf800000 - 0xbf9fffff]
[    0.327898] IOMMU: Setting identity map for device 0000:00:1d.0 [0xad6f5000 - 0xad721fff]
[    0.327915] IOMMU: Setting identity map for device 0000:00:1a.0 [0xad6f5000 - 0xad721fff]
[    0.327929] IOMMU: Setting identity map for device 0000:00:14.0 [0xad6f5000 - 0xad721fff]
[    0.327938] IOMMU: Prepare 0-16MiB unity mapping for LPC
[    0.327944] IOMMU: Setting identity map for device 0000:00:1f.0 [0x0 - 0xffffff]

dmesg | grep vfio
[   16.812467] vfio-pci 0000:01:00.0: enabling device (0000 -> 0003)

dmesg | grep DMAR
[    0.000000] ACPI: DMAR 00000000adafee70 000B8 (v01 INTEL      SNB  00000001 INTL 00000001)
[    0.326743] DMAR: No ATSR found
[    0.765236] dmar: DMAR:[DMA Read] Request device [0c:00.0] fault addr fffff000 
DMAR:[fault reason 02] Present bit in context entry is clear

Thx for your help! smile

A lot of people have reported problems using the intel igp as the host boot device you might want try to disable it and use one of your radeon cards for the host

Last edited by nbhs (2013-06-09 05:06:06)

Offline

#137 2013-06-09 10:51:23

thelatios
Member
Registered: 2013-06-09
Posts: 2

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Hello,

Thanks a lot for your guide, it helped a lot!

Even though VGA passthrough with VFIO is working (I got seabios output on my passthroughed card), a detail is bothering me.

I found this in my dmesg output:

[    0.277175] [Firmware Bug]: AMD-Vi: IOAPIC[9] not in IVRS table
[    0.277207] [Firmware Bug]: AMD-Vi: IOAPIC[10] not in IVRS table
[    0.277237] [Firmware Bug]: AMD-Vi: No southbridge IOAPIC found in IVRS table
[    0.277268] AMD-Vi: Disabling interrupt remapping due to BIOS Bug(s)

I guess I have to add something in my kernel command line like you did (ivrs_ioapic[9]=...), but I'm kind of confused on how you got the right values to put there. Could you give me a hint?

My motherboard is a Asus Sabertooth 990FX R2.0 with a AMD Athlon FX 8350 CPU.

Thanks again for the great guide!

Last edited by thelatios (2013-06-09 10:52:28)

Offline

#138 2013-06-09 11:39:56

nbhs
Member
From: Montevideo, Uruguay
Registered: 2013-05-02
Posts: 402

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

thelatios wrote:

Hello,

Thanks a lot for your guide, it helped a lot!

Even though VGA passthrough with VFIO is working (I got seabios output on my passthroughed card), a detail is bothering me.

I found this in my dmesg output:

[    0.277175] [Firmware Bug]: AMD-Vi: IOAPIC[9] not in IVRS table
[    0.277207] [Firmware Bug]: AMD-Vi: IOAPIC[10] not in IVRS table
[    0.277237] [Firmware Bug]: AMD-Vi: No southbridge IOAPIC found in IVRS table
[    0.277268] AMD-Vi: Disabling interrupt remapping due to BIOS Bug(s)

I guess I have to add something in my kernel command line like you did (ivrs_ioapic[9]=...), but I'm kind of confused on how you got the right values to put there. Could you give me a hint?

My motherboard is a Asus Sabertooth 990FX R2.0 with a AMD Athlon FX 8350 CPU.

Thanks again for the great guide!

You'll need to add "ivrs_ioapic[9]=00:14.0 ivrs_ioapic[10]=00:00.1" to your grub configuration, also you'll need either kernel 3.10 or 3.9 with this patch  amd_iommu_fixes.patch.tar.gz, this will enable interrupt remapping on your board so you wont need this line anymore:

vfio_iommu_type1.allow_unsafe_interrupts=1

Would you mind sharing which card you passed thru'?

Last edited by nbhs (2013-06-09 11:40:24)

Offline

#139 2013-06-09 12:44:04

thelatios
Member
Registered: 2013-06-09
Posts: 2

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

nbhs wrote:

You'll need to add "ivrs_ioapic[9]=00:14.0 ivrs_ioapic[10]=00:00.1" to your grub configuration, also you'll need either kernel 3.10 or 3.9 with this patch  amd_iommu_fixes.patch.tar.gz, this will enable interrupt remapping on your board so you wont need this line anymore:

vfio_iommu_type1.allow_unsafe_interrupts=1

Would you mind sharing which card you passed thru'?

Thanks for the patch. Do you know why 00:14.0 and 00:00.1 are the correct devices for those parameters?
On my motherboard,
- 00:14.0 is SMBus: Advanced Micro Devices [AMD] nee ATI SBx00 SMBus Controller (rev 42)
- 00:00.1 doesn't exist in lspci...

The passed thru' card is a Radeon HD5450.

Thanks!

Offline

#140 2013-06-09 13:10:14

nbhs
Member
From: Montevideo, Uruguay
Registered: 2013-05-02
Posts: 402

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

thelatios wrote:
nbhs wrote:

You'll need to add "ivrs_ioapic[9]=00:14.0 ivrs_ioapic[10]=00:00.1" to your grub configuration, also you'll need either kernel 3.10 or 3.9 with this patch  amd_iommu_fixes.patch.tar.gz, this will enable interrupt remapping on your board so you wont need this line anymore:

vfio_iommu_type1.allow_unsafe_interrupts=1

Would you mind sharing which card you passed thru'?

Thanks for the patch. Do you know why 00:14.0 and 00:00.1 are the correct devices for those parameters?
On my motherboard,
- 00:14.0 is SMBus: Advanced Micro Devices [AMD] nee ATI SBx00 SMBus Controller (rev 42)
- 00:00.1 doesn't exist in lspci...

The passed thru' card is a Radeon HD5450.

Thanks!

Yes they wont show up on lspci but those are the addresses of the sb ioapic and nb ioapic

Offline

#141 2013-06-10 01:56:49

jgott
Member
Registered: 2011-04-10
Posts: 21

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

For anyone else who's gotten video passthrough to work but is annoyed by stuttery or inconsistent video/audio performance, I may have some useful information.

Passing through real devices to virtual machines provides essentially full throughput (average performance over time), but the problem is that the worst-case latency can be very bad for important things like interrupt handlers and deferred procedure calls. (Interrupt handlers allow hardware to quickly inform its driver of something; DPCs are used by Windows drivers to delay time-consuming work that would otherwise be done in the int handler). If a hardware interrupt or DPC is delayed for some reason, then the hardware driver in question basically just has to wait it out until the delay is over, and then maybe try to catch up afterward. But for latency-sensitive things like audio or video, a delay of even a few milliseconds can result in a break in the audio stream or a long video frame. For storage hardware, it can result in longer access times, and therefore poor random access performance.

On real hardware, interrupt or DPC latency are almost always consistently good (unless you have a hardware or driver problem) because the kernel gives these tasks very high priority. Within a virtual machine, though, things can get weird because interrupts have to get handled by the host kernel before being passed to the guest kernel, and because interrupt handlers and DPCs that need to be handled immediately by the guest kernel may get preempted by unimportant tasks on the host.

Here are some tools for measuring DPC and interrupt latency in Windows that should allow you to get an idea if this is a problem for you:
LatencyMon
DPC Latency Checker
Windows Performance Analyzer (surprisingly powerful)

The single most effective thing I was able to do to make the guest's latency more consistent was to run the qemu process under one of Linux's realtime scheduler classes. This will probably make the biggest difference if you haven't dedicated CPU cores exclusively to the guest operating system, because the host and guest are in competition for the same CPU resources. Under normal conditions, the scheduler may allow processes on the host to preempt the guest and doesn't make any guarantees about how long the guest might have to wait to get a time slice.

Of the two realtime classes available, round-robin is preferable to fifo. Exactly how to run the qemu process with a realtime scheduling class is complicated by a number of things, including whether your VM is running in its own cgroup, so I can't give instructions that are universally applicable. Because I happen to run my VM as a systemd service, it involved adding this to the service file:

CPUSchedulingPolicy=rr

Another tweak that may be effective if your guest OS is Windows 8 is to turn off dynamic ticks. Linux has had dynamic ticks for a while (called tickless operation, NOHZ, etc.) but it's new in Windows 8 because of Microsoft's shift in focus toward mobile devices. Unfortunately, it's turned on by default. It's primarily a power optimization, and supposedly it shouldn't make any noticeable impact on the system's latency, but it has the potential to cause issues, so I recommend disabling it.

Also, be sure to read nbhs's performance tuning advice in the first two posts of this thread. In particular, if you're using your own kernel configuration, definitely try turning on kernel preemption, increasing the tick frequency, and turning off dynamic ticks. These host kernel tweaks all have the potential to improve latency for the guest.

Last edited by jgott (2013-06-10 02:02:02)

Offline

#142 2013-06-16 03:58:28

nbhs
Member
From: Montevideo, Uruguay
Registered: 2013-05-02
Posts: 402

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Intel users having problems with vfio grabbing a lot of devices like https://bbs.archlinux.org/viewtopic.php … 8#p1274378, might want to check this out: http://marc.info/?l=linux-kernel&m=136993923507507&w=2

Offline

#143 2013-06-16 06:59:57

apocolypse600
Member
Registered: 2013-05-26
Posts: 4

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Where does everyone keep up to date with news about VGA passthrough? I follow this thread, http://forums.linuxmint.com/viewtopic.php?f=42&t=112013 and http://www.overclock.net/t/1205216/guid … al-machine , although those are both aimed at passthrough via Xen rather than KVM. I also follow the xen-users, xen-devel and kvm general mailing lists. I notice that nbhs sometimes posts from a different mailing list (like the post directly before this one), but I can't find an easy way to navigate that list. Mailing lists aren't really my thing though, so that could be why. The KVM irc on freenode is pretty quiet most of the time, but I usually lurk there aswell.

It looks like Xen 4.3 is coming out very soon (wiki says release date of tomorrow, and they are definitely into the RC phase), and according to http://wiki.xen.org/wiki/Xen_Roadmap/4.3 it's defaulting to upstream Qemu (although it has partial in brackets next to it, so I'm not entirely sure what that means), so it'll be interesting to see how well it works. Although in the Linux Mint thread I posted, they were reporting issues doing the passthrough with newer versions of Xen 4.2, so who knows what a new version might bring.

I like the idea of using a NVIDIA card over an AMD card, so that if for some reason I can't get the passthrough to work, I can fall back on the propriety NVIDIA drivers under Linux, and just continue to use wine for games like I have been doing. That's why I was leaning towards KVM over Xen for my first stab at this. Also maybe it's the cynic in me, but I'm not sure many projects that the Linux Foundation has worked with have ever lasted all that long, so I'm assuming KVM (with the backing of Redhat) is a better option for support in the long run. No offence meant to the Linux Foundation by that, just an observation. Maybe I'm just still bitter about MeeGo.

I haven't actually bought any hardware yet, as I was hoping for passthrough techniques to mature a little before I try and get it to work. I tested using Synergy to share a keyboard and mouse between a laptop and a desktop, and that worked really well, so I imagine that should work find for sharing the keyboard and mouse with the virtual machine host and guest. I'm not entirely sure about audio yet, but I'd rather not have to passthrough a USB controller and switch my USB headset from one USB port to another when switching from host to guest, but I could probably live with it. Can anyone see any immediate problems with using Synergy to share the keyboard and mouse, either integrated graphics (I've seen some reports of issues with this in this thread) or my current GTS 250 to serve the Linux host and feeding a newer GPU through to a Windows guest (Probably Windows 7, but I do still have an install disk for XP)? I'd probably pass a hard-drive through directly. It's a bit hard to make comments without the exact hardware specifics I know, I just want to make sure I'm not missing something glaring when starting to plan for this build.

Offline

#144 2013-06-18 04:12:10

nbhs
Member
From: Montevideo, Uruguay
Registered: 2013-05-02
Posts: 402

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

apocolypse600 wrote:

Where does everyone keep up to date with news about VGA passthrough? I follow this thread, http://forums.linuxmint.com/viewtopic.php?f=42&t=112013 and http://www.overclock.net/t/1205216/guid … al-machine , although those are both aimed at passthrough via Xen rather than KVM. I also follow the xen-users, xen-devel and kvm general mailing lists. I notice that nbhs sometimes posts from a different mailing list (like the post directly before this one), but I can't find an easy way to navigate that list. Mailing lists aren't really my thing though, so that could be why. The KVM irc on freenode is pretty quiet most of the time, but I usually lurk there aswell.

It looks like Xen 4.3 is coming out very soon (wiki says release date of tomorrow, and they are definitely into the RC phase), and according to http://wiki.xen.org/wiki/Xen_Roadmap/4.3 it's defaulting to upstream Qemu (although it has partial in brackets next to it, so I'm not entirely sure what that means), so it'll be interesting to see how well it works. Although in the Linux Mint thread I posted, they were reporting issues doing the passthrough with newer versions of Xen 4.2, so who knows what a new version might bring.

I like the idea of using a NVIDIA card over an AMD card, so that if for some reason I can't get the passthrough to work, I can fall back on the propriety NVIDIA drivers under Linux, and just continue to use wine for games like I have been doing. That's why I was leaning towards KVM over Xen for my first stab at this. Also maybe it's the cynic in me, but I'm not sure many projects that the Linux Foundation has worked with have ever lasted all that long, so I'm assuming KVM (with the backing of Redhat) is a better option for support in the long run. No offence meant to the Linux Foundation by that, just an observation. Maybe I'm just still bitter about MeeGo.

I haven't actually bought any hardware yet, as I was hoping for passthrough techniques to mature a little before I try and get it to work. I tested using Synergy to share a keyboard and mouse between a laptop and a desktop, and that worked really well, so I imagine that should work find for sharing the keyboard and mouse with the virtual machine host and guest. I'm not entirely sure about audio yet, but I'd rather not have to passthrough a USB controller and switch my USB headset from one USB port to another when switching from host to guest, but I could probably live with it. Can anyone see any immediate problems with using Synergy to share the keyboard and mouse, either integrated graphics (I've seen some reports of issues with this in this thread) or my current GTS 250 to serve the Linux host and feeding a newer GPU through to a Windows guest (Probably Windows 7, but I do still have an install disk for XP)? I'd probably pass a hard-drive through directly. It's a bit hard to make comments without the exact hardware specifics I know, I just want to make sure I'm not missing something glaring when starting to plan for this build.

Well i read the mailing lists you can also subscribe if you want.
I dont think the xen devs are really interested in getting vga passthrough to work on non pro cards (quadro) atm, i havent seen any real activity since 4.0.
Right now most success reports have been from AMD users like me, you should expect this to change in the coming months since pci-assign is deprecated on kvm in favor of vfio-pci

Offline

#145 2013-06-20 20:39:01

rabcor
Banned
Registered: 2013-02-09
Posts: 500

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Thanks for this post! i was looking for this.

The default kernel settings in 3.9.6 are like this

CONFIG_VFIO_IOMMU_TYPE1=m
CONFIG_VFIO=m
CONFIG_VFIO_PCI=m
CONFIG_VFIO_PCI_VGA=y

Do i need to recompile it or is it enough to load that stuff as modules?

Seabios and Qemu conflict because Qemu already comes with seabios, what should i do in that regard? is it ok to just use the seabios version that came with qemu?

Last edited by rabcor (2013-06-20 20:39:30)

Offline

#146 2013-06-20 22:17:12

nbhs
Member
From: Montevideo, Uruguay
Registered: 2013-05-02
Posts: 402

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

rabcor wrote:

Thanks for this post! i was looking for this.

The default kernel settings in 3.9.6 are like this

CONFIG_VFIO_IOMMU_TYPE1=m
CONFIG_VFIO=m
CONFIG_VFIO_PCI=m
CONFIG_VFIO_PCI_VGA=y

Do i need to recompile it or is it enough to load that stuff as modules?

Seabios and Qemu conflict because Qemu already comes with seabios, what should i do in that regard? is it ok to just use the seabios version that came with qemu?

Yes its ok.

You'll need a patched seabios, this is the packages i currently use:

linux-mainline.zip
qemu.zip
seabios.zip

Offline

#147 2013-06-21 01:24:05

rabcor
Banned
Registered: 2013-02-09
Posts: 500

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

I can't build that qemu package, breaks on compilation.

Since the latest version of qemu (1.6) includes seabios wouldn't it be best to just patch that included seabios with this before compilation? (but i don't know how to do that, since i can't find the seabios files in there)

Last edited by rabcor (2013-06-21 03:38:32)

Offline

#148 2013-06-21 15:17:54

nbhs
Member
From: Montevideo, Uruguay
Registered: 2013-05-02
Posts: 402

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

rabcor wrote:

I can't build that qemu package, breaks on compilation.

Since the latest version of qemu (1.6) includes seabios wouldn't it be best to just patch that included seabios with this before compilation? (but i don't know how to do that, since i can't find the seabios files in there)

Have you updated your system? i just tried to build it ant it seems there was a problem with bluez dependency but it builds ok, i uploaded it again just in case

qemu.zip

I suppose you can patch the included seabios but on arch seabios comes as its own package, so you might break something

Offline

#149 2013-06-21 18:15:21

rabcor
Banned
Registered: 2013-02-09
Posts: 500

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

failed again, same error, and yes i updated my system only yesterday.

In file included from /usr/include/libfdt.h:55:0,
                 from /home/illustrious/qemu/src/qemu-1.5.0/device_tree.c:28:
/usr/include/fdt.h:58:2: error: unknown type name ‘fdt32_t’
  fdt32_t magic;    /* magic word FDT_MAGIC */
  ^
/usr/include/fdt.h:59:2: error: unknown type name ‘fdt32_t’
  fdt32_t totalsize;   /* total size of DT block */
  ^
/usr/include/fdt.h:60:2: error: unknown type name ‘fdt32_t’
  fdt32_t off_dt_struct;   /* offset to structure */
  ^
/usr/include/fdt.h:61:2: error: unknown type name ‘fdt32_t’
  fdt32_t off_dt_strings;   /* offset to strings */
  ^
/usr/include/fdt.h:62:2: error: unknown type name ‘fdt32_t’
  fdt32_t off_mem_rsvmap;   /* offset to memory reserve map */
  ^
/usr/include/fdt.h:63:2: error: unknown type name ‘fdt32_t’
  fdt32_t version;   /* format version */
  ^
/usr/include/fdt.h:64:2: error: unknown type name ‘fdt32_t’
  fdt32_t last_comp_version;  /* last compatible version */
  ^
/usr/include/fdt.h:67:2: error: unknown type name ‘fdt32_t’
  fdt32_t boot_cpuid_phys;  /* Which physical CPU id we're
  ^
/usr/include/fdt.h:70:2: error: unknown type name ‘fdt32_t’
  fdt32_t size_dt_strings;  /* size of the strings block */
  ^
/usr/include/fdt.h:73:2: error: unknown type name ‘fdt32_t’
  fdt32_t size_dt_struct;   /* size of the structure block */
  ^
/usr/include/fdt.h:77:2: error: unknown type name ‘fdt64_t’
  fdt64_t address;
  ^
/usr/include/fdt.h:78:2: error: unknown type name ‘fdt64_t’
  fdt64_t size;
  ^
/usr/include/fdt.h:82:2: error: unknown type name ‘fdt32_t’
  fdt32_t tag;
  ^
/usr/include/fdt.h:87:2: error: unknown type name ‘fdt32_t’
  fdt32_t tag;
  ^
/usr/include/fdt.h:88:2: error: unknown type name ‘fdt32_t’
  fdt32_t len;
  ^
/usr/include/fdt.h:89:2: error: unknown type name ‘fdt32_t’
  fdt32_t nameoff;
  ^
In file included from /home/illustrious/qemu/src/qemu-1.5.0/device_tree.c:28:0:
/usr/include/libfdt.h: In function ‘fdt_setprop_inplace_u32’:
/usr/include/libfdt.h:921:2: error: unknown type name ‘fdt32_t’
  fdt32_t tmp = cpu_to_fdt32(val);
  ^
/usr/include/libfdt.h: In function ‘fdt_setprop_inplace_u64’:
/usr/include/libfdt.h:956:2: error: unknown type name ‘fdt64_t’
  fdt64_t tmp = cpu_to_fdt64(val);
  ^
/usr/include/libfdt.h: In function ‘fdt_property_u32’:
/usr/include/libfdt.h:1032:2: error: unknown type name ‘fdt32_t’
  fdt32_t tmp = cpu_to_fdt32(val);
  ^
/usr/include/libfdt.h: In function ‘fdt_property_u64’:
/usr/include/libfdt.h:1037:2: error: unknown type name ‘fdt64_t’
  fdt64_t tmp = cpu_to_fdt64(val);
  ^
/usr/include/libfdt.h: In function ‘fdt_setprop_u32’:
/usr/include/libfdt.h:1193:2: error: unknown type name ‘fdt32_t’
  fdt32_t tmp = cpu_to_fdt32(val);
  ^
/usr/include/libfdt.h: In function ‘fdt_setprop_u64’:
/usr/include/libfdt.h:1228:2: error: unknown type name ‘fdt64_t’
  fdt64_t tmp = cpu_to_fdt64(val);
  ^
/usr/include/libfdt.h: In function ‘fdt_appendprop_u32’:
/usr/include/libfdt.h:1335:2: error: unknown type name ‘fdt32_t’
  fdt32_t tmp = cpu_to_fdt32(val);
  ^
/usr/include/libfdt.h: In function ‘fdt_appendprop_u64’:
/usr/include/libfdt.h:1370:2: error: unknown type name ‘fdt64_t’
  fdt64_t tmp = cpu_to_fdt64(val);
  ^
make[1]: *** [device_tree.o] Error 1
make: *** [subdir-i386-softmmu] Error 2
==> ERROR: A failure occurred in build().

if that's the case with seabios, can't the latest qemu version somehow be modified to not include seabios so it can be installed seperately like in older versions? (i would try to patch the seabios in qemu and hope nothing breaks in arch, but i don't know how i can find the right file, all i could find was an automated config script of some sort, but i'm at a loss of how i could force qemu to use an independantly installed version of seabios rather than installing it on its own, i figure it can't be that hard though, shouldn't it be possible to just copy old code and replace some of the new code to do that? or is it maybe more complicated than that?)

Last edited by rabcor (2013-06-22 04:10:29)

Offline

#150 2013-06-22 04:10:22

nbhs
Member
From: Montevideo, Uruguay
Registered: 2013-05-02
Posts: 402

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

rabcor wrote:

failed again, same error, and yes i updated my system only yesterday.

In file included from /usr/include/libfdt.h:55:0,
                 from /home/illustrious/qemu/src/qemu-1.5.0/device_tree.c:28:
/usr/include/fdt.h:58:2: error: unknown type name ‘fdt32_t’
  fdt32_t magic;    /* magic word FDT_MAGIC */
  ^
/usr/include/fdt.h:59:2: error: unknown type name ‘fdt32_t’
  fdt32_t totalsize;   /* total size of DT block */
  ^
/usr/include/fdt.h:60:2: error: unknown type name ‘fdt32_t’
  fdt32_t off_dt_struct;   /* offset to structure */
  ^
/usr/include/fdt.h:61:2: error: unknown type name ‘fdt32_t’
  fdt32_t off_dt_strings;   /* offset to strings */
  ^
/usr/include/fdt.h:62:2: error: unknown type name ‘fdt32_t’
  fdt32_t off_mem_rsvmap;   /* offset to memory reserve map */
  ^
/usr/include/fdt.h:63:2: error: unknown type name ‘fdt32_t’
  fdt32_t version;   /* format version */
  ^
/usr/include/fdt.h:64:2: error: unknown type name ‘fdt32_t’
  fdt32_t last_comp_version;  /* last compatible version */
  ^
/usr/include/fdt.h:67:2: error: unknown type name ‘fdt32_t’
  fdt32_t boot_cpuid_phys;  /* Which physical CPU id we're
  ^
/usr/include/fdt.h:70:2: error: unknown type name ‘fdt32_t’
  fdt32_t size_dt_strings;  /* size of the strings block */
  ^
/usr/include/fdt.h:73:2: error: unknown type name ‘fdt32_t’
  fdt32_t size_dt_struct;   /* size of the structure block */
  ^
/usr/include/fdt.h:77:2: error: unknown type name ‘fdt64_t’
  fdt64_t address;
  ^
/usr/include/fdt.h:78:2: error: unknown type name ‘fdt64_t’
  fdt64_t size;
  ^
/usr/include/fdt.h:82:2: error: unknown type name ‘fdt32_t’
  fdt32_t tag;
  ^
/usr/include/fdt.h:87:2: error: unknown type name ‘fdt32_t’
  fdt32_t tag;
  ^
/usr/include/fdt.h:88:2: error: unknown type name ‘fdt32_t’
  fdt32_t len;
  ^
/usr/include/fdt.h:89:2: error: unknown type name ‘fdt32_t’
  fdt32_t nameoff;
  ^
In file included from /home/illustrious/qemu/src/qemu-1.5.0/device_tree.c:28:0:
/usr/include/libfdt.h: In function ‘fdt_setprop_inplace_u32’:
/usr/include/libfdt.h:921:2: error: unknown type name ‘fdt32_t’
  fdt32_t tmp = cpu_to_fdt32(val);
  ^
/usr/include/libfdt.h: In function ‘fdt_setprop_inplace_u64’:
/usr/include/libfdt.h:956:2: error: unknown type name ‘fdt64_t’
  fdt64_t tmp = cpu_to_fdt64(val);
  ^
/usr/include/libfdt.h: In function ‘fdt_property_u32’:
/usr/include/libfdt.h:1032:2: error: unknown type name ‘fdt32_t’
  fdt32_t tmp = cpu_to_fdt32(val);
  ^
/usr/include/libfdt.h: In function ‘fdt_property_u64’:
/usr/include/libfdt.h:1037:2: error: unknown type name ‘fdt64_t’
  fdt64_t tmp = cpu_to_fdt64(val);
  ^
/usr/include/libfdt.h: In function ‘fdt_setprop_u32’:
/usr/include/libfdt.h:1193:2: error: unknown type name ‘fdt32_t’
  fdt32_t tmp = cpu_to_fdt32(val);
  ^
/usr/include/libfdt.h: In function ‘fdt_setprop_u64’:
/usr/include/libfdt.h:1228:2: error: unknown type name ‘fdt64_t’
  fdt64_t tmp = cpu_to_fdt64(val);
  ^
/usr/include/libfdt.h: In function ‘fdt_appendprop_u32’:
/usr/include/libfdt.h:1335:2: error: unknown type name ‘fdt32_t’
  fdt32_t tmp = cpu_to_fdt32(val);
  ^
/usr/include/libfdt.h: In function ‘fdt_appendprop_u64’:
/usr/include/libfdt.h:1370:2: error: unknown type name ‘fdt64_t’
  fdt64_t tmp = cpu_to_fdt64(val);
  ^
make[1]: *** [device_tree.o] Error 1
make: *** [subdir-i386-softmmu] Error 2
==> ERROR: A failure occurred in build().

if that's the case with seabios, can't the latest qemu version somehow be modified to not include seabios so it can be installed seperately like in older versions?


I managed to reproduce your error by installing dtc-git try this patched build:

qemu.zip

Offline

Board footer

Powered by FluxBB