KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Evonat · 2013-08-15 05:37:27

irtimmer wrote:

I have try this on my new computer (Nvdia GTX 660 and Intel Hasswell 4600). I have patched seabios, qemu and kernel mentioned in the opening post. It works while using the ArchLinux cd but I can't get it work under Seabios and Windows. It looks like there is still something missing in Seabios to properly initialize the GPU. Using ArchLinux to visualize Linux isn't very useful for me so I hope someone knows how to fix this. I could of course provide some more debug information if someone can tell me what is useful.
While running qemu with -vga none it is by default not possible to grab the keyboard and mouse in the Qemu window. After searching for an option to enable this while not using a emulated video card driver I have found somekind of a hack. If you add '-device qxl' to the Qemu command you would get a second video card which is showed in the Qemu window. In my testing it looks it won't cause conflicts (probably because it doesn't have a video bios).

I have a very similar problem: I manage to install windows using a cirrus/qxl/whatever card, and using pci-assign instead of vfio-pci my card works and I can even disable the "secondary display" from Windows so that only my display is on and the QEMU windows stays black.
But if I disable whatever graphic card I selected with -vga, Windows doesn't boot. The display stays black and nothing happens. If I try to gracefully power down QEMU with system_powerdown after a few seconds it turns off, so it's not like it's freezed or something.
I'd really like to know what is happening, so I even tried to add a secondary qxl adapter with:

-device qxl,bus=root.1,addr=01.0 \
-spice port=54321,disable-ticketing

But I can't see anything from remote-viewer and I noticed that Windows doesn't even have the drivers for it (it says it's an "unknown adapter" from the device manager), so at this point I'm at a loss.
I could be happy and start using it with a -vga adapter enabled but disabled from Windows, but it still bothers me.

nbhs · 2013-08-15 11:00:41

Have you guys tried blacklisting the intel driver on the host some people reported problems using it

Evonat · 2013-08-15 12:02:12

I haven't even compiled them in the kernel, let alone modprobed them!

Evonat · 2013-08-15 16:43:24

Oh, also, here's a funny thing: I can boot the guest arch with -vga none only when I'm using the unpatched seabios. I patch it, and the display doesn't turn on.
I guess there must be a problem with the identification of the primary VGA card in the seabios or qemu. I hope I can figure out how the code works and see if I can write me a patch.

GizmoChicken · 2013-08-16 15:17:59

GizmoChicken wrote:

IT WORKS!

I’ve made a few minor tweaks to my system since the time of my previous posts. For those interested, here’s my current setup, which seems to be running Windows 7 Ultimate (64-bit) guests nearly flawlessly:

MB: ASUS M5A99FX PRO R2.0 w/ BIOS v2005
CPU: AMD Phenom II X4 955
RAM: 20 GB DDR3 1600
Host GPU: Radeon HD6670 w/ Catalyst 13.4 driver
Guest GPU: Nvidia GTX550Ti w/ Nvidia 331.82 driver
Ubuntu 13.10 (fully updated)
Kernel 3.12.0 (from Ubuntu kernel-ppa/mainline, no patches) or 3.11.0-14 (from Ubuntu repository, no patches)
Qemu 1.7.0 (from ppa:jacob/virtualisation, no patches) or 1.5.0 (from Ubuntu repository, no patches)
Seabios 1.7.3  (from Ubuntu repository, no patches)

Using the Nvidia 331.82 driver, I have HDMI audio and a Windows experience score of 7.3 for both desktop graphics and gaming graphics. My Windows experience scores for processor and memory are 7.3 and 7.5, respectively. My Windows experience score for disk access is currently an unimpressive 6.0, but I'm sure that score could be improved through many means. Note that, although kernel 3.11.0-14 (from the Ubuntu repository, no patches) paired with Qemu 1.5.0 (from the Ubuntu repository, no patches) works well with my hardware, mainline kernel 3.12.0 (from Ubuntu kernel-ppa/mainline, no patches) paired with Qemu 1.7.0 (from ppa:jacob/virtualisation, no patches), likely works with a broader range of hardware since the latter combination supports bus reset. Between these two options, I recommend using mainline kernel 3.12 and Qemu 1.7.0.

Here are the start up commands that I'm using:

sudo vfio-bind 0000:03:00.0 0000:03:00.1 0000:00:16.0 0000:00:16.2 && \
sudo qemu-system-x86_64 \
-enable-kvm \
-M q35 \
-m 4096 \
-cpu host \
-daemonize \
-smp 4,sockets=1,cores=4,threads=1 \
-bios /usr/share/qemu/bios.bin \  
-vga none \
-device ioh3420,bus=pcie.0,addr=1c.0,multifunction=on,port=1,chassis=1,id=root \
-device vfio-pci,host=03:00.0,bus=root,addr=00.0,multifunction=on,x-vga=on \
-device vfio-pci,host=03:00.1,bus=root,addr=00.1 \
-device ahci,bus=pcie.0,id=ahci \
-drive file=/storage/images/Win7HVM.img,id=disk,format=raw \
-device ide-hd,bus=ahci.0,drive=disk \
-device vfio-pci,host=00:16.0,bus=pcie.0 \
-device vfio-pci,host=00:16.2,bus=pcie.0 \

Using BIOS v2005, IOMMU seems to work nearly flawlessly on my ASUS M5A99FX PRO R2.0 motherboard and "dmesg | grep AMD-Vi" gives me the following output:

$ dmesg | grep AMD-Vi
[    1.149303] AMD-Vi: Found IOMMU at 0000:00:00.2 cap 0x40
[    1.149305] AMD-Vi: Interrupt remapping enabled
[    1.159117] AMD-Vi: Lazy IO/TLB flushing enabled

Since I'm using a stock Ubuntu kernel, which is missing a few components needed for vfio, I added the following to /etc/modules:

pci_stub
vfio
vfio_iommu_type1
vfio_pci
# vfio_pci_vga  (For now, this component is in the stock Ubuntu kernel.)
kvm
kvm_amd

Note that if using an Intel CPU, you would want to replace "kvm_amd" with "kvm_intel" in the /etc/modules file described above.

After adding the above to /etc/modules, although probably not necessary, I updated initramfs using the following:

sudo update-initramfs -u

So that pci-stub will claim my Nvidia GTX550Ti GPU at boot, I edited my Grub cmdline found in /etc/default/grub to read as follows:

GRUB_CMDLINE_LINUX_DEFAULT="quiet splash pci-stub.ids=10de:1244,10de:0bee"

Note that I used lspci to determine the value "10de:1244,10de:0bee" that corresponds to my Nvidia GTX550Ti GPU.

After editing the Grub cmdline, I updated Grub using the following:

sudo update-grub

After booting with the Grub cmdline edited as above, "dmesg | grep pci-stub" gives me the following output:

$ dmesg | grep pci-stub
[    6.430514] pci-stub: add 10DE:1244 sub=FFFFFFFF:FFFFFFFF cls=00000000/00000000
[    6.430538] pci-stub 0000:03:00.0: claimed by stub
[    6.430554] pci-stub: add 10DE:0BEE sub=FFFFFFFF:FFFFFFFF cls=00000000/00000000
[    6.430566] pci-stub 0000:03:00.1: claimed by stub

For the sake of describing with completeness the steps that I took, I'll add that, following nbhs's tutorial, I created /usr/bin/vfio-bind, and added to it the following:

#!/bin/bash

modprobe vfio-pci

for dev in "$@"; do
        vendor=$(cat /sys/bus/pci/devices/$dev/vendor)
        device=$(cat /sys/bus/pci/devices/$dev/device)
        if [ -e /sys/bus/pci/devices/$dev/driver ]; then
                echo $dev > /sys/bus/pci/devices/$dev/driver/unbind
        fi
        echo $vendor $device > /sys/bus/pci/drivers/vfio-pci/new_id
done

I changed the file's permissions to 755:

chmod 755 /usr/bin/vfio-bind

Also following nbhs's tutorial, I created /etc/vfio-pci.cfg and added to it the following:

DEVICES="0000:03:00.0 0000:03:00.1 0000:00:16.0 0000:00:16.2"

I used lspci to determine which pci addresses to add to the above file. The "0000:03:00.0" and "0000:03:00.1" addresses correspond to my Nvidia GTX550Ti GPU. The "0000:00:16.0" and "0000:00:16.2" addresses correspond to an otherwise unused USB 2.0 controller, which allows me to pass through a separate keyboard and mouse. Of course, other addresses can be added in addition to, or in place of, the above mentioned addresses that I'm using. But note that, in my case, adding and binding any pci addresses corresponding to any USB 3.0 controller prevents the guest from booting, and at least in a few instances, corrupted my guests, rendering them unbootable. So use caution if attempting to pass through a USB 3.0 controller.

Finally, and also following nbhs's tutorial, I created /etc/systemd/system/binds-vfio-pci.service and added to it the following:

[Unit]
Description=Binds devices to vfio-pci
After=syslog.target

[Service]
EnvironmentFile=-/etc/vfio-pci.cfg
Type=oneshot
RemainAfterExit=yes
ExecStart=-/usr/bin/vfio-bind $DEVICES

[Install]
WantedBy=multi-user.target

As I said, the above setup seems to be running Windows 7 Ultimate (64-bit) guests nearly flawlessly.

I’ve also managed to passthrough my Nvidia GPU to an Ubuntu 13.10 guest. The Ubuntu VM starts up and runs reasonably well with GPU passthrough. However, unlike with the Windows VM, which consistently shuts down without incident for me, the Ubuntu VM often hangs on exit, sometimes requiring reboot of the host to clear the GPU. Or at least that was the case with Qemu 1.5. I’ll be testing Ubuntu VMs with Qemu 1.7 soon.

EDITS: Updated to reflect that BIOS v2005 cures previously reported IOMMU errors and other small tweaks, including the use of mainline kernel 3.12 and Qemu 1.7.0 from ppa:jacob/virtualisation.

Last edited by GizmoChicken (2013-12-08 05:22:25)

mikeserv · 2013-08-17 14:29:20

My config:

Gigabyte 970A-D3 v3.0 (latest FD firmware)
AMD Phenom II X6 1045T 2.7ghz
2x8GB DDR3 1600; 2X4GB DDR3 1600
2GB Nvidia GT640 (flashed with GOP compatible rom emailed me from eVGA upon request last year)
120GB Kingston SSD + 2X3TB WD Green; bcache + btrfs RAID0 data, RAID1 metadata array

I'd really like to run the linux hypervisor headless and passthrough the 1 video card to a Windows installation. I've seen the 2 gpu requirement a few times in this thread and am curious about it. Why is it necessary if you don't really care to view the host? Or, if x is a requirement, can I configure the host to use framebuffer only like with xvfb or xvnc?

-Mike

GizmoChicken · 2013-08-17 19:31:01

mikeserv wrote:

I'd really like to run the linux hypervisor headless and passthrough the 1 video card to a Windows installation. I've seen the 2 gpu requirement a few times in this thread and am curious about it. Why is it necessary if you don't really care to view the host? Or, if x is a requirement, can I configure the host to use framebuffer only like with xvfb or xvnc?

As to whether the VFIO component of KVM will work in a headless mode, I honestly don't know. So I'll leave that question for others to answer.

Although I can't directly answer your question, I can offer one possible alternative configuration: Have you considered installing a cheap video card in your (probably unused) pci slot for use as a "primary" card on your host?

In my case, I have a cheap "ATI Rage XL Pro" card (about $10 on ebay) installed in an otherwise unused pci slot of a XenServer setup. I only use the card to access xsconsole, which is a non-graphical interface, and the card works well for that purpose. To make this work, I set my bios to boot pci (rather than pcie) first.

If your bios allows for selecting pci to boot first, which I think it does, then although not truly headless, you could probably use an "ATI Rage XL Pro"pci graphics card (or possibly something a bit more potent, depending on your needs) as the primary card for your KVM w/VFIO host. Such a card would not only be cheap, but would also draw very little power. (Given your goals, you also might want to consider open source XenServer. But that's another topic.)

Last edited by GizmoChicken (2013-08-17 19:44:47)

mikeserv · 2013-08-17 20:02:31

I have had very little luck with Xen. Of course I last tried all of this Feb. At the time I couldn't find a single verifiable report of anyone successfully passing through a Fermi+ Nvidia to a Windows Vista+ guest. If I remember correctly it was due to Seabios memory space limitations and large Nvidia Roms. I requested the GOP ROM specifically in the hopes that qemu's UEFI firmware at the time could get past that but I got fed up and quit before I tried it.

Anyway, I kicked myself then (again!) for the GPU impulse buy (dammit Fry's) and figured I could just wait. I did some cool stuff with multiple lxc guests, vts and XDMCP, which I mean to do again as well... And I checked Google again today just to look for any possible improvements and found this!

The way I figure it I should be able to both. I was thinking that while the Windows guest ran I wouldn't have any need of the card because the tv is the machine's only local display and could power it down, rmmod nvidia, and modprobe pci-stub it, then when wanting to switch back I could reverse that. Too far-fetched?

Still, I probably should grab a cheap second GPU, I guess. Or maybe a PSU (gotta cheap diablotek 450 that came with the barebones kit now) and a nicer AMD... For Linux the 640 is more than adequate but it's a dog gaming board... I guess I'd probably need to verify I could keep it cool, too...

So thanks, Gizmo, I'll definitely consider your suggestion but would still love to hear from anyone else that may know of a workable single GPU config...

g.art · 2013-08-18 16:45:10

Hi guys!

Currently my setup is Host GPU - GTS250 (nouveau), Guest GPU which running Windows 7 Ultimate 64-bit - Radeon 6930. Everything run smooth with VFIO and your guide (thx to nbhs btw). But right now i got Radeon 5850 and i wanna use it as my host GPU, and i'm wondering is it possible to not blacklist the radeon(OSS) module in host? Cause i wanna use it simultanesly with my guest running Windows, like in my current config.

teekay · 2013-08-18 20:59:15

g.art wrote:

Hi guys!
Currently my setup is Host GPU - GTS250 (nouveau), Guest GPU which running Windows 7 Ultimate 64-bit - Radeon 6930. Everything run smooth with VFIO and your guide (thx to nbhs btw). But right now i got Radeon 5850 and i wanna use it as my host GPU, and i'm wondering is it possible to not blacklist the radeon(OSS) module in host? Cause i wanna use it simultanesly with my guest running Windows, like in my current config.

You could use pci-stub to "hide" the 6930 from the host's radeon driver. It's also explained in the guide how to do that.

teekay · 2013-08-21 20:09:52

Am I the only one who gets BSODs with a message like "your system BIOS is not fully ACPI compilant" when using Qemu 1.6.0 and seabios 1.7.2.2 patched?
Seabios 1.7.3-stable branch works fine here with Qemu 1.6.0 (and the patch is included there).

andy123 · 2013-08-21 22:27:30

teekay wrote:

Am I the only one who gets BSODs with a message like "your system BIOS is not fully ACPI compilant" when using Qemu 1.6.0 and seabios 1.7.2.2 patched?
Seabios 1.7.3-stable branch works fine here with Qemu 1.6.0 (and the patch is included there).

I can't say if you're the only one, but those packages work fine for me:

seabios 1.7.2.2-1
qemu-vga-current 1.6.0.g7b4b0e9-1

Chetyre · 2013-08-28 03:25:37

Hi,

I installed the patched kernel + qemu + seabios, but when I try intel_iommu=on, it gets stuck with a bunch of "buffer io error on device dm-3". with intel_iommu=off it boots correctly, but then I can't find the iommu groups or something and vfio-pci won't work.

What could be wrong?

nbhs · 2013-08-29 11:27:25

Chetyre wrote:

Hi,
I installed the patched kernel + qemu + seabios, but when I try intel_iommu=on, it gets stuck with a bunch of "buffer io error on device dm-3". with intel_iommu=off it boots correctly, but then I can't find the iommu groups or something and vfio-pci won't work.
What could be wrong?

It could be like in the case of andy123, the sata controller isnt working properly with iommu enabled.

Last edited by nbhs (2013-08-29 11:27:39)

Chetyre · 2013-08-29 11:54:47

nbhs wrote:

Chetyre wrote:
Hi,
I installed the patched kernel + qemu + seabios, but when I try intel_iommu=on, it gets stuck with a bunch of "buffer io error on device dm-3". with intel_iommu=off it boots correctly, but then I can't find the iommu groups or something and vfio-pci won't work.
What could be wrong?
It could be like in the case of andy123, the sata controller isnt working properly with iommu enabled.

Did he find a solution? Because if I can't turn iommu on then vfio won't work right? My motherboard is an Asrock Z77 Extreme4, with two controllers, an intel one and an asmedia one. The intel controller is bound to pci-back.

I'm not really sure where to begin with this.

nbhs · 2013-08-29 12:15:02

Chetyre wrote:

nbhs wrote:
Chetyre wrote:
Hi,
I installed the patched kernel + qemu + seabios, but when I try intel_iommu=on, it gets stuck with a bunch of "buffer io error on device dm-3". with intel_iommu=off it boots correctly, but then I can't find the iommu groups or something and vfio-pci won't work.
What could be wrong?
It could be like in the case of andy123, the sata controller isnt working properly with iommu enabled.

Did he find a solution? Because if I can't turn iommu on then vfio won't work right? My motherboard is an Asrock Z77 Extreme4, with two controllers, an intel one and an asmedia one. The intel controller is bound to pci-back.
I'm not really sure where to begin with this.

pci-back i a xen module, you shouldnt bind anything to it

Chetyre · 2013-08-29 12:17:46

nbhs wrote:

Chetyre wrote:
nbhs wrote:
It could be like in the case of andy123, the sata controller isnt working properly with iommu enabled.

Did he find a solution? Because if I can't turn iommu on then vfio won't work right? My motherboard is an Asrock Z77 Extreme4, with two controllers, an intel one and an asmedia one. The intel controller is bound to pci-back.
I'm not really sure where to begin with this.
pci-back i a xen module, you shouldnt bind anything to it

sorry, I'm still using xen so I got confused. I meant pci-stub.

nbhs · 2013-08-29 13:15:51

Chetyre wrote:

nbhs wrote:
Chetyre wrote:

Did he find a solution? Because if I can't turn iommu on then vfio won't work right? My motherboard is an Asrock Z77 Extreme4, with two controllers, an intel one and an asmedia one. The intel controller is bound to pci-back.
I'm not really sure where to begin with this.
pci-back i a xen module, you shouldnt bind anything to it

sorry, I'm still using xen so I got confused. I meant pci-stub.

Have you tried booting without binding the controller to pci-stub?

andy123 · 2013-08-29 14:49:02

I got different error messages, but if it is similar to my case you could try detaching all drives from one controller and booting and if you still get errors try the other controller.
My "solution" to the problem is: not using the second sata controller. If it turns out you really have the exact same problem, then you are lucky, because there is a patch for it for intel hardware (not for amd, which I use).

Chetyre · 2013-08-29 23:16:25

Ok, I tried both your suggestions, but still the same thing.

I managed to boot with intel_iommu=on ONCE, but only after I removed everything from pci-stub.ids. but as I said it only worked once, and then I just got a black screen with nothing in it. but when I do that I can change to tty1, but it just stays there doing nothing

Lekensteyn · 2013-08-31 20:52:05

I am new to this IOMMU thing, just installed an i7 3770 on a GA-Z68X-UD3H-B3 (rev 1.3), BIOS U1l. Using -device pci-assign,host=xx:yy.z successfully passes a USB controller and NIC (when ran as root).

After looking around, it seems that "VFIO" must be used for passing a VGA card, but I cannot simply unbind all devices under an IOMMU group, that will render me without networking and disk! Can you (desktop owners, not laptop) post the output of:

(for i in /sys/kernel/iommu_groups/*;do echo "IOMMU: $i";for dev in $i/devices/*;do lspci -s "${dev##*/}"; done; echo;done)

Output for my machine:

IOMMU: /sys/kernel/iommu_groups/0
00:00.0 Host bridge: Intel Corporation Xeon E3-1200 v2/3rd Gen Core processor DRAM Controller (rev 09)

IOMMU: /sys/kernel/iommu_groups/1
00:02.0 VGA compatible controller: Intel Corporation Xeon E3-1200 v2/3rd Gen Core processor Graphics Controller (rev 09)

IOMMU: /sys/kernel/iommu_groups/2
00:1a.0 USB controller: Intel Corporation 6 Series/C200 Series Chipset Family USB Enhanced Host Controller #2 (rev 05)

IOMMU: /sys/kernel/iommu_groups/3
00:1b.0 Audio device: Intel Corporation 6 Series/C200 Series Chipset Family High Definition Audio Controller (rev 05)

IOMMU: /sys/kernel/iommu_groups/4
00:1c.0 PCI bridge: Intel Corporation 6 Series/C200 Series Chipset Family PCI Express Root Port 1 (rev b5)
00:1c.3 PCI bridge: Intel Corporation 82801 PCI Bridge (rev b5)
00:1c.4 PCI bridge: Intel Corporation 6 Series/C200 Series Chipset Family PCI Express Root Port 5 (rev b5)
00:1c.5 PCI bridge: Intel Corporation 6 Series/C200 Series Chipset Family PCI Express Root Port 6 (rev b5)
00:1c.6 PCI bridge: Intel Corporation 6 Series/C200 Series Chipset Family PCI Express Root Port 7 (rev b5)
00:1c.7 PCI bridge: Intel Corporation 6 Series/C200 Series Chipset Family PCI Express Root Port 8 (rev b5)
02:00.0 PCI bridge: Intel Corporation 82801 PCI Bridge (rev 30)
03:00.0 Ethernet controller: Realtek Semiconductor Co., Ltd. RTL8169 PCI Gigabit Ethernet Controller (rev 10)
04:00.0 USB controller: Etron Technology, Inc. EJ168 USB 3.0 Host Controller (rev 01)
05:00.0 USB controller: Etron Technology, Inc. EJ168 USB 3.0 Host Controller (rev 01)
06:00.0 Ethernet controller: Realtek Semiconductor Co., Ltd. RTL8111/8168/8411 PCI Express Gigabit Ethernet Controller (rev 06)
07:00.0 SATA controller: Marvell Technology Group Ltd. 88SE9172 SATA 6Gb/s Controller (rev 11)

IOMMU: /sys/kernel/iommu_groups/5
00:1d.0 USB controller: Intel Corporation 6 Series/C200 Series Chipset Family USB Enhanced Host Controller #1 (rev 05)

IOMMU: /sys/kernel/iommu_groups/6
00:1f.0 ISA bridge: Intel Corporation Z68 Express Chipset Family LPC Controller (rev 05)
00:1f.2 SATA controller: Intel Corporation 6 Series/C200 Series Chipset Family SATA AHCI Controller (rev 05)
00:1f.3 SMBus: Intel Corporation 6 Series/C200 Series Chipset Family SMBus Controller (rev 05)

There is no discrete video card attached, I first need to get these groups more sane.

Edit:
I just noticed that the devices are either located under bus 00, device 1c or that they are the 00:1c.x device. From http://www.linux-kvm.org/page/How_to_as … T-d_in_KVM: VT-d spec specifies that all conventional PCI devices behind a PCIe-to PCI/PCI-X bridge or conventional PCI bridge can only be collectively assigned to the same guest. PCIe devices do not have this restriction.

Am I out of luck then with this motherboard?

Last edited by Lekensteyn (2013-08-31 21:30:09)

GizmoChicken · 2013-09-01 03:58:30

nbhs wrote:

GizmoChicken wrote:
nbhs wrote:
Mine does suffer from it fortunately there's a kernel ivrs override parameter (from kernel 3.10 up) in the kernel to work arround the interrupt remmaping issue
Yep, I saw your earlier post:
nbhs wrote:
You'll need to add "ivrs_ioapic[9]=00:14.0 ivrs_ioapic[10]=00:00.1" to your grub configuration, also you'll need either kernel 3.10 or 3.9 with this patch amd_iommu_fixes.patch.tar.gz, this will enable interrupt remapping on your board so you wont need this line anymore:
vfio_iommu_type1.allow_unsafe_interrupts=1
My motherboard's IVRS errors seem to be slightly different from your motherboard's IVRS errors. My board's errors include: 1) "ivrs_ioapic[5] not in IVRS table"; 2) "ivrs_ioapic[6] not in IVRS table" and 3) "No southbridge IOAPIC found" I'm not sure how to determine what addresses correspond to the "ivrs_ioapic[5]" and "ivrs_ioapic[6]" errors on my board.
How did you determine that ivrs_ioapic[9] corresponds to 00:14.0 and that ivrs_ioapic[10] corresponds to 00:00.1 on your board?
yeah its got to do with the number of cpu cores on your board just add ivrs_ioapic[5]=00:14.0 ivrs_ioapic[6]=00:00.1 as a kernel parameter to grub and you're done, once you get it working remove this line:
vfio_iommu_type1.allow_unsafe_interrupts=1
Im not going to write anything to Asus anymore this is my last board from them, they dont support linux, and they dont care fixing anything after the release, my next MB will be Gigabyte

In capitulation to my repeated nagging and pestering, ASUS finally sent me a patched BIOS (M5A99FX-PRO-R20-ASUS-9903.CAP) that attempts to address the IVRS errors that I was having with my ASUS M5A99FX PRO R2.0 motherboard.

Much to my surprise and satisfaction, the patched BIOS resolves ALL the IVRS errors that I had with my motherboard, eliminating the need for the kernel IVRS override. Yippee!

The support rep couldn't tell me when a version of the BIOS that includes the patch will be available for download from ASUS. Hopefully soon.

Last edited by GizmoChicken (2013-09-01 08:44:36)

nbhs · 2013-09-01 10:59:16

Lekensteyn wrote:

I am new to this IOMMU thing, just installed an i7 3770 on a GA-Z68X-UD3H-B3 (rev 1.3), BIOS U1l. Using -device pci-assign,host=xx:yy.z successfully passes a USB controller and NIC (when ran as root).

After looking around, it seems that "VFIO" must be used for passing a VGA card, but I cannot simply unbind all devices under an IOMMU group, that will render me without networking and disk! Can you (desktop owners, not laptop) post the output of:

(for i in /sys/kernel/iommu_groups/*;do echo "IOMMU: $i";for dev in $i/devices/*;do lspci -s "${dev##*/}"; done; echo;done)

Output for my machine:

IOMMU: /sys/kernel/iommu_groups/0
00:00.0 Host bridge: Intel Corporation Xeon E3-1200 v2/3rd Gen Core processor DRAM Controller (rev 09)

IOMMU: /sys/kernel/iommu_groups/1
00:02.0 VGA compatible controller: Intel Corporation Xeon E3-1200 v2/3rd Gen Core processor Graphics Controller (rev 09)

IOMMU: /sys/kernel/iommu_groups/2
00:1a.0 USB controller: Intel Corporation 6 Series/C200 Series Chipset Family USB Enhanced Host Controller #2 (rev 05)

IOMMU: /sys/kernel/iommu_groups/3
00:1b.0 Audio device: Intel Corporation 6 Series/C200 Series Chipset Family High Definition Audio Controller (rev 05)

IOMMU: /sys/kernel/iommu_groups/4
00:1c.0 PCI bridge: Intel Corporation 6 Series/C200 Series Chipset Family PCI Express Root Port 1 (rev b5)
00:1c.3 PCI bridge: Intel Corporation 82801 PCI Bridge (rev b5)
00:1c.4 PCI bridge: Intel Corporation 6 Series/C200 Series Chipset Family PCI Express Root Port 5 (rev b5)
00:1c.5 PCI bridge: Intel Corporation 6 Series/C200 Series Chipset Family PCI Express Root Port 6 (rev b5)
00:1c.6 PCI bridge: Intel Corporation 6 Series/C200 Series Chipset Family PCI Express Root Port 7 (rev b5)
00:1c.7 PCI bridge: Intel Corporation 6 Series/C200 Series Chipset Family PCI Express Root Port 8 (rev b5)
02:00.0 PCI bridge: Intel Corporation 82801 PCI Bridge (rev 30)
03:00.0 Ethernet controller: Realtek Semiconductor Co., Ltd. RTL8169 PCI Gigabit Ethernet Controller (rev 10)
04:00.0 USB controller: Etron Technology, Inc. EJ168 USB 3.0 Host Controller (rev 01)
05:00.0 USB controller: Etron Technology, Inc. EJ168 USB 3.0 Host Controller (rev 01)
06:00.0 Ethernet controller: Realtek Semiconductor Co., Ltd. RTL8111/8168/8411 PCI Express Gigabit Ethernet Controller (rev 06)
07:00.0 SATA controller: Marvell Technology Group Ltd. 88SE9172 SATA 6Gb/s Controller (rev 11)

IOMMU: /sys/kernel/iommu_groups/5
00:1d.0 USB controller: Intel Corporation 6 Series/C200 Series Chipset Family USB Enhanced Host Controller #1 (rev 05)

IOMMU: /sys/kernel/iommu_groups/6
00:1f.0 ISA bridge: Intel Corporation Z68 Express Chipset Family LPC Controller (rev 05)
00:1f.2 SATA controller: Intel Corporation 6 Series/C200 Series Chipset Family SATA AHCI Controller (rev 05)
00:1f.3 SMBus: Intel Corporation 6 Series/C200 Series Chipset Family SMBus Controller (rev 05)

There is no discrete video card attached, I first need to get these groups more sane.

Edit:
I just noticed that the devices are either located under bus 00, device 1c or that they are the 00:1c.x device. From http://www.linux-kvm.org/page/How_to_as … T-d_in_KVM: VT-d spec specifies that all conventional PCI devices behind a PCIe-to PCI/PCI-X bridge or conventional PCI bridge can only be collectively assigned to the same guest. PCIe devices do not have this restriction.

Am I out of luck then with this motherboard?

See: https://bbs.archlinux.org/viewtopic.php … 1#p1307091

nbhs · 2013-09-01 11:00:36

GizmoChicken wrote:

nbhs wrote:
GizmoChicken wrote:
Yep, I saw your earlier post:

My motherboard's IVRS errors seem to be slightly different from your motherboard's IVRS errors. My board's errors include: 1) "ivrs_ioapic[5] not in IVRS table"; 2) "ivrs_ioapic[6] not in IVRS table" and 3) "No southbridge IOAPIC found" I'm not sure how to determine what addresses correspond to the "ivrs_ioapic[5]" and "ivrs_ioapic[6]" errors on my board.
How did you determine that ivrs_ioapic[9] corresponds to 00:14.0 and that ivrs_ioapic[10] corresponds to 00:00.1 on your board?
yeah its got to do with the number of cpu cores on your board just add ivrs_ioapic[5]=00:14.0 ivrs_ioapic[6]=00:00.1 as a kernel parameter to grub and you're done, once you get it working remove this line:
vfio_iommu_type1.allow_unsafe_interrupts=1
Im not going to write anything to Asus anymore this is my last board from them, they dont support linux, and they dont care fixing anything after the release, my next MB will be Gigabyte
In capitulation to my repeated nagging and pestering, ASUS finally sent me a patched BIOS (M5A99FX-PRO-R20-ASUS-9903.CAP) that attempts to address the IVRS errors that I was having with my ASUS M5A99FX PRO R2.0 motherboard.
Much to my surprise and satisfaction, the patched BIOS resolves ALL the IVRS errors that I had with my motherboard, eliminating the need for the kernel IVRS override. Yippee!
The support rep couldn't tell me when a version of the BIOS that includes the patch will be available for download from ASUS. Hopefully soon.

That is good news! i wonder if they'll release an updated bios for the rest of their boards with this issue

Last edited by nbhs (2013-09-01 11:00:58)

Lekensteyn · 2013-09-01 14:09:08

nbhs wrote:

Lekensteyn wrote:
[..]
I just noticed that the devices are either located under bus 00, device 1c or that they are the 00:1c.x device. From http://www.linux-kvm.org/page/How_to_as … T-d_in_KVM: VT-d spec specifies that all conventional PCI devices behind a PCIe-to PCI/PCI-X bridge or conventional PCI bridge can only be collectively assigned to the same guest. PCIe devices do not have this restriction.
Am I out of luck then with this motherboard?
See: https://bbs.archlinux.org/viewtopic.php … 1#p1307091

Thank you, I encountered these patches earlier today. Perhaps someone could edit the first post to make a note of this patch?

Note: this may break the safety provided by IOMMU, devices can bypass the IOMMU and reach other devices in the same IOMMU domain. Though when compared with pci-assign, it is not that bad. (pci-assign requires you running as root to access the full PCI config space (which requires CAP_SYS_ADMIN)). See also http://www.linux-kvm.org/wiki/images/b/ … m-VFIO.pdf, page 29.

Last edited by Lekensteyn (2013-09-01 14:36:42)

Arch Linux

#426 2013-08-15 05:37:27

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

#427 2013-08-15 11:00:41

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

#428 2013-08-15 12:02:12

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

#429 2013-08-15 16:43:24

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

#430 2013-08-16 15:17:59

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

#431 2013-08-17 14:29:20

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

#432 2013-08-17 19:31:01

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

#433 2013-08-17 20:02:31

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

#434 2013-08-18 16:45:10

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

#435 2013-08-18 20:59:15

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

#436 2013-08-21 20:09:52

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

#437 2013-08-21 22:27:30

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

#438 2013-08-28 03:25:37

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

#439 2013-08-29 11:27:25

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

#440 2013-08-29 11:54:47

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

#441 2013-08-29 12:15:02

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

#442 2013-08-29 12:17:46

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

#443 2013-08-29 13:15:51

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

#444 2013-08-29 14:49:02

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

#445 2013-08-29 23:16:25

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

#446 2013-08-31 20:52:05

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

#447 2013-09-01 03:58:30

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

#448 2013-09-01 10:59:16

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

#449 2013-09-01 11:00:36

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

#450 2013-09-01 14:09:08

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Board footer