You are not logged in.
I think I figured out what is wrong, but not how to solve it.
Turns out I really shouldn`t have enabled the systemd service to vfio-bind... That was really stupid, and yet took me an eternity to notice.
anyway what happens is in my interpretation (I read very little on the subject): there is too much stuff in my iommu group for some reason. When the script runs it binds too many devices and makes the computer unusable. Is this really a thing that happens? I tried enabling only my ati card, but 4 devices were grabbed by vfio.
On the test script this produced a qemu window, but no output on the card.
When I try binding the rest of what I need, my system becomes pretty much unusable. Out of the 24 devices in my system it grabs 15 of them.
Last edited by Chetyre (2013-09-02 19:54:14)
Offline
I think I figured out what is wrong, but not how to solve it.
Turns out I really shouldn`t have enabled the systemd service to vfio-bind... That was really stupid, and yet took me an eternity to notice.
anyway what happens is in my interpretation (I read very little on the subject): there is too much stuff in my iommu group for some reason. When the script runs it binds too many devices and makes the computer unusable. Is this really a thing that happens? I tried enabling only my ati card, but 4 devices were grabbed by vfio.
On the test script this produced a qemu window, but no output on the card.
When I try binding the rest of what I need, my system becomes pretty much unusable. Out of the 24 devices in my system it grabs 15 of them.
Offline
Chetyre wrote:I think I figured out what is wrong, but not how to solve it.
Turns out I really shouldn`t have enabled the systemd service to vfio-bind... That was really stupid, and yet took me an eternity to notice.
anyway what happens is in my interpretation (I read very little on the subject): there is too much stuff in my iommu group for some reason. When the script runs it binds too many devices and makes the computer unusable. Is this really a thing that happens? I tried enabling only my ati card, but 4 devices were grabbed by vfio.
On the test script this produced a qemu window, but no output on the card.
When I try binding the rest of what I need, my system becomes pretty much unusable. Out of the 24 devices in my system it grabs 15 of them.
Thanks. I'll try this tonight. Maybe the first post could be edited to draw attention to the acs patch? I just realized that it was included in your patched kernel while trying to patch it myself.
Offline
Hello,
all steps before are succeed, but when i try to run vfio-bind it says
# vfio-bind 0000:01:00.0
ls: cannot access /sys/bus/pci/devices/0000:01:00.0/iommu_group/devices: No such file or directory
In all /sys/bus/pci/devices/XXXX:XX:XX.X/ directory "iommu_group" missed.
My distro is Ubuntu 13.10
Where I'm wrong?
Thanks in advance.
Offline
Hello,
all steps before are succeed, but when i try to run vfio-bind it says
# vfio-bind 0000:01:00.0 ls: cannot access /sys/bus/pci/devices/0000:01:00.0/iommu_group/devices: No such file or directory
In all /sys/bus/pci/devices/XXXX:XX:XX.X/ directory "iommu_group" missed.
My distro is Ubuntu 13.10
Where I'm wrong?
Thanks in advance.
Check if you cpu supports VT-D/AMD-VI and its enabled
Offline
Hello nbhs,
Thank you for quick reply!
My cpu is Intel Q9550 and support VT-d.
Mainboard is Asus Rampage Extreme with X48 chipset, who also support VT-d according Intel Chipsets
In my current BIOS VT-d enable/disable setting missing.
After activating Intel VT-d in the kernel by appending the intel_iommu=on parameter to the kernel line,
dmesg | grep -e DMAR -e IOMMU
return
[ 0.000000] Intel-IOMMU: enabled
Any ideas why "iommu_group" directories missed?
Thanks!
Offline
acs_override worked, but I encountered another problem: blank display that does nothing. If I use vga cirrus or std, windows boots fine and installs drivers for everything correctly, including the video card. But device manager says it can't free enough resources.
However, if I boot with vga none, it simply gets stuck with a blank qemu window in my monitor forever. No hdd activity or anything.
On xen, if I have a standard vga adaptor enabled and passthrough a video card, once windows boots it disables the qemu vga device and lets the gpu take over. Is there anyway to force that behavior? Disabling it on device manager does nothing, it resets during the reboot.
Offline
acs_override worked, but I encountered another problem: blank display that does nothing. If I use vga cirrus or std, windows boots fine and installs drivers for everything correctly, including the video card. But device manager says it can't free enough resources.
However, if I boot with vga none, it simply gets stuck with a blank qemu window in my monitor forever. No hdd activity or anything.
On xen, if I have a standard vga adaptor enabled and passthrough a video card, once windows boots it disables the qemu vga device and lets the gpu take over. Is there anyway to force that behavior? Disabling it on device manager does nothing, it resets during the reboot.
You aren't using a nvidia card with the proprietary driver on the host, are you?
i'm sorry for my poor english wirting skills…
Offline
Chetyre wrote:acs_override worked, but I encountered another problem: blank display that does nothing. If I use vga cirrus or std, windows boots fine and installs drivers for everything correctly, including the video card. But device manager says it can't free enough resources.
However, if I boot with vga none, it simply gets stuck with a blank qemu window in my monitor forever. No hdd activity or anything.
On xen, if I have a standard vga adaptor enabled and passthrough a video card, once windows boots it disables the qemu vga device and lets the gpu take over. Is there anyway to force that behavior? Disabling it on device manager does nothing, it resets during the reboot.
You aren't using a nvidia card with the proprietary driver on the host, are you?
No, I'm using the intel igp and trying to passthrough a radeon card
Offline
vfio_iommu_type1 won`t load
modprobe vfio_iommu_type1
return
ERROR: could not insert 'vfio_iommu_type1': No such device
Maybe MB (Rampage Extreme X48) not supported VT-d?
X48 is capable but Asus BIOS not have option to enable him.
Offline
Hi, so I just tried to get this working with linux 3.11 but recieved these error messages upon starting qemu
qemu-system-x86_64: -device vfio-pci,host=06:00.0,x-vga=on,addr=0.0,multifunction=on,bus=root.1: vfio: failed to set iommu for container: Operation not permitted
qemu-system-x86_64: -device vfio-pci,host=06:00.0,x-vga=on,addr=0.0,multifunction=on,bus=root.1: vfio: failed to setup container for group 18
qemu-system-x86_64: -device vfio-pci,host=06:00.0,x-vga=on,addr=0.0,multifunction=on,bus=root.1: vfio: failed to get group 18
qemu-system-x86_64: -device vfio-pci,host=06:00.0,x-vga=on,addr=0.0,multifunction=on,bus=root.1: Device initialization failed.
qemu-system-x86_64: -device vfio-pci,host=06:00.0,x-vga=on,addr=0.0,multifunction=on,bus=root.1: Device 'vfio-pci' could not be initialized
I simply applied the same patch to the kernel (I was kind of suprised that even worked).
Since 3.10 will be a lts kernel, should we simply stick with it?
i'm sorry for my poor english wirting skills…
Offline
Hi, so I just tried to get this working with linux 3.11 but recieved these error messages upon starting qemu
qemu-system-x86_64: -device vfio-pci,host=06:00.0,x-vga=on,addr=0.0,multifunction=on,bus=root.1: vfio: failed to set iommu for container: Operation not permitted qemu-system-x86_64: -device vfio-pci,host=06:00.0,x-vga=on,addr=0.0,multifunction=on,bus=root.1: vfio: failed to setup container for group 18 qemu-system-x86_64: -device vfio-pci,host=06:00.0,x-vga=on,addr=0.0,multifunction=on,bus=root.1: vfio: failed to get group 18 qemu-system-x86_64: -device vfio-pci,host=06:00.0,x-vga=on,addr=0.0,multifunction=on,bus=root.1: Device initialization failed. qemu-system-x86_64: -device vfio-pci,host=06:00.0,x-vga=on,addr=0.0,multifunction=on,bus=root.1: Device 'vfio-pci' could not be initialized
I simply applied the same patch to the kernel (I was kind of suprised that even worked).
Since 3.10 will be a lts kernel, should we simply stick with it?
Are you running it as root? im using kernel 3.11 + qemu 1.5.2 here without problems
here's my package:
Hopefully these patches wont be needed anymore for 3.12 and the next qemu release see: http://marc.info/?l=kvm&m=137865971705085&w=2 and http://qemu.11.n7.nabble.com/PATCH-vfio … 23928.html
i'm having some problems using qemu 1.6, it seems it requires seabios 1.7.3 and windows crashes the system at the login prompt, ill look into it when i get some time.
Last edited by nbhs (2013-09-10 03:55:19)
Offline
Are you running it as root? im using kernel 3.11 + qemu 1.5.2 here without problems
here's my package:
Hopefully these patches wont be needed anymore for 3.12 and the next qemu release see: http://marc.info/?l=kvm&m=137865971705085&w=2 and http://qemu.11.n7.nabble.com/PATCH-vfio … 23928.html
i'm having some problems using qemu 1.6, it seems it requires seabios 1.7.3 and windows crashes the system at the login prompt, ill look into it when i get some time.
I think I was running it as root, so my guess is that I was using the wrong patch or configured the kernel wrong somehow. I based my new config off of yours, used your patches and it works now. Thanks for the package.
It's good to hear that the patches might not be needed in 3.12.
About 1.6, my qemu package ist called "qemu-vga-current 1.6.0.g7b4b0e9-1", I sadly lost the pkgbuild recently, so I have no idea how that was created, but it works for me together with "seabios 1.7.2.2-1".
Thanks again for your help.
Last edited by andy123 (2013-09-11 15:19:20)
i'm sorry for my poor english wirting skills…
Offline
Greetings all.
Thanks for this how-to nbhs!
I've read every post here but can't get over something very simple. Im left believing that my board has buggy IOMMU support, pity since I bought it a few years ago wanting to do this very thing and options were limited.
System
i5-2500 (non-K)
ASRock Z68 Pro 3 Mobo
16gig ram
Intel IGPU on host Arch (with your custom built mainline kernel)
Some no name brand 7970 for the guest which currently might as well not exist
Im using syslinux, here is my kernel line:
root=UUID=069ddd96-02d7-4b5a-9b8c-d71403fb4579 ro quiet init=/usr/lib/systemd/systemd intel_iommu=on vfio_iommu_type1.allow_unsafe_interrupts=1 pcie_acs_override=downstream pci-stub.ids=1002:6798,1002:aaa0 initrd=../initramfs-linux-mainline.img BOOT_IMAGE=../vmlinuz-linux-mainline
Ive pretty much tried everything suggested in here so there's stuff in there that probably doesn't need to be, but that is how it was on my last attempt.
Other outputs:
@Shodan ~ $ dmesg |grep IOMMU
[ 0.000000] Warning: PCIe ACS overrides enabled; This may allow non-IOMMU protected peer-to-peer DMA
[ 0.000000] Intel-IOMMU: enabled
@Shodan ~ $ dmesg |grep pci-stub
[ 0.509170] pci-stub: add 1002:6798 sub=FFFFFFFF:FFFFFFFF cls=00000000/00000000
[ 0.509176] pci-stub 0000:01:00.0: claimed by stub
[ 0.509180] pci-stub: add 1002:AAA0 sub=FFFFFFFF:FFFFFFFF cls=00000000/00000000
[ 0.509183] pci-stub 0000:01:00.1: claimed by stub
@Shodan ~ $ lspci |grep VGA
00:02.0 VGA compatible controller: Intel Corporation 2nd Generation Core Processor Family Integrated Graphics Controller (rev 09)
01:00.0 VGA compatible controller: Advanced Micro Devices, Inc. [AMD/ATI] Tahiti XT [Radeon HD 7970]
Vt-d is enabled in the bios, however disabling it and rebooting still produced the dmesg output of it getting enabled so I don't know if it's lying or just referring to the module and not hardware (if thats even possible)
I compiled and installed from your first post with no issue.
The problem herein lies with the iommu_groups not being created, there were 3 other posters here with that same error but nothing they did has fixed it for me yet. I have blacklisted IGP, radeon as well as pci-stub'ing them.
For reference:
@Shodan ~ $ sudo vfio-bind 0000:01:00.0 0000:01:00.1
ls: cannot access /sys/bus/pci/devices/0000:01:00.0/iommu_group/devices: No such file or directory
ls: cannot access /sys/bus/pci/devices/0000:01:00.1/iommu_group/devices: No such file or directory
Your script doesn't lie...
@Shodan ~ $ sudo find /sys/bus |grep iommu_group
@Shodan ~ :( $
Any thoughts? Im just doing random stuff now, not sure where to go from here cept maybe buy a new motherboard.
Thanks, Joe
Offline
qemu-system-x86_64 -enable-kvm -M q35 -m 1024 -cpu host \
-smp 6,sockets=1,cores=6,threads=1 \
-bios /usr/share/qemu/bios.bin -vga none \
-device ioh3420,bus=pcie.0,addr=1c.0,multifunction=on,port=1,chassis=1,id=root.1 \
-device vfio-pci,host=05:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on \
-device vfio-pci,host=05:00.1,bus=root.1,addr=00.1
return
qemu-system-x86_64: -device vfio-pci,host=05:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on: vfio: error, group 1 is not viable, please ensure all devices within the iommu_group are bound to their vfio bus driver.
qemu-system-x86_64: -device vfio-pci,host=05:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on: vfio: failed to get group 1
qemu-system-x86_64: -device vfio-pci,host=05:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on: Device initialization failed.
qemu-system-x86_64: -device vfio-pci,host=05:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on: Device 'vfio-pci' could not be initialized
Any ideas?
Offline
Hello,
my primary and secondary VGA is in one IOMMU group, and i cant bind them to vfio.
I found in old posts here (https://bbs.archlinux.org/viewtopic.php … 1#p1307091) that must apply ACS-patch from here - http://www.spinics.net/lists/kvm/msg92163.html
Because i`m new in Linux world, can provide me step by step, how to apply this patch?
I'm using Ubuntu 13.10 - 3.11.0-7-generic
Thank you in advance!
Offline
GizmoChicken wrote:nbhs wrote:yeah its got to do with the number of cpu cores on your board just add ivrs_ioapic[5]=00:14.0 ivrs_ioapic[6]=00:00.1 as a kernel parameter to grub and you're done, once you get it working remove this line:
vfio_iommu_type1.allow_unsafe_interrupts=1
Im not going to write anything to Asus anymore this is my last board from them, they dont support linux, and they dont care fixing anything after the release, my next MB will be Gigabyte
In capitulation to my repeated nagging and pestering, ASUS finally sent me a patched BIOS (M5A99FX-PRO-R20-ASUS-9903.CAP) that attempts to address the IVRS errors that I was having with my ASUS M5A99FX PRO R2.0 motherboard.
Much to my surprise and satisfaction, the patched BIOS resolves ALL the IVRS errors that I had with my motherboard, eliminating the need for the kernel IVRS override. Yippee!
The support rep couldn't tell me when a version of the BIOS that includes the patch will be available for download from ASUS. Hopefully soon.
That is good news! i wonder if they'll release an updated bios for the rest of their boards with this issue
I recently learned that, in addition to the fix for my M5A99FX PRO R2.0 motherboard, ASUS issued a similar "test BIOS" for a Sabertooth 990FX R2.0. I don't know for certain what ASUS has planned for other motherboards, but I'm under the impression ASUS will at least make an effort to add the fix to future BIOS releases... perhaps not the next stable BIOS release, but eventually.
Anyone wanting a "test BIOS" for an ASUS motherboard that uses the AMD 990FX or 970FX chipsets should consider contacting ASUS and providing them with enough information to get to the correct person. ASUS might issue a custom "test BIOS" for other motherboards upon request, but no promises.
Here's some information that may help to get to the right person at ASUS:
Name of my primary contact at ASUS: lijay_quan
Name of "test BIOS" issued for the ASUS M5A99FX PRO R2: M5A99FX-PRO-R20-ASUS-9903.CAP
Name of "test BIOS" issued for the ASUS Sabertooth 990FX R2: SABERTOOTH-990FX-R20-ASUS-9901.CAP
Last edited by GizmoChicken (2013-09-16 04:38:44)
Offline
i'm having some problems using qemu 1.6, it seems it requires seabios 1.7.3 and windows crashes the system at the login prompt, ill look into it when i get some time.
So I wasn't the only one :-)
Am I the only one who gets BSODs with a message like "your system BIOS is not fully ACPI compilant" when using Qemu 1.6.0 and seabios 1.7.2.2 patched?
Seabios 1.7.3-stable branch works fine here with Qemu 1.6.0 (and the patch is included there).
Offline
Fully confirm. It works!
running on
Gigabyte AM3+ 990FXA-UD7/UD3
AMD FX-8150/8350
ATI 5450/Nvidia 210 - as primary.
ATI 7970/7850 - as passthrough.
Windows 7/8 fully works.
Qemu-git
Seabios from qemu
Freezing when installing, was solved. The problem was in image file permissions.
Now I want to try to pass another card in the guest machine, to make a crossfire.
Offline
Hi,
first of: thanks for this great topic!
I think about setting up a System with VGA-Passthrough, and i have still some questions:
Is it possible to use the System like in a normal Multi-monitor setup? I dont want to have 2 Mouses and Keyboards, one for the whole System would be great, is that possible?
Most of you guys with AMD-CPUs use a 990FX MB. Is the support better on that chipset? technichaly a 970 MB would work, too, wouldnt it?
Offline
Hi,
first of: thanks for this great topic!
I think about setting up a System with VGA-Passthrough, and i have still some questions:
Is it possible to use the System like in a normal Multi-monitor setup? I dont want to have 2 Mouses and Keyboards, one for the whole System would be great, is that possible?
Most of you guys with AMD-CPUs use a 990FX MB. Is the support better on that chipset? technichaly a 970 MB would work, too, wouldnt it?
Well if the 970 chipset supports iommu then yes you can, im using 1 monitor 1 keybaord and 1 mouse, my monitor supports 2 sources, so i can easly switch between windows and linux with 1 button, and for the mouse and keyboard you can use a usb switch, or like in my case i have 4 frontal usb ports 2 usb 2.0 and 2 usb 3.0 so i easly switch them (the usb 3 controller is on windows).
Offline
nbhs wrote:i'm having some problems using qemu 1.6, it seems it requires seabios 1.7.3 and windows crashes the system at the login prompt, ill look into it when i get some time.
So I wasn't the only one :-)
teekay some pages ago wrote:Am I the only one who gets BSODs with a message like "your system BIOS is not fully ACPI compilant" when using Qemu 1.6.0 and seabios 1.7.2.2 patched?
Seabios 1.7.3-stable branch works fine here with Qemu 1.6.0 (and the patch is included there).
Im gonna skip 1.6 and wait for kernel 3.12 and the next qemu release which like i mentioned before shouldnt need any extra patches.
Offline
SpacePirate wrote:Hi,
first of: thanks for this great topic!
I think about setting up a System with VGA-Passthrough, and i have still some questions:
Is it possible to use the System like in a normal Multi-monitor setup? I dont want to have 2 Mouses and Keyboards, one for the whole System would be great, is that possible?
Most of you guys with AMD-CPUs use a 990FX MB. Is the support better on that chipset? technichaly a 970 MB would work, too, wouldnt it?
Well if the 970 chipset supports iommu then yes you can, im using 1 monitor 1 keybaord and 1 mouse, my monitor supports 2 sources, so i can easly switch between windows and linux with 1 button, and for the mouse and keyboard you can use a usb switch, or like in my case i have 4 frontal usb ports 2 usb 2.0 and 2 usb 3.0 so i easly switch them (the usb 3 controller is on windows).
So theres no way of using the mouse for both systems like for example a shared soundcard?
Offline
nbhs wrote:SpacePirate wrote:Hi,
first of: thanks for this great topic!
I think about setting up a System with VGA-Passthrough, and i have still some questions:
Is it possible to use the System like in a normal Multi-monitor setup? I dont want to have 2 Mouses and Keyboards, one for the whole System would be great, is that possible?
Most of you guys with AMD-CPUs use a 990FX MB. Is the support better on that chipset? technichaly a 970 MB would work, too, wouldnt it?
Well if the 970 chipset supports iommu then yes you can, im using 1 monitor 1 keybaord and 1 mouse, my monitor supports 2 sources, so i can easly switch between windows and linux with 1 button, and for the mouse and keyboard you can use a usb switch, or like in my case i have 4 frontal usb ports 2 usb 2.0 and 2 usb 3.0 so i easly switch them (the usb 3 controller is on windows).
So theres no way of using the mouse for both systems like for example a shared soundcard?
Offline
SpacePirate wrote:nbhs wrote:Well if the 970 chipset supports iommu then yes you can, im using 1 monitor 1 keybaord and 1 mouse, my monitor supports 2 sources, so i can easly switch between windows and linux with 1 button, and for the mouse and keyboard you can use a usb switch, or like in my case i have 4 frontal usb ports 2 usb 2.0 and 2 usb 3.0 so i easly switch them (the usb 3 controller is on windows).
So theres no way of using the mouse for both systems like for example a shared soundcard?
That looks really nice! Thanks a lot for the help!
Offline