You are not logged in.
Used hardware
MoBo: ASUS X99-A
My greatest mistake since my virtualization journey started was to "upgrade" to X99-Deluxe from ASUS . period .
My previous 990FXA from GigaByte was rock stable for passing through GPUs . I got rid of it because the AMD FX-8350 was causing most heavy games to lag , and emulators were a nightmare to play on .
Offline
How so? What problems did you face? Maybe it was just bad luck and you received a bad board?
So far I've not had any real problems with mine. Unless one of the points I mentioned is caused by the board but I kinda doubt that tbh.
I wish Nouveau worked properly with my cards already! Or that nVidia releases a driver version for Wayland and which doesn't take possession of the VGA arbitration thingie.
I'd really like to give Wayland a shot to see how that fares with dynamic unbinding and rebinding of GPUs ![]()
Last edited by Omar007 (2015-05-10 01:09:23)
Offline
How so? What problems did you face? Maybe it was just bad luck and you received a bad board?
Well , I bought it 2 months after its release . It used to crash randomly ALOT during normal operation on my Windows VM . Secondly , when I moved to this platform my GT610 started to have reset issues and I am certain that it didn't have any issues running atop 990FXA .
Anyway , to be honest , recent ASUS updates fixed 99% of the stability issues (No more random crashing) .
My plan is IF (a big IF here) AMD's Zen performed near Haswell's single-threaded performance , I'll seriously consider moving to Zen+ (Zen's second gen.) platfrom .
Cheers ! ![]()
Offline
Has anyone been able to get SLI working? I have 2 gtx 780ti cards being passed through to a windows guest. The cards are recognize by the OS but the nvidia control panel does not have the option to enable SLI (however I can set one card as a PhysX processor). Any suggestions?
Hardware:
MoBo: Asus x99 Deluxe
Processor: Intel i7 5820k
Host Graphics: AMD R7 240
Guest Graphics: 2x Nvidia gtx 780ti
Currently using virt-manager to configure vm.
Offline
Has anyone been able to get SLI working? I have 2 gtx 780ti cards being passed through to a windows guest. The cards are recognize by the OS but the nvidia control panel does not have the option to enable SLI (however I can set one card as a PhysX processor). Any suggestions?
Hardware:
MoBo: Asus x99 Deluxe
Processor: Intel i7 5820k
Host Graphics: AMD R7 240
Guest Graphics: 2x Nvidia gtx 780tiCurrently using virt-manager to configure vm.
I can't remember anyone running SLI for, well, 75 pages at least.
Anyway, i think nvidia drivers detect your VM as non SLI-compatible motherboard and refuse to work.
It happens on physical machines too, so there's a crutch for that: HyperSLI. Maybe it will work if you'll enable nested virtualization. Or maybe it won't.
In the broad scale, i can't see any obvious technical reasons for it to not work - i've got my XDMA crossfire working since a long time. Maybe aw will tell something, i don't know.
Glad to see you have both cards running independently, at least.
The forum rules prohibit requesting support for distributions other than arch.
I gave up. It was too late.
What I was trying to do.
The reference about VFIO and KVM VGA passthrough.
Offline
I can't remember anyone running SLI for, well, 75 pages at least.
Anyway, i think nvidia drivers detect your VM as non SLI-compatible motherboard and refuse to work.
It happens on physical machines too, so there's a crutch for that: HyperSLI. Maybe it will work if you'll enable nested virtualization. Or maybe it won't.In the broad scale, i can't see any obvious technical reasons for it to not work - i've got my XDMA crossfire working since a long time. Maybe aw will tell something, i don't know.
Glad to see you have both cards running independently, at least.
Thanks for the response. I tried out nested virtualization with both HyperSLI and DifferentSLI (seems to be what most people were using in more recent posts from the thread you linked). Neither worked for me, it gave me another avenue to try at least. For now I will just settle for using one card.
Offline
For now I will just settle for using one card.
Or you could run two VMs simultaneously to try to "emulate" GRID K2 setup. Oh, btw, i think it's possible to quadrify your GPUs, check that. That may open you some possibilities.
Last edited by Duelist (2015-05-11 18:01:19)
The forum rules prohibit requesting support for distributions other than arch.
I gave up. It was too late.
What I was trying to do.
The reference about VFIO and KVM VGA passthrough.
Offline
Hello, this is my first post because I came to this forum because of this thread.
https://www.pugetsystems.com/labs/artic … 4-KVM-585/
https://www.pugetsystems.com/labs/artic … Setup-564/
I'm going to send some emails to AMD,Intel,GPU manufactures,and the motherboard manufactures.
I want to create a database of computer component compatibility to make things easier for my self,others,and to pad my resume.
What compatibility questions(Which of your mother board support X?) should I ask ?
Please list everything that I need to ask.
Offline
There is google spreadsheet of known working hardware in this thread.
Offline
Hey all,
After I showed my current setup around I now have the honor to play with a different config...
The system has two GTX 970. One is supposed to work with the host machine and one is for a Windows guest. Since both have the same vendor ID (10de:13c2,...) I'm expecting this to not work correctly, or is there a way to bind the specific device to linux after boot. Or can I filter with PCI stub for the spot ID in addition to the vendor one?
Pls excuse my grammar, in my the phone ...
Best regards,
Beerbelly
Offline
Hey all,
After I showed my current setup around I now have the honor to play with a different config...The system has two GTX 970. One is supposed to work with the host machine and one is for a Windows guest. Since both have the same vendor ID (10de:13c2,...) I'm expecting this to not work correctly, or is there a way to bind the specific device to linux after boot. Or can I filter with PCI stub for the spot ID in addition to the vendor one?
Pls excuse my grammar, in my the phone ...
http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses? Try https://www.redhat.com/mailman/listinfo/vfio-users
Offline
bierbauch wrote:Hey all,
After I showed my current setup around I now have the honor to play with a different config...The system has two GTX 970. One is supposed to work with the host machine and one is for a Windows guest. Since both have the same vendor ID (10de:13c2,...) I'm expecting this to not work correctly, or is there a way to bind the specific device to linux after boot. Or can I filter with PCI stub for the spot ID in addition to the vendor one?
Pls excuse my grammar, in my the phone ...
Y U NO XEN PCI-BACK?
Some users have found the xen-pciback module to be a suitable stand-in for pci-stub with the additional feature that the "hide" option for this module takes device addresses rather than device IDs. I can't load this module on Fedora, so here's my solution that I like a bit better.
Oh.. I guess you need to install xen for that, right? Is it possible to tear that module from xen and make it something like pci-id-stub or vfio-stub or something?
Last edited by Duelist (2015-05-12 21:57:26)
The forum rules prohibit requesting support for distributions other than arch.
I gave up. It was too late.
What I was trying to do.
The reference about VFIO and KVM VGA passthrough.
Offline
@aw :
I saw that you used a "generic" pattern to bind every NVIDIA & AMD devices to VFIO without the need to specify the exact IDs :
options vfio-pci ids=1002:ffffffff:ffffffff:ffffffff:00030000:ffff00ff,1002:ffffffff:ffffffff:ffffffff:00040300:ffffffff,10de:ffffffff:ffffffff:ffffffff:00030000:ffff00ff,10de:ffffffff:ffffffff:ffffffff:00040300:ffffffffCan I use that and SKIP the binding script altogether ? I know my question is confusing , basically what I'm doing now is this :
1 - Binding the devices to pci-stub.ids from the bootloader .
2 - Binding the same devices to VFIO using the binding script (from OP) like this :
vfio-bind 0000:01:00.0 0000:01:00.1 0000:02:00.0 0000:02:00.1 0000:05:00.0 0000:06:00.0 0000:07:00.0 0000:08:00.0 0000:00:1b.0 0000:00:19.03 - Starting VMs .
Wouldn't steps 1&2 become unnecessary if I use :
options vfio-pci ids=1002:ffffffff:ffffffff:ffffffff:00030000:ffff00ff,1002:ffffffff:ffffffff:ffffffff:00040300:ffffffff,10de:ffffffff:ffffffff:ffffffff:00030000:ffff00ff,10de:ffffffff:ffffffff:ffffffff:00040300:ffffffffin modprobe.d ?
A second question if I may :
Can I break the "options vfio-pci ids=" into multiple lines ? like this :
options vfio-pci ids=1002:ffffffff:ffffffff:ffffffff:00030000:ffff00ff,1002:ffffffff:ffffffff:ffffffff:00040300:ffffffff
options vfio-pci ids=10de:ffffffff:ffffffff:ffffffff:00030000:ffff00ff,10de:ffffffff:ffffffff:ffffffff:00040300:ffffffff
options vfio-pci ids=8086:8d20,1912:0015,8086:15a1Thanks !
Last edited by Denso (2015-05-12 22:22:49)
Offline
@aw :
I saw that you used a "generic" pattern to bind every NVIDIA & AMD devices to VFIO without the need to specify the exact IDs :
options vfio-pci ids=1002:ffffffff:ffffffff:ffffffff:00030000:ffff00ff,1002:ffffffff:ffffffff:ffffffff:00040300:ffffffff,10de:ffffffff:ffffffff:ffffffff:00030000:ffff00ff,10de:ffffffff:ffffffff:ffffffff:00040300:ffffffffCan I use that and SKIP the binding script altogether ? I know my question is confusing , basically what I'm doing now is this :
1 - Binding the devices to pci-stub.ids from the bootloader .
2 - Binding the same devices to VFIO using the binding script (from OP) like this :
vfio-bind 0000:01:00.0 0000:01:00.1 0000:02:00.0 0000:02:00.1 0000:05:00.0 0000:06:00.0 0000:07:00.0 0000:08:00.0 0000:00:1b.0 0000:00:19.03 - Starting VMs .
Wouldn't steps 1&2 become unnecessary if I use :
options vfio-pci ids=1002:ffffffff:ffffffff:ffffffff:00030000:ffff00ff,1002:ffffffff:ffffffff:ffffffff:00040300:ffffffff,10de:ffffffff:ffffffff:ffffffff:00030000:ffff00ff,10de:ffffffff:ffffffff:ffffffff:00040300:ffffffffin modprobe.d ?
Yes, vfio-pci.ids option added in 4.1 is meant to negate the need to use pci-stub and bind directly to vfio-pci. The only difference is that pci-stub is often built statically into the kernel while vfio-pci is more typically loaded as a module, so you'll need to configure your initramfs to include all the vfio modules and trigger it to load vfio-pci (the dependencies should all autoload the other modules).
A second question if I may :
Can I break the "options vfio-pci ids=" into multiple lines ? like this :
options vfio-pci ids=1002:ffffffff:ffffffff:ffffffff:00030000:ffff00ff,1002:ffffffff:ffffffff:ffffffff:00040300:ffffffff options vfio-pci ids=10de:ffffffff:ffffffff:ffffffff:00030000:ffff00ff,10de:ffffffff:ffffffff:ffffffff:00040300:ffffffff options vfio-pci ids=8086:8d20,1912:0015,8086:15a1Thanks !
AFAIK, no
http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses? Try https://www.redhat.com/mailman/listinfo/vfio-users
Offline
Y U NO XEN PCI-BACK?
Some users have found the xen-pciback module to be a suitable stand-in for pci-stub with the additional feature that the "hide" option for this module takes device addresses rather than device IDs. I can't load this module on Fedora, so here's my solution that I like a bit better.
Oh.. I guess you need to install xen for that, right? Is it possible to tear that module from xen and make it something like pci-id-stub or vfio-stub or something?
The problem that xen ignores with the pciback module is that the kernel has every right to re-number your PCI bus. What's at 01:00.0 now could be at 02:00.0 in the next kernel release if a tweak were made to bus enumeration. Things like the VT-d spec take this into account, providing paths to devices from known starting points, like the root bus where the bus number is fixed by the platform firmware. So while the xen-pciback method of picking devices usually works and nobody is likely to change PCI bus enumeration for the fun of it, it's flawed and I'm not willing to propose it as an alternative claiming method for vfio-pci. Besides, I showed in the blog post how trivial it is to create a script that moves the problem to userspace and makes use of a less problematic interface than binding drivers by PCI device IDs.
http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses? Try https://www.redhat.com/mailman/listinfo/vfio-users
Offline
Yes, vfio-pci.ids option added in 4.1 is meant to negate the need to use pci-stub and bind directly to vfio-pci. The only difference is that pci-stub is often built statically into the kernel while vfio-pci is more typically loaded as a module, so you'll need to configure your initramfs to include all the vfio modules and trigger it to load vfio-pci (the dependencies should all autoload the other modules).
Thanks , I did add vfio-pci to initramfs (mkinitcpio.conf in Arch) . Pci-stub is also a module in Arch's stock kernel . I had to add it to initramfs too .
AFAIK, no
Thanks ! ![]()
EDIT :
Breaking the "options vfio-pci ids=" to multiple lines is possible . I just broke mine to 3 lines , rebooted and everything is working like a charm :
options vfio-pci ids=1002:ffffffff:ffffffff:ffffffff:00030000:ffff00ff,1002:ffffffff:ffffffff:ffffffff:00040300:ffffffff
options vfio-pci ids=10de:ffffffff:ffffffff:ffffffff:00030000:ffff00ff,10de:ffffffff:ffffffff:ffffffff:00040300:ffffffff
options vfio-pci ids=8086:8d20,1912:0015,8086:15a1This makes adding devices and maintaining the binding list a lot easier and cleaner . ![]()
Last edited by Denso (2015-05-12 23:18:39)
Offline
EDIT :
Breaking the "options vfio-pci ids=" to multiple lines is possible . I just broke mine to 3 lines , rebooted and everything is working like a charm :
Cool
http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses? Try https://www.redhat.com/mailman/listinfo/vfio-users
Offline
I Arrived to PCI passthrough my Nvidia GTX 750 (from EVGA) on ASrock Z97 extrem 6 motherboard with intel 4470 proc by the linux-vfio dedied kernel and add: iommu, ACS, i915 options support on bootloader.
I just would like to add this material ability to the support list: https://docs.google.com/spreadsheet/ccc … _web#gid=0
But how to add it ?
Thank you.
Offline
Good morning
I've followed every hint I've found on the web but I can't make my VGA passthrough work
I'm on Debian testing with official kernel (3.16.0.4 - 3.16.7-ckt9-3), with ACS and i915 patches, distro's QEMU (2.1+dfsg-11) and distro's seabios (1.8.1-2)
This is my PC
Sapphire PURE Platinum H61
Intel i7 3770
Nvidia GTX660 Ti (Zotac, the one I want to passthrough)
8GB of DDR3
I can successfully load the vfio and vfio-pvi modules and associate the GPU and the HDMI audio to it with
modprobe vfio
modprobe vfio-pci
echo '0000:01:00.1' | sudo tee /sys/bus/pci/devices/0000:01:00.1/driver/unbind
echo 1002 6739 | sudo tee /sys/bus/pci/drivers/vfio-pci/new_id
echo 1002 aa88 | sudo tee /sys/bus/pci/drivers/vfio-pci/new_id
With lspci -nnv I can see that the two devices are used by the driver vfio-pci
With these qemu's argument I can successfully run Windows 7 and the GTX660 Ti is visible from there
#!/bin/sh
export QEMU_AUDIO_DRV=alsa
DISKIMG="/media/nicola/data/qemu/WindowsVM.img"
qemu-system-x86_64 --enable-kvm -M q35 -smp 8,sockets=1,cores=4,threads=2 -drive file=${DISKIMG},if=virtio -m 6144 \
-usbdevice tablet -soundhw ac97 -cpu host,kvm=off \
-net nic,model=virtio -net user,hostname=windowsvm \
-device ioh3420,bus=pcie.0,addr=1c.0,multifunction=on,port=1,chassis=1,id=root.1 \
-device piix4-ide,bus=pcie.0,id=piix4-ide \
-device vfio-pci,host=01:00.0,bus=root.1,multifunction=on,x-vga=on,romfile=/home/nicola/Desktop/Zotac.GTX660Ti.2048.120723.rom \
-device vfio-pci,host=01:00.1,bus=root.1 -vga vmware

But if I set -vga none I don't have any output on the discrete GPU. Do you have any hints?
Last edited by corna (2015-05-13 13:13:36)
Offline
Your i915 patch is not working/not enabled in kernel boot options.
edit: you got h61 mbo, i don't think it supports vt-d?
Last edited by slis (2015-05-13 07:35:22)
Offline
In my BIOS configuration there's a "VT-d" switch and dmesg does not output any IOMMU error; this should mean that the VT-d is supported, right?
When I'll get home I'll double-check the BIOS configuration and the GRUB kernel's boot options
Offline
I got this in my dmesg:
root@HOST:~# dmesg | grep IOMMU
[ 0.000000] Intel-IOMMU: enabled
[ 0.012702] dmar: IOMMU 0: reg_base_addr fed90000 ver 1:0 cap c0000020e60262 ecap f0101a
[ 0.012707] dmar: IOMMU 1: reg_base_addr fed91000 ver 1:0 cap c9008020660262 ecap f0105a
[ 0.012778] IOAPIC id 2 under DRHD base 0xfed91000 IOMMU 1
[ 0.398237] IOMMU: dmar0 using Queued invalidation
[ 0.398238] IOMMU: dmar1 using Queued invalidation
[ 0.398239] IOMMU: Setting RMRR:
[ 0.398248] IOMMU: Setting identity map for device 0000:00:02.0 [0xcb800000 - 0xcf9fffff]
[ 0.398526] IOMMU: Setting identity map for device 0000:00:14.0 [0xc95cb000 - 0xc95d7fff]
[ 0.398542] IOMMU: Setting identity map for device 0000:00:1a.0 [0xc95cb000 - 0xc95d7fff]
[ 0.398554] IOMMU: Setting identity map for device 0000:00:1d.0 [0xc95cb000 - 0xc95d7fff]
[ 0.398562] IOMMU: Prepare 0-16MiB unity mapping for LPC
[ 0.398568] IOMMU: Setting identity map for device 0000:00:1f.0 [0x0 - 0xffffff]
and this
root@HOST:~# dmesg | grep vgaarb
[ 0.000000] Kernel command line: BOOT_IMAGE=/boot/vmlinuz-3.19.0-mast root=UUID=d8c8db8d-2f38-479e-9aba-8b9e004cf0b0 ro transparent_hugepage=always intel_iommu=on i915.enable_hd_vgaarb=1 elevator=deadline quiet
[ 0.166672] vgaarb: setting as boot device: PCI:0000:00:02.0
[ 0.166673] vgaarb: device added: PCI:0000:00:02.0,decodes=io+mem,owns=io+mem,locks=none
[ 0.166676] vgaarb: device added: PCI:0000:01:00.0,decodes=io+mem,owns=none,locks=none
[ 0.166677] vgaarb: loaded
[ 0.166678] vgaarb: bridge control possible 0000:01:00.0
[ 0.166679] vgaarb: no bridge control possible 0000:00:02.0
[ 1.488098] vgaarb: device changed decodes: PCI:0000:00:02.0,olddecodes=io+mem,decodes=io:owns=io+mem
Offline
I've checked and both the "Intel Virtualization Technology" and "VT-d" switches are on in my BIOS
This is the output of your commands, almost identical
nicola@debianDesktop:~$ dmesg | grep IOMMU
[ 0.000000] Warning: PCIe ACS overrides enabled; This may allow non-IOMMU protected peer-to-peer DMA
[ 0.000000] Intel-IOMMU: enabled
[ 0.015015] dmar: IOMMU 0: reg_base_addr fed90000 ver 1:0 cap c0000020e60262 ecap f0101a
[ 0.015019] dmar: IOMMU 1: reg_base_addr fed91000 ver 1:0 cap c9008020660262 ecap f0105a
[ 0.015091] IOAPIC id 2 under DRHD base 0xfed91000 IOMMU 1
[ 0.506913] IOMMU 0 0xfed90000: using Queued invalidation
[ 0.506914] IOMMU 1 0xfed91000: using Queued invalidation
[ 0.506915] IOMMU: Setting RMRR:
[ 0.506924] IOMMU: Setting identity map for device 0000:00:02.0 [0xcb800000 - 0xcf9fffff]
[ 0.507212] IOMMU: Setting identity map for device 0000:00:1a.0 [0xca1fa000 - 0xca216fff]
[ 0.507230] IOMMU: Setting identity map for device 0000:00:1d.0 [0xca1fa000 - 0xca216fff]
[ 0.507241] IOMMU: Prepare 0-16MiB unity mapping for LPC
[ 0.507247] IOMMU: Setting identity map for device 0000:00:1f.0 [0x0 - 0xffffff]
[ 0.512879] AMD IOMMUv2 driver by Joerg Roedel <joerg.roedel@amd.com>
[ 0.512881] AMD IOMMUv2 functionality not available on this system
nicola@debianDesktop:~$ dmesg | grep vgaarb
[ 0.000000] Linux version 3.16.0-4-amd64 (debian-kernel@lists.debian.org) (gcc version 4.8.4 (Debian 4.8.4-1) ) #1 SMP Debian 3.16.7-ckt9-3~deb8u1+acsoverridei915vgaarbiter1 (2
[ 0.000000] Command line: BOOT_IMAGE=/vmlinuz-3.16.0-4-amd64 root=UUID=cf3edd70-0d97-4e7a-af69-a5b6beff2e6e ro intel_iommu=on pcie_acs_override=downstream i915.enable_hd_vgaarb=1 quiet
[ 0.000000] Kernel command line: BOOT_IMAGE=/vmlinuz-3.16.0-4-amd64 root=UUID=cf3edd70-0d97-4e7a-af69-a5b6beff2e6e ro intel_iommu=on pcie_acs_override=downstream i915.enable_hd_vgaarb=1 quiet
[ 0.256781] vgaarb: setting as boot device: PCI:0000:00:02.0
[ 0.256782] vgaarb: device added: PCI:0000:00:02.0,decodes=io+mem,owns=io+mem,locks=none
[ 0.256785] vgaarb: device added: PCI:0000:01:00.0,decodes=io+mem,owns=none,locks=none
[ 0.256786] vgaarb: loaded
[ 0.256787] vgaarb: bridge control possible 0000:01:00.0
[ 0.256788] vgaarb: no bridge control possible 0000:00:02.0
[ 2.203712] vgaarb: device changed decodes: PCI:0000:00:02.0,olddecodes=io+mem,decodes=io:owns=io
I'm running the qemu's script with kdesudo; is there a way to run it without sudo privileges?
Ideas?
Last edited by corna (2015-05-13 13:20:06)
Offline
Yeah it all looks good, you should have output on your gpu... old qemu maybe? i don't know.
edit: try ovmf? try libvirt? check aw's blog.
Last edited by slis (2015-05-13 13:42:19)
Offline
The output of rom-parser (http://vfio.blogspot.it/2014/08/does-my … t-efi.html) is this one
Valid ROM signature found @0h, PCIR offset 190h
PCIR: type 0, vendor: 10de, device: 1183, class: 030000
PCIR: revision 0, vendor revision: 1
Last imageso I suppose I can't use ovmf...
What about libvirt? Can you link me the blog?
QEMU's version is 2.1, a recent one
Last edited by corna (2015-05-13 22:07:24)
Offline