You are not logged in.
Hey, so I have been poking around the thread trying to get this working, is the first page's guide still considered correct? I have seen mention of the linux-vfio aur package, but no other posts about how it works, and what needs to be configured on top of it. I guess my confusion revolves specifically around the pci-vfio bind script as I don't remember having to use that or that it was a dead end last time I got this working.
I have gotten this working before, but it's been at least 6 months and a hundred pages since I last read through the thread.
Offline
I would not recommend the vfio bind script to anyone. The better option is to use libvirt to manage the VM. If you're using OVMF for your guest (preferred), you can manage everything natively within the libvirt xml. If you need to go the seabios route due to old guest or old hardware, use a wrapper script to add the x-vga option to the vfio-pci device in the resulting QEMU commandline. If you insist on using q35 (not recommended except for Linux guests), you can also use the wrapper to move the GPU to the root complex or create and use root ports. This usage is really only a workaround until libvirt gets sane support for q35 topology though. Using <qemu:arg> options in the xml is generally not recommended or necessary and causes significant headaches if used to specify the assigned device since it's effectively hidden from libvirt.
http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses? Try https://www.redhat.com/mailman/listinfo/vfio-users
Offline
Thanks for the fast reply!
Ok, so I am planning on using OVMF(I have used it in the past in conjunction with libvirt and know the gotchas that apply to nvidia cards) but before I get there, is there anything with the kernel that I need to do? I have downloaded and compiled linux-vfio, and am about to add a boot configuration for it, but this is where my memory fails me. I don't remember if this is something I had to do to get OVMF to attach to the video card correctly, or if it was something I did that turned out to not apply. And is the required boot option just to enable iommu in the boot parameters on a stock kernel?
Setup:
Intel 4790k(Supports VT-d)
Intel IGD
Nvidia GTX760(Supports EFI)
Offline
@Punkbob
Given your hardware and intended (non-VGA) usage, the only potential patch you'd need vs stock is the ACS override. You can probably avoid that patch as well by being careful not to share processor root ports with other devices or worst case, moving the Nvidia card to a PCH root port. I'm sorry, I don't know what the linux-vfio package is, I don't run Arch myself. Personally I'd try to use the stock kernel for your configuration. I run a GTX750 guest (UEFI) with host IGD on stock Fedora.
http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses? Try https://www.redhat.com/mailman/listinfo/vfio-users
Offline
Bronek wrote:is it independent from http://article.gmane.org/gmane.linux.kernel.pci/40663 , or should I patch my kernel 3.18.12 as well? Or perhaps try with kernel 4.0?
Not at all, it should be independent of kernel version. I've also decided that the reset is not reliable enough for the kernel, so this is the patch proposed for QEMU: https://lists.gnu.org/archive/html/qemu … 03128.html
EDIT: Note that this version only enables the reset for specific device IDs. If you're looking to try it on something else, you'll need to add the ID for your hardware to the list.
@aw
I added my device to your patch and it seems to work, I was able to restart guest (without crashing host) although tried only once. More testing needed because after first restart I noticed brief artifacts on screen of this card, until driver was loaded.
More details of my card, in case you are interested
root@gdansk ~ # lspci -nnv -s 0000:82:00.0
82:00.0 VGA compatible controller [0300]: Advanced Micro Devices, Inc. [AMD/ATI] Tonga PRO GL [FirePro W7100] [1002:692b] (prog-if 00 [VGA controller])
Subsystem: Advanced Micro Devices, Inc. [AMD/ATI] Device [1002:030c]
Flags: fast devsel, IRQ 11
Memory at e0000000 (64-bit, prefetchable) [disabled] [size=256M]
Memory at f0000000 (64-bit, prefetchable) [disabled] [size=2M]
I/O ports at f000 [disabled] [size=256]
Memory at fbd00000 (32-bit, non-prefetchable) [disabled] [size=256K]
Expansion ROM at fbd40000 [disabled] [size=128K]
Capabilities: [48] Vendor Specific Information: Len=08 <?>
Capabilities: [50] Power Management version 3
Capabilities: [58] Express Legacy Endpoint, MSI 00
Capabilities: [a0] MSI: Enable- Count=1/1 Maskable- 64bit+
Capabilities: [100] Vendor Specific Information: ID=0001 Rev=1 Len=010 <?>
Capabilities: [150] Advanced Error Reporting
Capabilities: [200] #15
Capabilities: [270] #19
Capabilities: [2b0] Address Translation Service (ATS)
Capabilities: [2c0] #13
Capabilities: [2d0] #1b
Capabilities: [328] Alternative Routing-ID Interpretation (ARI)
root@gdansk ~ # lspci -nnv -s 0000:82:00.1
82:00.1 Audio device [0403]: Advanced Micro Devices, Inc. [AMD/ATI] Device [1002:aad8]
Subsystem: Advanced Micro Devices, Inc. [AMD/ATI] Device [1002:aad8]
Flags: bus master, fast devsel, latency 0, IRQ 10
Memory at fbd60000 (64-bit, non-prefetchable) [size=16K]
Capabilities: [48] Vendor Specific Information: Len=08 <?>
Capabilities: [50] Power Management version 3
Capabilities: [58] Express Legacy Endpoint, MSI 00
Capabilities: [a0] MSI: Enable- Count=1/1 Maskable- 64bit+
Capabilities: [100] Vendor Specific Information: ID=0001 Rev=1 Len=010 <?>
Capabilities: [150] Advanced Error Reporting
Capabilities: [328] Alternative Routing-ID Interpretation (ARI)
Kernel modules: snd_hda_intelHere is change of your patch:
+ case 0x67B9:
+ case 0x67BA:
+ case 0x67BE:
+ /* Tonga */
+ case 0x692B: /* Tonga PRO GL [Radeon W7100] */
+ vdev->resetfn = vfio_radeon_reset;
+ break;
+ }
+ break;
+ }
+}FWIW, I noticed that I am able to cleanly (probably - didn't try to build yet) apply your patches from mainline kernel on 3.18.12 which I'm using, would you recommend or discourage applying any of those? Assuming I am not merging any other patches from mainline
bronek@gdansk ~/Sources/linux-stable (git)-[v3.18.12] % git log --oneline --reverse -n 21
43f497a Linux 3.18.12
8659c14 PCI: Add flag for devices that don't reset on D3hot->D0 transition
d3aa82b PCI: Mark AMD/ATI VGA devices that don't reset on D3hot->D0 transition
92e46be vfio/type1: DMA unmap chunking
def9cee vfio/type1: Chunk contiguous reserved/invalid page mappings
f7fd50e vfio/type1: Add conditional rescheduling
87498ef vfio: Add device tracking during unbind
afe7fd7 vfio: Tie IOMMU group reference to vfio group
e22eb3a vfio: Add and use device request op for vfio bus drivers
1d8e0e3 vfio-pci: Generalize setup of simple eventfds
e0919875 vfio-pci: Add device request interface
65c825f iommu/vt-d: Detach domain *only* from attached iommus
05b36e6 PCI: Add ACS quirks for Intel 1G NICs
6988212 iommu/vt-d: Remove unused variable
9971924 vgaarb: Stub vga_set_legacy_decoding()
7f56323 vfio-pci: Add module option to disable VGA region access
b7316ee vfio-pci: Add VGA arbiter client
b3d9280 vfio-pci: Allow PCI IDs to be specified as module options
0802afb vfio-pci: Remove warning if try-reset fails
42ee7c2 vfio-pci: Move idle devices to D3hot power state
faadced vfio-pci: Fix use after freeThank you for your hard work on vfio and PCI subsystem :)
Last edited by Bronek (2015-04-27 21:30:28)
Offline
I would not recommend the vfio bind script to anyone.
Thanks for reminding me to remove it from my systemd and system overall - i've moved to libvirt ever since and totally forgot about that script.
The forum rules prohibit requesting support for distributions other than arch.
I gave up. It was too late.
What I was trying to do.
The reference about VFIO and KVM VGA passthrough.
Offline
I'm interested in building a new system for this, with the Code 43 for Nvidia issue and the Radeon Hawaii/Bonaire reset issue, what is the recommended performant card to go with? Wait for the next line of Radeon cards?
Last edited by blakx (2015-04-28 00:16:26)
Offline
Thanks for the help, just one more quick question:
Anyone have experience with getting code 10 errors on the HDMI driver for an Nvidia card?
Edit: NVM, just fixed it.
Last edited by Punkbob (2015-04-28 00:30:48)
Offline
mqddb wrote:Hello everyone:
Will this vga-passthrough can support spice protocol? currently, spice only support qxl vga.No, an assigned GPU does not support any of the -display methods. Remote displays with GPU assignment requires guest-based solutions.
@aw
If any solutions to use spice remote display when an GPU assignment. I am thinking cp the assignment GPU's FB to qxl vga, if windows guest support two vga working at the same time? If you want add this VGA-Passthrough feature to ovirt or openstack, this spice problem must be resolve, right? Do you have any plan to do this?
Offline
Has anybody actually had success following the Ubuntu guide? Seems like a flood of people following whatever misinformation it contains and landing here.
I've had success with Ubuntu. Though I did not follow any guide, my configurations are based on documentation and general experiences from users in various forums. I'm planning to evaluate other hardware configurations and document my method in a HowTo article once satisfied with the results. For those interested in seeing Ubuntu in action, recently I published a video which demonstrates Windows Gaming with QEMU/KVM on Ubuntu Linux.
Offline
aw wrote:Has anybody actually had success following the Ubuntu guide? Seems like a flood of people following whatever misinformation it contains and landing here.
I've had success with Ubuntu. Though I did not follow any guide, my configurations are based on documentation and general experiences from users in various forums. I'm planning to evaluate other hardware configurations and document my method in a HowTo article once satisfied with the results. For those interested in seeing Ubuntu in action, recently I published a video which demonstrates Windows Gaming with QEMU/KVM on Ubuntu Linux.
R9 280 also has reset problems? That ought to be Tahiti based. Have you tried updating the Bonaire/Hawaii patch as Bronek did to add your ID? https://lists.gnu.org/archive/html/qemu … 03128.html
http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses? Try https://www.redhat.com/mailman/listinfo/vfio-users
Offline
aw wrote:mqddb wrote:Hello everyone:
Will this vga-passthrough can support spice protocol? currently, spice only support qxl vga.No, an assigned GPU does not support any of the -display methods. Remote displays with GPU assignment requires guest-based solutions.
@aw
If any solutions to use spice remote display when an GPU assignment. I am thinking cp the assignment GPU's FB to qxl vga, if windows guest support two vga working at the same time? If you want add this VGA-Passthrough feature to ovirt or openstack, this spice problem must be resolve, right? Do you have any plan to do this?
Nope. Scraping the framebuffer requires knowing where the framebuffer lives, which requires far more knowledge of proprietary drivers than I'm interested in. There are client based solutions for this, NICE DCV, HP RGS, etc...
http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses? Try https://www.redhat.com/mailman/listinfo/vfio-users
Offline
@Bronek
List looks like a pretty complete sync to v4.1-rc1. There's one more fix in the queue for e0919875, it's been posted, but not yet pushed. I'll try to get it in for rc2. Please update with further testing on Tonga.
http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses? Try https://www.redhat.com/mailman/listinfo/vfio-users
Offline
I'm interested in building a new system for this, with the Code 43 for Nvidia issue and the Radeon Hawaii/Bonaire reset issue, what is the recommended performant card to go with? Wait for the next line of Radeon cards?
Radeon R9 290X (more specifically, Saphire Tri-X UEFI enabled) worked very well for me, but now I'm running two virtual machines and need space for 2 GPUs and a bunch of other cards, so it was simply too large. It is slightly larger than two slots, so it had to go, for only this reason. I do not have problems with AMD FirePro V series but they are quite expensive for what they do, similarly more expensive model of NVidia Quaddro should work for you since they are "enabled" for GPU pass-through (e.g. K2200 , K4200 etc) but again they are very expensive for what they do. I'd guess Radeon R9 should work for you and is economical choice, but this is also a question of lucky/good choice of vendor and model.
Last edited by Bronek (2015-04-28 09:05:24)
Offline
@Bronek
List looks like a pretty complete sync to v4.1-rc1. There's one more fix in the queue for e0919875, it's been posted, but not yet pushed. I'll try to get it in for rc2. Please update with further testing on Tonga.
Something is not right, but I suspect it's not with qemu but vfio (or something else in kernel). Occasionally when I shutdown VM guest and then start it up again, it seems like its trying to use wrong PCI device at the moment when driver gets loaded. Not a GPU card, something entirely different. At first Windows start appears as normal, then it blanks and a cursor appears on black background (which is normal for short time when driver starts), at which moment "dmesg -w" starts showing errors from a different PCIe card and its driver (slot 84:00.0 is Intel P3700, driver nvme) and Windows guest remains with blank screen.
[80149.889741] dmar: DRHD: handling fault status reg 2
[80149.894627] dmar: DMAR:[DMA Read] Request device [84:00.0] fault addr ffe0e000
DMAR:[fault reason 06] PTE Read access is not set
[80149.908554] dmar: DRHD: handling fault status reg 102
[80149.913614] dmar: DMAR:[DMA Write] Request device [84:00.0] fault addr ffff9000
DMAR:[fault reason 05] PTE Write access is not set
[80150.052146] usb 3-1: reset low-speed USB device number 2 using xhci_hcd
[80150.340244] xhci_hcd 0000:08:00.0: xHCI xhci_drop_endpoint called with disabled ep ffff8810266a4600
[80150.340251] xhci_hcd 0000:08:00.0: xHCI xhci_drop_endpoint called with disabled ep ffff8810266a4540
[80150.340259] usb 3-1: ep 0x81 - rounding interval to 64 microframes, ep desc says 80 microframes
[80150.340265] usb 3-1: ep 0x82 - rounding interval to 64 microframes, ep desc says 80 microframes
[80150.485412] dmar: DRHD: handling fault status reg 202
[80150.490475] dmar: DMAR:[DMA Read] Request device [84:00.0] fault addr fff24000
DMAR:[fault reason 06] PTE Read access is not set
[80150.622368] usb 3-1: reset low-speed USB device number 2 using xhci_hcd
[80150.910482] xhci_hcd 0000:08:00.0: xHCI xhci_drop_endpoint called with disabled ep ffff8810266a4600
[80150.910489] xhci_hcd 0000:08:00.0: xHCI xhci_drop_endpoint called with disabled ep ffff8810266a4540
[80150.910497] usb 3-1: ep 0x81 - rounding interval to 64 microframes, ep desc says 80 microframes
[80150.910502] usb 3-1: ep 0x82 - rounding interval to 64 microframes, ep desc says 80 microframes
[80150.966996] dmar: DRHD: handling fault status reg 302
[80150.972052] dmar: DMAR:[DMA Read] Request device [84:00.0] fault addr ffeca000
DMAR:[fault reason 06] PTE Read access is not set
[80151.560200] dmar: DRHD: handling fault status reg 402
[80151.565259] dmar: DMAR:[DMA Read] Request device [84:00.0] fault addr fffca000
DMAR:[fault reason 06] PTE Read access is not set
[80155.227534] dmar: DRHD: handling fault status reg 502
[80155.232595] dmar: DMAR:[DMA Read] Request device [84:00.0] fault addr fffad000
DMAR:[fault reason 06] PTE Read access is not set
[80155.246330] dmar: DRHD: handling fault status reg 602
[80155.251410] dmar: DMAR:[DMA Read] Request device [84:00.0] fault addr fff02000
DMAR:[fault reason 06] PTE Read access is not set
[80170.201208] dmar: DRHD: handling fault status reg 702
[80170.206278] dmar: DMAR:[DMA Write] Request device [84:00.0] fault addr ffff9000
DMAR:[fault reason 05] PTE Write access is not set
[80180.114179] nvme 0000:84:00.0: Aborting I/O 0 QID 1
[80180.114186] nvme 0000:84:00.0: Aborting I/O 1 QID 1
[80180.114194] nvme 0000:84:00.0: Aborting I/O 0 QID 16
[80180.114197] nvme 0000:84:00.0: Aborting I/O 1 QID 16
[80180.114279] dmar: DRHD: handling fault status reg 2
[80180.119169] dmar: DMAR:[DMA Write] Request device [84:00.0] fault addr fffff000
DMAR:[fault reason 05] PTE Write access is not setFWIW my current kernel does not any of your newer patches beyond 3.18.12
Last edited by Bronek (2015-04-28 19:24:19)
Offline
@Bronek
Are you using the ACS override patch? What's the topology (lspci -tv)?
http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses? Try https://www.redhat.com/mailman/listinfo/vfio-users
Offline
Linuxlookup wrote:aw wrote:Has anybody actually had success following the Ubuntu guide? Seems like a flood of people following whatever misinformation it contains and landing here.
I've had success with Ubuntu. Though I did not follow any guide, my configurations are based on documentation and general experiences from users in various forums. I'm planning to evaluate other hardware configurations and document my method in a HowTo article once satisfied with the results. For those interested in seeing Ubuntu in action, recently I published a video which demonstrates Windows Gaming with QEMU/KVM on Ubuntu Linux.
R9 280 also has reset problems? That ought to be Tahiti based. Have you tried updating the Bonaire/Hawaii patch as Bronek did to add your ID? https://lists.gnu.org/archive/html/qemu … 03128.html
Yes, I too have experienced a reset issue with AMD R9 280. Few months back I saw your Qemu-devel post from 12/02/14 and experimented with information provided. Unsuccessful, I haven't pursued any further solution myself.
Offline
I have tried several more things trying to fix this freezing/bsod issue. Somewhere along the line it seems it's changed so that I no longer get freezes, just BSOD. At one point I did get a host freeze, unsure if this was caused by the VM or not.
Made a new clean UEFI install of Windows 7, GPT on LVM on LUKS on mdadm raid1. BSOD now reboots the VM, likely the default behavior for windows. Event viewer says code is 0x116.
Changed to kvm-pci-assign. Issue still persisted, changed back to vfio-pci after.
Changed some settings to match closer to Duelist's posted configuration.
root=UUID=4f2e65ef-ef21-4628-bbd7-219d7855732b rw quiet cryptdevice=UUID=17b9c41f-ab59-4a68-a5ec-b7b3e2776bf6:calue pci-stub.ids=1002:67b1,1002:aac8 iommu=pt acpi_enforce_resources=laxoptions vfio_iommu_type1 disable_hugepages=1
options kvm-amd npt=1 nested=0#!/bin/sh
export QEMU_AUDIO_DRV=sdl
export QEMU_AUDIO_DAC_FIXED_SETTINGS=1
export QEMU_AUDIO_DAC_FIXED_FREQ=44100
export QEMU_AUDIO_DAC_FIXED_FMT=S16
export QEMU_SDL_SAMPLES=1024
qemu-system-x86_64 \
-machine accel=kvm \
-machine pc-i440fx-2.1,accel=kvm,usb=off \
-cpu Opteron_G5,+bmi1,+perfctr_nb,+perfctr_core,+topoext,+nodeid_msr,+tce,+lwp,+wdt,+skinit,+ibs,+osvw,+cr8legacy,+extapic,+cmp_legacy,+fxsr_opt,+mmxext,+osxsave,+monitor,+ht,+vme \
-drive file=OVMF-newinstall.fd,format=raw,if=pflash \
-drive file=/dev/mapper/urw-win,format=raw,if=ide \
-m 4096 \
-realtime mlock=on \
-smp 3 \
-no-user-config \
-nodefaults \
-rtc base=localtime,driftfix=slew \
-global kvm-pit.lost_tick_policy=discard \
-no-hpet \
-global PIIX4_PM.disable_s3=1 \
-global PIIX4_PM.disable_s4=1 \
-device ich9-usb-ehci1,id=usb,bus=pci.0,addr=0x5.0x7 \
-device ich9-usb-uhci1,masterbus=usb.0,firstport=0,bus=pci.0,multifunction=on,addr=0x5 \
-device ich9-usb-uhci2,masterbus=usb.0,firstport=2,bus=pci.0,addr=0x5.0x1 \
-device ich9-usb-uhci3,masterbus=usb.0,firstport=4,bus=pci.0,addr=0x5.0x2 \
-device usb-host,id=usbkbdms,bus=usb.0,vendorid=0x13ba,productid=0x0018 \
-netdev user,id=net \
-device rtl8139,netdev=net,bus=pci.0,addr=0x6 \
-device vfio-pci,host=01:00.0,bus=pci.0,addr=0x8 \
-device vfio-pci,host=01:00.1,bus=pci.0,addr=0x9 \
-vga none \
-soundhw hda \
$*(I did manage to mostly fix my audio issues by using sdl instead of alsa, and changing asound.conf to use 44100 Hz for dmix.)
Issue still persists.
Removed Radeon 9250 PCI card from motherboard, wasn't using it anyway. Didn't fix.
Removed "blacklist radeon" from modprobe.d. Guest card already stubbed through pci-stub.
Ran windows update on guest. Didn't fix issue.
Started running VM from remote system over SSH with -nographic. Didn't fix issue.
Removed host's card (GeForce 9800 GT), no longer running X on host, running VM over ssh. Didn't fix issue.
I am going now to move the guest-assigned card to another PCI-e x16 slot and attempt again, not expecting any success. I have a powered x1 to x16 riser which I had been running this card on previously on an older motherboard, I may try that later, not expecting any success.
I don't really understand why this is an issue. I can even shutdown the guest, unbind the card from vfio and bind it to radeon, then log in and launch X on the host and use it without an issue. I'm thinking this could be a bug in the guest radeon drivers. Perhaps I might try some older versions of the driver. I'm running out of things left to test. I could run linux within the VM and attempt to get anything on the guest's dmesg output, but I don't know.
It's really frustrating being this close to success and not being able to do anything about it.
Offline
You have a Quadro 2000, just assigned it as a secondary device and be done with it. https://access.redhat.com/documentation … e-GPU.html
If it doesn't work, update kernel and/or QEMU. Note that only K-series Quadros are actually supported, but Fermi are likely to work. Quadro cards do not work well when configured as the primary display for a VM.
I am trying to build a machine where the host is a server so basically no UI except SSH. Anyway, I was pointing to the fact that there is a bug somewhere that is causing the host to crash/freeze. Given the advanced experimentation going on, not really surprised, just wondering on what is the best action, if there is a bug tracker, etc.
I am planning to get a UEFI bios for my card, PNY technical support claims that they have a version of the BIOS with UEFI support, so trying that route. Thank you very much, I am impressed by how much of this stuff works anyway!!
Offline
aw wrote:You have a Quadro 2000, just assigned it as a secondary device and be done with it. https://access.redhat.com/documentation … e-GPU.html
If it doesn't work, update kernel and/or QEMU. Note that only K-series Quadros are actually supported, but Fermi are likely to work. Quadro cards do not work well when configured as the primary display for a VM.
I am trying to build a machine where the host is a server so basically no UI except SSH. Anyway, I was pointing to the fact that there is a bug somewhere that is causing the host to crash/freeze. Given the advanced experimentation going on, not really surprised, just wondering on what is the best action, if there is a bug tracker, etc.
Your kernel is too old to be interesting for upstream bugs. You can see if the problem still occurs on current upstream or file a bug with your distro.
I am planning to get a UEFI bios for my card, PNY technical support claims that they have a version of the BIOS with UEFI support, so trying that route. Thank you very much, I am impressed by how much of this stuff works anyway!!
That doesn't solve the problem that Quadro cards don't generally behave as VM primary displays, if you use the secondary display approach, it doesn't matter whether you have a legacy or UEFI BIOS for the card.
http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses? Try https://www.redhat.com/mailman/listinfo/vfio-users
Offline
That doesn't solve the problem that Quadro cards don't generally behave as VM primary displays
Where can I find more info about this? If it boots, runs and handle Premier 6 CUDA acceleration, I am happy. Investigating about the last part, but I am interested all there is to know about this. Thanks a lot!
Offline
That doesn't solve the problem that Quadro cards don't generally behave as VM primary displays
Where can I find more info about this? If it boots, runs and handle Premier 6 CUDA acceleration, I am happy. Investigating about the last part, but I am interested all there is to know about this. Thanks a lot!
Try it, you'll see. Given that you have the hardware to run in a mode that Nvidia actually claims to support, I'm not sure why you wouldn't want to run in that mode. Some people just like to make things more difficult...
http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses? Try https://www.redhat.com/mailman/listinfo/vfio-users
Offline
@Bronek
Are you using the ACS override patch? What's the topology (lspci -tv)?
No ACS override patch here, only patch on top of 3.18.12 I have at this moment is from "git diff c143c2333c4..3ccb354d641" (VFS change in kernel mainline), which is a workaround for ZFS issue https://github.com/zfsonlinux/zfs/issues/3030
Here is topology (with list of vfio devices):
root@gdansk ~ # cat /proc/cmdline
BOOT_IMAGE=../vmlinuz-linux zfs=zroot rw spl.spl_hostid=0xa8c08502 console=ttyS0,115200N8R nomodeset udev.children-max=32 edac_core.edac_mc_panic_on_ue=1 intel_iommu=on pci-stub.ids=8086:1d2d,8086:1d20,8086:1d26,1912:0014,1002:692b,1002:aad8,1002:6707,1002:aa80 isolcpus=2,3,4,5,6,7,8,9,10,11,14,15,16,17,18,19,20,21,22,23 initrd=../initramfs-linux.img
root@gdansk ~ # lspci -nnvt
-+-[0000:ff]-+-08.0 Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 QPI Link 0 [8086:0e80]
| +-09.0 Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 QPI Link 1 [8086:0e90]
| +-0a.0 Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 Power Control Unit 0 [8086:0ec0]
| +-0a.1 Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 Power Control Unit 1 [8086:0ec1]
| +-0a.2 Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 Power Control Unit 2 [8086:0ec2]
| +-0a.3 Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 Power Control Unit 3 [8086:0ec3]
| +-0b.0 Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 UBOX Registers [8086:0e1e]
| +-0b.3 Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 UBOX Registers [8086:0e1f]
| +-0c.0 Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 Unicast Registers [8086:0ee0]
| +-0c.1 Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 Unicast Registers [8086:0ee2]
| +-0c.2 Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 Unicast Registers [8086:0ee4]
| +-0d.0 Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 Unicast Registers [8086:0ee1]
| +-0d.1 Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 Unicast Registers [8086:0ee3]
| +-0d.2 Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 Unicast Registers [8086:0ee5]
| +-0e.0 Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 Home Agent 0 [8086:0ea0]
| +-0e.1 Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 Home Agent 0 [8086:0e30]
| +-0f.0 Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 Integrated Memory Controller 0 Target Address/Thermal Registers [8086:0ea8]
| +-0f.1 Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 Integrated Memory Controller 0 RAS Registers [8086:0e71]
| +-0f.2 Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 Integrated Memory Controller 0 Channel Target Address Decoder Registers [8086:0eaa]
| +-0f.3 Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 Integrated Memory Controller 0 Channel Target Address Decoder Registers [8086:0eab]
| +-0f.4 Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 Integrated Memory Controller 0 Channel Target Address Decoder Registers [8086:0eac]
| +-0f.5 Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 Integrated Memory Controller 0 Channel Target Address Decoder Registers [8086:0ead]
| +-10.0 Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 Integrated Memory Controller 1 Channel 0-3 Thermal Control 0 [8086:0eb0]
| +-10.1 Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 Integrated Memory Controller 1 Channel 0-3 Thermal Control 1 [8086:0eb1]
| +-10.2 Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 Integrated Memory Controller 1 Channel 0-3 ERROR Registers 0 [8086:0eb2]
| +-10.3 Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 Integrated Memory Controller 1 Channel 0-3 ERROR Registers 1 [8086:0eb3]
| +-10.4 Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 Integrated Memory Controller 1 Channel 0-3 Thermal Control 2 [8086:0eb4]
| +-10.5 Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 Integrated Memory Controller 1 Channel 0-3 Thermal Control 3 [8086:0eb5]
| +-10.6 Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 Integrated Memory Controller 1 Channel 0-3 ERROR Registers 2 [8086:0eb6]
| +-10.7 Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 Integrated Memory Controller 1 Channel 0-3 ERROR Registers 3 [8086:0eb7]
| +-13.0 Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 R2PCIe [8086:0e1d]
| +-13.1 Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 R2PCIe [8086:0e34]
| +-13.4 Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 QPI Ring Registers [8086:0e81]
| +-13.5 Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 QPI Ring Performance Ring Monitoring [8086:0e36]
| +-16.0 Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 System Address Decoder [8086:0ec8]
| +-16.1 Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 Broadcast Registers [8086:0ec9]
| \-16.2 Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 Broadcast Registers [8086:0eca]
+-[0000:80]-+-01.0-[81]----00.0 Renesas Technology Corp. uPD720201 USB 3.0 Host Controller [1912:0014]
| +-02.0-[82]--+-00.0 Advanced Micro Devices, Inc. [AMD/ATI] Tonga PRO GL [FirePro W7100] [1002:692b]
| | \-00.1 Advanced Micro Devices, Inc. [AMD/ATI] Device [1002:aad8]
| +-03.0-[83]----00.0 LSI Logic / Symbios Logic MegaRAID SAS 2208 [Thunderbolt] [1000:005b]
| +-03.2-[84]----00.0 Intel Corporation PCIe Data Center SSD [8086:0953]
| +-04.0 Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 Crystal Beach DMA Channel 0 [8086:0e20]
| +-04.1 Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 Crystal Beach DMA Channel 1 [8086:0e21]
| +-04.2 Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 Crystal Beach DMA Channel 2 [8086:0e22]
| +-04.3 Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 Crystal Beach DMA Channel 3 [8086:0e23]
| +-04.4 Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 Crystal Beach DMA Channel 4 [8086:0e24]
| +-04.5 Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 Crystal Beach DMA Channel 5 [8086:0e25]
| +-04.6 Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 Crystal Beach DMA Channel 6 [8086:0e26]
| +-04.7 Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 Crystal Beach DMA Channel 7 [8086:0e27]
| +-05.0 Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 VTd/Memory Map/Misc [8086:0e28]
| +-05.2 Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 IIO RAS [8086:0e2a]
| \-05.4 Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 IOAPIC [8086:0e2c]
+-[0000:7f]-+-08.0 Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 QPI Link 0 [8086:0e80]
| +-09.0 Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 QPI Link 1 [8086:0e90]
| +-0a.0 Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 Power Control Unit 0 [8086:0ec0]
| +-0a.1 Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 Power Control Unit 1 [8086:0ec1]
| +-0a.2 Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 Power Control Unit 2 [8086:0ec2]
| +-0a.3 Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 Power Control Unit 3 [8086:0ec3]
| +-0b.0 Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 UBOX Registers [8086:0e1e]
| +-0b.3 Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 UBOX Registers [8086:0e1f]
| +-0c.0 Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 Unicast Registers [8086:0ee0]
| +-0c.1 Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 Unicast Registers [8086:0ee2]
| +-0c.2 Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 Unicast Registers [8086:0ee4]
| +-0d.0 Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 Unicast Registers [8086:0ee1]
| +-0d.1 Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 Unicast Registers [8086:0ee3]
| +-0d.2 Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 Unicast Registers [8086:0ee5]
| +-0e.0 Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 Home Agent 0 [8086:0ea0]
| +-0e.1 Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 Home Agent 0 [8086:0e30]
| +-0f.0 Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 Integrated Memory Controller 0 Target Address/Thermal Registers [8086:0ea8]
| +-0f.1 Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 Integrated Memory Controller 0 RAS Registers [8086:0e71]
| +-0f.2 Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 Integrated Memory Controller 0 Channel Target Address Decoder Registers [8086:0eaa]
| +-0f.3 Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 Integrated Memory Controller 0 Channel Target Address Decoder Registers [8086:0eab]
| +-0f.4 Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 Integrated Memory Controller 0 Channel Target Address Decoder Registers [8086:0eac]
| +-0f.5 Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 Integrated Memory Controller 0 Channel Target Address Decoder Registers [8086:0ead]
| +-10.0 Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 Integrated Memory Controller 1 Channel 0-3 Thermal Control 0 [8086:0eb0]
| +-10.1 Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 Integrated Memory Controller 1 Channel 0-3 Thermal Control 1 [8086:0eb1]
| +-10.2 Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 Integrated Memory Controller 1 Channel 0-3 ERROR Registers 0 [8086:0eb2]
| +-10.3 Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 Integrated Memory Controller 1 Channel 0-3 ERROR Registers 1 [8086:0eb3]
| +-10.4 Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 Integrated Memory Controller 1 Channel 0-3 Thermal Control 2 [8086:0eb4]
| +-10.5 Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 Integrated Memory Controller 1 Channel 0-3 Thermal Control 3 [8086:0eb5]
| +-10.6 Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 Integrated Memory Controller 1 Channel 0-3 ERROR Registers 2 [8086:0eb6]
| +-10.7 Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 Integrated Memory Controller 1 Channel 0-3 ERROR Registers 3 [8086:0eb7]
| +-13.0 Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 R2PCIe [8086:0e1d]
| +-13.1 Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 R2PCIe [8086:0e34]
| +-13.4 Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 QPI Ring Registers [8086:0e81]
| +-13.5 Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 QPI Ring Performance Ring Monitoring [8086:0e36]
| +-16.0 Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 System Address Decoder [8086:0ec8]
| +-16.1 Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 Broadcast Registers [8086:0ec9]
| \-16.2 Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 Broadcast Registers [8086:0eca]
\-[0000:00]-+-00.0 Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 DMI2 [8086:0e00]
+-01.0-[01]----00.0 LSI Logic / Symbios Logic SAS2308 PCI-Express Fusion-MPT SAS-2 [1000:0086]
+-02.0-[02]----00.0 Renesas Technology Corp. uPD720201 USB 3.0 Host Controller [1912:0014]
+-03.0-[03]--+-00.0 Advanced Micro Devices, Inc. [AMD/ATI] Cayman LE GL [FirePro V5900] [1002:6707]
| \-00.1 Advanced Micro Devices, Inc. [AMD/ATI] Cayman/Antilles HDMI Audio [Radeon HD 6900 Series] [1002:aa80]
+-04.0 Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 Crystal Beach DMA Channel 0 [8086:0e20]
+-04.1 Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 Crystal Beach DMA Channel 1 [8086:0e21]
+-04.2 Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 Crystal Beach DMA Channel 2 [8086:0e22]
+-04.3 Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 Crystal Beach DMA Channel 3 [8086:0e23]
+-04.4 Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 Crystal Beach DMA Channel 4 [8086:0e24]
+-04.5 Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 Crystal Beach DMA Channel 5 [8086:0e25]
+-04.6 Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 Crystal Beach DMA Channel 6 [8086:0e26]
+-04.7 Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 Crystal Beach DMA Channel 7 [8086:0e27]
+-05.0 Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 VTd/Memory Map/Misc [8086:0e28]
+-05.2 Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 IIO RAS [8086:0e2a]
+-05.4 Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 IOAPIC [8086:0e2c]
+-11.0-[04]----00.0 Intel Corporation C602 chipset 4-Port SATA Storage Control Unit [8086:1d6b]
+-16.0 Intel Corporation C600/X79 series chipset MEI Controller #1 [8086:1d3a]
+-16.1 Intel Corporation C600/X79 series chipset MEI Controller #2 [8086:1d3b]
+-1a.0 Intel Corporation C600/X79 series chipset USB2 Enhanced Host Controller #2 [8086:1d2d]
+-1b.0 Intel Corporation C600/X79 series chipset High Definition Audio Controller [8086:1d20]
+-1c.0-[05-06]--+-00.0 Intel Corporation I350 Gigabit Network Connection [8086:1521]
| \-00.1 Intel Corporation I350 Gigabit Network Connection [8086:1521]
+-1c.5-[07]----00.0 NEC Corporation uPD720200 USB 3.0 Host Controller [1033:0194]
+-1c.6-[08]----00.0 NEC Corporation uPD720200 USB 3.0 Host Controller [1033:0194]
+-1d.0 Intel Corporation C600/X79 series chipset USB2 Enhanced Host Controller #1 [8086:1d26]
+-1e.0-[09]--
+-1f.0 Intel Corporation C600/X79 series chipset LPC Controller [8086:1d41]
+-1f.2 Intel Corporation C600/X79 series chipset 6-Port SATA AHCI Controller [8086:1d02]
+-1f.3 Intel Corporation C600/X79 series chipset SMBus Host Controller [8086:1d22]
\-1f.6 Intel Corporation C600/X79 series chipset Thermal Management Controller [8086:1d24]I think I had these occasional problems with 84:00 before, but I do not remember whether these happened when I was using other graphic card (I also have FirePro V4900) or not.
Last edited by Bronek (2015-04-29 18:24:03)
Offline
Hey everyone! I have been trying to set this up on my system for hours now and have came to a standstill, lol.
After I add the kernel parameter
"pci-stub.ids=10de:13c0,10de:0fbb" my system fails to get passed the "Reached target Graphical Interface" tab while booting. If I remove that parameter it boots fine.
Also when running,
"qemu-system-x86_64 -enable-kvm -m 8000 -cpu host,kvm=off -vga none -device vfio-pci,host=07:00.0 -smp 4,sockets=1,cores=4,threads=2 -hda /games/win.cow"I get the error
"qemu-system-x86_64: -device vfio-pci,host=07:00.0: vfio: error opening /dev/vfio/23: No such file or directory
qemu-system-x86_64: -device vfio-pci,host=07:00.0: vfio: failed to get group 23
qemu-system-x86_64: -device vfio-pci,host=07:00.0: Device initialization failed.
qemu-system-x86_64: -device vfio-pci,host=07:00.0: Device 'vfio-pci' could not be initialized"I realize that the error for qemu is probably from the system not booting with the pci-stub kernel parameter, but I can't boot while its on.
Last edited by Zerqz (2015-04-30 01:21:40)
Offline
Hey everyone! I have been trying to set this up on my system for hours now and have came to a standstill, lol.
After I add the kernel parameter
"pci-stub.ids=10de:13c0,10de:0fbb"my system fails to get passed the "Reached target Graphical Interface" tab while booting. If I remove that parameter it boots fine.
Do you have.. a second GPU that should be left for host?
What distrib, kernel and hardware are you using? You sure that IOMMU is on?
The forum rules prohibit requesting support for distributions other than arch.
I gave up. It was too late.
What I was trying to do.
The reference about VFIO and KVM VGA passthrough.
Offline