You are not logged in.
There are really very, very few reasons that anyone here should be using <qemu:arg> options any longer. If you are, you're probably doing something wrong.
Correct me if I'm wrong, but Win 7 guests and below who use libvirt will still use qemu:arg since apparently OVMF + EFI + Win 7 seems pretty fragile, at least from what I've read here. So you're referring to Win 8+
No, you should be using a wrapper script around the qemu binary to add the x-vga=on option so that libvirt can manage the vfio device itself.
http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses? Try https://www.redhat.com/mailman/listinfo/vfio-users
Offline
Hmm obviously I missed that somewhere, could you give a link to the page number? EDIT: never mind, found it post 3262... whoops too late thanks though
Last edited by mostlyharmless (2015-01-31 00:05:32)
Offline
Hmm obviously I missed that somewhere, could you give a link to the page number?
https://bbs.archlinux.org/viewtopic.php … 1#p1475541
EDIT: PS, letting libvirt manage the device is how you avoid needing to elevate permissions for the VM or run it as root. You should not need to touch anything in /etc/libvirt/qemu.conf
Last edited by aw (2015-01-31 01:32:41)
http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses? Try https://www.redhat.com/mailman/listinfo/vfio-users
Offline
Guys, GIVE THIS A READ.
Checking the Device Manager in my Windows guest, I have noticed that my GPU was not using MSI. I have followed the guide linked to Alex's blog post and I had made the registry changes in Windows and I was able to get around 10% improvement in benchmarks and the overall performance has definitely improved. I will link the benchmark results later.
Thanks "aw" for your hardwork!
Offline
...you're probably doing something wrong.
You're absolutely right!
I'd try the latter set, but you're missing an update to those instructions here:
http://vfio.blogspot.com/2014/09/libvir … -ovmf.html
My slides from KVM Forum include a further updated setup (hit space a couple times):
http://awilliam.github.io/presentations … 2014/#/4/2
Thanks for this direction Alex, I thought I had read most things but my comprehension of what it all means is somewhat lacking.
I have set up the VM using virt-manager and am trying to edit the domain xml. I have installed the ovmf-svn package from the AUR but I am confused to what I should edit these lines to:
<domain type='kvm'>
...
<os>
<loader readonly='yes' type='pflash'>/usr/share/edk2.git/ovmf-x64/OVMF_CODE-pure-efi.fd</loader>
<nvram template='/usr/share/edk2.git/ovmf-x64/OVMF_VARS-pure-efi.fd'/>
...
</os>
</domain>
I am confused because under /usr/share I do not have an edk2.git directory and:
# find / -name edk2*
returns nothing.
*Edit*
$ pacman -Ql ovmf-svn
ovmf-svn /usr/
ovmf-svn /usr/share/
ovmf-svn /usr/share/ovmf/
ovmf-svn /usr/share/ovmf/ia32/
ovmf-svn /usr/share/ovmf/ia32/ovmf_code_ia32.bin
ovmf-svn /usr/share/ovmf/ia32/ovmf_ia32.bin
ovmf-svn /usr/share/ovmf/ia32/ovmf_vars_ia32.bin
ovmf-svn /usr/share/ovmf/x64/
ovmf-svn /usr/share/ovmf/x64/ovmf_code_x64.bin
ovmf-svn /usr/share/ovmf/x64/ovmf_vars_x64.bin
ovmf-svn /usr/share/ovmf/x64/ovmf_x64.bin
Which of these, if any, do I need to point to in the domain xml list?
Last edited by cdrjameson (2015-01-31 11:44:16)
Offline
$ pacman -Ql ovmf-svn ovmf-svn /usr/ ovmf-svn /usr/share/ ovmf-svn /usr/share/ovmf/ ovmf-svn /usr/share/ovmf/ia32/ ovmf-svn /usr/share/ovmf/ia32/ovmf_code_ia32.bin ovmf-svn /usr/share/ovmf/ia32/ovmf_ia32.bin ovmf-svn /usr/share/ovmf/ia32/ovmf_vars_ia32.bin ovmf-svn /usr/share/ovmf/x64/ ovmf-svn /usr/share/ovmf/x64/ovmf_code_x64.bin <--- CODE ovmf-svn /usr/share/ovmf/x64/ovmf_vars_x64.bin <--- VARS ovmf-svn /usr/share/ovmf/x64/ovmf_x64.bin
http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses? Try https://www.redhat.com/mailman/listinfo/vfio-users
Offline
Bronek wrote:Ok so I've finally setup serial console and my host is finally headless. I was able to capture this when restarting virtual machine, which is Windows 7 (no EFI). This happens when VM is started after guest reboot
[ 250.730914] kernel BUG at drivers/pci/ats.c:138!
My kernel is 3.17.8 , I'm planning upgrade to 3.18 "any moment now" but due to zfs used as a root filesystem this requires little work. This only happens if I restart virtual machine which is using FirePro W7100 - restart of the other virtual machine using FirePro V4900 does not trigger this bug. After hitting this bug it is not possible to gracefully shutdown qemu.
That's saying that the device had ATS (Address Translation Services) enabled before reset, but when we went to restore the state of it after reset, we couldn't even find the capability. Maybe the device never really recovers from reset?
Got it. I suppose I should complain to AMD support ...
Offline
Good morning everyone! First of all, thank you nbhs for your awesome KVM guide and giving me a hope of using my GPU with KVM!
However, I'm having a little problem here with my setup.
I've managed to 'unbind' the GPU from the host and do the vfio-bind stuff and it seems that QEMU picks it up and tries to use it, but I don't see anything on the screen: it's either "no signal" warning and then powersave mode, or just the black screen (the monitor, however, acts as if there is something, no powersave).
When I tried to boot archiso in guest, it detected the GPU properly (when nouveau module had started). Everything else gives me a black screen: qemu-bios, OVMF, Windows, etc.
When I was playing with qemu options, I've got Windows to display its bootloader (but no bios) on the screen, but then it gave me a BSOD. After that, nothing again.
The only thing I think I'm doing wrong is that I'm trying to build a system with totally headless host - 1 GPU for guest only (I'm doing everything from my laptop over SSH).
Here's my config:
Motherboard: ASUS M5A97 (with fixed IVRS table -- see boot options)
CPU: AMD FX-4100
RAM: 8GB DDR3, trying to pass 4-6GB to the VM
GPU: Gigabyte NV-65TOC1GI (nVidia GeForce GTX 650Ti 1GB), VBIOS F21
Arch Linux x86_64, linux 3.18.4-1-ARCH
Boot cmdline: dolvm root=/dev/mapper/roccatbase-baseroot rw ivrs_ioapic[5]=00:14.0 ivrs_ioapic[6]=00:00.1 pci-stub.ids=10de:11c6,10de:0e0b video=efifb:off
lspci:
01:00.0 VGA compatible controller [0300]: NVIDIA Corporation GK106 [GeForce GTX 650 Ti] [10de:11c6] (rev a1)
01:00.1 Audio device [0403]: NVIDIA Corporation GK106 HDMI Audio Controller [10de:0e0b] (rev a1)
cat /etc/vfio-pci.cfg
DEVICES="0000:01:00.0 0000:01:00.1"
I use QEMU-git 2.2.r36653.ga46b3aa-1.
Here are some qemu scripts that I'm trying to boot.
Is there anything else I can do to make it work, or should I give up the idea of using only one GPU?
ADD: I've just boot Fedora 21 x86_64 live iso with my testvm5.sh script and it works flawlessly! I've added my USB keyboard and mouse to the script and now I can use it pretty much like if it was bare-metal installation. But still, I didn't see OVMF bios on my screen, instead it showed up in text console I was running testvm5.sh from. O_o
ADD2:I replaced 2 usb device entries in my script with one USB controller, which I'm passing through via VFIO-PCI. Now I can see the (possible?) cause of my problem.
In system logs, when the devices are initialised, I can see some vfio-pci messages about reserving IRQs for PCI devices. When I used Fedora ISO, I got this:
Feb 01 12:43:12 roccatbase kernel: vfio-pci 0000:03:00.0: enabling device (0400 -> 0402)
Feb 01 12:43:15 roccatbase kernel: kvm: zapping shadow pages for mmio generation wraparound
Feb 01 12:43:24 roccatbase kernel: kvm [807]: vcpu0 ignored rdmsr: 0xc0010112
Feb 01 12:43:24 roccatbase kernel: kvm [807]: vcpu0 unimplemented perfctr wrmsr: 0xc0010004 data 0xffff
Feb 01 12:43:25 roccatbase kernel: vfio-pci 0000:03:00.0: irq 33 for MSI/MSI-X
Feb 01 12:43:25 roccatbase kernel: vfio-pci 0000:03:00.0: irq 33 for MSI/MSI-X
Feb 01 12:43:25 roccatbase kernel: vfio-pci 0000:03:00.0: irq 34 for MSI/MSI-X
Feb 01 12:43:25 roccatbase kernel: vfio-pci 0000:03:00.0: irq 33 for MSI/MSI-X
Feb 01 12:43:25 roccatbase kernel: vfio-pci 0000:03:00.0: irq 34 for MSI/MSI-X
Feb 01 12:43:25 roccatbase kernel: vfio-pci 0000:03:00.0: irq 35 for MSI/MSI-X
Feb 01 12:43:26 roccatbase kernel: vfio-pci 0000:01:00.0: irq 36 for MSI/MSI-X
01:00.0 is the pci-id of my GPU and 03:00.0 is my USB controller. The GPU starts up shortly after I get the last message.
But when trying to boot Windows, I get:
Feb 01 13:22:39 roccatbase kernel: kvm: zapping shadow pages for mmio generation wraparound
Feb 01 13:23:17 roccatbase kernel: vfio-pci 0000:03:00.0: irq 33 for MSI/MSI-X
...
Feb 01 13:23:17 roccatbase kernel: vfio-pci 0000:03:00.0: irq 39 for MSI/MSI-X
Feb 01 13:23:17 roccatbase kernel: vfio-pci 0000:03:00.0: irq 40 for MSI/MSI-X
And there's no pci-id of the GPU. The only thing I've changed was the cdrom entry.
Last edited by spijet (2015-02-01 05:30:10)
Offline
3. Dude, check this out! A whole table with stats and graphics and stuff!
https://docs.google.com/spreadsheet/ccc … _web#gid=0
What the...? I am speechless! Why isn't this awesome list on the first page??? I spent months shopping for compatible hardware and never ever saw this! Damn you, gianormous thread!
On a side note: I fixed my audio cracking/popping issues by forcing the same format (16bit x 44100Hz) in both the guest and the host. Hooray!
Last edited by nythrix (2015-02-01 09:27:25)
Offline
I fixed my "boot entries problem for OVMF" now the next problem is "ACS".... QEMU nonUEFI worked flawlessly for me but I had to compile kernel because of ACS patch thats why I'm trying OVMF. Now the problem is that OVMF is not working without ACS patch.
Am I missing something here? Wasnt OVMF fixing the ACS problem in the first place? Are there any commands/options/settings that I'm missing here?
Please help!
Offline
I fixed my "boot entries problem for OVMF" now the next problem is "ACS".... QEMU nonUEFI worked flawlessly for me but I had to compile kernel because of ACS patch thats why I'm trying OVMF. Now the problem is that OVMF is not working without ACS patch.
Am I missing something here? Wasnt OVMF fixing the ACS problem in the first place? Are there any commands/options/settings that I'm missing here?
OVMF does not change anything about ACS. ACS is a hardware isolation issue.
http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses? Try https://www.redhat.com/mailman/listinfo/vfio-users
Offline
devianceluka wrote:I fixed my "boot entries problem for OVMF" now the next problem is "ACS".... QEMU nonUEFI worked flawlessly for me but I had to compile kernel because of ACS patch thats why I'm trying OVMF. Now the problem is that OVMF is not working without ACS patch.
Am I missing something here? Wasnt OVMF fixing the ACS problem in the first place? Are there any commands/options/settings that I'm missing here?
OVMF does not change anything about ACS. ACS is a hardware isolation issue.
I understand, but im 100% positive that I saw numerous times that with OVMF one does not need ACS override patch?
Offline
aw wrote:devianceluka wrote:I fixed my "boot entries problem for OVMF" now the next problem is "ACS".... QEMU nonUEFI worked flawlessly for me but I had to compile kernel because of ACS patch thats why I'm trying OVMF. Now the problem is that OVMF is not working without ACS patch.
Am I missing something here? Wasnt OVMF fixing the ACS problem in the first place? Are there any commands/options/settings that I'm missing here?
OVMF does not change anything about ACS. ACS is a hardware isolation issue.
I understand, but im 100% positive that I saw numerous times that with OVMF one does not need ACS override patch?
You are 100% wrong. With OVMF you don't need the i915 VGA arbitration patch.
http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses? Try https://www.redhat.com/mailman/listinfo/vfio-users
Offline
devianceluka wrote:aw wrote:OVMF does not change anything about ACS. ACS is a hardware isolation issue.
I understand, but im 100% positive that I saw numerous times that with OVMF one does not need ACS override patch?
You are 100% wrong. With OVMF you don't need the i915 VGA arbitration patch.
So your saying to passthrough bus 2 GPU and bus 3 GPU to 2 different VMs - it does not matter whether OVMF or legacy ?
Offline
Is possible to create vfio on Asrock Z68 Pro3, i5 3570 and G650TI?
Offline
Is possible to create vfio on Asrock Z68 Pro3, i5 3570 and G650TI?
Does your motherboard and CPU support IOMMU and VT-D?
http://ark.intel.com/products/65702/Int … o-3_80-GHz
It looks like your CPU does. Explore your BIOS to see if there is an option to enable VT-D for it.
Offline
Now I have 2500k and will buy 3570.
Last bios and there is option VT-d Capability Unsupported. I think that's because my cpu (i5 2500k) is not supported.
Is possible to have problems?
Offline
Ok, i've run into an issue. The ultimate goal of my system is to combine both my Gaming and FreeNAS systems into one. Over the last two days, with help from others here and a bunch of googling, i've been able to get the Gaming portion of my system to run. I have been able to successfully passthrough my USB Controller/Createive SB Z/Ethernet Controller and my GTX 760 with no code 43. When I went to pass my LSI RAID Card through, it complained about that my Ethertnet Adapter as already assigned. This makes sense as my Ethernet Adaoter,SBZ, and the LSI RAID Card are in the same IOMMU Group. I was hoping that multiple devices in a single IOMMU Group could be shared out to different VMs, but this doesn't appear to be the case - correct?
My system has3 IOMMU Groups which I can assign devices from. Group 1 is PCIe Slot 1 which contains an old single slot Quadro NVS graphics card for console access. Group 2 is PCIe Slot 2 which has my GTX760. As I don't need console access once I get everything setup, I figured I could free up Group 1 and place my LSI Card in instead - pulling it out of the group with my USB Contooler/Creative SBZ/Ethernet. Now with my console is being supplied by my GTX 760. I am able to boot my FreeNAS VM and have it see my LSI RAID Card. However, when I boot my Gaming VM, the 'virsh start gamingvm' will return my prompt, the monitor attached to my GTX 760 goes blank, then shortly after the system will hard lock. Is there a setting i'm missing to enable my host to run headless?
03:00.0 SCSI storage controller [0100]: LSI Logic / Symbios Logic SAS1064ET PCI-Express Fusion-MPT SAS [1000:0056] (rev 02)
04:00.0 VGA compatible controller [0300]: NVIDIA Corporation GK104 [GeForce GTX 760] [10de:1187] (rev a1)
04:00.1 Audio device [0403]: NVIDIA Corporation GK104 HDMI Audio Controller [10de:0e0a] (rev a1)
I defined my GTX 760 device-id in: /etc/default/grub
GRUB_CMDLINE_LINUX_DEFAULT="intel_iommu=on vfio_iommu_type1.allow_unsafe_interrupts=1 pci-stub.ids=8086:3a37,8086:3a38,8086:3a39,8086:3a3c,10de:1187,10de:0e0a,1033:0194,8086:3a40,8086:3a42,8086:3a44,8086:3a48,8086:3a4a,11ab:4364,1102:12:00,11ab:4364,1000:0056 quiet"
As well as listed the slot which it resides in: /etc/vfio-pci.cfg
DEVICES="0000:00:1a.0 0000:00:1a.1 0000:00:1a.2 0000:00:1a.7 0000:04:00.0 0000:04:00.1 0000:02:00.0 0000:00:1c.0 0000:00:1c.1 0000:00:1c.2 0000:00:1c.4 0000:00:1c.5 0000:05:00.0 0000:06:00.0 0000:07:00.0 0000:03:00.0"
NOUVEAU is blacklisted:
[user@xenhost1 ~]$ cat /etc/modprobe.d/blacklist.conf
blacklist nouveau
My GTX 760 does support EFI:
[user@xenhost1 ~]$ rom-parser/rom-parser GTX760EV.ROM
Valid ROM signature found @600h, PCIR offset 190h
PCIR: type 0, vendor: 10de, device: 1187, class: 030000
PCIR: revision 0, vendor revision: 1
Valid ROM signature found @fc00h, PCIR offset 1ch
PCIR: type 3, vendor: 10de, device: 1187, class: 030000
PCIR: revision 3, vendor revision: 0
EFI: Signature Valid
Last image
I'm at a loss as to how to get around this, if even possible...
Last edited by The_Moves (2015-02-01 16:49:21)
Offline
Do not attach PCIe root ports to pci-stub or bind them to vfio-pci. Leave them alone
http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses? Try https://www.redhat.com/mailman/listinfo/vfio-users
Offline
Does there exist ACS override patch for kernel 3.18.3 (Fedora 21) ? Trying the patch from OP fails...
EDIT:
Patch16000: override_for_missing_acs_capabilities.patch
+ case "$patch" in
+ patch -p1 -F1 -s
2 out of 3 hunks FAILED -- saving rejects to file drivers/pci/quirks.c.rej
napaka: Bad exit status from /var/tmp/rpm-tmp.sCnxM8 (%prep)
RPM build errors:
Bad exit status from /var/tmp/rpm-tmp.sCnxM8 (%prep)
Last edited by devianceluka (2015-02-01 23:24:27)
Offline
Does there exist ACS override patch for kernel 3.18.3 (Fedora 21) ? Trying the patch from OP fails...
EDIT:
Patch16000: override_for_missing_acs_capabilities.patch
+ case "$patch" in
+ patch -p1 -F1 -s
2 out of 3 hunks FAILED -- saving rejects to file drivers/pci/quirks.c.rej
napaka: Bad exit status from /var/tmp/rpm-tmp.sCnxM8 (%prep)RPM build errors:
Bad exit status from /var/tmp/rpm-tmp.sCnxM8 (%prep)
It applies just fine to stock 3.18.5, so you probably have to patch it hand
Offline
devianceluka wrote:Does there exist ACS override patch for kernel 3.18.3 (Fedora 21) ? Trying the patch from OP fails...
EDIT:
Patch16000: override_for_missing_acs_capabilities.patch
+ case "$patch" in
+ patch -p1 -F1 -s
2 out of 3 hunks FAILED -- saving rejects to file drivers/pci/quirks.c.rej
napaka: Bad exit status from /var/tmp/rpm-tmp.sCnxM8 (%prep)RPM build errors:
Bad exit status from /var/tmp/rpm-tmp.sCnxM8 (%prep)It applies just fine to stock 3.18.5, so you probably have to patch it hand
I'm doing the exact same thing as on Fedora 20 (~3.17) where it was compiling flawlessly for months. Patch it by hand?
Offline
Do not attach PCIe root ports to pci-stub or bind them to vfio-pci. Leave them alone
I went through an made sure my XML made sense after reading the below thread in which your a part of, as well as looked at a number of other XMLs:
https://www.redhat.com/archives/libvir- … 00881.html
But are you talking actual physical PCIe Root Ports? What is considered a root port? I apologize as I am still learning here.
Offline
aw wrote:Do not attach PCIe root ports to pci-stub or bind them to vfio-pci. Leave them alone
I went through an made sure my XML made sense after reading the below thread in which your a part of, as well as looked at a number of other XMLs:
Way off track, that's guest configuration, vfio-pci and pci-stub are host drivers.
But are you talking actual physical PCIe Root Ports? What is considered a root port? I apologize as I am still learning here.
Those things that say Root Port when you run lspci | grep -i "root port". You list some of them in both your kernel command line and your DEVICES.
http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses? Try https://www.redhat.com/mailman/listinfo/vfio-users
Offline
The_Moves wrote:aw wrote:Do not attach PCIe root ports to pci-stub or bind them to vfio-pci. Leave them alone
I went through an made sure my XML made sense after reading the below thread in which your a part of, as well as looked at a number of other XMLs:
Way off track, that's guest configuration, vfio-pci and pci-stub are host drivers.
But are you talking actual physical PCIe Root Ports? What is considered a root port? I apologize as I am still learning here.
Those things that say Root Port when you run lspci | grep -i "root port". You list some of them in both your kernel command line and your DEVICES.
Wow, thank you do much! I can't believe I did that. i thought those were my other set of USB devices, no wonder why they weren't working as expected. I also had a typo for my Sound Card "1102:12:00". I'm surprised I was able to get it passed through and working.
What happens when I have the root ports assigned? Freezes? Panics?
Last edited by The_Moves (2015-02-02 04:53:56)
Offline