Arch Linux

aw

Member

Registered: 2013-10-04

Posts: 921

Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

@garnerlogan65

See post #5365 and #5367.  I'm not interested in guessing the details of the problem, repeating request that were not followed, or holding hands through reboots.

---

http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Blind Tree Frog

Member

Registered: 2013-12-31

Posts: 27

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

FWIW, the kernel handles the boot VGA ROM differently than secondary ROMs.  You actually get the shadow copy that might be modified during the execution of the ROM code.  So, somehow getting a copy of the real, untouched ROM and passing it as a file would be encouraged for this scenario.

You're assigning an AMD card to a 440FX VM running Linux.  That's pretty much the one case where I recommend Q35.  I've posted patches for this, but I lost track of whether they've all been accepted.  The problem is that the radeon driver blindly assumes that a downstream port exists above the device because that's the way it appears on real hardware.  It then goes trying to read link speed info on a device that doesn't exist and you get the above oops.  You need to switch to Q35, add a root port and put the GPU below the root port.

And now it seems to be working.  I downloaded the ROM from a site upstream but using it didn't work, so I just dropped that and I finally managed to log into a guest system ("yay!!").

qemu command:

#!/bin/bash

# archbang guest
qemu-system-x86_64 \
    -enable-kvm -M q35 -m 6144 \
    -smp 5,sockets=1,cores=5,threads=1 -cpu host,kvm=off \
    -device ioh3420,bus=pcie.0,addr=1c.0,multifunction=on,port=1,chassis=1,id=fakeRoot.1 \
    -device vfio-pci,host=03:00.0,bus=fakeRoot.1,addr=00.0,multifunction=on,x-vga=on \
    -device vfio-pci,host=03:00.1,bus=fakeRoot.1,addr=00.1 \
    -cdrom /root/archbang-150516-i686.iso \
    -usb -usbdevice host:045e:0768 -usbdevice host:3842:2410 \
    -rtc base=localtime \
    -vga none -display none \
    $@

There are two minor hiccups I've noticed thus far:
1) the arch guest I previously set up has an ethernet device, but didn't automatically grab a IP address.  The archbang one did however, so I'm assuming a reinstall will fix it.
2) Previously the keyboard was supported at the boot loader screen, now it doesn't seem to be. May be a non-issue in the end but I figured I should write down for my later tinkering.

Anyhow, now to get sound, networking, and a handful of other things set up (passing the bluray player directly to the vm for one) and then toss it into virsh.  Thanks for your help, aw; I owe you a beer if you are ever in the RTP area.

garnerlogan65

Member

Registered: 2014-11-22

Posts: 26

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

aw, I already gave you the information you asked for (whether vfio was static or not), and I'm not sure what else to do from here, if I knew what to do I would have done it, I can't really do anything with vfio files missing. The last thing I tried to do was near the end of part 3 of your guide, and install options to dracut and generating an initramfs, which did not work. It said that it was unable to install vfio and such. Not sure how to proceed from here.

The_Moves

Member

Registered: 2015-01-06

Posts: 59

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

aw, I already gave you the information you asked for (whether vfio was static or not), and I'm not sure what else to do from here, if I knew what to do I would have done it, I can't really do anything with vfio files missing. The last thing I tried to do was near the end of part 3 of your guide, and install options to dracut and generating an initramfs, which did not work. It said that it was unable to install vfio and such. Not sure how to proceed from here.

If your having difficulties with a certain part, there are other guides out there that are not made specefically for VFIO, but should give you a hint into what your doing wrong and how you can correct it.

Also, don't forget to do you 'man' page pushups.  Like:

man grep

aw

Member

Registered: 2013-10-04

Posts: 921

Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

@garnerlogan65

Here's a trivial example:

I can't really do anything with vfio files missing. The last thing I tried to do was near the end of part 3 of your guide, and install options to dracut and generating an initramfs, which did not work. It said that it was unable to install vfio and such. Not sure how to proceed from here.

What vfio files are missing?  What exactly are you trying to do "near the end of part 3 of your guide"?  What exactly didn't work generating an initramfs?  Give the error message, screen log, anything!  If you can't bother to provide enough background on what you've done, what you're trying to do, what you're expecting and what actually happened, then I can't be bothered to spend time on your problems.  Sorry.

---

http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Denso

Member

Registered: 2014-08-30

Posts: 179

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

@Denso

Are you rebuilding your initramfs between each of these changes?  What shows up in dmesg if you run 'modprobe -r vfio-pci; modprobe vfio-pci' with modprobe.d as in the previous post?

Yes . Whenever I do changes to initramfs config file , I rebuild it .

modprobe -r vfio-pci

modprobe: FATAL: Module vfio_pci is in use.

Of course , VFIO is in use already (being bound to Intel Audio , Intel NIC for my VM) . It just won't get bound to GPUs using modprobe.d .

aw

Member

Registered: 2013-10-04

Posts: 921

Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

@Denso

Well obviously you can't remove the module while a VM is running and using it...  I don't know what to tell you, I've got a system running with exactly this right now:

pci-stub.ids=10de:ffffffff:ffffffff:ffffffff:00030000:ffff00ff,10de:ffffffff:ffffffff:ffffffff:00040300:ffffffff,6549:ffffffff,11c1:5811

It works exactly like it's supposed to, claiming all the Nvidia VGA and audio devices, plus all Teradici devices, and then a random firewire controller that shares a group with a NIC.  Double check everything.

---

http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Denso

Member

Registered: 2014-08-30

Posts: 179

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

@aw :

Umm , I'm not using pci-stub.ids .

I'm using "options vfio-pci ids =" inside a modprobe.d conf file (Kernel 4.1-rc7) .

I'm skipping pci-stub altogether and binding the devices directly to vfio . It works for the Intel Audio , Intel NIC and a PCI-E USB3 controller . Just not the GPUs .

I think I'll compile a 4.1-rc1 and see if it would work . It used to work on that version like a charm .

I'll keep you posted !

aw

Member

Registered: 2013-10-04

Posts: 921

Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

vfio-pci and pci-stub use pretty much the same code for the ids option, the difference is that pci-stub is usually builtin and vfio-pci is usually a module.  So, if you get different results, it probably has something to do with that.

---

http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Denso

Member

Registered: 2014-08-30

Posts: 179

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

@aw :

Even 4.0-rc1 didn't work , which was working perfectly fine before I install this motherboard (X99-E WS) . Perhaps it is because it uses PLX chips ? I read a previous post that vfio-bind can bind bridges while modprobe.d can't right ?

Here is the tree :

root ~ lspci -t
-+-[0000:ff]-+-0b.0
 |           +-0b.1
 |           +-0b.2
 |           +-0c.0
 |           +-0c.1
 |           +-0c.2
 |           +-0c.3
 |           +-0c.4
 |           +-0c.5
 |           +-0f.0
 |           +-0f.1
 |           +-0f.4
 |           +-0f.5
 |           +-0f.6
 |           +-10.0
 |           +-10.1
 |           +-10.5
 |           +-10.6
 |           +-10.7
 |           +-12.0
 |           +-12.1
 |           +-13.0
 |           +-13.1
 |           +-13.2
 |           +-13.3
 |           +-13.4
 |           +-13.5
 |           +-13.6
 |           +-13.7
 |           +-14.0
 |           +-14.1
 |           +-14.2
 |           +-14.3
 |           +-14.6
 |           +-14.7
 |           +-15.0
 |           +-15.1
 |           +-15.2
 |           +-15.3
 |           +-16.0
 |           +-16.6
 |           +-16.7
 |           +-17.0
 |           +-17.4
 |           +-17.5
 |           +-17.6
 |           +-17.7
 |           +-1e.0
 |           +-1e.1
 |           +-1e.2
 |           +-1e.3
 |           +-1e.4
 |           +-1f.0
 |           \-1f.2
 \-[0000:00]-+-00.0
             +-01.0-[01]--
             +-01.1-[02]--
             +-02.0-[07-11]----00.0-[08-11]--+-08.0-[0b]----00.0
             |                               +-09.0-[0c-11]----00.0-[0d-11]--+-01.0-[0e]----00.0
             |                               |                               +-05.0-[0f]----00.0
             |                               |                               +-07.0-[10]----00.0
             |                               |                               \-09.0-[11]----00.0
             |                               +-10.0-[09]--+-00.0
             |                               |            \-00.1
             |                               \-11.0-[0a]--
             +-03.0-[03-06]----00.0-[04-06]--+-08.0-[06]--
             |                               \-10.0-[05]--+-00.0
             |                                            \-00.1
             +-05.0
             +-05.1
             +-05.2
             +-05.4
             +-11.0
             +-11.4
             +-14.0
             +-16.0
             +-19.0
             +-1a.0
             +-1b.0
             +-1c.0-[12]--
             +-1c.3-[13-16]----00.0-[14-16]--+-01.0-[15]----00.0
             |                               \-05.0-[16]----00.0
             +-1c.4-[17]----00.0
             +-1c.6-[18]----00.0
             +-1d.0
             +-1f.0
             +-1f.2
             \-1f.3

root ~ lspci
00:00.0 Host bridge: Intel Corporation Xeon E5 v3/Core i7 DMI2 (rev 02)
00:01.0 PCI bridge: Intel Corporation Xeon E5 v3/Core i7 PCI Express Root Port 1 (rev 02)
00:01.1 PCI bridge: Intel Corporation Xeon E5 v3/Core i7 PCI Express Root Port 1 (rev 02)
00:02.0 PCI bridge: Intel Corporation Xeon E5 v3/Core i7 PCI Express Root Port 2 (rev 02)
00:03.0 PCI bridge: Intel Corporation Xeon E5 v3/Core i7 PCI Express Root Port 3 (rev 02)
00:05.0 System peripheral: Intel Corporation Xeon E5 v3/Core i7 Address Map, VTd_Misc, System Management (rev 02)
00:05.1 System peripheral: Intel Corporation Xeon E5 v3/Core i7 Hot Plug (rev 02)
00:05.2 System peripheral: Intel Corporation Xeon E5 v3/Core i7 RAS, Control Status and Global Errors (rev 02)
00:05.4 PIC: Intel Corporation Xeon E5 v3/Core i7 I/O APIC (rev 02)
00:11.0 Unassigned class [ff00]: Intel Corporation C610/X99 series chipset SPSR (rev 05)
00:11.4 SATA controller: Intel Corporation C610/X99 series chipset sSATA Controller [AHCI mode] (rev 05)
00:14.0 USB controller: Intel Corporation C610/X99 series chipset USB xHCI Host Controller (rev 05)
00:16.0 Communication controller: Intel Corporation C610/X99 series chipset MEI Controller #1 (rev 05)
00:19.0 Ethernet controller: Intel Corporation Ethernet Connection (2) I218-LM (rev 05)
00:1a.0 USB controller: Intel Corporation C610/X99 series chipset USB Enhanced Host Controller #2 (rev 05)
00:1b.0 Audio device: Intel Corporation C610/X99 series chipset HD Audio Controller (rev 05)
00:1c.0 PCI bridge: Intel Corporation C610/X99 series chipset PCI Express Root Port #1 (rev d5)
00:1c.3 PCI bridge: Intel Corporation C610/X99 series chipset PCI Express Root Port #4 (rev d5)
00:1c.4 PCI bridge: Intel Corporation C610/X99 series chipset PCI Express Root Port #5 (rev d5)
00:1c.6 PCI bridge: Intel Corporation C610/X99 series chipset PCI Express Root Port #7 (rev d5)
00:1d.0 USB controller: Intel Corporation C610/X99 series chipset USB Enhanced Host Controller #1 (rev 05)
00:1f.0 ISA bridge: Intel Corporation C610/X99 series chipset LPC Controller (rev 05)
00:1f.2 SATA controller: Intel Corporation C610/X99 series chipset 6-Port SATA Controller [AHCI mode] (rev 05)
00:1f.3 SMBus: Intel Corporation C610/X99 series chipset SMBus Controller (rev 05)
03:00.0 PCI bridge: PLX Technology, Inc. PEX 8747 48-Lane, 5-Port PCI Express Gen 3 (8.0 GT/s) Switch (rev ca)
04:08.0 PCI bridge: PLX Technology, Inc. PEX 8747 48-Lane, 5-Port PCI Express Gen 3 (8.0 GT/s) Switch (rev ca)
04:10.0 PCI bridge: PLX Technology, Inc. PEX 8747 48-Lane, 5-Port PCI Express Gen 3 (8.0 GT/s) Switch (rev ca)
05:00.0 VGA compatible controller: NVIDIA Corporation GK104 [GeForce GTX 770] (rev a1)
05:00.1 Audio device: NVIDIA Corporation GK104 HDMI Audio Controller (rev a1)
07:00.0 PCI bridge: PLX Technology, Inc. PEX 8747 48-Lane, 5-Port PCI Express Gen 3 (8.0 GT/s) Switch (rev ca)
08:08.0 PCI bridge: PLX Technology, Inc. PEX 8747 48-Lane, 5-Port PCI Express Gen 3 (8.0 GT/s) Switch (rev ca)
08:09.0 PCI bridge: PLX Technology, Inc. PEX 8747 48-Lane, 5-Port PCI Express Gen 3 (8.0 GT/s) Switch (rev ca)
08:10.0 PCI bridge: PLX Technology, Inc. PEX 8747 48-Lane, 5-Port PCI Express Gen 3 (8.0 GT/s) Switch (rev ca)
08:11.0 PCI bridge: PLX Technology, Inc. PEX 8747 48-Lane, 5-Port PCI Express Gen 3 (8.0 GT/s) Switch (rev ca)
09:00.0 VGA compatible controller: NVIDIA Corporation GM107 [GeForce GTX 750] (rev a2)
09:00.1 Audio device: NVIDIA Corporation Device 0fbc (rev a1)
0b:00.0 Serial Attached SCSI controller: LSI Logic / Symbios Logic SAS2008 PCI-Express Fusion-MPT SAS-2 [Falcon] (rev 03)
0c:00.0 PCI bridge: PLX Technology, Inc. PEX 8608 8-lane, 8-Port PCI Express Gen 2 (5.0 GT/s) Switch (rev ba)
0d:01.0 PCI bridge: PLX Technology, Inc. PEX 8608 8-lane, 8-Port PCI Express Gen 2 (5.0 GT/s) Switch (rev ba)
0d:05.0 PCI bridge: PLX Technology, Inc. PEX 8608 8-lane, 8-Port PCI Express Gen 2 (5.0 GT/s) Switch (rev ba)
0d:07.0 PCI bridge: PLX Technology, Inc. PEX 8608 8-lane, 8-Port PCI Express Gen 2 (5.0 GT/s) Switch (rev ba)
0d:09.0 PCI bridge: PLX Technology, Inc. PEX 8608 8-lane, 8-Port PCI Express Gen 2 (5.0 GT/s) Switch (rev ba)
0e:00.0 USB controller: Renesas Technology Corp. uPD720202 USB 3.0 Host Controller (rev 02)
0f:00.0 USB controller: Renesas Technology Corp. uPD720202 USB 3.0 Host Controller (rev 02)
10:00.0 USB controller: Renesas Technology Corp. uPD720202 USB 3.0 Host Controller (rev 02)
11:00.0 USB controller: Renesas Technology Corp. uPD720202 USB 3.0 Host Controller (rev 02)
13:00.0 PCI bridge: ASMedia Technology Inc. Device 1184
14:01.0 PCI bridge: ASMedia Technology Inc. Device 1184
14:05.0 PCI bridge: ASMedia Technology Inc. Device 1184
15:00.0 Ethernet controller: Intel Corporation I210 Gigabit Network Connection (rev 03)
16:00.0 SATA controller: ASMedia Technology Inc. ASM1062 Serial ATA Controller (rev 02)
17:00.0 SATA controller: ASMedia Technology Inc. ASM1062 Serial ATA Controller (rev 02)
18:00.0 USB controller: ASMedia Technology Inc. ASM1042A USB 3.0 Host Controller
ff:0b.0 System peripheral: Intel Corporation Xeon E5 v3/Core i7 R3 QPI Link 0 & 1 Monitoring (rev 02)
ff:0b.1 Performance counters: Intel Corporation Xeon E5 v3/Core i7 R3 QPI Link 0 & 1 Monitoring (rev 02)
ff:0b.2 Performance counters: Intel Corporation Xeon E5 v3/Core i7 R3 QPI Link 0 & 1 Monitoring (rev 02)
ff:0c.0 System peripheral: Intel Corporation Xeon E5 v3/Core i7 Unicast Registers (rev 02)
ff:0c.1 System peripheral: Intel Corporation Xeon E5 v3/Core i7 Unicast Registers (rev 02)
ff:0c.2 System peripheral: Intel Corporation Xeon E5 v3/Core i7 Unicast Registers (rev 02)
ff:0c.3 System peripheral: Intel Corporation Xeon E5 v3/Core i7 Unicast Registers (rev 02)
ff:0c.4 System peripheral: Intel Corporation Xeon E5 v3/Core i7 Unicast Registers (rev 02)
ff:0c.5 System peripheral: Intel Corporation Xeon E5 v3/Core i7 Unicast Registers (rev 02)
ff:0f.0 System peripheral: Intel Corporation Xeon E5 v3/Core i7 Buffered Ring Agent (rev 02)
ff:0f.1 System peripheral: Intel Corporation Xeon E5 v3/Core i7 Buffered Ring Agent (rev 02)
ff:0f.4 System peripheral: Intel Corporation Xeon E5 v3/Core i7 System Address Decoder & Broadcast Registers (rev 02)
ff:0f.5 System peripheral: Intel Corporation Xeon E5 v3/Core i7 System Address Decoder & Broadcast Registers (rev 02)
ff:0f.6 System peripheral: Intel Corporation Xeon E5 v3/Core i7 System Address Decoder & Broadcast Registers (rev 02)
ff:10.0 System peripheral: Intel Corporation Xeon E5 v3/Core i7 PCIe Ring Interface (rev 02)
ff:10.1 Performance counters: Intel Corporation Xeon E5 v3/Core i7 PCIe Ring Interface (rev 02)
ff:10.5 System peripheral: Intel Corporation Xeon E5 v3/Core i7 Scratchpad & Semaphore Registers (rev 02)
ff:10.6 Performance counters: Intel Corporation Xeon E5 v3/Core i7 Scratchpad & Semaphore Registers (rev 02)
ff:10.7 System peripheral: Intel Corporation Xeon E5 v3/Core i7 Scratchpad & Semaphore Registers (rev 02)
ff:12.0 System peripheral: Intel Corporation Xeon E5 v3/Core i7 Home Agent 0 (rev 02)
ff:12.1 Performance counters: Intel Corporation Xeon E5 v3/Core i7 Home Agent 0 (rev 02)
ff:13.0 System peripheral: Intel Corporation Xeon E5 v3/Core i7 Integrated Memory Controller 0 Target Address, Thermal & RAS Registers (rev 02)
ff:13.1 System peripheral: Intel Corporation Xeon E5 v3/Core i7 Integrated Memory Controller 0 Target Address, Thermal & RAS Registers (rev 02)
ff:13.2 System peripheral: Intel Corporation Xeon E5 v3/Core i7 Integrated Memory Controller 0 Channel Target Address Decoder (rev 02)
ff:13.3 System peripheral: Intel Corporation Xeon E5 v3/Core i7 Integrated Memory Controller 0 Channel Target Address Decoder (rev 02)
ff:13.4 System peripheral: Intel Corporation Xeon E5 v3/Core i7 Integrated Memory Controller 0 Channel Target Address Decoder (rev 02)
ff:13.5 System peripheral: Intel Corporation Xeon E5 v3/Core i7 Integrated Memory Controller 0 Channel Target Address Decoder (rev 02)
ff:13.6 System peripheral: Intel Corporation Xeon E5 v3/Core i7 DDRIO Channel 0/1 Broadcast (rev 02)
ff:13.7 System peripheral: Intel Corporation Xeon E5 v3/Core i7 DDRIO Global Broadcast (rev 02)
ff:14.0 System peripheral: Intel Corporation Xeon E5 v3/Core i7 Integrated Memory Controller 0 Channel 0 Thermal Control (rev 02)
ff:14.1 System peripheral: Intel Corporation Xeon E5 v3/Core i7 Integrated Memory Controller 0 Channel 1 Thermal Control (rev 02)
ff:14.2 System peripheral: Intel Corporation Xeon E5 v3/Core i7 Integrated Memory Controller 0 Channel 0 ERROR Registers (rev 02)
ff:14.3 System peripheral: Intel Corporation Xeon E5 v3/Core i7 Integrated Memory Controller 0 Channel 1 ERROR Registers (rev 02)
ff:14.6 System peripheral: Intel Corporation Xeon E5 v3/Core i7 DDRIO (VMSE) 0 & 1 (rev 02)
ff:14.7 System peripheral: Intel Corporation Xeon E5 v3/Core i7 DDRIO (VMSE) 0 & 1 (rev 02)
ff:15.0 System peripheral: Intel Corporation Xeon E5 v3/Core i7 Integrated Memory Controller 0 Channel 2 Thermal Control (rev 02)
ff:15.1 System peripheral: Intel Corporation Xeon E5 v3/Core i7 Integrated Memory Controller 0 Channel 3 Thermal Control (rev 02)
ff:15.2 System peripheral: Intel Corporation Xeon E5 v3/Core i7 Integrated Memory Controller 0 Channel 2 ERROR Registers (rev 02)
ff:15.3 System peripheral: Intel Corporation Xeon E5 v3/Core i7 Integrated Memory Controller 0 Channel 3 ERROR Registers (rev 02)
ff:16.0 System peripheral: Intel Corporation Xeon E5 v3/Core i7 Integrated Memory Controller 1 Target Address, Thermal & RAS Registers (rev 02)
ff:16.6 System peripheral: Intel Corporation Xeon E5 v3/Core i7 DDRIO Channel 2/3 Broadcast (rev 02)
ff:16.7 System peripheral: Intel Corporation Xeon E5 v3/Core i7 DDRIO Global Broadcast (rev 02)
ff:17.0 System peripheral: Intel Corporation Xeon E5 v3/Core i7 Integrated Memory Controller 1 Channel 0 Thermal Control (rev 02)
ff:17.4 System peripheral: Intel Corporation Xeon E5 v3/Core i7 DDRIO (VMSE) 2 & 3 (rev 02)
ff:17.5 System peripheral: Intel Corporation Xeon E5 v3/Core i7 DDRIO (VMSE) 2 & 3 (rev 02)
ff:17.6 System peripheral: Intel Corporation Xeon E5 v3/Core i7 DDRIO (VMSE) 2 & 3 (rev 02)
ff:17.7 System peripheral: Intel Corporation Xeon E5 v3/Core i7 DDRIO (VMSE) 2 & 3 (rev 02)
ff:1e.0 System peripheral: Intel Corporation Xeon E5 v3/Core i7 Power Control Unit (rev 02)
ff:1e.1 System peripheral: Intel Corporation Xeon E5 v3/Core i7 Power Control Unit (rev 02)
ff:1e.2 System peripheral: Intel Corporation Xeon E5 v3/Core i7 Power Control Unit (rev 02)
ff:1e.3 System peripheral: Intel Corporation Xeon E5 v3/Core i7 Power Control Unit (rev 02)
ff:1e.4 System peripheral: Intel Corporation Xeon E5 v3/Core i7 Power Control Unit (rev 02)
ff:1f.0 System peripheral: Intel Corporation Xeon E5 v3/Core i7 VCU (rev 02)
ff:1f.2 System peripheral: Intel Corporation Xeon E5 v3/Core i7 VCU (rev 02)

I'm sorry for bothering you this much Alex

aw

Member

Registered: 2013-10-04

Posts: 921

Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

It was a mistake for vfio-bind to try to bind bridges and a bug in vfio-pci to allow bridges.  The latter has since been corrected in vfio-pci.  That means that if you use vfio-bind, you may end up with bridges without a driver since vfio-pci won't let them bind.

---

http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Denso

Member

Registered: 2014-08-30

Posts: 179

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

@ aw :

After using vfio-bind and starting all of my VMs , checking all of the bridges' drivers in use , they are all using "pcieport" . So I guess I'm fine for now

Let's see if I can hunt and fix the modprobe.d ids issue .

supertrampx

Member

Registered: 2015-06-07

Posts: 11

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

I have checked MSI enable on my GTx 970. In linux it reports MSI: enable-    in windows when I view the connection type pci, i see the card after driver install using msi.

I am getting this error occasionally after a freeze and vm crashes, usually when I am installing, or accessing the graphics card.

qemu-system-x86_64: vfio_err_notifier_handler(0000:04:00.0) Unrecoverable error detected.  Please collect any data possible and then kill the guest

any thoughts or ideas?

Last edited by supertrampx (2015-06-20 18:35:05)

DanaGoyette

Member

Registered: 2014-01-03

Posts: 46

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

The latest time I tried to reset my radeon via suspend (because the workaround seemed to have failed), I got this message when trying to start the VM after resume:
Error starting domain: Requested operation is not valid: PCI device 0000:08:00.0 is not assignable

Syslog showed the following message:
libvirtd[2036]: internal error: Device 0000:08:00.0 is behind a switch lacking ACS and cannot be assigned

That's coming from libvirt, not the kernel.  Oddly, it didn't complain about the same device before suspend.
I know I can just set relaxed_acs_check=1 in libvirtd.conf, but it would be good to find out why it suddenly decides to complain.

lspci -tv (with * denoting devices I'm forwarding to this VM):

-[0000:00]-+-00.0  Intel Corporation Xeon E3-1200 v3 Processor DRAM Controller
           +-01.0-[01]--+-00.0  Advanced Micro Devices, Inc. [AMD/ATI] Hawaii PRO [Radeon R9 290]
           |            \-00.1  Advanced Micro Devices, Inc. [AMD/ATI] Device aac8
           +-02.0  Intel Corporation Xeon E3-1200 v3 Processor Integrated Graphics Controller
           +-03.0  Intel Corporation Xeon E3-1200 v3/4th Gen Core Processor HD Audio Controller
           +-14.0  Intel Corporation 8 Series/C220 Series Chipset Family USB xHCI
           +-16.0  Intel Corporation 8 Series/C220 Series Chipset Family MEI Controller #1
           +-16.3  Intel Corporation 8 Series/C220 Series Chipset Family KT Controller
           +-19.0  Intel Corporation Ethernet Connection I217-LM
           +-1a.0  Intel Corporation 8 Series/C220 Series Chipset Family USB EHCI #2
           *-1b.0  Intel Corporation 8 Series/C220 Series Chipset High Definition Audio Controller
           *-1c.0-[02]----00.0  ASMedia Technology Inc. ASM1062 Serial ATA Controller
           +-1c.1-[03-0a]----00.0-[04-0a]--+-01.0-[05]--
           |                               +-04.0-[06]--
           |                               +-05.0-[07]--
           |                               *-07.0-[08]----00.0  Renesas Technology Corp. uPD720201 USB 3.0 Host Controller
           |                               \-09.0-[09-0a]----00.0-[0a]----00.0  Texas Instruments XIO2213A/B/XIO2221 IEEE-1394b OHCI Controller [Cheetah Express]
           *-1c.3-[0b]----00.0  Intel Corporation I210 Gigabit Network Connection
           +-1c.4-[0c-44]--
           +-1d.0  Intel Corporation 8 Series/C220 Series Chipset Family USB EHCI #1
           +-1f.0  Intel Corporation C226 Series Chipset Family Server Advanced SKU LPC Controller
           +-1f.2  Intel Corporation 8 Series/C220 Series Chipset Family 6-port SATA Controller 1 [AHCI mode]
           +-1f.3  Intel Corporation 8 Series/C220 Series Chipset Family SMBus Controller
           \-1f.6  Intel Corporation 8 Series Chipset Family Thermal Management Controller

lspci -nnv for the path involving that device, after suspend/resume:

00:00.0 Host bridge [0600]: Intel Corporation Xeon E3-1200 v3 Processor DRAM Controller [8086:0c08] (rev 06)
	Subsystem: Super Micro Computer Inc Device [15d9:0818]
	Flags: bus master, fast devsel, latency 0
	Capabilities: [e0] Vendor Specific Information: Len=0c <?>
	Kernel driver in use: ie31200_edac

00:1c.1 PCI bridge [0604]: Intel Corporation 8 Series/C220 Series Chipset Family PCI Express Root Port #2 [8086:8c12] (rev d5) (prog-if 00 [Normal decode])
	Flags: bus master, fast devsel, latency 0
	Bus: primary=00, secondary=03, subordinate=0a, sec-latency=0
	Memory behind bridge: ee800000-eebfffff
	Capabilities: [40] Express Root Port (Slot+), MSI 00
	Capabilities: [80] MSI: Enable- Count=1/1 Maskable- 64bit-
	Capabilities: [90] Subsystem: Super Micro Computer Inc Device [15d9:0818]
	Capabilities: [a0] Power Management version 3
	Kernel driver in use: pcieport

03:00.0 PCI bridge [0604]: PLX Technology, Inc. PEX 8606 6 Lane, 6 Port PCI Express Gen 2 (5.0 GT/s) Switch [10b5:8606] (rev ba) (prog-if 00 [Normal decode])
	Flags: bus master, fast devsel, latency 0
	Memory at eeb00000 (32-bit, non-prefetchable) [size=128K]
	Bus: primary=03, secondary=04, subordinate=0a, sec-latency=0
	Memory behind bridge: ee800000-eeafffff
	Capabilities: [40] Power Management version 3
	Capabilities: [48] MSI: Enable+ Count=1/4 Maskable+ 64bit+
	Capabilities: [68] Express Upstream Port, MSI 00
	Capabilities: [a4] Subsystem: PLX Technology, Inc. PEX 8606 6 Lane, 6 Port PCI Express Gen 2 (5.0 GT/s) Switch [10b5:8606]
	Capabilities: [100] Device Serial Number ba-86-01-10-b5-df-0e-00
	Capabilities: [fb4] Advanced Error Reporting
	Capabilities: [138] Power Budgeting <?>
	Capabilities: [148] Virtual Channel
	Capabilities: [448] Vendor Specific Information: ID=0000 Rev=0 Len=0cc <?>
	Capabilities: [950] Vendor Specific Information: ID=0001 Rev=0 Len=010 <?>
	Kernel driver in use: pcieport

04:07.0 PCI bridge [0604]: PLX Technology, Inc. PEX 8606 6 Lane, 6 Port PCI Express Gen 2 (5.0 GT/s) Switch [10b5:8606] (rev ba) (prog-if 00 [Normal decode])
	Flags: bus master, fast devsel, latency 0
	Bus: primary=04, secondary=08, subordinate=08, sec-latency=0
	Memory behind bridge: eea00000-eeafffff
	Capabilities: [40] Power Management version 3
	Capabilities: [48] MSI: Enable+ Count=1/4 Maskable+ 64bit+
	Capabilities: [68] Express Downstream Port (Slot+), MSI 00
	Capabilities: [a4] Subsystem: PLX Technology, Inc. PEX 8606 6 Lane, 6 Port PCI Express Gen 2 (5.0 GT/s) Switch [10b5:8606]
	Capabilities: [100] Device Serial Number ba-86-01-10-b5-df-0e-00
	Capabilities: [fb4] Advanced Error Reporting
	Capabilities: [148] Virtual Channel
	Capabilities: [520] Access Control Services
	Capabilities: [950] Vendor Specific Information: ID=0001 Rev=0 Len=010 <?>
	Kernel driver in use: pcieport

08:00.0 USB controller [0c03]: Renesas Technology Corp. uPD720201 USB 3.0 Host Controller [1912:0014] (rev 03) (prog-if 30 [XHCI])
	Subsystem: Super Micro Computer Inc Device [15d9:0818]
	Flags: bus master, fast devsel, latency 0, IRQ 16
	Memory at eea00000 (64-bit, non-prefetchable) [size=8K]
	Capabilities: [50] Power Management version 3
	Capabilities: [70] MSI: Enable- Count=1/8 Maskable- 64bit+
	Capabilities: [90] MSI-X: Enable+ Count=8 Masked-
	Capabilities: [a0] Express Endpoint, MSI 00
	Capabilities: [100] Advanced Error Reporting
	Capabilities: [150] Latency Tolerance Reporting
	Kernel driver in use: xhci_hcd

aw

Member

Registered: 2013-10-04

Posts: 921

Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

I have checked MSI enable on my GTx 970. In linux it reports MSI: enable-    in windows when I view the connection type pci, i see the card after driver install using msi.

I am getting this error occasionally after a freeze and vm crashes, usually when I am installing, or accessing the graphics card.

qemu-system-x86_64: vfio_err_notifier_handler(0000:04:00.0) Unrecoverable error detected.  Please collect any data possible and then kill the guest

any thoughts or ideas?

You got a fatal AER error.  Are you overclocking?  You can boot the host with pci=noaer to disable it, but that's no guarantee that the you're not still getting bus errors.

---

http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Denso

Member

Registered: 2014-08-30

Posts: 179

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Hi ...

Anyone else is having trouble with their beidged networking ? Both of my VMs are now unreachable , and they can't reach the outer network .

Commands I use to set the bridge up :

ip link add name br0 type bridge
ip link set dev br0 up
ip link set dev eth0 promisc on
ip link set dev eth0 up
ip link set dev eth0 master br0
ip link set dev br0 up
ip addr add 10.0.1.240/24 broadcast 10.0.1.255 dev br0
ip route add default via 10.0.1.1

The part of code to start the VM (the net bit) :

-netdev bridge,id=br0 -device virtio-net-pci,netdev=br0,id=nic0,mac=B8:A7:2A:6E:09:B2

Even using the legacy method doesn't work :

-net nic,model=virtio,macaddr=B8:A7:2A:6E:09:B2 -net bridge,br=br0

Windows guests can't identify the network or connect to the outside world , I even set the adapter's IP/Subnet/Gateway/DNS manually (Yes , I know what I'm doing) .

Dmesg doesn't show anything not familier :

[Sun Jun 21 02:53:51 2015] br0: port 1(eth0) entered forwarding state
[Sun Jun 21 02:53:51 2015] br0: port 1(eth0) entered forwarding state
[Sun Jun 21 02:54:06 2015] br0: port 1(eth0) entered forwarding state
[Sun Jun 21 02:56:03 2015] device tap0 entered promiscuous mode
[Sun Jun 21 02:56:03 2015] br0: port 2(tap0) entered forwarding state
[Sun Jun 21 02:56:03 2015] br0: port 2(tap0) entered forwarding state
[Sun Jun 21 02:56:07 2015] kvm: zapping shadow pages for mmio generation wraparound
[Sun Jun 21 02:56:18 2015] br0: port 2(tap0) entered forwarding state
[Sun Jun 21 02:57:12 2015] br0: port 2(tap0) entered disabled state
[Sun Jun 21 02:57:12 2015] device tap0 left promiscuous mode
[Sun Jun 21 02:57:12 2015] br0: port 2(tap0) entered disabled state
[Sun Jun 21 03:03:04 2015] device tap0 entered promiscuous mode
[Sun Jun 21 03:03:04 2015] br0: port 2(tap0) entered forwarding state
[Sun Jun 21 03:03:04 2015] br0: port 2(tap0) entered forwarding state
[Sun Jun 21 03:03:08 2015] kvm: zapping shadow pages for mmio generation wraparound
[Sun Jun 21 03:03:19 2015] br0: port 2(tap0) entered forwarding state
[Sun Jun 21 03:04:33 2015] br0: port 2(tap0) entered disabled state
[Sun Jun 21 03:04:33 2015] device tap0 left promiscuous mode
[Sun Jun 21 03:04:33 2015] br0: port 2(tap0) entered disabled state
[Sun Jun 21 03:15:35 2015] device tap0 entered promiscuous mode
[Sun Jun 21 03:15:35 2015] br0: port 2(tap0) entered forwarding state
[Sun Jun 21 03:15:35 2015] br0: port 2(tap0) entered forwarding state
[Sun Jun 21 03:15:39 2015] kvm: zapping shadow pages for mmio generation wraparound
[Sun Jun 21 03:15:50 2015] br0: port 2(tap0) entered forwarding state
[Sun Jun 21 03:16:37 2015] br0: port 2(tap0) entered disabled state
[Sun Jun 21 03:16:37 2015] device tap0 left promiscuous mode
[Sun Jun 21 03:16:37 2015] br0: port 2(tap0) entered disabled state
[Sun Jun 21 03:48:07 2015] device tap0 entered promiscuous mode
[Sun Jun 21 03:48:07 2015] br0: port 2(tap0) entered forwarding state
[Sun Jun 21 03:48:07 2015] br0: port 2(tap0) entered forwarding state
[Sun Jun 21 03:48:12 2015] kvm: zapping shadow pages for mmio generation wraparound
[Sun Jun 21 03:48:22 2015] br0: port 2(tap0) entered forwarding state
[Sun Jun 21 04:33:41 2015] br0: port 2(tap0) entered disabled state
[Sun Jun 21 04:33:41 2015] device tap0 left promiscuous mode
[Sun Jun 21 04:33:41 2015] br0: port 2(tap0) entered disabled state

Note that the "disabled state" messages are when I actually shutdown the VM to test , so it doesn't go to the disabled state out of spite .

Kernel is 4.1-rc7 with Qemu-git compiled 7 hours ago . (It didn't work with the pervious version compiled a week ago either) .

Any light ?

Last edited by Denso (2015-06-21 01:54:17)

ghormoon

Member

Registered: 2015-05-20

Posts: 17

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

what if you manually create the tap? did you try? I start my vm with
-netdev tap,id=netdev0,ifname=vmtap10,script=no,downscript=no
-device virtio-net-pci,netdev=netdev0,id=net0,mac=xx:xx:xx:xx:xx:xx,bus=pci.2,addr=0x1
the tap has to exist of course

Denso

Member

Registered: 2014-08-30

Posts: 179

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

@ghormoon :

Thanks , I've never tried to create taps manually because QEMU was doing it automatically . And I still prefer to set up one bridge and let QEMU create as many taps as it needs when I run new VMs .

Regards

ghormoon

Member

Registered: 2015-05-20

Posts: 17

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

it's ok to make it automatically if it works and suits you. the point was to test out, where the problem is, if the creation, or something entirely different

Denso

Member

Registered: 2014-08-30

Posts: 179

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

@ ghormoon :

I tried what you suggested just now , and the result is the same .

I can't believe that just changing the motherboard would result in this much hassle . -_-

EDIT :

THe first thing I did after installing this motherboard was compiling Qemu + OVMF + Kernel using GCC 5.1 which was in the stable repos for Arch at that time . Could it be the reason for these unusual issues I'm having thus far ?!

Last edited by Denso (2015-06-21 07:01:37)

scoobydog

Member

Registered: 2014-04-14

Posts: 5

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

hi,

after some experiences with my socket 2011 CPU and x79 chipset I checked the new 1150 Socket CPU with a z97 chipset, and it works with the i915 patch and seabios and without patching only with OVMF on Debian 8 Jessie.

I was so happy that I create some Tutorials for all how want to start with this.

my system is build on:
Intel(R) Core(TM) i5-4590 CPU @ 3.30GHz socket 1150
Gigabyte GA-Z97X-UD3H-BK
16GB DDR3 Memory
512GB SSD
NVIDIA GeForce GTX 780 (ASUS GTX780-DC2OC-3GD5) for passthrugh

Debian 8.1 Jessie

Linux  3.16.0-4-amd64 #1 SMP Debian 3.16.7-ckt11-1 (2015-05-24) x86_64 GNU/Linux
QEMU emulator version 2.1.2 (Debian 1:2.1+dfsg-12+deb8u1), Copyright (c) 2003-2008 Fabrice Bellard
ovmf    0~20150106.5c2d456b-1 from SID repository

English tutorial:
Tutorial Debian 8, passthrough you physical video card to your virtual Windows7 with KVM and OVMF

German Tutorial:
Tutorial: Debian 8, mit KVM und OVMF die Physikalische Grafikkarte im virtuellen Windows7 benutzen

for Kernel Patching
HOWTO Debian Jessie 8 Kernel patch i915 VGA arbiter

kvm-exp.blogspot.de

Last edited by scoobydog (2015-06-21 21:44:28)

Duelist

Member

Registered: 2014-09-22

Posts: 358

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

EDIT :

THe first thing I did after installing this motherboard was compiling Qemu + OVMF + Kernel using GCC 5.1 which was in the stable repos for Arch at that time . Could it be the reason for these unusual issues I'm having thus far ?!

The gentooist within me speaks: "Look for the build log, luke".

---

The forum rules prohibit requesting support for distributions other than arch.
I gave up. It was too late.
What I was trying to do.
The reference about VFIO and KVM VGA passthrough.

Denso

Member

Registered: 2014-08-30

Posts: 179

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

hi,

after some experiences with my socket 2011 CPU and x79 chipset I checked the new 1150 Socket CPU with a z97 chipset, and it works with the i915 patch and seabios and without patching only with OVMF on Debian 8 Jessie.

I was so happy that I create some Tutorials for all how want to start with this.

my system is build on:
Intel(R) Core(TM) i5-4590 CPU @ 3.30GHz socket 1150
Gigabyte GA-Z97X-UD3H-BK
16GB DDR3 Memory
512GB SSD
NVIDIA GeForce GTX 780 (ASUS GTX780-DC2OC-3GD5) for passthrugh

Debian 8.1 Jessie

Linux  3.16.0-4-amd64 #1 SMP Debian 3.16.7-ckt11-1 (2015-05-24) x86_64 GNU/Linux
QEMU emulator version 2.1.2 (Debian 1:2.1+dfsg-12+deb8u1), Copyright (c) 2003-2008 Fabrice Bellard
ovmf    0~20150106.5c2d456b-1 from SID repository

English tutorial:
Tutorial Debian 8, passthrough you physical video card to your virtual Windows7 with KVM and OVMF

German Tutorial:
Tutorial: Debian 8, mit KVM und OVMF die Physikalische Grafikkarte im virtuellen Windows7 benutzen

for Kernel Patching
HOWTO Debian Jessie 8 Kernel patch i915 VGA arbiter

kvm-exp.blogspot.de

Hi ! Thank you for spending the time writing those posts !

@Duelist :

I am certain that if there were problems with the building proccess , the whole thing would throw and error and ask if you would like to start from the begining .

Anyone here compiled these packages using GCC 5.1 ?

Duelist

Member

Registered: 2014-09-22

Posts: 358

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

@Duelist :

I am certain that if there were problems with the building proccess , the whole thing would throw and error and ask if you would like to start from the begining .

Gentooist in me speaks again: "Nope. If you're missing some dependencies, make could just say something "oh, you don't have that software, well, i'll just drop it's support out completely" and continue building. Moreover, in gentoo there's a feature called USE-flags to specifically tell the toolchain that you want some select features."

And also, you can check your iptables.

---

The forum rules prohibit requesting support for distributions other than arch.
I gave up. It was too late.
What I was trying to do.
The reference about VFIO and KVM VGA passthrough.

Denso

Member

Registered: 2014-08-30

Posts: 179

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Gentooist in me speaks again: "Nope. If you're missing some dependencies, make could just say something "oh, you don't have that software, well, i'll just drop it's support out completely" and continue building. Moreover, in gentoo there's a feature called USE-flags to specifically tell the toolchain that you want some select features."

And also, you can check your iptables.

Well , I'll build em packages again using yaourt . How can I save the build log for examining it later ?

Oh , and I don't use iptaples or any other firewalls for the time being , so its not the cause for this issue .

Cheers !