Arch Linux

Denso

Member

Registered: 2014-08-30

Posts: 179

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

@Bronek :

I am passing my primary GPU to a Windows VM successfully . I boot the host with UEFI , and also use OVMF for booting VM (Complete EFI setup) . What I did :

1 - Boot the host with 

video=efifb:off

2 - The GPU you want to passthrough must be first installed as a seconday GPU on the host , then use nvagetbios to dump the ROM from it .
3 - Edit your VM to use the ROM you just dumped .
4 - Shutdown the system and move the card to the first slot to become primary (In case your motherboard doesn't allow you to set which card to use as a primary) .
5 - Boot the host and test the VM .

I passed my GTX770 to a VM thus making my host headless .

Good luck .

nbhs

Member

From: Montevideo, Uruguay

Registered: 2013-05-02

Posts: 402

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

...

OVMF doesn't boot from sata controllers yet,  there are some patches posted on the edk2 mailing list where you can find them, I also posted a link to a patched binary a few pages back, there's also ovmf-fishman-git on AUR which contains said patches

EDIT: OVMF-sata.tar.gz

Last edited by nbhs (2015-04-09 19:09:01)

Bronek

Member

From: London

Registered: 2014-02-14

Posts: 123

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

@Bronek :
I am passing my primary GPU to a Windows VM successfully . I boot the host with UEFI , and also use OVMF for booting VM (Complete EFI setup) . What I did :

1 - Boot the host with 

video=efifb:off

that's a good hint. I am not using EFI, but I will try all of

nofb nomodeset video=vesafb:off

and see if any of those make any difference. As for the other points, I found that my card works just fine with its own ROM as initialised by computer BIOS, thus no need to fiddle.

Basically my problem is not that passthrough does not work on primary GPU (it does work, most of the time), but that despite this

 pci-stub.ids=. . .,1002:6749,1002:aa90,1002:6707,1002:aa80

the card is not claimed by pci-stub at boot, thus allowing other modules to claim it which caused me some trouble.

My dmesg says

[    0.963355] pci 0000:03:00.0: [1002:6707] type 00 class 0x030000
[    0.963367] pci 0000:03:00.0: reg 0x10: [mem 0xc0000000-0xcfffffff 64bit pref]
[    0.963376] pci 0000:03:00.0: reg 0x18: [mem 0xdfd20000-0xdfd3ffff 64bit]
[    0.963382] pci 0000:03:00.0: reg 0x20: [io  0x7000-0x70ff]
[    0.963393] pci 0000:03:00.0: reg 0x30: [mem 0xdfd00000-0xdfd1ffff pref]
[    0.963426] pci 0000:03:00.0: supports D1 D2
[    0.963468] pci 0000:03:00.1: [1002:aa80] type 00 class 0x040300
[    0.963479] pci 0000:03:00.1: reg 0x10: [mem 0xdfd40000-0xdfd43fff 64bit]
[    0.963534] pci 0000:03:00.1: supports D1 D2
. . .
[    1.024920] vgaarb: setting as boot device: PCI:0000:03:00.0
[    1.024922] vgaarb: device added: PCI:0000:03:00.0,decodes=io+mem,owns=io+mem,locks=none
[    1.024933] vgaarb: device added: PCI:0000:82:00.0,decodes=io+mem,owns=none,locks=none
[    1.024938] vgaarb: loaded
[    1.024939] vgaarb: bridge control possible 0000:82:00.0
[    1.024940] vgaarb: bridge control possible 0000:03:00.0
. . .
[    1.082113] pci 0000:03:00.0: Video device with shadowed ROM
. . .
[    1.185892] pci-stub: add 1002:6749 sub=FFFFFFFF:FFFFFFFF cls=00000000/00000000
[    1.185917] pci-stub 0000:82:00.0: claimed by stub
[    1.185929] pci-stub: add 1002:AA90 sub=FFFFFFFF:FFFFFFFF cls=00000000/00000000
[    1.185947] pci-stub 0000:82:00.1: claimed by stub

Note that 1002:6707,1002:aa80 are not claimed. I assumed its vgaarb preventing pci-stub from claiming the card. Incorrectly as it turns out, but now I do not know what does it.

Good luck .

Thanks

Last edited by Bronek (2015-04-09 20:31:22)

Duelist

Member

Registered: 2014-09-22

Posts: 358

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

@Bronek:
Can you give us lspci -v output?

Because...

[    0.373329] vgaarb: device added: PCI:0000:01:00.0,decodes=io+mem,owns=io+mem,locks=none
[    0.373447] vgaarb: device added: PCI:0000:02:00.0,decodes=io+mem,owns=none,locks=none
[    0.373563] vgaarb: device added: PCI:0000:04:00.0,decodes=io+mem,owns=none,locks=none
[    0.373677] vgaarb: loaded
[    0.373751] vgaarb: bridge control possible 0000:04:00.0
[    0.373825] vgaarb: bridge control possible 0000:02:00.0
[    0.373900] vgaarb: bridge control possible 0000:01:00.0

and then...

[    0.914831] pci-stub: add 1022:780D sub=FFFFFFFF:FFFFFFFF cls=00000000/00000000
[    0.914951] pci-stub 0000:00:14.2: claimed by stub
[    0.915030] pci-stub: add 1002:683F sub=FFFFFFFF:FFFFFFFF cls=00000000/00000000
[    0.915156] pci-stub 0000:01:00.0: claimed by stub
[    0.915236] pci-stub 0000:02:00.0: claimed by stub
[    0.915312] pci-stub: add 1002:AAB0 sub=FFFFFFFF:FFFFFFFF cls=00000000/00000000
[    0.915432] pci-stub 0000:01:00.1: claimed by stub
[    0.915511] pci-stub 0000:02:00.1: claimed by stub
[    0.915588] pci-stub: add 1002:683F sub=FFFFFFFF:FFFFFFFF cls=00000000/00000000
[    0.915703] pci-stub: add 1002:683F sub=FFFFFFFF:FFFFFFFF cls=00000000/00000000

is what i see in logs on my system.
And it gets bound to pci-stub fine. Then vfio-bind systemd service starts, binding the devices to vfio-pci

pci-stub.ids=1022:780d,1002:683f,1002:aab0,1002:683f,1002:683f

Note: I have 2 identical cards, BUT, as you can see, i've made a typo(just noted, lol), you don't need to enter it multiple times. It's not possible to passthrough one of two identical cards without using a special pci-stub version which binds by PCI BDF address rather than ven_id:dev_id pair. Even if it didn't bind for the first time for some weird reason, pci-stub tries to bind it again.

Therefore, i'm not sure that your devices aren't bound to pci-stub or vfio-pci. Maybe they are?

BTW:

01:00.1 Audio device: Advanced Micro Devices, Inc. [AMD/ATI] Cape Verde/Pitcairn HDMI Audio [Radeon HD 7700/7800 Series]
        Subsystem: ASUSTeK Computer Inc. Device aab0
        Flags: bus master, fast devsel, latency 0, IRQ 15
        Memory at fe360000 (64-bit, non-prefetchable) [size=16K]
        Capabilities: <access denied>
        Kernel driver in use: vfio-pci
        Kernel modules: snd_hda_intel

As you can see, i've got audio device bound as well.

EDIT:
An idea struck me suddenly.
Maybe you didn't update your grub config, containing the new kernel command-line?

su -c "grub2-mkconfig > /boot/grub/grub.cfg"

Try doing cat /proc/cmdline to find out what's your current kernel commandline.

Last edited by Duelist (2015-04-09 21:25:58)

---

The forum rules prohibit requesting support for distributions other than arch.
I gave up. It was too late.
What I was trying to do.
The reference about VFIO and KVM VGA passthrough.

Marius_Linux

Member

Registered: 2015-04-07

Posts: 3

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Hi, after a full week of trying to get this working, I don't know what to do anymore right now.
I'm running Fedora 21(latest Updates, Kernel 3.19.3-200.fc21.x86_64, QEMU qemu-2.1.3-3.fc21), trying to passthrough my two NVIDIA GTX 680(one is a Palit Jetstream, the other one a normal Gainward) to a VM, and I'm using OVMF.
My CPU is an Intel i7 4770 (without K). supporting VT-d/x.
Now, the problem is, I can boot and install Windows 8.1, it detects the GPU, I then install driver version 314.xx (so without the hypervisor detection), but after reboot of the VM, Windows reports them with Code 43, and they are not working without NVIDIA drivers.
This is my lspci -nn:

00:00.0 Host bridge [0600]: Intel Corporation Xeon E3-1200 v2/3rd Gen Core processor DRAM Controller [8086:0150] (rev 09)
00:01.0 PCI bridge [0604]: Intel Corporation Xeon E3-1200 v2/3rd Gen Core processor PCI Express Root Port [8086:0151] (rev 09)
00:01.1 PCI bridge [0604]: Intel Corporation Xeon E3-1200 v2/3rd Gen Core processor PCI Express Root Port [8086:0155] (rev 09)
00:02.0 VGA compatible controller [0300]: Intel Corporation Xeon E3-1200 v2/3rd Gen Core processor Graphics Controller [8086:0162] (rev 09)
00:14.0 USB controller [0c03]: Intel Corporation 7 Series/C210 Series Chipset Family USB xHCI Host Controller [8086:1e31] (rev 04)
00:16.0 Communication controller [0780]: Intel Corporation 7 Series/C210 Series Chipset Family MEI Controller #1 [8086:1e3a] (rev 04)
00:1a.0 USB controller [0c03]: Intel Corporation 7 Series/C210 Series Chipset Family USB Enhanced Host Controller #2 [8086:1e2d] (rev 04)
00:1c.0 PCI bridge [0604]: Intel Corporation 7 Series/C210 Series Chipset Family PCI Express Root Port 1 [8086:1e10] (rev c4)
00:1c.3 PCI bridge [0604]: Intel Corporation 7 Series/C210 Series Chipset Family PCI Express Root Port 4 [8086:1e16] (rev c4)
00:1c.4 PCI bridge [0604]: Intel Corporation 7 Series/C210 Series Chipset Family PCI Express Root Port 5 [8086:1e18] (rev c4)
00:1c.5 PCI bridge [0604]: Intel Corporation 7 Series/C210 Series Chipset Family PCI Express Root Port 6 [8086:1e1a] (rev c4)
00:1c.6 PCI bridge [0604]: Intel Corporation 7 Series/C210 Series Chipset Family PCI Express Root Port 7 [8086:1e1c] (rev c4)
00:1c.7 PCI bridge [0604]: Intel Corporation 7 Series/C210 Series Chipset Family PCI Express Root Port 8 [8086:1e1e] (rev c4)
00:1d.0 USB controller [0c03]: Intel Corporation 7 Series/C210 Series Chipset Family USB Enhanced Host Controller #1 [8086:1e26] (rev 04)
00:1f.0 ISA bridge [0601]: Intel Corporation Z77 Express Chipset LPC Controller [8086:1e44] (rev 04)
00:1f.2 SATA controller [0106]: Intel Corporation 7 Series/C210 Series Chipset Family 6-port SATA Controller [AHCI mode] [8086:1e02] (rev 04)
00:1f.3 SMBus [0c05]: Intel Corporation 7 Series/C210 Series Chipset Family SMBus Controller [8086:1e22] (rev 04)
01:00.0 VGA compatible controller [0300]: NVIDIA Corporation GK104 [GeForce GTX 680] [10de:1180] (rev a1)
01:00.1 Audio device [0403]: NVIDIA Corporation GK104 HDMI Audio Controller [10de:0e0a] (rev a1)
02:00.0 VGA compatible controller [0300]: NVIDIA Corporation GK104 [GeForce GTX 680] [10de:1180] (rev a1)
02:00.1 Audio device [0403]: NVIDIA Corporation GK104 HDMI Audio Controller [10de:0e0a] (rev a1)
04:00.0 SATA controller [0106]: ASMedia Technology Inc. ASM1062 Serial ATA Controller [1b21:0612] (rev 01)
05:00.0 Ethernet controller [0200]: Broadcom Corporation NetLink BCM57781 Gigabit Ethernet PCIe [14e4:16b1] (rev 10)
06:00.0 PCI bridge [0604]: ASMedia Technology Inc. ASM1083/1085 PCIe to PCI Bridge [1b21:1080] (rev 01)
08:00.0 PCI bridge [0604]: PLX Technology, Inc. PEX8112 x1 Lane PCI Express-to-PCI Bridge [10b5:8112] (rev aa)
09:04.0 Multimedia audio controller [0401]: C-Media Electronics Inc CMI8788 [Oxygen HD Audio] [13f6:8788]
0a:00.0 USB controller [0c03]: ASMedia Technology Inc. ASM1042 SuperSpeed USB Host Controller [1b21:1042] 

I'm using virt-manager, so this is my config XML:

<domain type='kvm' xmlns:qemu='http://libvirt.org/schemas/domain/qemu/1.0'>
  <name>win8.1</name>
  <uuid>e7d551c9-a77e-4a90-8974-9c881f7662ad</uuid>
  <memory unit='KiB'>4194304</memory>
  <currentMemory unit='KiB'>4194304</currentMemory>
  <vcpu placement='static'>8</vcpu>
  <os>
    <type arch='x86_64' machine='pc-i440fx-2.1'>hvm</type>
    <loader type='pflash'>/var/lib/libvirt/images/win81-OVMF.fd</loader>
  </os>
  <features>
    <acpi/>
    <apic/>
    <pae/>
    <kvm>
      <hidden state='on'/>
    </kvm>
  </features>
  <cpu mode='host-model'>
    <model fallback='allow'/>
  </cpu>
  <clock offset='localtime'>
    <timer name='rtc' tickpolicy='catchup'/>
    <timer name='pit' tickpolicy='delay'/>
    <timer name='hpet' present='no'/>
    <timer name='hypervclock' present='no'/>
  </clock>
  <on_poweroff>destroy</on_poweroff>
  <on_reboot>restart</on_reboot>
  <on_crash>restart</on_crash>
  <pm>
    <suspend-to-mem enabled='no'/>
    <suspend-to-disk enabled='no'/>
  </pm>
  <devices>
    <emulator>/usr/bin/qemu-kvm</emulator>
    <disk type='file' device='disk'>
      <driver name='qemu' type='qcow2'/>
      <source file='/var/lib/libvirt/images/win8.1.qcow2'/>
      <target dev='hda' bus='ide'/>
      <boot order='2'/>
      <address type='drive' controller='0' bus='0' target='0' unit='0'/>
    </disk>
    <disk type='file' device='cdrom'>
      <driver name='qemu' type='raw'/>
      <source file='/home/marius/Downloads/de_windows_8_1_x64_dvd_2707227.iso'/>
      <target dev='hdb' bus='ide'/>
      <readonly/>
      <boot order='1'/>
      <address type='drive' controller='0' bus='0' target='0' unit='1'/>
    </disk>
    <disk type='file' device='disk'>
      <driver name='qemu' type='raw'/>
      <source file='/var/lib/libvirt/images/win81-OVMF.fd'/>
      <target dev='sda' bus='usb'/>
    </disk>
    <controller type='usb' index='0' model='ich9-ehci1'>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x05' function='0x7'/>
    </controller>
    <controller type='usb' index='0' model='ich9-uhci1'>
      <master startport='0'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x05' function='0x0' multifunction='on'/>
    </controller>
    <controller type='usb' index='0' model='ich9-uhci2'>
      <master startport='2'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x05' function='0x1'/>
    </controller>
    <controller type='usb' index='0' model='ich9-uhci3'>
      <master startport='4'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x05' function='0x2'/>
    </controller>
    <controller type='pci' index='0' model='pci-root'/>
    <controller type='ide' index='0'>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x1'/>
    </controller>
    <controller type='virtio-serial' index='0'>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x06' function='0x0'/>
    </controller>
    <serial type='pty'>
      <target port='0'/>
    </serial>
    <console type='pty'>
      <target type='serial' port='0'/>
    </console>
    <input type='tablet' bus='usb'/>
    <sound model='ich6'>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x04' function='0x0'/>
    </sound>
    <hostdev mode='subsystem' type='pci' managed='yes'>
      <source>
        <address domain='0x0000' bus='0x01' slot='0x00' function='0x0'/>
      </source>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x07' function='0x0'/>
    </hostdev>
    <hostdev mode='subsystem' type='pci' managed='yes'>
      <source>
        <address domain='0x0000' bus='0x01' slot='0x00' function='0x1'/>
      </source>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x08' function='0x0'/>
    </hostdev>
    <hostdev mode='subsystem' type='pci' managed='yes'>
      <source>
        <address domain='0x0000' bus='0x02' slot='0x00' function='0x0'/>
      </source>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x09' function='0x0'/>
    </hostdev>
    <hostdev mode='subsystem' type='pci' managed='yes'>
      <source>
        <address domain='0x0000' bus='0x02' slot='0x00' function='0x1'/>
      </source>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x0a' function='0x0'/>
    </hostdev>
    <redirdev bus='usb' type='spicevmc'>
    </redirdev>
    <redirdev bus='usb' type='spicevmc'>
    </redirdev>
    <memballoon model='virtio'>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x0b' function='0x0'/>
    </memballoon>
  </devices>
  <qemu:commandline>
    <qemu:arg value='-vga'/>
    <qemu:arg value='none'/>
  </qemu:commandline>
</domain>

It produces this command:

/usr/bin/qemu-system-x86_64 -machine accel=kvm -name win8.1 -S -machine pc-i440fx-2.1,accel=kvm,usb=off -cpu SandyBridge,+erms,+smep,+fsgsbase,+rdrand,+f16c,+osxsave,+pcid,+pdcm,+xtpr,+tm2,+est,+smx,+vmx,+ds_cpl,+monitor,+dtes64,+pbe,+tm,+ht,+ss,+acpi,+ds,+vme,kvm=off -drive file=/var/lib/libvirt/images/win81-OVMF.fd,if=pflash,format=raw,unit=0 -m 4096 -realtime mlock=off -smp 8,sockets=8,cores=1,threads=1 -uuid e7d551c9-a77e-4a90-8974-9c881f7662ad -nographic -no-user-config -nodefaults -chardev socket,id=charmonitor,path=/var/lib/libvirt/qemu/win8.1.monitor,server,nowait -mon chardev=charmonitor,id=monitor,mode=control -rtc base=localtime,driftfix=slew -global kvm-pit.lost_tick_policy=discard -no-hpet -no-shutdown -global PIIX4_PM.disable_s3=1 -global PIIX4_PM.disable_s4=1 -boot strict=on -device ich9-usb-ehci1,id=usb,bus=pci.0,addr=0x5.0x7 -device ich9-usb-uhci1,masterbus=usb.0,firstport=0,bus=pci.0,multifunction=on,addr=0x5 -device ich9-usb-uhci2,masterbus=usb.0,firstport=2,bus=pci.0,addr=0x5.0x1 -device ich9-usb-uhci3,masterbus=usb.0,firstport=4,bus=pci.0,addr=0x5.0x2 -device virtio-serial-pci,id=virtio-serial0,bus=pci.0,addr=0x6 -drive file=/var/lib/libvirt/images/win8.1.qcow2,if=none,id=drive-ide0-0-0,format=qcow2 -device ide-hd,bus=ide.0,unit=0,drive=drive-ide0-0-0,id=ide0-0-0,bootindex=2 -drive file=/home/marius/Downloads/de_windows_8_1_x64_dvd_2707227.iso,if=none,id=drive-ide0-0-1,readonly=on,format=raw -device ide-cd,bus=ide.0,unit=1,drive=drive-ide0-0-1,id=ide0-0-1,bootindex=1 -drive file=/var/lib/libvirt/images/win81-OVMF.fd,if=none,id=drive-usb-disk0,format=raw -device usb-storage,drive=drive-usb-disk0,id=usb-disk0,removable=off -chardev pty,id=charserial0 -device isa-serial,chardev=charserial0,id=serial0 -device usb-tablet,id=input0 -device intel-hda,id=sound0,bus=pci.0,addr=0x4 -device hda-duplex,id=sound0-codec0,bus=sound0.0,cad=0 -chardev spicevmc,id=charredir0,name=usbredir -device usb-redir,chardev=charredir0,id=redir0 -chardev spicevmc,id=charredir1,name=usbredir -device usb-redir,chardev=charredir1,id=redir1 -device vfio-pci,host=01:00.0,id=hostdev0,bus=pci.0,addr=0x7 -device vfio-pci,host=01:00.1,id=hostdev1,bus=pci.0,addr=0x8 -device vfio-pci,host=02:00.0,id=hostdev2,bus=pci.0,addr=0x9 -device vfio-pci,host=02:00.1,id=hostdev3,bus=pci.0,addr=0xa -device virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0xb -vga none -msg timestamp=on 

Anybody who could help me please?
Many thanks in advance
Marius

Bronek

Member

From: London

Registered: 2014-02-14

Posts: 123

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

@Bronek:
Can you give us lspci -v output?

Here

# lspci -vn -s 0000:03:00.0
03:00.0 0300: 1002:6707 (prog-if 00 [VGA controller])
        Subsystem: 1002:0b06
        Flags: bus master, fast devsel, latency 0, IRQ 162
        Memory at c0000000 (64-bit, prefetchable) [size=256M]
        Memory at dfd20000 (64-bit, non-prefetchable) [size=128K]
        I/O ports at 7000 [size=256]
        Expansion ROM at dfd00000 [disabled] [size=128K]
        Capabilities: [50] Power Management version 3
        Capabilities: [58] Express Legacy Endpoint, MSI 00
        Capabilities: [a0] MSI: Enable+ Count=1/1 Maskable- 64bit+
        Capabilities: [100] Vendor Specific Information: ID=0001 Rev=1 Len=010 <?>
        Capabilities: [150] Advanced Error Reporting
        Kernel driver in use: vfio-pci
        Kernel modules: radeon

as you can see, I'm running guest at this moment which is why module in use is vfio-pci. Before I started guest there was no driver (any) in use for primary card (radeon is blacklisted), while for the other card it was "Kernel driver in use: pci-stub" (consistent with dmesg)

Because...

[    0.373329] vgaarb: device added: PCI:0000:01:00.0,decodes=io+mem,owns=io+mem,locks=none
[    0.373447] vgaarb: device added: PCI:0000:02:00.0,decodes=io+mem,owns=none,locks=none
[    0.373563] vgaarb: device added: PCI:0000:04:00.0,decodes=io+mem,owns=none,locks=none
[    0.373677] vgaarb: loaded
[    0.373751] vgaarb: bridge control possible 0000:04:00.0
[    0.373825] vgaarb: bridge control possible 0000:02:00.0
[    0.373900] vgaarb: bridge control possible 0000:01:00.0

that interesting, you do not have " vgaarb: setting as boot device". I guess that's because booting from EFI does not need vgaarb.

As for audio, I do not use HDMI (I dont even passthrough function .1), I have USB ODAC instead attached to one of USB cards passed through to guest. I like this sound much better

EDIT:
An idea struck me suddenly.
Maybe you didn't update your grub config, containing the new kernel command-line?

that's exactly why I'm not using GRUB Syslinux here, much simpler.

Last edited by Bronek (2015-04-09 21:40:59)

Duelist

Member

Registered: 2014-09-22

Posts: 358

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

@Bronek:
Can you give us lspci -v output?

Here

# lspci -vn -s 0000:03:00.0
03:00.0 0300: 1002:6707 (prog-if 00 [VGA controller])
        Subsystem: 1002:0b06
        Flags: bus master, fast devsel, latency 0, IRQ 162
        Memory at c0000000 (64-bit, prefetchable) [size=256M]
        Memory at dfd20000 (64-bit, non-prefetchable) [size=128K]
        I/O ports at 7000 [size=256]
        Expansion ROM at dfd00000 [disabled] [size=128K]
        Capabilities: [50] Power Management version 3
        Capabilities: [58] Express Legacy Endpoint, MSI 00
        Capabilities: [a0] MSI: Enable+ Count=1/1 Maskable- 64bit+
        Capabilities: [100] Vendor Specific Information: ID=0001 Rev=1 Len=010 <?>
        Capabilities: [150] Advanced Error Reporting
        Kernel driver in use: vfio-pci
        Kernel modules: radeon

as you can see, I'm running guest at this moment which is why module in use is vfio-pci. Before I started guest there was no driver (any) in use for primary card (radeon is blacklisted), while for the other card it was "Kernel driver in use: pci-stub" (consistent with dmesg)

Because...

[    0.373329] vgaarb: device added: PCI:0000:01:00.0,decodes=io+mem,owns=io+mem,locks=none
[    0.373447] vgaarb: device added: PCI:0000:02:00.0,decodes=io+mem,owns=none,locks=none
[    0.373563] vgaarb: device added: PCI:0000:04:00.0,decodes=io+mem,owns=none,locks=none
[    0.373677] vgaarb: loaded
[    0.373751] vgaarb: bridge control possible 0000:04:00.0
[    0.373825] vgaarb: bridge control possible 0000:02:00.0
[    0.373900] vgaarb: bridge control possible 0000:01:00.0

that interesting, you do not have " vgaarb: setting as boot device". I guess that's because booting from EFI does not need vgaarb.

As for audio, I do not use HDMI (I dont even passthrough function .1), I have USB ODAC instead attached to one of USB cards passed through to guest. I like this sound much better

EDIT:
An idea struck me suddenly.
Maybe you didn't update your grub config, containing the new kernel command-line?

that's exactly why I'm not using GRUB Syslinux here, much simpler.

[    0.373252] vgaarb: setting as boot device: PCI:0000:01:00.0

Apparently, it got clipped. I do have vgaarb and i'm booting host with csm. Too lazy to redo.

Do cat /proc/cmdline. Maybe it's syslinux's fault. Also, i think, you can make a file in /etc/modprobe.d/ and put pci-stub options there.

---

The forum rules prohibit requesting support for distributions other than arch.
I gave up. It was too late.
What I was trying to do.
The reference about VFIO and KVM VGA passthrough.

Duelist

Member

Registered: 2014-09-22

Posts: 358

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Hi, after a full week of trying to get this working, I don't know what to do anymore right now.
I'm running Fedora 21(latest Updates, Kernel 3.19.3-200.fc21.x86_64, QEMU qemu-2.1.3-3.fc21), trying to passthrough my two NVIDIA GTX 680(one is a Palit Jetstream, the other one a normal Gainward) to a VM, and I'm using OVMF.
My CPU is an Intel i7 4770 (without K). supporting VT-d/x.
Now, the problem is, I can boot and install Windows 8.1, it detects the GPU, I then install driver version 314.xx (so without the hypervisor detection), but after reboot of the VM, Windows reports them with Code 43, and they are not working without NVIDIA drivers.
This is my lspci -nn:

..snip..
01:00.0 VGA compatible controller [0300]: NVIDIA Corporation GK104 [GeForce GTX 680] [10de:1180] (rev a1)
01:00.1 Audio device [0403]: NVIDIA Corporation GK104 HDMI Audio Controller [10de:0e0a] (rev a1)
02:00.0 VGA compatible controller [0300]: NVIDIA Corporation GK104 [GeForce GTX 680] [10de:1180] (rev a1)
02:00.1 Audio device [0403]: NVIDIA Corporation GK104 HDMI Audio Controller [10de:0e0a] (rev a1)
..snip..

I'm using virt-manager, so this is my config XML:
..snip..

It produces this command:
..snip..

Anybody who could help me please?
Many thanks in advance
Marius

First of all - if you're going to try using SLI - there may be problems with 440fx being "compatible" with SLI. In the real, hardware world you must have "SLI-compatible" certified motherboard for SLI to work. Moreover, back in the day, you must've use HyperSLI (based on VT-d) third party software to make SLI work on non-SLI-certified motherboards. The software is outdated, but still available, i wonder if it will work inside a VM using nested virtualization. That would be crazy.
List of things you can do:
1. Try Q35 chipset model.
2. Provide us dmesg to see if there's correct interrupt assigning messages and all the needed stuff.
3. Feed GTX680s with ROMs dumped from them in case of "Invalid ROM contents" messages. Also you could get the needed ROMs here.
4. "Quadrify" your GPUs using modified vfio-pci with fake PCI IDs and (maybe) ROMs. You could quadrify it hardware-way, but that's too hardcore and pointless if the culprit is somewhere else. That'll make all the problems with drivers detecting hypervisor go away, as it's 100% legit to passthrough a Quadro device. That way you could use more fresh drivers.
5. As i can see, you have Z77 chipset. Lurk a few pages back, there was some guy with that chipset and some problems with nvidia. I don't remember what the problem was, actually.
6. Try using legacy KVM pci passthrough: here's an example of using it(driver option). It doesn't work with VGA, but since you're using OVMF anyway...
7. Wait for aw to come, heh.

Last edited by Duelist (2015-04-09 22:26:38)

---

The forum rules prohibit requesting support for distributions other than arch.
I gave up. It was too late.
What I was trying to do.
The reference about VFIO and KVM VGA passthrough.

Bronek

Member

From: London

Registered: 2014-02-14

Posts: 123

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Do cat /proc/cmdline.

Thanks, that was a very good advice. It turns out I am an idiot (don't tell my family, they already know), I wrote new boot parameters in the wrong file.

Duelist

Member

Registered: 2014-09-22

Posts: 358

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

I wrote new boot parameters in the wrong file.

that's exactly why I'm not using GRUB Syslinux here, much simpler.

Kek.

---

The forum rules prohibit requesting support for distributions other than arch.
I gave up. It was too late.
What I was trying to do.
The reference about VFIO and KVM VGA passthrough.

aw

Member

Registered: 2013-10-04

Posts: 921

Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Now, the problem is, I can boot and install Windows 8.1, it detects the GPU, I then install driver version 314.xx (so without the hypervisor detection), ...

Is this a typo?  You only need pre-338.77 to avoid all hypervisor detection and pre-344.11 for the hyper-v stuff.  Try a newer driver.  Start with a single card for the guest.  Dump your <qemu:args> section, you don't need it.

---

http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Slabity

Member

Registered: 2013-12-29

Posts: 40

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

So I managed to successfully pass through my GPU using the OVMF method, but I'm struggling to get audio to work over ALSA. I am using libvirt, and have added the following to my VM config:

<qemu:commandline>
  <qemu:env name='QEMU_AUDIO_DRV' value='alsa'/>
  <qemu:env name='QEMU_ALSA_DAC_PERIOD_SIZE' value='170'/>
  <qemu:env name='QEMU_ALSA_DAC_BUFFER_SIZE' value='512'/>
</qemu:commandline>

But my host does not show any ALSA client appearing. Any ideas where I should look?

Marius_Linux

Member

Registered: 2015-04-07

Posts: 3

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Hi, after a full week of trying to get this working, I don't know what to do anymore right now.
I'm running Fedora 21(latest Updates, Kernel 3.19.3-200.fc21.x86_64, QEMU qemu-2.1.3-3.fc21), trying to passthrough my two NVIDIA GTX 680(one is a Palit Jetstream, the other one a normal Gainward) to a VM, and I'm using OVMF.
My CPU is an Intel i7 4770 (without K). supporting VT-d/x.
Now, the problem is, I can boot and install Windows 8.1, it detects the GPU, I then install driver version 314.xx (so without the hypervisor detection), but after reboot of the VM, Windows reports them with Code 43, and they are not working without NVIDIA drivers.
This is my lspci -nn:

..snip..
01:00.0 VGA compatible controller [0300]: NVIDIA Corporation GK104 [GeForce GTX 680] [10de:1180] (rev a1)
01:00.1 Audio device [0403]: NVIDIA Corporation GK104 HDMI Audio Controller [10de:0e0a] (rev a1)
02:00.0 VGA compatible controller [0300]: NVIDIA Corporation GK104 [GeForce GTX 680] [10de:1180] (rev a1)
02:00.1 Audio device [0403]: NVIDIA Corporation GK104 HDMI Audio Controller [10de:0e0a] (rev a1)
..snip..

I'm using virt-manager, so this is my config XML:
..snip..

It produces this command:
..snip..

Anybody who could help me please?
Many thanks in advance
Marius

First of all - if you're going to try using SLI - there may be problems with 440fx being "compatible" with SLI. In the real, hardware world you must have "SLI-compatible" certified motherboard for SLI to work. Moreover, back in the day, you must've use HyperSLI (based on VT-d) third party software to make SLI work on non-SLI-certified motherboards. The software is outdated, but still available, i wonder if it will work inside a VM using nested virtualization. That would be crazy.
List of things you can do:
1. Try Q35 chipset model.
2. Provide us dmesg to see if there's correct interrupt assigning messages and all the needed stuff.
3. Feed GTX680s with ROMs dumped from them in case of "Invalid ROM contents" messages. Also you could get the needed ROMs here.
4. "Quadrify" your GPUs using modified vfio-pci with fake PCI IDs and (maybe) ROMs. You could quadrify it hardware-way, but that's too hardcore and pointless if the culprit is somewhere else. That'll make all the problems with drivers detecting hypervisor go away, as it's 100% legit to passthrough a Quadro device. That way you could use more fresh drivers.
5. As i can see, you have Z77 chipset. Lurk a few pages back, there was some guy with that chipset and some problems with nvidia. I don't remember what the problem was, actually.
6. Try using legacy KVM pci passthrough: here's an example of using it(driver option). It doesn't work with VGA, but since you're using OVMF anyway...
7. Wait for aw to come, heh.

Now, the problem is, I can boot and install Windows 8.1, it detects the GPU, I then install driver version 314.xx (so without the hypervisor detection), ...

Is this a typo?  You only need pre-338.77 to avoid all hypervisor detection and pre-344.11 for the hyper-v stuff.  Try a newer driver.  Start with a single card for the guest.  Dump your <qemu:args> section, you don't need it.

Thanks for your help you two, I was surprised how quick you answered :-)
I've just checked my Palit card using your tool, and as it seems, it doesn't have EFI support:

Valid ROM signature found @0h, PCIR offset 190h
        PCIR: type 0, vendor: 10de, device: 1180, class: 030000
        PCIR: revision 0, vendor revision: 1
        Last image
 

Currently, I am looking for a BIOS for this Card(and my Gainward GTX 680), which support EFI. Does anybody know where to look, or how to check if the ROM is compatible with my card?

Last edited by Marius_Linux (2015-04-10 14:45:36)

Duelist

Member

Registered: 2014-09-22

Posts: 358

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Currently, I am looking for a BIOS for this Card(and my Gainward GTX 680), which support EFI. Does anybody know where to look, or how to check if the ROM is compatible with my card?

3. Feed GTX680s with ROMs dumped from them in case of "Invalid ROM contents" messages. Also you could get the needed ROMs here.

TechPowerUp GPU BIOS database.
Official site of your GPU vendor is also a good place to look for GPU bios updates. Since my GT610 does support UEFI, it's very strange that your GTX680 doesn't.
Something hints me that all 680 are alike to each other being reference designs, and they all can be quadrified and/or their VBIOS images are compatible.
Of course, vendor may change the reference design, ruining compatibility, but i haven't heard of that for 680s.

How to check.. Well, add it and observe. If the card will(for some weird and unlikely reason) start smoking - quick CTRL-C will save it. If you see Tianocore boot splash - everything's working fine.

<rom bar='on' file='/path/to/your/romfile.rom'/>

P.S.
AFAIR, SLI won't double your video memory, but XDMA CF will.

Last edited by Duelist (2015-04-10 17:06:08)

---

The forum rules prohibit requesting support for distributions other than arch.
I gave up. It was too late.
What I was trying to do.
The reference about VFIO and KVM VGA passthrough.

lersek

Member

Registered: 2015-03-15

Posts: 38

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Yes, you can theoretically pass through any PCI endpoint device.  How well it works depends on the device.  Whether it's supported as a pre-boot device depends on whether there's a boot ROM for the device or built-in support or standard compatibility in the VM firmware.  IOW, try it and find out.  OTOH, my personal opinion is that assigning PCI HBAs often provides a marginal performance improvement and significantly more headaches vs paravirtual solutions.

I gave this a shot today just to experiment. I connected a spare 1TB SATA drive to the motherboard's Marvell SATA controller (non-chipset 2nd controller) and passed that through qemu to play around with. OVMF shows no signs of seeing a drive/controller (not sure if I should even be seeing anything really), but booting off a Windows install ISO shows the hard drive perfectly fine, and allows me to install to it. However, after the install completes and the guest system reboots, OVMF just dumps back at the Shell> prompt. OVMF's Boot Manager shows a "Windows Boot Manager" entry with device path: HD(2,GPT,UUID#)/\EFI\Microsoft\Boot\bootmgfw.efi so it seems like Windows at least installed boot information to the drive and to the firmware VARS I think? Not sure what to do from here or if it is a dead end. Maybe others have experimented with trying something like this (this is just all out of curiosity's sake). Probably has something to do with HD index showing as being at 2, wonder if this could be manually edited.

I try the "same" thing with my Motherbord Sata Controller it's a SB7x0/SB8x0/SB9x0 SATA Controller [AHCI mode]. The Windows 8.1 DVD won't boot and the Windows 10 Tech Preview freezes.

Ovmf has no boot option for this controller, even when i use uefi-raid mode.

I hope someone can help with that.

Try virtio-scsi block passthrough. I found a concise (libvirt based) example here: https://blueprints.launchpad.net/nova/+ … ce-mapping

From http://libvirt.org/formatdomain.html, "... generic SCSI commands from the guest are accepted and passed through to the physical device ...". This can be used for CD-ROM/DVD burning passthrough etc (http://wiki.qemu.org/Features/VirtioSCSI).

Also, OVMF's builtin virtio-scsi driver will be able to handle & boot off such devices. The qemu command line is:

  -device virtio-scsi-pci,id=scsi0 \
  -drive file=/dev/disk/by-id/...,if=none,id=virtio-scsi-passthrough0,format=raw \
  -device scsi-block,bus=scsi0.0,drive=virtio-scsi-passthrough0

Helios747

Member

Registered: 2012-11-17

Posts: 12

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Hello, I'm on ubuntu 14.10 x64 and I'm trying to get this to work. So far I run the qemu script and I get a small console where I can input commands to qemu, but no output on my display hooked up to my guest GPU, nor any errors.

My specs:

Gigabyte GA-Z97-HD3 (VT-d enabled)
Intel Core i7-4790K (intel ark says VT-d compatible)
Host GPU: Intel HD Graphics 4600
Guest GPU: MSI TwinFrozr GTX 770

output of "lspci -nn | grep NVIDIA"

01:00.0 VGA compatible controller [0300]: NVIDIA Corporation GK104 [GeForce GTX 770] [10de:1184] (rev a1)
01:00.1 Audio device [0403]: NVIDIA Corporation GK104 HDMI Audio Controller [10de:0e0a] (rev a1)

This is my grub command line options

GRUB_CMDLINE_LINUX_DEFAULT="quiet splash pcie_acs_override=downstream intel_iommu=on,igfx_on i915.enable_hd_vgaarb=1"

I have nouveau blacklisted, I have the following in /etc/initramfs-tools/modules (And I updated initramfs after)

pci_stub ids=10de:1184,10de:0e0a

I ran the vfio-bind script with no errors. I ran it with

sudo ./vfio-bind 0000:01:00.0 0000:01:00.1

These are the contents of it

#!/bin/bash

modprobe vfio-pci

for dev in "$@"; do
        vendor=$(cat /sys/bus/pci/devices/$dev/vendor)
        device=$(cat /sys/bus/pci/devices/$dev/device)
        if [ -e /sys/bus/pci/devices/$dev/driver ]; then
                echo $dev > /sys/bus/pci/devices/$dev/driver/unbind
        fi
        echo $vendor $device > /sys/bus/pci/drivers/vfio-pci/new_id
done

I run qemu with this script

qemu-system-x86_64 -enable-kvm -m 1024 -cpu host,kvm=off \
-smp 4,sockets=1,cores=4,threads=1 \
-device vfio-pci,host=01:00.0,x-vga=on -device vfio-pci,host=01:00.1 \
-vga none -device virtio-scsi-pci,id=scsi -drive file=/home/anthony/windows.img,id=disk,format=raw,if=none \
-device scsi-hd,drive=disk -drive file=/home/anthony/win8.iso,id=isocd,if=none -device scsi-cd,drive=isocd \
-drive file=/home/anthony/virtio.iso,id=virtiocd,if=none -device ide-cd,bus=ide.1,drive=virtiocd

I don't get any visible errors. Just a blank, usable qemu console and no output at all on the display hooked up to the guest GPU.

Am I missing something?

Duelist

Member

Registered: 2014-09-22

Posts: 358

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Hello, I'm on ubuntu 14.10 x64 and I'm trying to get this to work. So far I run the qemu script and I get a small console where I can input commands to qemu, but no output on my display hooked up to my guest GPU, nor any errors.

My specs:

Gigabyte GA-Z97-HD3 (VT-d enabled)
Intel Core i7-4790K (intel ark says VT-d compatible)
Host GPU: Intel HD Graphics 4600
Guest GPU: MSI TwinFrozr GTX 770

output of "lspci -nn | grep NVIDIA"

01:00.0 VGA compatible controller [0300]: NVIDIA Corporation GK104 [GeForce GTX 770] [10de:1184] (rev a1)
01:00.1 Audio device [0403]: NVIDIA Corporation GK104 HDMI Audio Controller [10de:0e0a] (rev a1)

This is my grub command line options

GRUB_CMDLINE_LINUX_DEFAULT="quiet splash pcie_acs_override=downstream intel_iommu=on,igfx_on i915.enable_hd_vgaarb=1"

..snip..
I run qemu with this script

qemu-system-x86_64 -enable-kvm -m 1024 -cpu host,kvm=off \
-smp 4,sockets=1,cores=4,threads=1 \
-device vfio-pci,host=01:00.0,x-vga=on -device vfio-pci,host=01:00.1 \
-vga none -device virtio-scsi-pci,id=scsi -drive file=/home/anthony/windows.img,id=disk,format=raw,if=none \
-device scsi-hd,drive=disk -drive file=/home/anthony/win8.iso,id=isocd,if=none -device scsi-cd,drive=isocd \
-drive file=/home/anthony/virtio.iso,id=virtiocd,if=none -device ide-cd,bus=ide.1,drive=virtiocd

I don't get any visible errors. Just a blank, usable qemu console and no output at all on the display hooked up to the guest GPU.

Am I missing something?

Try checking dmesg. If there are any errors in KVM or VFIO work - it'll be there.

---

The forum rules prohibit requesting support for distributions other than arch.
I gave up. It was too late.
What I was trying to do.
The reference about VFIO and KVM VGA passthrough.

aw

Member

Registered: 2013-10-04

Posts: 921

Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Am I missing something?

Whenever it's not explicitly mentioned, my first suspicion is whether the i915 patch is actually applied to the kernel.

---

http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Denso

Member

Registered: 2014-08-30

Posts: 179

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

These keep piling in dmesg :

[Fri Apr 10 05:53:03 2015] pcieport 0000:00:03.0: AER: Multiple Corrected error received: id=0018
[Fri Apr 10 05:53:03 2015] pcieport 0000:00:03.0: PCIe Bus Error: severity=Corrected, type=Data Link Layer, id=0018(Receiver ID)
[Fri Apr 10 05:53:03 2015] pcieport 0000:00:03.0:   device [8086:2f08] error status/mask=00000040/00002000
[Fri Apr 10 05:53:03 2015] pcieport 0000:00:03.0:    [ 6] Bad TLP

Using 4.0-rc5 and also happened on previous kernels . When it starts appearing , any VM rebooting or even host rebooting causes the host to hard crash . This is on X99 platform , where 00:03.0 is a :

00:03.0 PCI bridge [0604]: Intel Corporation Xeon E5 v3/Core i7 PCI Express Root Port 3 [8086:2f08] (rev 02) (prog-if 00 [Normal decode])

aw

Member

Registered: 2013-10-04

Posts: 921

Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

These keep piling in dmesg :

[Fri Apr 10 05:53:03 2015] pcieport 0000:00:03.0: AER: Multiple Corrected error received: id=0018
[Fri Apr 10 05:53:03 2015] pcieport 0000:00:03.0: PCIe Bus Error: severity=Corrected, type=Data Link Layer, id=0018(Receiver ID)
[Fri Apr 10 05:53:03 2015] pcieport 0000:00:03.0:   device [8086:2f08] error status/mask=00000040/00002000
[Fri Apr 10 05:53:03 2015] pcieport 0000:00:03.0:    [ 6] Bad TLP

You're getting bad packets on the link, the root port is the receiver of the packet.  This could indicate some sort of hardware problem.  Are you overclocking?  If you move the device behind this root port to another slot, do the errors follow it?  It could be a bad endpoint device.

---

http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Denso

Member

Registered: 2014-08-30

Posts: 179

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

You're getting bad packets on the link, the root port is the receiver of the packet.  This could indicate some sort of hardware problem.  Are you overclocking?  If you move the device behind this root port to another slot, do the errors follow it?  It could be a bad endpoint device.

So it is the GPU and not the port itself that causes the problem .

Let me understand this correctly . I have these root ports :

00:01.0 PCI bridge: Intel Corporation Xeon E5 v3/Core i7 PCI Express Root Port 1 (rev 02)
00:01.1 PCI bridge: Intel Corporation Xeon E5 v3/Core i7 PCI Express Root Port 1 (rev 02)
00:02.0 PCI bridge: Intel Corporation Xeon E5 v3/Core i7 PCI Express Root Port 2 (rev 02)
00:03.0 PCI bridge: Intel Corporation Xeon E5 v3/Core i7 PCI Express Root Port 3 (rev 02)

Are they numbered in a relative way to their physical position on the motherboard ? If that's the case , then 00:03.0 is the fourth slot from the top which is occupied by my GT610 (the card of legendary headaches thus far) .

aw

Member

Registered: 2013-10-04

Posts: 921

Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

You're getting bad packets on the link, the root port is the receiver of the packet.  This could indicate some sort of hardware problem.  Are you overclocking?  If you move the device behind this root port to another slot, do the errors follow it?  It could be a bad endpoint device.

So it is the GPU and not the port itself that causes the problem .

Let me understand this correctly . I have these root ports :

00:01.0 PCI bridge: Intel Corporation Xeon E5 v3/Core i7 PCI Express Root Port 1 (rev 02)
00:01.1 PCI bridge: Intel Corporation Xeon E5 v3/Core i7 PCI Express Root Port 1 (rev 02)
00:02.0 PCI bridge: Intel Corporation Xeon E5 v3/Core i7 PCI Express Root Port 2 (rev 02)
00:03.0 PCI bridge: Intel Corporation Xeon E5 v3/Core i7 PCI Express Root Port 3 (rev 02)

Are they numbered in a relative way to their physical position on the motherboard ? If that's the case , then 00:03.0 is the fourth slot from the top which is occupied by my GT610 (the card of legendary headaches thus far) .

The slot to root port mapping depends on how your motherboard is wired.  There are ways that the firmware can provide more human readable slot names via ACPI, but they're often not used on desktop boards.  lspci -t is useful for seeing a tree view of the hardware for more easily mapping which endpoint address is behind which root port.

---

http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Denso

Member

Registered: 2014-08-30

Posts: 179

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

The slot to root port mapping depends on how your motherboard is wired.  There are ways that the firmware can provide more human readable slot names via ACPI, but they're often not used on desktop boards.  lspci -t is useful for seeing a tree view of the hardware for more easily mapping which endpoint address is behind which root port.

Forgive me , but I couldn't make sense of this output :

-+-[0000:ff]-+-0b.0
 |           +-0b.1
 |           +-0b.2
 |           +-0c.0
 |           +-0c.1
 |           +-0c.2
 |           +-0c.3
 |           +-0c.4
 |           +-0c.5
 |           +-0f.0
 |           +-0f.1
 |           +-0f.4
 |           +-0f.5
 |           +-0f.6
 |           +-10.0
 |           +-10.1
 |           +-10.5
 |           +-10.6
 |           +-10.7
 |           +-12.0
 |           +-12.1
 |           +-13.0
 |           +-13.1
 |           +-13.2
 |           +-13.3
 |           +-13.4
 |           +-13.5
 |           +-13.6
 |           +-13.7
 |           +-14.0
 |           +-14.1
 |           +-14.2
 |           +-14.3
 |           +-14.6
 |           +-14.7
 |           +-15.0
 |           +-15.1
 |           +-15.2
 |           +-15.3
 |           +-16.0
 |           +-16.6
 |           +-16.7
 |           +-17.0
 |           +-17.4
 |           +-17.5
 |           +-17.6
 |           +-17.7
 |           +-1e.0
 |           +-1e.1
 |           +-1e.2
 |           +-1e.3
 |           +-1e.4
 |           +-1f.0
 |           \-1f.2
 \-[0000:00]-+-00.0
             +-01.0-[03-08]----00.0-[04-08]--+-01.0-[05]----00.0
             |                               +-05.0-[06]----00.0
             |                               +-07.0-[07]----00.0
             |                               \-09.0-[08]----00.0
             +-01.1-[09]--
             +-02.0-[02]--+-00.0
             |            \-00.1
             +-03.0-[01]--+-00.0
             |            \-00.1
             +-05.0
             +-05.1
             +-05.2
             +-05.4
             +-11.0
             +-11.4
             +-14.0
             +-16.0
             +-19.0
             +-1a.0
             +-1b.0
             +-1c.0-[0a]--
             +-1c.3-[0b-10]----00.0-[0c-10]--+-01.0-[0d]----00.0
             |                               +-02.0-[0e]----00.0
             |                               +-03.0-[0f]--
             |                               \-04.0-[10]--
             +-1c.4-[11]----00.0
             +-1c.6-[12]----00.0
             +-1d.0
             +-1f.0
             +-1f.2
             \-1f.3

aw

Member

Registered: 2013-10-04

Posts: 921

Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

 \-[0000:00]-+-00.0
             +-01.0-[03-08]----00.0-[04-08]--+-01.0-[05]----00.0
             |                               +-05.0-[06]----00.0
             |                               +-07.0-[07]----00.0
             |                               \-09.0-[08]----00.0
             +-01.1-[09]--
             +-02.0-[02]--+-00.0
             |            \-00.1
             +-03.0-[01]--+-00.0
             |            \-00.1
             ...         
             +-1c.0-[0a]--
             +-1c.3-[0b-10]----00.0-[0c-10]--+-01.0-[0d]----00.0
             |                               +-02.0-[0e]----00.0
             |                               +-03.0-[0f]--
             |                               \-04.0-[10]--
             +-1c.4-[11]----00.0
             +-1c.6-[12]----00.0

Devices 01:00.0 and 01:00.1 are behind root port 00:03.0

---

http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Denso

Member

Registered: 2014-08-30

Posts: 179

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

@aw :

Ohhh I get it now ! The first number is the Port number , the numbers after it are the devices numbers . So devices 02:00.0 , 02:00.1 are behind Port 00:02.0 . Device 11:00.0 is behind 00:1c.4 , and so on .

Thanks Alex ! It is GTX770 then , which is the host's primary GPU . I'll try to troubleshoot it and post my findings if I ever got it fixed hopefully .

EDIT :

Rebooted the host , entered BIOS and loaded optimized defaults , also I added "nomodeset" &  "nofb" to the boot parameters . Uptil now , no errors in dmesg . Will see if it can survive overnight .

Last edited by Denso (2015-04-11 16:13:30)