Arch Linux

mauorrizze

Member

Registered: 2014-08-22

Posts: 18

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Unfortunately the audio there stutters while playing sound.

In my case all sound problems were latency related. If the problem gets worse when you pass more CPU cores and better when you pass less, then it's propably the same. Other phenomenons: tools like LatencyMon let the windows guest freeze really soon.
What helps: cpu pinning (for each guest and the host!), in case of HDMI: enable MSI in registry
Unconfirmed: hugepages

thearcherblog

Member

Registered: 2014-10-30

Posts: 165

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Hello

I've never tried the CPU pinning so I will make a newbie question... is something that I can only make with libvirt? My windows 8.1 guest is working perfectly and I'm running it from the command line directly... is therre any way to improve (even more) the performance? Or I need libvirt and be able to convert form my script to the XML format?

Thanks a in advance

aw

Member

Registered: 2013-10-04

Posts: 921

Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

thanks a lot.

when I configure the device as secondary graphics for the VM and do not use the x-vga option, the driver report code 12.

Use 440fx, not q35

EDIT: You might just be able to move the GPU to the root complex, but behind a root port is known to cause the problem right now.

when I changed to i440fx, the driver report code 43.

my qemu commandline:

qemu-system-x86_64 -cpu SandyBridge,kvm=off -smp 1,sockets=1,cores=1,threads=1 \
-m 8192 -enable-kvm \
-rtc base=localtime,clock=host \
-acpitable file=/var/lib/libvirt/images/LENOVO-TC-90-MSFT-2.1.BIN \
-device virtio-scsi-pci,id=scsi \
-drive file=/var/lib/libvirt/images/win7_nvidia_k2_clean.img,cache=writeback,\
if=none,format=qcow2,aio=native,id=virtio-scsi-disk0 \
-device scsi-hd,drive=virtio-scsi-disk0 \
-net nic,model=virtio,macaddr=52:54:00:1a:2b:3c \
-net tap,ifname=tap0,script=/etc/qemu-ifup,downscript=/etc/qemu-ifdown \
-vga std -vnc :3 \
-device vfio-pci,host=04:00.0,addr=05.0,multifunction=on \
-usb -device usb-tablet \
-monitor stdio

kvm=off is also not necessary with supported nvidia cards, but I don't see anything else off.  What kernel/qemu are you using?

---

http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

gnoma_86

Member

Registered: 2015-02-10

Posts: 1

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Hello,

I'm new in KVM and I'm trying to do a VGA pass through.

I've done the vfio grouping and I attached my HD5770 to the virtual machine. Installation goes well. After install I see my video card in the device manager and when I install the driver they are both - VGA and HDMI visible with exact model. However it says that I have to reboot in order to get the device online. Tested and no picture coming from the physical VGA before reboot. After I install the driver and reboot I got BSOD on the spice during boot. I can hear during boot that the FAN of the HD5770 rotates faster, but when I connect a monitor it instantly goes in power saving. Tried with AMD driver, tried also with the driver from ASUS, I got the same result.

I removed the virtual video card and spice. What I noticed is that even before OS startup I cannot see the boot at all, the display goes to power saving instantly. During power on the VM it doesn't even blink or go online for a second, it stays constantly off.

From what I saw in youtube, I should be able to see my VM booting on the VGA passed through to the VM.

Did anybody face the same issue?
Please help.

Thank you.

valnare

Member

Registered: 2015-02-09

Posts: 3

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

In my case all sound problems were latency related. If the problem gets worse when you pass more CPU cores and better when you pass less, then it's propably the same. Other phenomenons: tools like LatencyMon let the windows guest freeze really soon.
What helps: cpu pinning (for each guest and the host!), in case of HDMI: enable MSI in registry
Unconfirmed: hugepages

I tried playing audio with only a single core pinned on the host and the guest. Still the same result. The stuttering also does not change with more load on the CPU.
Latencymon worked while only using one cpu core, when I tried running it with more than one it frooze after 2 seconds. MSI was enabled, didn't change anything for the stuttering. I'll experiment a bit more with cpu pinning though.
Enabling or disabling hugepages didn't do anything for me.

Thank you for your suggestions

gessert

Member

Registered: 2015-02-09

Posts: 5

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Goog evening,

I've been getting strange results and need guidance.

Graphicscards:
AMD Radeon r5 230
GeForce GTX 970
GeForce GTX 970

I'm trying to get two different configurations to work:
Host: AMD Radeon r5 230
Guest1: Nvidia GeForce GTX 970
Guest2: Nvidia GeForce GTX 970

Host: AMD Radeon r5 230
Guest1: Nvidia GeForce GTX 970
Guest1: Nvidia GeForce GTX 970

Basically I want to use the Nvidia cards either for two different guests or as an SLI configuration in a single guest.

I have successfully passed the second geforce card to a guest, but the first one will not work. The first card happens to be the card used by the host's uefi, grub and terminal booting messages. I assume the problem stems from here. I can not change the cards' order, nor set a primary grapicsadapter in the host's uefi. The first geforce card's monitor stays active after host boot and shows a single underscore. When I run the guest with the first card configured, the monitor goes blank and I get a warning in the qemu-window:

-device vfio-pci,host=04:00.0,multifunction=on,x-vga=on: VFIO 0000:04:00.0 BAR 3 mmap unsupported. Performance may be slow

Info:
Distribution: Debian Wheezy (running on wheezy-backports)
qemu/kvm 2.1
Seabios
3.16.0-0.bpo.4-amd64

lspci | grep NVIDIA

03:00.0 VGA compatible controller [0300]: NVIDIA Corporation Device [10de:13c2] (rev a1)
03:00.1 Audio device [0403]: NVIDIA Corporation Device [10de:0fbb] (rev a1)
04:00.0 VGA compatible controller [0300]: NVIDIA Corporation Device [10de:13c2] (rev a1)
04:00.1 Audio device [0403]: NVIDIA Corporation Device [10de:0fbb] (rev a1)

GRUB_CMDLINE_LINUX_DEFAULT
in /etc/default/grub.cfg

GRUB_CMDLINE_LINUX_DEFAULT="quiet intel_iommu=on vfio_iommu_type1.allow_unsafe_interrupts=1"

run via "update-grub"

/etc/modules

pci_stub
vfio
vfio_iommu_type1
vfio_pci
kvm
kvm_intel

/etc/initramfs-tools/modules

pci_stub ids=10de:13c2,10de:0fbb

run via "update-initramfs -u"

dmesg | pci-stub

[    1.449636] pci-stub: add 10DE:13C2 sub=FFFFFFFF:FFFFFFFF cls=00000000/00000000
[    1.449654] pci-stub 0000:03:00.0: claimed by stub
[    1.449662] pci-stub 0000:04:00.0: claimed by stub
[    1.449664] pci-stub: add 10DE:0FBB sub=FFFFFFFF:FFFFFFFF cls=00000000/00000000
[    1.449675] pci-stub 0000:03:00.1: claimed by stub
[    1.449682] pci-stub 0000:04:00.1: claimed by stub

Working:

/etc/vfio-pci1.cfg

0000:03:00.0
0000:03:00.1

/home/me/vm1

#!/bin/bash

configfile=/etc/vfio-pci1.cfg

vfiobind() {
    dev="$1"
        vendor=$(cat /sys/bus/pci/devices/$dev/vendor)
        device=$(cat /sys/bus/pci/devices/$dev/device)
        if [ -e /sys/bus/pci/devices/$dev/driver ]; then
                echo $dev > /sys/bus/pci/devices/$dev/driver/unbind
        fi
        echo $vendor $device > /sys/bus/pci/drivers/vfio-pci/new_id
   
}

modprobe vfio-pci

cat $configfile | while read line;do
    echo $line | grep ^# >/dev/null 2>&1 && continue
        vfiobind $line
done

export QEMU_AUDIO_DRV=alsa QEMU_AUDIO_TIMER_PERIOD=0
sudo qemu-system-x86_64 -enable-kvm -M pc -m 8192 -cpu host,kvm=off \
-smp 4,sockets=1,cores=4,threads=1 \
-bios /usr/share/seabios/bios.bin -vga none \
-device vfio-pci,host=03:00.0,multifunction=on,x-vga=on \
-device vfio-pci,host=03:00.1 \
-usb -usbdevice host:04d9:a086 -usbdevice host:046a:0023 \
-drive file=/home/me/windows10.img,format=raw,media=disk \
-drive file=/home/me/win10.iso,media=cdrom \
-soundhw ac97 \
-boot menu=on

exit 0

not working:

/etc/vfio-pci2.cfg

0000:04:00.0
0000:04:00.1

/home/me/vm2

#!/bin/bash

configfile=/etc/vfio-pci2.cfg

vfiobind() {
    dev="$1"
        vendor=$(cat /sys/bus/pci/devices/$dev/vendor)
        device=$(cat /sys/bus/pci/devices/$dev/device)
        if [ -e /sys/bus/pci/devices/$dev/driver ]; then
                echo $dev > /sys/bus/pci/devices/$dev/driver/unbind
        fi
        echo $vendor $device > /sys/bus/pci/drivers/vfio-pci/new_id
   
}

modprobe vfio-pci

cat $configfile | while read line;do
    echo $line | grep ^# >/dev/null 2>&1 && continue
        vfiobind $line
done

export QEMU_AUDIO_DRV=alsa QEMU_AUDIO_TIMER_PERIOD=0
sudo qemu-system-x86_64 -enable-kvm -M pc -m 8192 -cpu host,kvm=off \
-smp 4,sockets=1,cores=4,threads=1 \
-bios /usr/share/seabios/bios.bin -vga none \
-device vfio-pci,host=04:00.0,multifunction=on,x-vga=on \
-device vfio-pci,host=04:00.1 \
-usb -usbdevice host:04d9:a086 -usbdevice host:046a:0023 \
-drive file=/home/me/windows10.img,format=raw,media=disk \
-drive file=/home/me/win10.iso,media=cdrom \
-soundhw ac97 \
-boot menu=on

exit 0

The "broken" guest's boot attempt creates additional messages:

dmesg | grep 0000:04

vfio-pci 0000:04:00.0: Invalid ROM contents

I dumped the graphicscard's rom and added a romfile parameter to the config, but I guess it's just a follow up problem.

-device vfio-pci,host=04:00.0,multifunction=on,x-vga=on,romfile=/home/me/vbios.rom \

This does get rid of the Invalid ROM contents error, but still no output on the screen.

I have already dumped around 70 hours into this project and I'm afraid my linux-knowledge is rather limited. I'm part of the raspberry-pi generation of linux users. So help is vastly appreciated.

I would offer beer, if you were in throwing range.

aw

Member

Registered: 2013-10-04

Posts: 921

Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

-device vfio-pci,host=04:00.0,multifunction=on,x-vga=on: VFIO 0000:04:00.0 BAR 3 mmap unsupported. Performance may be slow

As Duelist figured out, this means a host driver is attached to the device.  A common problem when using the boot graphics device.  Look in /proc/iomem, figure out what it is, and disable it.

---

http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

erganzi

Member

Registered: 2014-07-09

Posts: 19

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Use 440fx, not q35

EDIT: You might just be able to move the GPU to the root complex, but behind a root port is known to cause the problem right now.

when I changed to i440fx, the driver report code 43.

my qemu commandline:

qemu-system-x86_64 -cpu SandyBridge,kvm=off -smp 1,sockets=1,cores=1,threads=1 \
-m 8192 -enable-kvm \
-rtc base=localtime,clock=host \
-acpitable file=/var/lib/libvirt/images/LENOVO-TC-90-MSFT-2.1.BIN \
-device virtio-scsi-pci,id=scsi \
-drive file=/var/lib/libvirt/images/win7_nvidia_k2_clean.img,cache=writeback,\
if=none,format=qcow2,aio=native,id=virtio-scsi-disk0 \
-device scsi-hd,drive=virtio-scsi-disk0 \
-net nic,model=virtio,macaddr=52:54:00:1a:2b:3c \
-net tap,ifname=tap0,script=/etc/qemu-ifup,downscript=/etc/qemu-ifdown \
-vga std -vnc :3 \
-device vfio-pci,host=04:00.0,addr=05.0,multifunction=on \
-usb -device usb-tablet \
-monitor stdio

kvm=off is also not necessary with supported nvidia cards, but I don't see anything else off.  What kernel/qemu are you using?

Hi, aw:

my kernel/qemu version info as following:

linux-3.12.5 from kernel.org with following patchs.
pci/iommu: Quirk non-compliant PCIe-to-PCI bridges
i915_fixes[1]
i915_fixes[2]
Enable overrides for missing ACS capabilities

qemu-2.0.0 with patch
vfio-pci: Disallow device from using NoSnoop transactions

aw

Member

Registered: 2013-10-04

Posts: 921

Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Hi, aw:

my kernel/qemu version info as following:

linux-3.12.5 from kernel.org with following patchs.
pci/iommu: Quirk non-compliant PCIe-to-PCI bridges
i915_fixes[1]
i915_fixes[2]
Enable overrides for missing ACS capabilities

qemu-2.0.0 with patch
vfio-pci: Disallow device from using NoSnoop transactions

Oh jeez, ping me again when you're using something reasonable.  I'd suggest at least 3.16 and QEMU 2.2.  You don't need any patches unless you have ACS issues with IOMMU groups.

---

http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

erganzi

Member

Registered: 2014-07-09

Posts: 19

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Hi, aw:

my kernel/qemu version info as following:

linux-3.12.5 from kernel.org with following patchs.
pci/iommu: Quirk non-compliant PCIe-to-PCI bridges
i915_fixes[1]
i915_fixes[2]
Enable overrides for missing ACS capabilities

qemu-2.0.0 with patch
vfio-pci: Disallow device from using NoSnoop transactions

Oh jeez, ping me again when you're using something reasonable.  I'd suggest at least 3.16 and QEMU 2.2.  You don't need any patches unless you have ACS issues with IOMMU groups.

OK, I will try.
but yesterday I tried RHEL 7.0, it worked well with GRID K2. and the kernel version is 3.10.0-121.el7.x86_64 , qemu version is 1.5.3-60.el7.
had they work with the patched ?

aw

Member

Registered: 2013-10-04

Posts: 921

Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Hi, aw:

my kernel/qemu version info as following:

linux-3.12.5 from kernel.org with following patchs.
pci/iommu: Quirk non-compliant PCIe-to-PCI bridges
i915_fixes[1]
i915_fixes[2]
Enable overrides for missing ACS capabilities

qemu-2.0.0 with patch
vfio-pci: Disallow device from using NoSnoop transactions

Oh jeez, ping me again when you're using something reasonable.  I'd suggest at least 3.16 and QEMU 2.2.  You don't need any patches unless you have ACS issues with IOMMU groups.

OK, I will try.
but yesterday I tried RHEL 7.0, it worked well with GRID K2. and the kernel version is 3.10.0-121.el7.x86_64 , qemu version is 1.5.3-60.el7.
had they work with the patched ?

If only you knew how much effort we put in to backporting fixes and features while maintaining ABI compatibility for those older base versions...

---

http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

erganzi

Member

Registered: 2014-07-09

Posts: 19

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Oh jeez, ping me again when you're using something reasonable.  I'd suggest at least 3.16 and QEMU 2.2.  You don't need any patches unless you have ACS issues with IOMMU groups.

OK, I will try.
but yesterday I tried RHEL 7.0, it worked well with GRID K2. and the kernel version is 3.10.0-121.el7.x86_64 , qemu version is 1.5.3-60.el7.
had they work with the patched ?

If only you knew how much effort we put in to backporting fixes and features while maintaining ABI compatibility for those older base versions...

Thanks for those work.  I will try what you suggest and give a feedback.

hotfunction

Member

Registered: 2015-02-05

Posts: 10

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Hi..

That black window is the sdl graphics head for the VM which appears because you haven't used the -nographic option.  The GPU output should be on a monitor connected to the assigned GPU.  I'd also recommend, like I almost always do, using 440fx rather than q35 for the VM.  It looks like you're using a Linux guest, but afaik, the nvidia driver does not have an issue with PCIe topology in Linux (until you get to lots of GPUs and that may already be fixed too).

I added -nographics option and -vga none, mixed together with the script changed into 440fx, but none came up out of the window, the session is there and running, but the qemu operation is not cancelable with ctrl + c and i dont see any window output. i went into another session and checked it with "top" command, and the qemu is still there...

also, i tried to remove my nographics and vga none, and just normally run that VM without those options, but all i get is just a black monitor of qemu, not even booting into seabios and it stucks there forever...

here is my script for 440fx:

#!/bin/bash

configfile=/etc/vfio-pci1.cfg

vfiobind() {
    dev="$1"
        vendor=$(cat /sys/bus/pci/devices/$dev/vendor)
        device=$(cat /sys/bus/pci/devices/$dev/device)
        if [ -e /sys/bus/pci/devices/$dev/driver ]; then
                echo $dev > /sys/bus/pci/devices/$dev/driver/unbind
        fi
        echo $vendor $device > /sys/bus/pci/drivers/vfio-pci/new_id

}

modprobe vfio-pci

cat $configfile | while read line;do
    echo $line | grep ^# >/dev/null 2>&1 && continue
        vfiobind $line
done

sudo qemu-system-x86_64 -enable-kvm -M pc -m 2048 -cpu host \
-smp 2,sockets=1,cores=2,threads=1 \
-bios /usr/share/seabios/bios.bin \
-device vfio-pci,host=05:00.0,multifunction=on,x-vga=on,romfile=/home/sslab719/MSI.GTX560Ti.1024.110825.rom \
-device vfio-pci,host=05:00.1 \
-drive file=/home/sslab719/VMimages/VM.img,format=qcow2,media=disk \
#-drive file=/home/me/win10.iso,media=cdrom \
-boot menu=on

exit 0

any ideas..?

Last edited by hotfunction (2015-02-11 03:45:21)

aw

Member

Registered: 2013-10-04

Posts: 921

Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

any ideas..?

I already suggested that the ROM might be the problem and you don't seem to have much confidence that it's the correct one.  Why haven't you tried anything further on that path?

---

http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

erganzi

Member

Registered: 2014-07-09

Posts: 19

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

OK, I will try.
but yesterday I tried RHEL 7.0, it worked well with GRID K2. and the kernel version is 3.10.0-121.el7.x86_64 , qemu version is 1.5.3-60.el7.
had they work with the patched ?

If only you knew how much effort we put in to backporting fixes and features while maintaining ABI compatibility for those older base versions...

Thanks for those work.  I will try what you suggest and give a feedback.

Hi, aw:

I just change the kernel version to 3.18.6 from kernel.org. It now work for me.  qemu version is still 2.0.

I compare the result of the command "lspci -vvv -k -s 04:00.0" between linux-3.12.5 and linux-3.18.6, there are just a little different.
If it can give you some idea.

kernel: 3.12.5

04:00.0 VGA compatible controller: nVidia Corporation Device 11bf (rev a1) (prog-if 00 [VGA controller])
    Subsystem: nVidia Corporation Device 100d
    Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
    Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
    Interrupt: pin A routed to IRQ 11
    Region 0: Memory at c9000000 (32-bit, non-prefetchable) [disabled] [size=16M]
    Region 1: Memory at b8000000 (64-bit, prefetchable) [disabled] [size=128M]
    Region 3: Memory at c0000000 (64-bit, prefetchable) [disabled] [size=32M]
    Region 5: I/O ports at 9000 [disabled] [size=128]
    Expansion ROM at ca000000 [disabled] [size=512K]
    Capabilities: [60] Power Management version 3
        Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0-,D1-,D2-,D3hot-,D3cold-)
        Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=0 PME-
    Capabilities: [68] MSI: Enable- Count=1/1 Maskable- 64bit+
        Address: 0000000000000000  Data: 0000
    Capabilities: [78] Express (v2) Endpoint, MSI 00
        DevCap:    MaxPayload 256 bytes, PhantFunc 0, Latency L0s unlimited, L1 <64us
            ExtTag+ AttnBtn- AttnInd- PwrInd- RBE+ FLReset-
        DevCtl:    Report errors: Correctable- Non-Fatal- Fatal- Unsupported-
            RlxdOrd- ExtTag- PhantFunc- AuxPwr- NoSnoop+
            MaxPayload 256 bytes, MaxReadReq 512 bytes
        DevSta:    CorrErr- UncorrErr- FatalErr- UnsuppReq- AuxPwr- TransPend-
        LnkCap:    Port #8, Speed unknown, Width x16, ASPM L0s L1, Latency L0 <512ns, L1 <4us
            ClockPM+ Surprise- LLActRep- BwNot-
        LnkCtl:    ASPM Disabled; RCB 64 bytes Disabled- Retrain- CommClk-
            ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
        LnkSta:    Speed 2.5GT/s, Width x16, TrErr- Train- SlotClk+ DLActive- BWMgmt- ABWMgmt-
        DevCap2: Completion Timeout: Range AB, TimeoutDis+
        DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis-
        LnkCtl2: Target Link Speed: Unknown, EnterCompliance- SpeedDis-, Selectable De-emphasis: -6dB
             Transmit Margin: Normal Operating Range, EnterModifiedCompliance- ComplianceSOS-
             Compliance De-emphasis: -6dB
        LnkSta2: Current De-emphasis Level: -6dB
    Capabilities: [b4] Vendor Specific Information <?>
    Capabilities: [100] Virtual Channel <?>
    Capabilities: [128] Power Budgeting <?>
    Capabilities: [420] Advanced Error Reporting
        UESta:    DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
        UEMsk:    DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq+ ACSViol-
        UESvrt:    DLP+ SDES+ TLP- FCP+ CmpltTO- CmpltAbrt- UnxCmplt- RxOF+ MalfTLP+ ECRC- UnsupReq- ACSViol-
        CESta:    RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr-
        CEMsk:    RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr+
        AERCap:    First Error Pointer: 00, GenCap- CGenEn- ChkCap- ChkEn-
    Capabilities: [600] Vendor Specific Information <?>
    Capabilities: [900] #19
    Kernel driver in use: vfio-pci
    Kernel modules: nouveau, nvidiafb

kernel: 3.18.6

04:00.0 VGA compatible controller: nVidia Corporation Device 11bf (rev a1) (prog-if 00 [VGA controller])
    Subsystem: nVidia Corporation Device 100d
    Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx+
    Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
    Interrupt: pin A routed to IRQ 27
    Region 0: Memory at c9000000 (32-bit, non-prefetchable) [disabled] [size=16M]
    Region 1: Memory at b8000000 (64-bit, prefetchable) [disabled] [size=128M]
    Region 3: Memory at c0000000 (64-bit, prefetchable) [disabled] [size=32M]
    Region 5: I/O ports at 9000 [disabled] [size=128]
    Expansion ROM at ca000000 [disabled] [size=512K]
    Capabilities: [60] Power Management version 3
        Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0-,D1-,D2-,D3hot-,D3cold-)
        Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=0 PME-
    Capabilities: [68] MSI: Enable- Count=1/1 Maskable- 64bit+
        Address: 0000000000000000  Data: 0000
    Capabilities: [78] Express (v2) Endpoint, MSI 00
        DevCap:    MaxPayload 256 bytes, PhantFunc 0, Latency L0s unlimited, L1 <64us
            ExtTag+ AttnBtn- AttnInd- PwrInd- RBE+ FLReset-
        DevCtl:    Report errors: Correctable- Non-Fatal- Fatal- Unsupported-
            RlxdOrd+ ExtTag+ PhantFunc- AuxPwr- NoSnoop-
            MaxPayload 256 bytes, MaxReadReq 512 bytes
        DevSta:    CorrErr- UncorrErr- FatalErr- UnsuppReq- AuxPwr- TransPend-
        LnkCap:    Port #8, Speed unknown, Width x16, ASPM L0s L1, Latency L0 <512ns, L1 <4us
            ClockPM+ Surprise- LLActRep- BwNot-
        LnkCtl:    ASPM Disabled; RCB 64 bytes Disabled- Retrain- CommClk-
            ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
        LnkSta:    Speed 2.5GT/s, Width x16, TrErr- Train- SlotClk+ DLActive- BWMgmt- ABWMgmt-
        DevCap2: Completion Timeout: Range AB, TimeoutDis+
        DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis-
        LnkCtl2: Target Link Speed: Unknown, EnterCompliance- SpeedDis-, Selectable De-emphasis: -6dB
             Transmit Margin: Normal Operating Range, EnterModifiedCompliance- ComplianceSOS-
             Compliance De-emphasis: -6dB
        LnkSta2: Current De-emphasis Level: -6dB
    Capabilities: [b4] Vendor Specific Information <?>
    Capabilities: [100] Virtual Channel <?>
    Capabilities: [128] Power Budgeting <?>
    Capabilities: [420] Advanced Error Reporting
        UESta:    DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
        UEMsk:    DLP- SDES- TLP- FCP- CmpltTO- CmpltAbrt- UnxCmplt- RxOF- MalfTLP- ECRC- UnsupReq- ACSViol-
        UESvrt:    DLP+ SDES+ TLP- FCP+ CmpltTO- CmpltAbrt- UnxCmplt- RxOF+ MalfTLP+ ECRC- UnsupReq- ACSViol-
        CESta:    RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr-
        CEMsk:    RxErr- BadTLP- BadDLLP- Rollover- Timeout- NonFatalErr+
        AERCap:    First Error Pointer: 00, GenCap- CGenEn- ChkCap- ChkEn-
    Capabilities: [600] Vendor Specific Information <?>
    Capabilities: [900] #19
    Kernel driver in use: vfio-pci

aw

Member

Registered: 2013-10-04

Posts: 921

Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

If only you knew how much effort we put in to backporting fixes and features while maintaining ABI compatibility for those older base versions...

Thanks for those work.  I will try what you suggest and give a feedback.

Hi, aw:

I just change the kernel version to 3.18.6 from kernel.org. It now work for me.  qemu version is still 2.0.

I compare the result of the command "lspci -vvv -k -s 04:00.0" between linux-3.12.5 and linux-3.18.6, there are just a little different.
If it can give you some idea.

There's really not much question about what the problem is/was, the kvm-vfio device didn't go in until 3.13.  That's the code that toggles whether kvm emulates or ignores operations that invalidate the cache when devices are allowed to perform operations that can bypass it, for example NoSnoop.  That was an early cause of Code 43 errors.  The NoSnoop patch you were using was an early, and fairly broken, attempt to fix that.  The new code is far more reliable, as you can see.

---

http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

erganzi

Member

Registered: 2014-07-09

Posts: 19

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Thanks for those work.  I will try what you suggest and give a feedback.

Hi, aw:

I just change the kernel version to 3.18.6 from kernel.org. It now work for me.  qemu version is still 2.0.

I compare the result of the command "lspci -vvv -k -s 04:00.0" between linux-3.12.5 and linux-3.18.6, there are just a little different.
If it can give you some idea.

There's really not much question about what the problem is/was, the kvm-vfio device didn't go in until 3.13.  That's the code that toggles whether kvm emulates or ignores operations that invalidate the cache when devices are allowed to perform operations that can bypass it, for example NoSnoop.  That was an early cause of Code 43 errors.  The NoSnoop patch you were using was an early, and fairly broken, attempt to fix that.  The new code is far more reliable, as you can see.

OK, thanks for reply. I will use the version 3.18.6.

kainet

Member

Registered: 2014-12-22

Posts: 9

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

In my case all sound problems were latency related. If the problem gets worse when you pass more CPU cores and better when you pass less, then it's propably the same. Other phenomenons: tools like LatencyMon let the windows guest freeze really soon.
What helps: cpu pinning (for each guest and the host!), in case of HDMI: enable MSI in registry
Unconfirmed: hugepages

I tried playing audio with only a single core pinned on the host and the guest. Still the same result. The stuttering also does not change with more load on the CPU.
Latencymon worked while only using one cpu core, when I tried running it with more than one it frooze after 2 seconds. MSI was enabled, didn't change anything for the stuttering. I'll experiment a bit more with cpu pinning though.
Enabling or disabling hugepages didn't do anything for me.

Thank you for your suggestions

Have same problems with ASUS GTX 670.  With GA GTX770 sound seems ok but that card spins FAN at maximum speed when VM was turned off, so she does not fit me. With 670 i turning off when turning on vm, till i not achieve stable sound, but as long as i do so it seems that getting stable sound requires even more reboots. I am very interested in a way you solve this problem.

mmm

Member

Registered: 2014-11-06

Posts: 4

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Hi,
I'm having trouble passing through the third and fourth graphics card to VMs while passing through the first and second one works without any issues.
The VMs just dont give a signal on the GPU's outputs and they dont seem to boot up since no network services are starting from inside the VM.
Also neither qemu itself nor dmesg seem to report any errors which leaves me wondering how I can even get information about what the problem is.
All cards are GTX 780Ti, the motherboard is a dual-socket board with both sockets equipped and all cards are apparently not in an iommu group with any other devices.

Does anyone have suggestions how to approach this?

gessert

Member

Registered: 2015-02-09

Posts: 5

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

-device vfio-pci,host=04:00.0,multifunction=on,x-vga=on: VFIO 0000:04:00.0 BAR 3 mmap unsupported. Performance may be slow

As Duelist figured out, this means a host driver is attached to the device.  A common problem when using the boot graphics device.  Look in /proc/iomem, figure out what it is, and disable it.

excerpt from cat /proc/iomem

90000000-fbffbfff : PCI Bus 0000:00
  a0000000-b1ffffff : PCI Bus 0000:04
    a0000000-afffffff : 0000:04:00.0
    b0000000-b1ffffff : 0000:04:00.0
      b1000000-b12fffff : BOOTFB

From what I could gather with a google search: BOOTFB is the framebuffer that sends terminal messages during boot. Could I just "disable" it? How do I proceed? Please advise.

complete cat /proc/iomem

00000000-00000fff : reserved
00001000-0009ffff : System RAM
000a0000-000bffff : PCI Bus 0000:00
000c0000-000cebff : Video ROM
000f0000-000fffff : System ROM
00100000-790c7fff : System RAM
  01000000-0154d5b2 : Kernel code
  0154d5b3-018ea13f : Kernel data
  01a1e000-01aeffff : Kernel bss
790c8000-79c36fff : reserved
79c37000-79ef3fff : System RAM
79ef4000-7aa7cfff : ACPI Non-volatile Storage
7aa7d000-7afbefff : reserved
7afbf000-7afe5fff : System RAM
7afe6000-7afe6fff : reserved
7afe7000-7afe9fff : System RAM
7afea000-7afeafff : reserved
7afeb000-7afebfff : System RAM
7afec000-7b071fff : reserved
7b072000-7b289fff : System RAM
7b28a000-7bff8fff : reserved
7bff9000-7bffffff : System RAM
80000000-8fffffff : PCI MMCONFIG 0000 [bus 00-ff]
  80000000-8fffffff : reserved
90000000-fbffbfff : PCI Bus 0000:00
  a0000000-b1ffffff : PCI Bus 0000:04
    a0000000-afffffff : 0000:04:00.0
    b0000000-b1ffffff : 0000:04:00.0
      b1000000-b12fffff : BOOTFB
  c0000000-d1ffffff : PCI Bus 0000:03
    c0000000-cfffffff : 0000:03:00.0
    d0000000-d1ffffff : 0000:03:00.0
  e0000000-efffffff : PCI Bus 0000:02
    e0000000-efffffff : 0000:02:00.0
  f8000000-f90fffff : PCI Bus 0000:04
    f8000000-f8ffffff : 0000:04:00.0
    f9000000-f907ffff : 0000:04:00.0
    f9080000-f9083fff : 0000:04:00.1
  f9100000-f911ffff : 0000:00:19.0
    f9100000-f911ffff : e1000e
  f9120000-f912ffff : 0000:00:14.0
    f9120000-f912ffff : xhci_hcd
  f9130000-f9133fff : 0000:00:1b.0
    f9130000-f9133fff : ICH HD audio
  f9135000-f91350ff : 0000:00:1f.3
  f9136000-f91367ff : 0000:00:1f.2
    f9136000-f91367ff : ahci
  f9137000-f91373ff : 0000:00:1d.0
    f9137000-f91373ff : ehci_hcd
  f9138000-f91383ff : 0000:00:1a.0
    f9138000-f91383ff : ehci_hcd
  f9139000-f9139fff : 0000:00:19.0
    f9139000-f9139fff : e1000e
  f913c000-f913c00f : 0000:00:16.0
    f913c000-f913c00f : mei_me
  f913d000-f913d7ff : 0000:00:11.4
    f913d000-f913d7ff : ahci
  f913e000-f913efff : 0000:00:05.4
  fa000000-fb0fffff : PCI Bus 0000:03
    fa000000-faffffff : 0000:03:00.0
    fb000000-fb07ffff : 0000:03:00.0
    fb080000-fb083fff : 0000:03:00.1
  fb200000-fb2fffff : PCI Bus 0000:02
    fb200000-fb21ffff : 0000:02:00.0
    fb220000-fb23ffff : 0000:02:00.0
    fb240000-fb243fff : 0000:02:00.1
      fb240000-fb243fff : ICH HD audio
fbffc000-fbffcfff : dmar1
fbffd000-fbffdfff : dmar0
fec00000-fecfffff : PNP0003:00
  fec00000-fec003ff : IOAPIC 0
  fec01000-fec013ff : IOAPIC 1
fed00000-fed003ff : HPET 0
  fed00000-fed003ff : PNP0103:00
fed12000-fed1200f : pnp 00:01
fed12010-fed1201f : pnp 00:01
fed1b000-fed1bfff : pnp 00:01
fed1c000-fed1ffff : reserved
  fed1f410-fed1f414 : iTCO_wdt
fed45000-fed8bfff : pnp 00:01
fee00000-feefffff : pnp 00:01
  fee00000-fee00fff : Local APIC
ff000000-ffffffff : reserved
  ff000000-ffffffff : pnp 00:01
100000000-47fffffff : System RAM

Last edited by gessert (2015-02-11 17:53:00)

evilsephiroth

Member

Registered: 2014-09-16

Posts: 17

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

According to this thread, no solution yet for drivers > 340.52. I'm stuck at that version too.(gtx 560Ti on Windows 7 machine)

What are you talking about?!  I'm running 347.25 here with GTX750.  KVM and Hyper-V extensions need to be hidden/disabled to avoid Code 43 on GeForce cards.

Exactly...
It s a steam in home streaming vm so performance would decrease also...

gessert

Member

Registered: 2015-02-09

Posts: 5

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

I have changed the grub command line and added:

GRUB_CMDLINE_LINUX_DEFAULT="quiet intel_iommu=on vfio_iommu_type1.allow_unsafe_interrupts=1 nofb"
GRUB_GFXPAYLOAD_LINUX=text

run via update-grub

cat /proc/iomem now looks clean.

During reboot I noticed the terminal booting messages will now stop earlier. While loading the ramdisk. It keeps displaying it's last screen.

The guest does not generate the "-device vfio-pci,host=04:00.0,multifunction=on,x-vga=on: VFIO 0000:04:00.0 BAR 3 mmap unsupported. Performance may be slow" error anymore.

"dmesg | grep 0000:04" still shows "vfio-pci 0000:04:00.0: Invalid ROM contents". So I added the romfile again. This gets rid of this problem.

When I run the vm now, the screen turns off. I still do not have graphics output. Please advise.

Len

Member

Registered: 2015-01-06

Posts: 23

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

When starting

qemu-system-x86_64 -enable-kvm -M q35 -m 2048 -cpu host \
-smp 4,sockets=1,cores=4,threads=1 \
-bios /usr/share/qemu/bios.bin -vga none \
-device ioh3420,bus=pcie.0,addr=1c.0,multifunction=on,port=1,chassis=1,id=root.1 \
-device vfio-pci,host=01:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on \
-device vfio-pci,host=01:00.1,bus=root.1,addr=00.1 \
-device ahci,id=ahci0 \
-drive file=/dev/sda,format=raw,if=none,id=sata0-0-0,media=disk \
-drive file=/home/len/win.iso,if=none,id=sata0-0-1,media=cdrom
-device ide-drive,bus=ahci0.0,unit=0,drive=drive-sata0-0-0,id=drive-sata0-0-0 \
-device ide-cd,bus=ahci0.1,unit=0,drive=drive-sata0-0-1,id=drive-sata0-0-1 \

getting this error

[   40.960654] vfio-pci 0000:01:00.0: enabling device (0000 -> 0003)
[   40.960757] vfio_ecap_init: 0000:01:00.0 hiding ecap 0x1e@0x258
[   40.960763] vfio_ecap_init: 0000:01:00.0 hiding ecap 0x19@0x900
[   43.955151] [drm:intel_uncore_check_errors] *ERROR* Unclaimed register before interrupt
[   43.955202] [drm:intel_uncore_check_errors] *ERROR* Unclaimed register before interrupt

I'm using kernel provided in this thread (uname checked),

[root@holo ~]# cat /etc/modules-load.d/kvm_virt.conf
pci_stub
vfio
vfio_iommu_type1
vfio_pci
kvm
kvm_intel

[root@holo ~]# l /sys/bus/pci/drivers/vfio-pci/
total 0
drwxr-xr-x  2 root root    0 Feb 12 03:47 .
drwxr-xr-x 21 root root    0 Feb 12 03:40 ..
lrwxrwxrwx  1 root root    0 Feb 12 03:46 0000:01:00.0 -> ../../../../devices/pci0000:00/0000:00:01.0/0000:01:00.0
lrwxrwxrwx  1 root root    0 Feb 12 03:46 0000:01:00.1 -> ../../../../devices/pci0000:00/0000:00:01.0/0000:01:00.1
--w-------  1 root root 4096 Feb 12 03:46 bind
lrwxrwxrwx  1 root root    0 Feb 12 03:46 module -> ../../../../module/vfio_pci
--w-------  1 root root 4096 Feb 12 03:40 new_id
--w-------  1 root root 4096 Feb 12 03:46 remove_id
--w-------  1 root root 4096 Feb 12 03:40 uevent
--w-------  1 root root 4096 Feb 12 03:46 unbind

GRUB_CMDLINE_LINUX_DEFAULT="quiet intel_iommu=on pci-stub.ids=10de:13c2,10de:0fbb i915.enable_hd_vgaarb=1"

Any clues guys?

Last edited by Len (2015-02-12 02:48:33)

hotfunction

Member

Registered: 2015-02-05

Posts: 10

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

If 440fx and -nographic don't help and you're looking at the right output from the GPU, I'd next be suspicious of the ROM file.  It looks like one from techpowerup.  Does it actually match your card?  Can you dump the ROM from your card instead?  Do you even need a dump of the ROM?

hi sir aw.. i tried downloading various ROMs from other sites too, but none of them seems to get me working, they still gave me black screen output on the sscreen... and it seemed to be hard to find nvidia's ROM corresponding on that... so is it okay if i change into another GPU card? will it help me to work?

That black window is the sdl graphics head for the VM which appears because you haven't used the -nographic option.  The GPU output should be on a monitor connected to the assigned GPU.  I'd also recommend, like I almost always do, using 440fx rather than q35 for the VM.  It looks like you're using a Linux guest, but afaik, the nvidia driver does not have an issue with PCIe topology in Linux (until you get to lots of GPUs and that may already be fixed too).

and also about this part... i actually still confused, after some reading and stuff, i redone the steps from nbhs's page one, after making the single script and others.. i still see the black qemu command line window when adding -vga none... and also without a rom setting, i also tried rombar=0 option and -nographic still didnt give me output.. can you give me further explanation on this.. ?? i am very confused about the black qemu command line, and how does people use -vga none without -nographics to go straight to the VMS without the black qemu command line?

Edit : i tried adding -curses instead of -nographics, and its still giving me blank output, which i can type "quit" and i can go back to the qemu, which is that i am guessing that is still the black qemu command line output again.. removing that -vga none boots -curses to the VM.. so is the black qemu command line from the -vga none output is showing because of my GPU rom incorrect? or am i missing something..?

Last edited by hotfunction (2015-02-12 05:33:08)

valnare

Member

Registered: 2015-02-09

Posts: 3

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Have same problems with ASUS GTX 670.  With GA GTX770 sound seems ok but that card spins FAN at maximum speed when VM was turned off, so she does not fit me. With 670 i turning off when turning on vm, till i not achieve stable sound, but as long as i do so it seems that getting stable sound requires even more reboots. I am very interested in a way you solve this problem.

I still had an USB sound card lying around that I passed through, where I do not get any suttering. This is only a workaround for me and it would be nice to use the HDMI ouput, but I have no idea how to solve the stuttering.