Arch Linux

aw

Member

Registered: 2013-10-04

Posts: 921

Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

@aw :

It seems that DMAR issues made it into 3.18.3 too .

3.18.3 contains 2 IOMMU commits :

iommu/vt-d: Fix dmar_domain leak in iommu_attach_device
iommu/vt-d: Fix an off-by-one bug in __domain_mapping()

Which one do you think is to blame ?

By that, do you mean that it is not present in 3.18.2?  I'm familiar with the first patch, I doubt it's to blame.  If you can revert one of those patches and toggle the problem let me know and I'll investigate further.  Easiest way to do a quick revert test is git show <commit ID> | patch -p1 -R  It's also possible that the problem is a side-effect of other code changes.

---

http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

mostlyharmless

Member

Registered: 2014-01-16

Posts: 72

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Hmm, here's an update, in case anyone is interested.  The cd line that allows host access is now

-drive file=/dev/sr0,if=none,id=drive-ide0-0-0,readonly=on,format=raw -device ide-cd,bus=ide.0,unit=0,drive=drive-ide0-0-0,id=ide0-0-0

Why qemu now requires that instead of the previous

 -drive file=/dev/sr0,id=cddisk -device ide-cd,bus=ide.1,drive=cddisk 

is beyond me. I presume that is a recent qemu development not an arch vs Slackware thing.

Speaking of which, of course the pulseaudio warning was eliminated by my installing pulseaudio.. heh heh

However, the bluescreen with error 0x0000007b continues.  Works without passthrough under arch... works under qemu 2.0.0 under Slackware, but not passthrough under arch.  Clearly the screen gets passed through, as I get to see all the futile Windows recovery options.  Perhaps if I uninstall the nvidia driver from Windows while not passing through, and then, assuming I can boot Windows passedthrough with the default Windows VGA driver I can reinstall the NVIDIA driver.  Supposedly 0x0000007b is a drive problem though.

Any other insight into the problem would be welcome.

Last edited by mostlyharmless (2015-01-28 19:59:50)

aw

Member

Registered: 2013-10-04

Posts: 921

Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Hmm, here's an update, in case anyone is interested.  The cd line that allows host access is now

-drive file=/dev/sr0,if=none,id=drive-ide0-0-0,readonly=on,format=raw -device ide-cd,bus=ide.0,unit=0,drive=drive-ide0-0-0,id=ide0-0-0

Why qemu now requires that instead of the previous

 -drive file=/dev/sr0,id=cddisk -device ide-cd,bus=ide.1,drive=cddisk 

is beyond me. I presume that is a recent qemu development not an arch vs Slackware thing.

The difference is if=none.  Without that QEMU assumes an interface for the device, then when you attempt to use the same interface later it's already in use.

Speaking of which, of course the pulseaudio warning was eliminated by my installing pulseaudio.. heh heh

However, the bluescreen with error 0x0000007b continues.  Works without passthrough under arch... works under qemu 2.0.0 under Slackware, but not passthrough under arch.  Clearly the screen gets passed through, as I get to see all the futile Windows recovery options.  Perhaps if I uninstall the nvidia driver from Windows while not passing through, and then, assuming I can boot Windows passedthrough with the default Windows VGA driver I can reinstall the NVIDIA driver.  Supposedly 0x0000007b is a drive problem though.

Any other insight into the problem would be welcome.

If you only get the problem when you attempt to pass through the host drive like this, then apparently it's related to the drive...

---

http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

dhiru1602

Member

Registered: 2015-01-28

Posts: 13

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Hi guys,

I am in need of desperate help. I have followed all of the posts on this topic and I am stuck with no solution to my problem. I was able to passthrough my GPU in Linux guest but on Windows guests I get Code 43.

Current Setup:
Core I7 920
Intel Desktop Board DX58SO (No Integrated Graphics)
Host GPU: Nvidia 9600 GT
Guest GPU: Nvidia 9600 GT

The Intel Processor page doesn't show support for VT-d, but instead mentions that the processor supports VT-x. I have passed "intel_iommu=on"  in the commandline to check if IOMMU worked and it did. This is when I have decided to give this method a try.

$ dmesg | grep "IOMMU"
[    0.000000] Intel-IOMMU: enabled
[    0.022507] dmar: IOMMU 0: reg_base_addr fe711000 ver 1:0 cap c9008010e60262 ecap f0207a
[    0.022512] dmar: IOMMU 1: reg_base_addr fe710000 ver 1:0 cap c90780106f0462 ecap f020fa
[    0.468207] IOMMU 0 0xfe711000: using Queued invalidation
[    0.468210] IOMMU 1 0xfe710000: using Queued invalidation
[    0.468214] IOMMU: Setting RMRR:
[    0.468222] IOMMU: Setting identity map for device 0000:00:1d.0 [0xec000 - 0xeefff]
[    0.468245] IOMMU: Setting identity map for device 0000:00:1a.7 [0xbf4a1000 - 0xbf4a1fff]
[    0.468264] IOMMU: Setting identity map for device 0000:00:1a.2 [0xbf4ac000 - 0xbf4acfff]
[    0.468282] IOMMU: Setting identity map for device 0000:00:1a.1 [0xbf4aa000 - 0xbf4aafff]
[    0.468300] IOMMU: Setting identity map for device 0000:00:1a.0 [0xbf4a7000 - 0xbf4a7fff]
[    0.468318] IOMMU: Setting identity map for device 0000:00:1d.7 [0xbf4a5000 - 0xbf4a5fff]
[    0.468335] IOMMU: Setting identity map for device 0000:00:1d.2 [0xbf4b0000 - 0xbf4b0fff]
[    0.468353] IOMMU: Setting identity map for device 0000:00:1d.1 [0xbf4af000 - 0xbf4affff]
[    0.468364] IOMMU: Setting identity map for device 0000:00:1d.0 [0xbf4ad000 - 0xbf4adfff]
[    0.468373] IOMMU: Prepare 0-16MiB unity mapping for LPC
[    0.468379] IOMMU: Setting identity map for device 0000:00:1f.0 [0x0 - 0xffffff]

I have downloaded the kernel from the OP, built and installed it. I am using the following commandline:

intel_iommu=on pcie_acs_override=downstream

and confirmed that ACS override was enabled

[    0.000000] Warning: PCIe ACS overrides enabled; This may allow non-IOMMU protected peer-to-peer DMA

Since I had 2 identical cards, they had identical vendor and device ID's. If I used the pci-stub, it would end up stubbing both the graphic cards which will render my host useless.

$ lspci -nn | grep VGA
02:00.0 VGA compatible controller [0300]: NVIDIA Corporation G94 [GeForce 9600 GT] [10de:0622] (rev a1)
03:00.0 VGA compatible controller [0300]: NVIDIA Corporation G94 [GeForce 9600 GT] [10de:0622] (rev a1)

Thanks to @cjn! I was using his workaround to unbind the primary card after the pci-stub module has loaded.

$ cat /etc/modprobe.d/pci-stub.conf 
options pci-stub ids=10de:0622
install pci-stub /sbin/modprobe --ignore-install pci-stub $CMDLINE_OPTS; /usr/local/sbin/pci-stub-unbind 0000:02:00.0
softdep drm pre: pci-stub

$ cat /usr/local/sbin/pci-stub-unbind
#!/bin/sh
for dev in "$@"
do
  echo $dev > /sys/bus/pci/drivers/pci-stub/unbind
done

The first GPU is now used by the Nvidia 340 driver on the host and the GPU is assigned to pci-stub. I have then assigned the GPU to the vfio-pci driver and to confirm the same

03:00.0 VGA compatible controller: NVIDIA Corporation G94 [GeForce 9600 GT] (rev a1) (prog-if 00 [VGA controller])
        Subsystem: XFX Pine Group Inc. Device 234b
        Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR+ FastB2B- DisINTx-
        Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
        Latency: 0, Cache Line Size: 64 bytes
        Interrupt: pin A routed to IRQ 16
        Region 0: Memory at c2000000 (32-bit, non-prefetchable) [size=16M]
        Region 1: Memory at e0000000 (64-bit, prefetchable) [size=256M]
        Region 3: Memory at c0000000 (64-bit, non-prefetchable) [size=32M]
        Region 5: I/O ports at 2000 [size=128]
        Expansion ROM at <ignored> [disabled]
        Capabilities: <access denied>
        Kernel driver in use: vfio-pci
        Kernel modules: nouveau, nvidia

I have set the following module options.

options kvm ignore_msrs=1
options vfio_iommu_type1 allow_unsafe_interrupts=1

#options kvm_intel emulate_invalid_guest_state=0 [With this set, I had no VGA output even in Virt-Manager console]

<domain type='kvm' xmlns:qemu='http://libvirt.org/schemas/domain/qemu/1.0'>
  <name>Windows7</name>
  <uuid>e9a9c56a-0708-4539-9d77-130283cbb354</uuid>
  <memory unit='KiB'>8388608</memory>
  <currentMemory unit='KiB'>8388608</currentMemory>
  <vcpu placement='static'>4</vcpu>
  <os>
    <type arch='x86_64' machine='pc-i440fx-2.2'>hvm</type>
    <boot dev='hd'/>
  </os>
  <features>
    <acpi/>
    <apic/>
    <pae/>
    <kvm>
        <hidden state='on'/>
    </kvm>
  </features>
  <cpu mode='custom' match='exact'>
    <model fallback='allow'>Nehalem</model>
  </cpu>
  <clock offset='utc'>
    <timer name='rtc' tickpolicy='catchup'/>
    <timer name='pit' tickpolicy='delay'/>
    <timer name='hpet' present='no'/>
  </clock>
  <on_poweroff>destroy</on_poweroff>
  <on_reboot>restart</on_reboot>
  <on_crash>restart</on_crash>
  <pm>
    <suspend-to-mem enabled='no'/>
    <suspend-to-disk enabled='no'/>
  </pm>
  <devices>
    <emulator>/usr/sbin/qemu-system-x86_64</emulator>
    <disk type='file' device='disk'>
      <driver name='qemu' type='qcow2'/>
      <source file='/mnt/VMs/qemu/Windows7.qcow2'/>
      <target dev='hda' bus='ide'/>
      <address type='drive' controller='0' bus='0' target='0' unit='0'/>
    </disk>
    <disk type='block' device='cdrom'>
      <driver name='qemu' type='raw'/>
      <target dev='hdb' bus='ide'/>
      <readonly/>
      <address type='drive' controller='0' bus='0' target='0' unit='1'/>
    </disk>
    <controller type='usb' index='0' model='ich9-ehci1'>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x06' function='0x7'/>
    </controller>
    <controller type='usb' index='0' model='ich9-uhci1'>
      <master startport='0'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x06' function='0x0' multifunction='on'/>
    </controller>
    <controller type='usb' index='0' model='ich9-uhci2'>
      <master startport='2'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x06' function='0x1'/>
    </controller>
    <controller type='usb' index='0' model='ich9-uhci3'>
      <master startport='4'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x06' function='0x2'/>
    </controller>
    <controller type='pci' index='0' model='pci-root'/>
    <controller type='ide' index='0'>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x1'/>
    </controller>
    <controller type='virtio-serial' index='0'>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x05' function='0x0'/>
    </controller>
    <interface type='direct'>
      <mac address='52:54:00:91:20:1c'/>
      <source dev='enp0s25' mode='bridge'/>
      <model type='rtl8139'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x0'/>
    </interface>
    <serial type='pty'>
      <target port='0'/>
    </serial>
    <console type='pty'>
      <target type='serial' port='0'/>
    </console>
    <channel type='spicevmc'>
      <target type='virtio' name='com.redhat.spice.0'/>
      <address type='virtio-serial' controller='0' bus='0' port='1'/>
    </channel>
    <input type='mouse' bus='ps2'/>
    <input type='keyboard' bus='ps2'/>
    <graphics type='spice' autoport='yes'/>
    <sound model='ich6'>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x04' function='0x0'/>
    </sound>
    <video>
      <model type='qxl' ram='65536' vram='65536' vgamem='16384' heads='1'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x0'/>
    </video>
    <redirdev bus='usb' type='spicevmc'>
    </redirdev>
    <redirdev bus='usb' type='spicevmc'>
    </redirdev>
    <redirdev bus='usb' type='spicevmc'>
    </redirdev>
    <redirdev bus='usb' type='spicevmc'>
    </redirdev>
    <memballoon model='virtio'>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x07' function='0x0'/>
    </memballoon>
  </devices>
  <qemu:commandline>
    <qemu:arg value='-device'/>
    <qemu:arg value='vfio-pci,host=03:00.0,multifunction=on'/>
    <qemu:arg value='-vga'/>
    <qemu:arg value='none'/>
    <qemu:arg value='-cpu'/>
    <qemu:arg value='host,kvm=off'/>
  </qemu:commandline>
</domain>

With all the above changes, I was able to boot into Ubuntu and install the Nvidia graphic drivers and everything worked fine. I was able to get the display out of the 2nd monitor. However, with Windows, I always get Code 43 error.

I have done extensive reading and I made sure that I was using <hidden state='on'/> and kvm=off to make sure that the newer Nvidia driver started properly. I have even tried using an older 304 Nvidia driver in the guest and it didn't change a thing.

While looking at the Host Kernel log, I was able to make a comparision.

Kernel log for Windows booting

<6>[ 3916.375945] vfio-pci 0000:03:00.0: enabling device (0400 -> 0403)
<6>[ 3922.353501] kvm: zapping shadow pages for mmio generation wraparound

Kernel log for Linux booting

<6>[ 5821.639931] vfio-pci 0000:03:00.0: enabling device (0400 -> 0403)
<6>[ 5827.818355] kvm: zapping shadow pages for mmio generation wraparound
<3>[ 5839.699364] kvm [30427]: vcpu0 ignored rdmsr: 0x1c9
<3>[ 5839.699368] kvm [30427]: vcpu0 ignored wrmsr: 0x1c9 data 3
<3>[ 5839.699370] kvm [30427]: vcpu0 ignored rdmsr: 0x1c9
<3>[ 5839.699372] kvm [30427]: vcpu0 ignored rdmsr: 0x1a6
<3>[ 5839.699373] kvm [30427]: vcpu0 ignored wrmsr: 0x1a6 data 1ff
<3>[ 5839.699375] kvm [30427]: vcpu0 ignored rdmsr: 0x1a6
<3>[ 5839.699378] kvm [30427]: vcpu0 ignored rdmsr: 0x3f6
<3>[ 5839.699380] kvm [30427]: vcpu0 ignored wrmsr: 0x3f6 data 1ff
<3>[ 5839.699381] kvm [30427]: vcpu0 ignored rdmsr: 0x3f6
<7>[ 5844.410667] vfio-pci 0000:03:00.0: irq 36 for MSI/MSI-X

"vfio-pci 0000:03:00.0: irq 36 for MSI/MSI-X" doesn't occur with windows. Probably the interrupt's are not working?

Please advice.

mostlyharmless

Member

Registered: 2014-01-16

Posts: 72

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

@aw Interesting about the if=none; apparently a new requirement?

As to the other, I'm not passing through a drive, just my NVIDIA card. It passes through great under Slackware and the older qemu. The same VM with the video pass through stripped out boots fine under Arch and Slackware with their respective qemus.  The CD drive line has nothing to do with that either, as the same problem exists regardless of whether I include the CD host drive line in the qemu arguments.

@dhiru1602

Interesting, the intel ark http://ark.intel.com/search/advanced?s= … s&VTD=true

doesn't list the i7-920 as having Vt-d support, yet you apparently had success with a linux guest.

[Edit] I see the 920XM on this list http://ark.intel.com/search/advanced?s=t&VTD=true

Last edited by mostlyharmless (2015-01-28 21:56:22)

Denso

Member

Registered: 2014-08-30

Posts: 179

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

By that, do you mean that it is not present in 3.18.2?  I'm familiar with the first patch, I doubt it's to blame.  If you can revert one of those patches and toggle the problem let me know and I'll investigate further.  Easiest way to do a quick revert test is git show <commit ID> | patch -p1 -R  It's also possible that the problem is a side-effect of other code changes.

Yes , those errors didn't happen with 3.18.2 . I noticed that a system upgrade that happened today pulled 3.18.3 . Again , DMAR issues + Code 43 in one Windows VM .

I didn't know that I can revert a patch from an already compiled \ running kernel ?! Is that what  git show <commit ID> | patch -p1 -R does ?

EDIT : Sorry , I'm just stupid . I'll pull the source and revert those patches one by one .

Thanks !

Last edited by Denso (2015-01-28 21:59:26)

aw

Member

Registered: 2013-10-04

Posts: 921

Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

I didn't know that I can revert a patch from an already compiled \ running kernel ?! Is that what  git show <commit ID> | patch -p1 -R does ?

No, that reverts it from the source, you'd need to rebuild.

---

http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

Denso

Member

Registered: 2014-08-30

Posts: 179

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

No, that reverts it from the source, you'd need to rebuild.

Yup . I realized how stupid that question was , maybe I was dreaming of runtime dynamic patching / reverting !

By the way , did I go blind or is git REALLY pulling a 3.9GB tree ?! lol

http://i.imgur.com/H8JPe2c.png

nythrix

Member

Registered: 2015-01-22

Posts: 4

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Thanks to the awesome people helping out in this thread I've been able to successfully pass through a 9600GT to a Windows 7 guest.

Unfortunately, linux guests don't work. I tried my oldish Debian VM, which has been running on "-vga std" since forever but I loose graphics output around the time the console switches fonts during boot (not sure how to describe it). Everything happens very quickly so the last messages I'm able to make out are stuff about intel RAPL and something about BARs ..? I wiped out the VM and reinstalled Debian but the problem persisted. The installation itself was flawless though (with passthrough). Twice.

Then I tried installing Fedora 21 Workstation. Couldn't get much further than some sort of greeting menu. It seems the signal is again lost during attempts to fiddle with the graphics/framebuffer.

Is this some sort of device reset problems I'm observing?

PS: I'm running 3.17 with the i915 arbiter patch. Qemu 2.1.2.

For future reference:
Nvidia 9600GT works great with Windows 7 guests (w/o additional ROM file) but not with Linux guests. I plugged in a GTX 460 for those and it worked (requires ROM and guest restart is still kinda hairy).
My motherboard is a Gigabyte GA-Z97X-UD3H (rev. 1.0, bios F7). Offers two usable IOMMU groups (i.e. you can safely run 2 independently accelerated VMs w/o the ACS hack): CPU-based, PCIe v3.0 slots in x16 or x8+x8 configurations and PCH-based, PCIe v2.1 slots in x4 or x1+x1+x1 configurations. The primary graphics can be the iGFX or a dGPU plugged into any physical PCIe x16 slot or even the friggin legacy PCI slot (hanging of the PCH-based IOMMU group). Any RIVA TNT owners should be satisfied with this one
Everything is powered by an i5-4590S. I still think AMD is way superior when it comes to virtualization. I hate Intel's market segmentation and purposeful technology crippling. Were it not for the atrocious TDP, I would've rather gone for an FX-8xxx.
Kernel 3.17 + VGA patch.
Qemu 2.1.2.

My launcher script:

#!/bin/bash


vfiobind()
{
	pci_addr="$1"
	vendor=$(cat /sys/bus/pci/devices/$pci_addr/vendor)
	device=$(cat /sys/bus/pci/devices/$pci_addr/device)
	if [ -e /sys/bus/pci/devices/$pci_addr/driver ]; then
		echo $pci_addr > /sys/bus/pci/devices/$pci_addr/driver/unbind
	fi
	echo $vendor $device > /sys/bus/pci/drivers/vfio-pci/new_id
}


modprobe vfio-pci


vfiobind 0000:01:00.0 # PCI Express x16 slot
#vfiobind 0000:01:00.1
#vfiobind 0000:02:00.0 # PCI Express x8 slot
#vfiobind 0000:05:00.0 # PCI Express x4 slot
#vfiobind 0000:05:00.1


export QEMU_AUDIO_DRV=pa
pulseaudio --start
pacmd set-sink-mute 1 false
pacmd set-sink-volume 1 8192


qemu_args=""
qemu_args=$qemu_args" -machine type=q35,accel=kvm"
qemu_args=$qemu_args" -cpu host"
qemu_args=$qemu_args" -smp cpus=3,cores=1,threads=1"
qemu_args=$qemu_args" -boot order=dc,menu=on"
qemu_args=$qemu_args" -m size=10G"
qemu_args=$qemu_args" -soundhw hda"
qemu_args=$qemu_args" -device ahci,id=ahci-id"
#qemu_args=$qemu_args" -device ide-cd,bus=ahci-id.0,drive=CD-id"
qemu_args=$qemu_args" -device ide-hd,bus=ahci-id.1,drive=HD-id"
qemu_args=$qemu_args" -device ioh3420,bus=pcie.0,addr=1c.0,multifunction=on,port=1,chassis=1,id=ioh3420-id"
qemu_args=$qemu_args" -device vfio-pci,host=01:00.0,bus=ioh3420-id,addr=00.0,multifunction=on,x-vga=on"
qemu_args=$qemu_args" -usb"
qemu_args=$qemu_args" -nographic"
#qemu_args=$qemu_args" -drive if=none,id=CD-id,file=/dev/cdrom"
qemu_args=$qemu_args" -drive if=none,id=HD-id,file=./vm.img"
#qemu_args=$qemu_args" -vga none"
qemu_args=$qemu_args" -net nic"
qemu_args=$qemu_args" -net user,restrict=on"
qemu_args=$qemu_args" -enable-kvm"
qemu_args=$qemu_args" -nodefaults"
#qemu_args=$qemu_args" -runas user"


qemu-system-x86_64 $qemu_args

I'm still fighting with access permissions here and there. "-runas user" doesn't really help.

mostlyharmless

Member

Registered: 2014-01-16

Posts: 72

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

OK, so removing the NVIDIA driver from within Windows didn't change the 0x0000007b error.  Still mystified. Here's more data:

qemu under Slackware is qemu 2.1.0, under Arch qemu 2.2.0 

Under qemu 2.2.0, removing the

-drive file=/dev/mapper/cryptvg2-win7,id=disk,if=virtio 

line allows the system to boot from the install CD for Windows 7... 

Something has changed in qemu's hard disk handling, akin to the if=none for the CD, perhaps with virtio?

but not from a Windows XPSP2 install disk that fails with bluescreen error 0x000000A5 Bios not fully ACPI complaint. Pressing F7 to bypass ACPI leads to the bluescreen with 0x0000007b.

Now let's see what happens under Slackware.... nah same thing, obviously that's a red herring secondary to WinXPSP2.  Hmm, though it runs under libvirt through under both Slackware and Arch, maybe that's a -M q35 thing since libvirt defaults to 440.

Last edited by mostlyharmless (2015-01-29 15:13:55)

vx

Member

Registered: 2015-01-29

Posts: 1

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Hi guys,

Thanks for this thread. Been playing with this for about a week. I got passthrough working on the first day, but been having one big issue since then...

Windows installs fine. The first time I get to the desktop, it works fine. I can reboot multiple times and it brings me back to the desktop, until any one of the following happens:

1) I get near the the end of the catalyst driver installation
2) Windows power saving kicks in and turns off the monitor connected to the passthrough GPU
3) The host!!! monitor turns off because I haven't touched that keyboard in a while

When any of these happens, the passthrough monitor turns off, and can't be turned back on by touching the passthrough keyboard/mouse. If I kill the qemu process and restart it, it shows me SeaBIOS, the windows splash screen, and then the passthrough monitor goes off again. Can't recover from this point without reinstalling windows, and then I'm back to square one.

Apologies if this has been covered already. The closest I can find are a few posts by syllopsium and firewing1 way back on page 37. syllopsium eventually used the NoSnoop patch, which apparently has been deprecated since, so I'm not sure if to try it.

I've been trying a bunch of different stuff (detailed below), and literally every working combination that gets the VM to start ends up with the same issue. Any help or guidance on what to try next would be appreciated. Thanks in advance.

Setup:
I got here via this Puget Systems how-to: http://www.pugetsystems.com/labs/articl … 4-KVM-585/, so my configuration is somewhat based on their instructions, but I have since updated to a newer Ubuntu version & kernel, applied the i915 arbitration patch and am using qemu 2.1.0.

Problem was the same in previous setup (14.04 distro / 3.13 kernel / qemu 2.0). Only the arbitration patch made a small difference. My host monitor does not turn blue now when I start the VM. Host monitor just displays Ubuntu Server console, no X. Could not having X installed be a problem?

GPU ROM was dumped using instructions from the virtio blog. It doesn't have a UEFI bios, so OVMF is not an option. Whether the romfile is passed to qemu or not makes no difference.

I've tried with both the q35 and 440fx machines (see below for qemu params). Same problem on both of them.

I'm using the Windows 8.1 90 day eval to make sure it works before I buy it. Not sure if this will make a difference from the retail version. Or maybe I need to use an older version instead? Have not tried that.

I've not tried attaching a cirrus adapter and installing catalyst over VNC yet. Since it happens whether catalyst is installed or not, I'm guessing this is not the issue. Will try it later today anyway though.

qemu startup (q35):

sudo qemu-system-x86_64 \
-enable-kvm \
-daemonize \
-M q35 \
-m 4096 \
-cpu host \
-smp 2,sockets=1,cores=2 \
-vga none \
-display none \
-serial none -parallel none \
-device ioh3420,bus=pcie.0,addr=1c.0,multifunction=on,port=1,chassis=1,id=root.1 \
-device vfio-pci,host=0000:01:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on,romfile=$HOME/r9270.rom \
-device vfio-pci,host=0000:01:00.1,bus=root.1,addr=00.1 \
-usb -usbdevice host:046d:c52e -usbdevice host:1bcf:0007 \
-drive file=$HOME/kvm/gaming.img,id=disk,format=raw -device ide-hd,bus=ide.0,drive=disk \
-drive file=$HOME/win8.iso,id=isocd -device ide-cd,bus=ide.1,drive=isocd \
-boot order='dc' \

qemu startup (440fx):

sudo qemu-system-x86_64 \
-enable-kvm \
-daemonize \
-M pc \
-m 4096 \
-cpu host \
-smp 2,sockets=1,cores=2 \
-vga none \
-display none \
-serial none \
-parallel none \
-device vfio-pci,host=0000:01:00.0,multifunction=on,x-vga=on,romfile=$HOME/r9270.rom \
-device vfio-pci,host=0000:01:00.1 \
-usb -usbdevice host:046d:c52e -usbdevice host:1bcf:0007 \
-drive file=$HOME/kvm/gaming.img,media=disk,format=raw \
-drive file=$HOME/win8.iso,media=cdrom \
-boot order='dc' \

Hardware:
CPU:                 Intel i5 3470S - http://ark.intel.com/products/68315/Int … o-3_60-GHz
Motherboard:    MSI H77MA-G43 - http://www.msi.com/product/mb/H77MA-G43 … o-overview
GPU Host:        Intel HD 2500 from CPU
GPU VM:          Sapphire Radeon R9 270 - http://www.sapphiretech.com/presentatio … 2089&lid=1

Software:
Host OS:           Ubuntu 14.10 Server
Kernel:              3.16.0-29 (+i915 arbitration patch)
qemu version:   2.1.0
Guest OS:         Windows 8.1 (90 day eval)

Wooots

Member

Registered: 2015-01-29

Posts: 3

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Hello there

I've read this article and some others and got my Nvidia working at the moment using the following command:

qemu-system-x86_64 -enable-kvm -m 1024 -cpu host,kvm=off -smp 6,sockets=1,cores=6,threads=2 -vga none -device vfio-pci,host=03:00.0,x-vga=on -device vfio-pci,host=03:00.1  -drive file=/opt/VirtualMachines/Passthrough-test/win7.qcow2,id=disk,format=qcow2,if=none -device ide-hd,drive=disk -usb -device usb-host,vendorid=0x046d,productid=0xc531 -usb -device usb-host,vendorid=0x046d,productid=0xc24d

But I have one little question do you all start qemu as root? Isn't this a security problem?

Last edited by Wooots (2015-01-29 18:17:24)

dhiru1602

Member

Registered: 2015-01-28

Posts: 13

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

I was finally able to get the passthrough working on Windows but I had to use a dirty hack to start the VM everytime.

As I have mentioned earlier, my GPU was working fine in passthrough on Linux but I got Code 43 error on Windows. This was because I don't get ""vfio-pci 0000:03:00.0: irq 36 for MSI/MSI-X" on Windows.

I created a Linux Virtual Machine and after booting it, whenever I got the irq message in the kernel log, I would immediately turn off the Linux VM and boot the Windows VM. The GPU gets detected and works perfectly on Windows without Code 43. I have even installed the latest 340 nvidia driver and games work great. However, once I shutdown or reboot the VM, I will have to start the Linux VM first and force stop it when the interrupt line is registered and then boot the windows virtual machine. This works like about 5 out of 10 times and it's really a pain.

Any thoughts on what might be preventing the interrupts from working on a Windows guest?

@dhiru1602

Interesting, the intel ark http://ark.intel.com/search/advanced?s= … s&VTD=true

doesn't list the i7-920 as having Vt-d support, yet you apparently had success with a linux guest.

[Edit] I see the 920XM on this list http://ark.intel.com/search/advanced?s=t&VTD=true

Yes. Interestingly enough, the passthrough works even without the patched ACS kernel.

Last edited by dhiru1602 (2015-01-29 18:49:53)

The_Moves

Member

Registered: 2015-01-06

Posts: 59

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Is there a good post where someone converted from a QEMU command to an XML file? I'm having issues getting my xml file created. I have no issues with multiple device passthrough, GTX760/USB/Ethernet/Creative Sounds Blaster Z, I just want to be able to manage using an XML and LibVirt/Virt-Manager

mostlyharmless

Member

Registered: 2014-01-16

Posts: 72

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

@Wooots;  I usually run as root or use sudo; I'm pretty sure somewhere in the last 160 pages there's some detail on how to fudge the permissions to avoid that...

@The_Moves on that note, see page 94

devianceluka

Member

Registered: 2014-05-19

Posts: 44

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Using Fedora 21 and Virt-Manager, I successfully installed a VM with UEFI/OVMF by editing this:

/etc/libvirt/qemu.conf
nvram = [ "/usr/share/edk2.git/ovmf-x64/OVMF-pure-efi.fd:/usr/share/edk2.git/ovmf-x64/OVMF_VARS-pure-efi.fd" ]

Everything works flawlessly except my VM is NOT saving EFI Boot entry (Ubuntu/Fedora for now, later Windows). What could be the problem? I tried manually adding "by file" but on reboot it resets and forgets the entry. Also tried "sudo grub-install" in Ubuntu so Ubuntu makes the entry with no success either.

Please help!

Last edited by devianceluka (2015-01-30 00:12:21)

maxexcloo

Member

Registered: 2009-10-14

Posts: 177

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Is it at all possible to use the dedicated card on my optimus enabled laptop as a passthrough? I'm thinking it could be possible (although probably not implemented) as the output from the card could then be copied to the normal output (as optimus usually works), providing an awesome experience.

(Just thinking about it, unless the card is different somehow or has some feature disabled why isn't it possible?)

fatfingers

Member

Registered: 2015-01-30

Posts: 1

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Tried to get it working for me too, but with no success.

My system:
Radeon 7950
i5 3470 (non-K) incl Intel GPU for Host
AsRock Z77 Extreme4
8GB Ram

First thing I was trying was the old way, with pci-assign. I was able to first boot with VNC, installing ATI drivers. After reboot the screen woke up at login prompt and the Performance was ok/good (I wasn't trying too much. Windows Performance Index at 7.5, so it should have worked ). But at reboot of VM the host crashed....

So I was looking around the internet and found this thread. Tried many hours to get it working, but I don't get any picture to the screen. No Seabios Texts and nothing in Windows after installing ATI drivers with vnc (and removing VNC again) like it was working with pci-assign.

Have tried Qemu-git from AUR, Qemu-1.5-rc1, seabios from your link, stock kernel (3.9.2-1) and kernel 3.10-rc1.
Did not change anything .

The only thing I havn't tried till now is to dump the VGA bios and put it in. Will do this today, when I´m at home.

IOMMU is enabled. logfiles look good (for me ). The only thing is "DMAR: No ATSR found". You have a Idea about this?
Full logfiles I can add later. 

Bind everything to vfio-pci seems to work. Also there isn't any error message at KVM-Start
I also tried all with pci-stub entries at grub and without, same with the interrupt remapping entry.

Do you have any Idea?

Thanks
apo

---------------------------------------------------------------------------------------------------------------------------------
I have the same mainboard except its the extreme4-M.

Brief setup ,kernel 3.18 mainline -recompiled,NO ACS PATCH ,went back to distro re-compiled kernel as issues with mainline.
ati 7950 and nvidia q570,xeon 1245v2
ATI is the boot card which then swaps to the nvidia.
ATI on SLOT1 and the Nvidia on slot3
No Xorg.conf
Nouveau on nvidia
Non UEFI / legacy kernel 
Using the cards extracted Rom ,and the Rom Bios option in the qemu config .

Some testing with the bridges as i did have a fully working pci passthrough till ubuntu went from
13.12 to 14.04 and xen to 4.4 i have had issues ,not sure if its xorg or the kernel,still testing
and havent had passthrough in xen since 4.3 and earlier kernels,hence the testing with
qemu again.xen does utilise the qemu platform.
xen 4.5 has no pci passthrough,and iam trying to get it working on qemu.

eg:

chmod 660 /sys/bus/pci/devices/0000\:00\:01.0/driver/new_id
chmod 660 /sys/bus/pci/devices/0000\:00\:01.0/driver/unbind

echo 8086 0151 | sudo tee /sys/bus/pci/drivers/vfio-pci/new_id

I also detach and re-attach in the qemu script

virsh nodedev-reattach pci_8086_1e31
virsh nodedev-detach pci_8086_1e31
virsh nodedev-detach   pci_8086_1e31 # bridge # Be careful ! this can cause major issues if the groups of the ati card and the bridge are not on their own.
virsh nodedev-reattach pci_8086_1e31 # bridge # Be careful ! this can cause major issues if the groups of the ati card and the bridge are not on their own.

# Grub Linux Default
GRUB_CMDLINE_LINUX_DEFAULT="intel_iommu=on,pt,igfx_off vfio_iommu_type1.allow_unsafe_interrupts=1  \
kvm_intel.emulate_invalid_guest_state=0 radeon.blacklist=1 radeon.modeset=0 rd.driver.blacklist=radeon rd.blacklist=radeon"

I also have a modified xorg.conf to ignore and remove fbdev and vesa from attaching itself via Xorg ( takes over).
Also removed xorg ati and xorg fglrx from the packages.

I know this isnt a xen forum but i have found that pciback does bind to the bridge although i dont pass it through.

Regards

I followed some of the intel vfio ddk ,fortunately the machine didnt fall over like i expected when unbinding this bus,intel vfio ddk -notes page 14

Last edited by fatfingers (2015-01-31 13:04:45)

The_Moves

Member

Registered: 2015-01-06

Posts: 59

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

@Wooots;  I usually run as root or use sudo; I'm pretty sure somewhere in the last 160 pages there's some detail on how to fudge the permissions to avoid that...

@The_Moves on that note, see page 94

I've gone through page 94, however my goal is to follow Alex's lead and get rid of all 'qemu' calls. I'll have some dedicated time this weekend to sit and figure it out, but anything to reference will help.

flack

Member

Registered: 2014-06-12

Posts: 7

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Hi all i have some troubles with my KVM host computer. I dont know what can happened. So sorry for my english.

I using KVM VGA passthrough for some time i think more than six mounths. All works fine but sometimes i see only some error msq with libusb in console where i run my qemu image. I think this libusb problems is not too much big. I have usb mouse, keyobard, webcam, flashdrive and usb soundcard, usb network card attach to my qemu virtual machine. In virtual machine runs Windows 7 and here is my setting for qemu kvm.

sudo vfio-bind 0000:01:00.0 0000:01:00.1 && sudo qemu-system-x86_64 -rtc clock=host -cpu host,hv-time -smp 4,sockets=1,cores=2,threads=2 -M q35 -enable-kvm -m 5120 -bios /usr/share/qemu/bios.bin -vga none -device ahci,bus=pcie.0,id=ahci -drive file=/home/back/KVM/windows.iso,id=isocd -device ide-cd,bus=ahci.0,drive=isocd -drive file=/home/back/KVM/virtio.iso,id=isocd2 -device ide-cd,bus=ahci.1,drive=isocd2 -drive file=disk-preallc.qcow2,if=virtio -drive file=/media/back/btrfs-test400/ntfsDisk.qcow2,if=virtio -drive file=/media/back/btrfs-test400/ntfsDisk-preallocated.qcow2,if=virtio -drive file=/media/back/btrfs-test400/ntfs-100-Disk-preallocated.qcow2,if=virtio -boot menu=on -device ioh3420,bus=pcie.0,addr=1c.0,multifunction=on,port=1,chassis=1,id=root.1 -device vfio-pci,host=01:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=off,romfile=vbios2.rom,x-vga=on -device vfio-pci,host=01:00.1,bus=pcie.0 -usb -usbdevice host:13ba:0018 -usbdevice host:093a:2521 -usbdevice host:1a2c:0002 -usbdevice host:9710:7830 -usbdevice host:0d8c:000c -usbdevice host:041e:4064

Three day ago i shutdown PC becouse i want to clean up the room and relocate table to near window-I have dark room. I plug off all cables from back (Monitor, mouse, power..). With this opportunity i want to try my older VGA card if still working or not. For this i go to BIOS to change primary VGA output from intelGFX(on board) to PCIe card. SHUTDOWN and remove my GTX 560 from PCIe slot and istead put older VGA card and then. Putt the all cables back to PC and POWERON ... i saw POST output on LCD i open BIOS and change primary graphic to intelGFX and shutdown PC. Remove old but workin VGA card and insert back my GTX 560 card.

Boot computer ubuntu 14.04 start and then i open browser. Site not found? whats wrong. Ohh then i see my ethernet card ETH0 missing in ifconfig. I fast reboot PC just for test. And my card still not registred like eth0. I see only eth1-this is my USB network card for qemu machine. I run my virtual machine they boot verry slow. I also try next restart of PC but pc verry slow boot and hangs. I must more time reboot pc. Now PC boot work but somethimes freeze and network card still not work. Also not blink leds and also lspci dont see this network card-like no exist. I think integrated on board network card is now die- maybe my motherboard go die.

at this moment i run dmesg and see too much this lines..

[34913.005352] DMAR:[fault reason 06] PTE Read access is not set
[35395.484547] dmar: DRHD: handling fault status reg 3
[35395.484553] dmar: DMAR:[DMA Read] Request device [05:00.0] fault addr ffbfc000 
[35395.484553] DMAR:[fault reason 06] PTE Read access is not set
[35395.957843] dmar: DRHD: handling fault status reg 3
[35395.957851] dmar: DMAR:[DMA Read] Request device [05:00.0] fault addr ffbfc000 
[35395.957851] DMAR:[fault reason 06] PTE Read access is not set
[36830.943853] dmar: DRHD: handling fault status reg 3
[36830.943858] dmar: DMAR:[DMA Read] Request device [05:00.0] fault addr ffbfc000 
[36830.943858] DMAR:[fault reason 06] PTE Read access is not set
[37196.245627] dmar: DRHD: handling fault status reg 3
[37196.245633] dmar: DMAR:[DMA Read] Request device [05:00.0] fault addr ffbfc000 

still increasing

Yesterday i backup all files in /var/log/ but i am not guru for this. In syslog.3 there is last information about module atl1c -this was network module for my integrated network card.
There are syslog files http://www.mediafire.com/download/57314 … yslogs.zip
I have also PCI NETWORK CARD which i use at this moment instead integrated.

This is my lspci output:

00:00.0 Host bridge: Intel Corporation Xeon E3-1200 v2/3rd Gen Core processor DRAM Controller (rev 09)
00:01.0 PCI bridge: Intel Corporation Xeon E3-1200 v2/3rd Gen Core processor PCI Express Root Port (rev 09)
00:02.0 VGA compatible controller: Intel Corporation Xeon E3-1200 v2/3rd Gen Core processor Graphics Controller (rev 09)
00:14.0 USB controller: Intel Corporation 7 Series/C210 Series Chipset Family USB xHCI Host Controller (rev 04)
00:16.0 Communication controller: Intel Corporation 7 Series/C210 Series Chipset Family MEI Controller #1 (rev 04)
00:1a.0 USB controller: Intel Corporation 7 Series/C210 Series Chipset Family USB Enhanced Host Controller #2 (rev 04)
00:1b.0 Audio device: Intel Corporation 7 Series/C210 Series Chipset Family High Definition Audio Controller (rev 04)
00:1c.0 PCI bridge: Intel Corporation 7 Series/C210 Series Chipset Family PCI Express Root Port 1 (rev c4)
00:1c.4 PCI bridge: Intel Corporation 7 Series/C210 Series Chipset Family PCI Express Root Port 5 (rev c4)
00:1c.5 PCI bridge: Intel Corporation 82801 PCI Bridge (rev c4)
00:1c.7 PCI bridge: Intel Corporation 7 Series/C210 Series Chipset Family PCI Express Root Port 8 (rev c4)
00:1d.0 USB controller: Intel Corporation 7 Series/C210 Series Chipset Family USB Enhanced Host Controller #1 (rev 04)
00:1f.0 ISA bridge: Intel Corporation Z77 Express Chipset LPC Controller (rev 04)
00:1f.2 SATA controller: Intel Corporation 7 Series/C210 Series Chipset Family 6-port SATA Controller [AHCI mode] (rev 04)
00:1f.3 SMBus: Intel Corporation 7 Series/C210 Series Chipset Family SMBus Controller (rev 04)
01:00.0 VGA compatible controller: NVIDIA Corporation GF114 [GeForce GTX 560] (rev a1)
01:00.1 Audio device: NVIDIA Corporation GF114 HDMI Audio Controller (rev a1)
03:00.0 USB controller: VIA Technologies, Inc. VL80x xHCI USB 3.0 Controller (rev 03)
04:00.0 PCI bridge: Intel Corporation 82801 PCI Bridge (rev 30)
05:00.0 Ethernet controller: Realtek Semiconductor Co., Ltd. RTL8169 PCI Gigabit Ethernet Controller (rev 10)
06:00.0 SATA controller: Marvell Technology Group Ltd. 88SE9172 SATA 6Gb/s Controller (rev 11)

kernel:

Linux KVM-Ubuntu 3.15.1 #1 SMP Thu Jun 19 00:44:25 CEST 2014 x86_64 x86_64 x86_64 GNU/Linux

   
CPU: i7 3770
MB: Gigabyte Z77X-D3H ver. 1.0
VGA: Asus GTX 560 DCII
------------------------------------------
RAM: 12Gb=3*4Gb Crucial (i dont know which one)
SSD1: intel 530 256Gb  (Ubuntu 14.04)
SSD2: samsung evo 840 256Gb (using only for qemu images)
HDD3: junk
-------------------------------------------

I think this can be related with virtualization, bios or just my mainboard is broken.

Bronek

Member

From: London

Registered: 2014-02-14

Posts: 119

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Ok so I've finally setup serial console and my host is finally headless. I was able to capture this when restarting virtual machine, which is Windows 7 (no EFI). This happens when VM is started after guest reboot

[  250.726291] ------------[ cut here ]------------
[  250.730914] kernel BUG at drivers/pci/ats.c:138!
[  250.735535] invalid opcode: 0000 [#1] PREEMPT SMP 
[  250.740379] Modules linked in: vfio_pci vfio_iommu_type1 vfio iTCO_wdt iTCO_vendor_support ext4 crc16 mbcache jbd2 mxm_wmi coretemp x86_pkg_temp_thermal intel_powerclamp kvm_intel kvm crct10dif_pclmul crc32_pclmul crc32c_intel ghash_clmulni_intel aesni_intel aes_x86_64 lrw gf128mul glue_helper ablk_helper cryptd pcspkr serio_raw sb_edac edac_core igb joydev mousedev evdev mac_hid lpc_ich i2c_i801 ptp pps_core hwmon snd_hda_intel i2c_algo_bit snd_hda_controller i2c_core snd_hda_codec mei_me mei snd_hwdep snd_pcm snd_timer snd soundcore tpm_tis ioatdma tpm dca wmi processor shpchp button sch_fq_codel zfs(PO) zunicode(PO) zavl(PO) zcommon(PO) znvpair(PO) spl(O) hid_generic usbhid hid sd_mod crc_t10dif crct10dif_common atkbd libps2 megaraid_sas ahci isci libahci libsas ehci_pci ehci_hcd xhci_hcd libata mpt2sas raid_class scsi_transport_sas usbcore scsi_mod usb_common i8042 serio bridge stp llc vhost_net tun vhost macvtap macvlan
[  250.824223] CPU: 0 PID: 1305 Comm: qemu:win171 Tainted: P           O   3.17.8-2-ARCH #1
[  250.832308] Hardware name: Supermicro X9DA7/E/X9DA7/E, BIOS 3.0a 07/02/2014
[  250.839270] task: ffff882027203250 ti: ffff880c3facc000 task.ti: ffff880c3facc000
[  250.846752] RIP: 0010:[<ffffffff812dd5ff>]  [<ffffffff812dd5ff>] pci_restore_ats_state+0x6f/0x80
[  250.855562] RSP: 0018:ffff880c3facfd80  EFLAGS: 00010246
[  250.860885] RAX: 0000000000000000 RBX: ffff8820283cb000 RCX: 0000000000000ffc
[  250.868019] RDX: 00000000000000ff RSI: 0000000000000246 RDI: 0000000000000246
[  250.875163] RBP: ffff880c3facfd88 R08: 0000000000000004 R09: ffff880c3facfd04
[  250.882309] R10: 0000000000000000 R11: 0000000000000246 R12: ffff8820283cb000
[  250.889443] R13: 0000000000000001 R14: 0000000000000000 R15: 0000000000000000
[  250.896572] FS:  00007f7df0a78940(0000) GS:ffff88103fc00000(0000) knlGS:0000000000000000
[  250.904675] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[  250.910417] CR2: 00007f79437fdc88 CR3: 0000002027240000 CR4: 00000000001427f0
[  250.917562] Stack:
[  250.919579]  0000000000000000 ffff880c3facfda8 ffffffff812bde59 ffff8820283cb000
[  250.927035]  ffff88202844ac28 ffff880c3facfdc0 ffffffff812be0a5 ffff8820283cb000
[  250.934510]  ffff880c3facfde0 ffffffff812be0d8 ffff88202844ac00 ffff880c3facfe50
[  250.941967] Call Trace:
[  250.944426]  [<ffffffff812bde59>] pci_restore_state.part.31+0x29/0x210
[  250.950951]  [<ffffffff812be0a5>] pci_dev_restore+0x45/0x50
[  250.956543]  [<ffffffff812be0d8>] pci_bus_restore+0x28/0x50
[  250.962120]  [<ffffffff812c028e>] pci_try_reset_bus+0x3e/0x70
[  250.967895]  [<ffffffffa04333bc>] vfio_pci_ioctl+0x9ac/0xa00 [vfio_pci]
[  250.974508]  [<ffffffffa04323b0>] ? vfio_pci_mmap+0x1f0/0x1f0 [vfio_pci]
[  250.981239]  [<ffffffff811ea3d8>] ? fsnotify+0x228/0x2f0
[  250.986552]  [<ffffffffa0421283>] vfio_device_fops_unl_ioctl+0x23/0x30 [vfio]
[  250.993708]  [<ffffffff811c01d0>] do_vfs_ioctl+0x2e0/0x4c0
[  250.999192]  [<ffffffff811ca08e>] ? __fget+0x6e/0xb0
[  251.004175]  [<ffffffff811c0431>] SyS_ioctl+0x81/0xa0
[  251.009239]  [<ffffffff81502129>] system_call_fastpath+0x16/0x1b
[  251.015263] Code: 8b 73 38 48 8b 7b 10 83 c2 06 e8 4d a4 fd ff 5b 5d c3 66 2e 0f 1f 84 00 00 00 00 00 0f b7 70 04 8d 4e f4 83 e1 1f 80 cd 80 eb d3 <0f> 0b 66 66 66 66 66 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 
[  251.035239] RIP  [<ffffffff812dd5ff>] pci_restore_ats_state+0x6f/0x80
[  251.041697]  RSP <ffff880c3facfd80>
[  251.045286] ---[ end trace ff4dde6dedc315f3 ]---

My kernel is 3.17.8 , I'm planning upgrade to 3.18 "any moment now" but due to zfs used as a root filesystem this requires little work. This only happens if I restart virtual machine which is using FirePro W7100 - restart of the other virtual machine using FirePro V4900 does not trigger this bug. After hitting this bug it is not possible to gracefully shutdown qemu.

Last edited by Bronek (2015-01-30 22:21:58)

aw

Member

Registered: 2013-10-04

Posts: 921

Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

Ok so I've finally setup serial console and my host is finally headless. I was able to capture this when restarting virtual machine, which is Windows 7 (no EFI). This happens when VM is started after guest reboot

[  250.730914] kernel BUG at drivers/pci/ats.c:138!

My kernel is 3.17.8 , I'm planning upgrade to 3.18 "any moment now" but due to zfs used as a root filesystem this requires little work. This only happens if I restart virtual machine which is using FirePro W7100 - restart of the other virtual machine using FirePro V4900 does not trigger this bug. After hitting this bug it is not possible to gracefully shutdown qemu.

That's saying that the device had ATS (Address Translation Services) enabled before reset, but when we went to restore the state of it after reset, we couldn't even find the capability.  Maybe the device never really recovers from reset?

---

http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

cdrjameson

Member

Registered: 2013-05-19

Posts: 46

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

I am attempting to follow aw's blog and have been following this thread on and off for a long time.

When attempting to follow this part of his instructions:

Make the first line of the XML look like this:

<domain type='kvm' xmlns:qemu='http://libvirt.org/schemas/domain/qemu/1.0'>

The xmlns specification is the part that needs to be added.  Next we need to add OVMF, just before the </domain> close at the end of the file, add something like this:

  <qemu:commandline>
    <qemu:arg value='-drive'/>
    <qemu:arg value='if=pflash,format=raw,readonly,file=/usr/share/edk2.git/ovmf-x64/OVMF-pure-efi.fd'/>
  </qemu:commandline>

Adjust the path as necessary if you're using a different installation.

I cannot find a file called OVMF-pure-efi.fd anywhere on my system. I have tried:

sudo find / -name 'OVMF*'
/home/.bigstore/Downloads/ovmf/ovmf-svn/src/tianocore-edk2-svn_build/Build/OvmfIa32/RELEASE_GCC49/FV/OVMF.fd
/home/.bigstore/Downloads/ovmf/ovmf-svn/src/tianocore-edk2-svn_build/Build/OvmfIa32/RELEASE_GCC49/FV/OVMF_CODE.fd
/home/.bigstore/Downloads/ovmf/ovmf-svn/src/tianocore-edk2-svn_build/Build/OvmfIa32/RELEASE_GCC49/FV/OVMF_VARS.fd
/home/.bigstore/Downloads/ovmf/ovmf-svn/src/tianocore-edk2-svn_build/Build/OvmfX64/RELEASE_GCC49/FV/OVMF.fd
/home/.bigstore/Downloads/ovmf/ovmf-svn/src/tianocore-edk2-svn_build/Build/OvmfX64/RELEASE_GCC49/FV/OVMF_CODE.fd
/home/.bigstore/Downloads/ovmf/ovmf-svn/src/tianocore-edk2-svn_build/Build/OvmfX64/RELEASE_GCC49/FV/OVMF_VARS.fd

Have I missed something in the installation of the ovmf-svn package? Or which of my installed files do I need to point it to?
P.S I am new to this game if my basic question didn't give it away.

Last edited by cdrjameson (2015-01-30 23:25:53)

aw

Member

Registered: 2013-10-04

Posts: 921

Website

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

I am attempting to follow aw's blog and have been following this thread on and off for a long time.

When attempting to follow this part of his instructions:

Make the first line of the XML look like this:

<domain type='kvm' xmlns:qemu='http://libvirt.org/schemas/domain/qemu/1.0'>

The xmlns specification is the part that needs to be added.  Next we need to add OVMF, just before the </domain> close at the end of the file, add something like this:

  <qemu:commandline>
    <qemu:arg value='-drive'/>
    <qemu:arg value='if=pflash,format=raw,readonly,file=/usr/share/edk2.git/ovmf-x64/OVMF-pure-efi.fd'/>
  </qemu:commandline>

Adjust the path as necessary if you're using a different installation.

I cannot find a file called OVMF-pure-efi.fd anywhere on my system. I have tried:

sudo find / -name 'OVMF*'
/home/.bigstore/Downloads/ovmf/ovmf-svn/src/tianocore-edk2-svn_build/Build/OvmfIa32/RELEASE_GCC49/FV/OVMF.fd
/home/.bigstore/Downloads/ovmf/ovmf-svn/src/tianocore-edk2-svn_build/Build/OvmfIa32/RELEASE_GCC49/FV/OVMF_CODE.fd
/home/.bigstore/Downloads/ovmf/ovmf-svn/src/tianocore-edk2-svn_build/Build/OvmfIa32/RELEASE_GCC49/FV/OVMF_VARS.fd
/home/.bigstore/Downloads/ovmf/ovmf-svn/src/tianocore-edk2-svn_build/Build/OvmfX64/RELEASE_GCC49/FV/OVMF.fd
/home/.bigstore/Downloads/ovmf/ovmf-svn/src/tianocore-edk2-svn_build/Build/OvmfX64/RELEASE_GCC49/FV/OVMF_CODE.fd
/home/.bigstore/Downloads/ovmf/ovmf-svn/src/tianocore-edk2-svn_build/Build/OvmfX64/RELEASE_GCC49/FV/OVMF_VARS.fd

Have I missed something in the installation of the ovmf-svn package? Or which of my installed files do I need to point it to?

I'd try the latter set, but you're missing an update to those instructions here:

http://vfio.blogspot.com/2014/09/libvir … -ovmf.html

My slides from KVM Forum include a further updated setup (hit space a couple times):

http://awilliam.github.io/presentations … 2014/#/4/2

The "pure" name indicates that it doesn't include the CSM (Compatibility Support Module), which would provide the legacy support that we're trying to avoid.  Upstream and your distro provided packages may name it differently.  There are really very, very few reasons that anyone here should be using <qemu:arg> options any longer.  If you are, you're probably doing something wrong.

---

http://vfio.blogspot.com
Looking for a more open forum to discuss vfio related uses?  Try https://www.redhat.com/mailman/listinfo/vfio-users

mostlyharmless

Member

Registered: 2014-01-16

Posts: 72

Re: KVM VGA-Passthrough using the new vfio-vga support in kernel =>3.9

There are really very, very few reasons that anyone here should be using <qemu:arg> options any longer.  If you are, you're probably doing something wrong.

Correct me if I'm wrong, but Win 7 guests and below who use libvirt will still use qemu:arg since apparently OVMF + EFI + Win 7 seems pretty fragile, at least from what I've read here.  So you're referring to Win 8+

Last edited by mostlyharmless (2015-01-30 23:51:29)