You are not logged in.

#1 2016-07-10 16:30:45

maltejahn
Member
Registered: 2016-07-10
Posts: 13

Setting up KVM VGA Passthrough

Hi,

im quite new to Arch and im trying to set up a Server with multiple VMs.
One VM should be WIN10 and the second graphic adapter should only handle this VM.

Sorry for this long post, but i tried to give as much information to as possible.

I tried to follow this:
https://wiki.archlinux.org/index.php/PC … h_via_OVMF

I can boot Win10, but i wasnt able to install the graphic driver in Win10 (no Error 43). The Device Manager shows "Microsoft basic display adapter"
I hope i can later use the patch mentioned on the Wiki to get the new graphic adapter to work. Or is this already my problem?

I tried to install the drivers for the card, the installation process started but ended in something like "Install wasnt successful". A right click on this standard graphic adapter showed me the corret id 10de:1b81. So i thought the graphic adapter was passthroughed correctly.

To be honest, i found a lot of tutorials where pci-stub and vfio-pci is mixed? For my "latest" Arch installation only vfio-pci should be used?
I hope there are a few tweaks possible. Using a Benchmark, writing to the disk is about 30 mb/s. But thats not my biggest problem right now.


What i did so far

1. Enable IOMMU

dmesg|grep -e DMAR -e IOMMU
[    0.000000] ACPI: DMAR 0x000000003E941B50 0000F0 (v01 INTEL  SKL      00000001 INTL 00000001)
[    0.000000] DMAR: IOMMU enabled
[    0.045982] DMAR: Host address width 39
[    0.045983] DMAR: DRHD base: 0x000000fed90000 flags: 0x0
[    0.045988] DMAR: dmar0: reg_base_addr fed90000 ver 1:0 cap 1c0000c40660462 ecap 7e3ff0505e
[    0.045989] DMAR: DRHD base: 0x000000fed91000 flags: 0x1
[    0.045991] DMAR: dmar1: reg_base_addr fed91000 ver 1:0 cap d2008c40660462 ecap f050da
[    0.045992] DMAR: RMRR base: 0x0000003dd6d000 end: 0x0000003dd8cfff
[    0.045993] DMAR: RMRR base: 0x0000003f800000 end: 0x0000007fffffff
[    0.045994] DMAR: ANDD device: 1 name: \_SB.PCI0.I2C0
[    0.045995] DMAR: ANDD device: 2 name: \_SB.PCI0.I2C1
[    0.045996] DMAR-IR: IOAPIC id 2 under DRHD base  0xfed91000 IOMMU 1
[    0.045996] DMAR-IR: HPET id 0 under DRHD base 0xfed91000
[    0.045997] DMAR-IR: Queued invalidation will be enabled to support x2apic and Intr-remapping.
[    0.047336] DMAR-IR: Enabled IRQ remapping in x2apic mode
[    0.989908] DMAR: ACPI device "device:6f" under DMAR at fed91000 as 00:15.0
[    0.989911] DMAR: ACPI device "device:70" under DMAR at fed91000 as 00:15.1
[    0.989917] DMAR: No ATSR found
[    0.989973] DMAR: dmar0: Using Queued invalidation
[    0.990154] DMAR: dmar1: Using Queued invalidation
[    0.990158] DMAR: Setting RMRR:
[    0.990185] DMAR: Setting identity map for device 0000:00:02.0 [0x3f800000 - 0x7fffffff]
[    0.990195] DMAR: Setting identity map for device 0000:00:14.0 [0x3dd6d000 - 0x3dd8cfff]
[    0.990200] DMAR: Prepare 0-16MiB unity mapping for LPC
[    0.990205] DMAR: Setting identity map for device 0000:00:1f.0 [0x0 - 0xffffff]
[    0.990216] DMAR: Intel(R) Virtualization Technology for Directed I/O


-> so far, VT-d seems to be working

2. Groups?

for iommu_group in $(find /sys/kernel/iommu_groups/ -maxdepth 1 -mindepth 1 -type d); do echo "IOMMU group $(basename "$iommu_group")"; for device in $(ls -1 "$iommu_group"/devices/); do echo -n $'\t'; lspci -nns "$device"; done; done
IOMMU group 0
	00:00.0 Host bridge [0600]: Intel Corporation Skylake Host Bridge/DRAM Registers [8086:1918] (rev 07)
IOMMU group 1
	00:01.0 PCI bridge [0604]: Intel Corporation Skylake PCIe Controller (x16) [8086:1901] (rev 07)
	01:00.0 VGA compatible controller [0300]: NVIDIA Corporation Device [10de:1b81] (rev a1)
	01:00.1 Audio device [0403]: NVIDIA Corporation Device [10de:10f0] (rev a1)
....

This is a new NVIDIA 1070. So audio and graphic is in this group

3. So try if all worked fine

modinfo vfio-pci
filename:       /lib/modules/4.6.3-1-ARCH/kernel/drivers/vfio/pci/vfio-pci.ko.gz
description:    VFIO PCI - User Level meta-driver
author:         Alex Williamson <alex.williamson@redhat.com>

4. Add to /etc/modprobe.d/vfio.conf

options vfio-pci ids=10de:1b81,10de:10f0

5. going on...:

/etc/mkinitcpio.conf
Modify: MODULES="... vfio vfio_iommu_type1 vfio_pci vfio_virqfd ..."
Modify: HOOKS="... modconf ..."

Do: mkinitcpio -p linux

6. Check

dmesg | grep -i vfio 
[    1.182553] VFIO - User Level meta-driver version: 0.3
[    1.198908] vfio_pci: add [10de:1b81[ffff:ffff]] class 0x000000/00000000
[    1.212182] vfio_pci: add [10de:10f0[ffff:ffff]] class 0x000000/00000000
[   37.152439] vfio-pci 0000:01:00.0: enabling device (0000 -> 0003)
[   37.152663] vfio_ecap_init: 0000:01:00.0 hiding ecap 0x19@0x900


lspci -nnk -d 10de:1b81
01:00.0 VGA compatible controller [0300]: NVIDIA Corporation Device [10de:1b81] (rev a1)
	Subsystem: Micro-Star International Co., Ltd. [MSI] Device [1462:3302]
	Kernel driver in use: vfio-pci
	Kernel modules: nouveau

lspci -nnk -d 10de:10f0
01:00.1 Audio device [0403]: NVIDIA Corporation Device [10de:10f0] (rev a1)
	Subsystem: Micro-Star International Co., Ltd. [MSI] Device [1462:3302]
	Kernel driver in use: vfio-pci
	Kernel modules: snd_hda_intel

-> is it ok that one "nouveau" appears under 1:00.0?

7. I added OVMF into /etc/libvirt/qemu.conf

nvram = [
	"/usr/share/ovmf/x64/ovmf_x64.bin:/usr/share/ovmf/x64/ovmf_vars_x64.bin"
]

8. I modified my libvirt conf file after i created a basic one using virt-manager. I also added a Rom of the graphic adapter

<!--
WARNING: THIS IS AN AUTO-GENERATED FILE. CHANGES TO IT ARE LIKELY TO BE
OVERWRITTEN AND LOST. Changes to this xml configuration should be made using:
  virsh edit Windows10
or other application using the libvirt API.
-->

<domain type='kvm' xmlns:qemu='http://libvirt.org/schemas/domain/qemu/1.0'>
  <name>Windows10</name>
  <uuid>037f2d00-67d2-401a-ae2f-eaa75404fd67</uuid>
  <memory unit='KiB'>16384000</memory>
  <currentMemory unit='KiB'>16384000</currentMemory>
  <vcpu placement='static'>6</vcpu>
  <os>
    <type arch='x86_64' machine='pc-i440fx-2.6'>hvm</type>
    <loader readonly='yes' type='pflash'>usr/share/ovmf/x64/ovmf_x64.bin</loader>
    <nvram>/var/lib/libvirt/qemu/nvram/Windows10_VARS.fd</nvram>
    <boot dev='hd'/>
  </os>
  <features>
    <acpi/>
    <apic/>
    <vmport state='off'/>
  </features>
  <cpu mode='host-model'>
    <model fallback='allow'/>
  </cpu>
  <clock offset='utc'>
    <timer name='rtc' tickpolicy='catchup'/>
    <timer name='pit' tickpolicy='delay'/>
    <timer name='hpet' present='no'/>
  </clock>
  <on_poweroff>destroy</on_poweroff>
  <on_reboot>restart</on_reboot>
  <on_crash>restart</on_crash>
  <pm>
    <suspend-to-mem enabled='no'/>
    <suspend-to-disk enabled='no'/>
  </pm>
  <devices>
    <emulator>/usr/sbin/qemu-system-x86_64</emulator>
    <disk type='file' device='disk'>
      <driver name='qemu' type='qcow2'/>
      <source file='/var/lib/libvirt/images/Windows10.qcow2'/>
      <target dev='sda' bus='scsi'/>
      <address type='drive' controller='0' bus='0' target='0' unit='0'/>
    </disk>
    <disk type='file' device='cdrom'>
      <driver name='qemu' type='raw'/>
      <source file='/home/malte/Dokumente/ISOs/de_windows_10_multiple_editions_x64_dvd_6846954.iso'/>
      <target dev='hdb' bus='ide'/>
      <readonly/>
      <address type='drive' controller='0' bus='0' target='0' unit='1'/>
    </disk>
    <disk type='file' device='cdrom'>
      <driver name='qemu' type='raw'/>
      <source file='/home/malte/Downloads/virtio-win-0.1.102.iso'/>
      <target dev='hdc' bus='ide'/>
      <readonly/>
      <address type='drive' controller='0' bus='1' target='0' unit='0'/>
    </disk>
    <disk type='file' device='cdrom'>
      <driver name='qemu' type='raw'/>
      <source file='/home/malte/Downloads/virtio-win-0.1.118.iso'/>
      <target dev='hdd' bus='ide'/>
      <readonly/>
      <address type='drive' controller='0' bus='1' target='0' unit='1'/>
    </disk>
    <controller type='usb' index='0' model='ich9-ehci1'>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x05' function='0x7'/>
    </controller>
    <controller type='usb' index='0' model='ich9-uhci1'>
      <master startport='0'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x05' function='0x0' multifunction='on'/>
    </controller>
    <controller type='usb' index='0' model='ich9-uhci2'>
      <master startport='2'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x05' function='0x1'/>
    </controller>
    <controller type='usb' index='0' model='ich9-uhci3'>
      <master startport='4'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x05' function='0x2'/>
    </controller>
    <controller type='scsi' index='0' model='virtio-scsi'>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x04' function='0x0'/>
    </controller>
    <controller type='pci' index='0' model='pci-root'/>
    <controller type='ide' index='0'>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x1'/>
    </controller>
    <controller type='virtio-serial' index='0'>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x06' function='0x0'/>
    </controller>
    <interface type='direct'>
      <mac address='52:54:00:a8:91:64'/>
      <source dev='eth0' mode='bridge'/>
      <model type='rtl8139'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x0'/>
    </interface>
    <serial type='pty'>
      <target port='0'/>
    </serial>
    <console type='pty'>
      <target type='serial' port='0'/>
    </console>
    <channel type='spicevmc'>
      <target type='virtio' name='com.redhat.spice.0'/>
      <address type='virtio-serial' controller='0' bus='0' port='1'/>
    </channel>
    <input type='mouse' bus='ps2'/>
    <input type='keyboard' bus='ps2'/>
    <sound model='ich6'>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x0'/>
    </sound>
    <hostdev mode='subsystem' type='pci' managed='yes'>
      <source>
        <address domain='0x0000' bus='0x01' slot='0x00' function='0x0'/>
      </source>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x07' function='0x0'/>
    </hostdev>
    <hostdev mode='subsystem' type='pci' managed='yes'>
      <source>
        <address domain='0x0000' bus='0x01' slot='0x00' function='0x1'/>
      </source>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x08' function='0x0'/>
    </hostdev>
    <hostdev mode='subsystem' type='pci' managed='yes'>
      <source>
        <address domain='0x0000' bus='0x03' slot='0x00' function='0x0'/>
      </source>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x09' function='0x0'/>
    </hostdev>
    <redirdev bus='usb' type='spicevmc'>
    </redirdev>
    <redirdev bus='usb' type='spicevmc'>
    </redirdev>
    <memballoon model='virtio'>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x0a' function='0x0'/>
    </memballoon>
  </devices>
<qemu:commandline>
    <qemu:arg value='-device'/>
    <qemu:arg value='vfio-pci,host=01:00.0,multifunction=on,x-vga=on,romfile=/home/malte/Dokumente/ISOs/MSI.GTX1070.8192.160602.rom'/>
    <qemu:arg value='-device'/>
    <qemu:arg value='vfio-pci,host=01:00.1'/>
</qemu:commandline>
</domain>

Last edited by maltejahn (2016-07-10 16:32:08)

Offline

#2 2016-07-11 09:42:09

Omar007
Member
Registered: 2015-04-09
Posts: 357

Re: Setting up KVM VGA Passthrough

On that wiki page is a section specifically about nVidia issues (Error 43). Did you try that anyway or not at all?
https://wiki.archlinux.org/index.php/PC … indows_VMs

And for 6), yes it is normal for nouveau to show as an item in the kernel modules property, since that is the kernel module for that card. However, as you can see the currently loaded driver is vfio_pci and not nouveau so everything seems good.
(The same goes for the sound; kernel module snd_hda_intel but vfio_pci is used)


EDIT: Btw, about 8), I believe you can use virt-manager to assign the PCI-e device as well. No need to manually edit iirc. Furthermore, I've personally never had to assign a ROM file before. Have you tried without?
Lastly, if you are getting output on the GPU, you could remove the SPICE/VGA/VNC GPU components from the VM (if you haven't done so already).

Last edited by Omar007 (2016-07-11 09:52:11)

Offline

#3 2016-07-11 16:53:59

maltejahn
Member
Registered: 2016-07-10
Posts: 13

Re: Setting up KVM VGA Passthrough

Hi,

i would be happy to get an error "43", but till now the device manager in Win10 only shows me a "Microsoft... adapter". I would have expected something like "NVIDIA 1070..." where a get under Details a "43" error

Maybe im confused by the amount of "tutorials" and i am a bit lost here...
I hoped to tell Windows that there is a Nvidia Adapter if i use a rom file.

I already removed unnecessary things like you mentioned. Things are going well since i only get an output on the 1070 graphic adapter

I even dont know if i should use OVMF or SeaBios as i found some websites saying it would be better to use Seabios for vga passthrough. But i have no idea how to switch from OVMF to SeaBios. Is it just the flag x-vga=on?
And.. I dont even know - do i need a Script to unbind/bind as the other output say vfio-pci is configured well.

First, the CPU doesnt support ACS.  But since the NVidia adapter is on its own group, there shouldnt be a problem (as i understand). I got a old "FX 1800" which i wanted to try. But this card appears also in group 1 with the other NVIDIA Card. A quick "pcie_acs_override = downstream" in syslinux hasnt changed the output.

Finally. Maybe its just not possible (till now) to use the card and i have to wait a while till hopefully it is supported

Maybe a few words about the PC and what im trying to do
E3-1245V5
Intel Graphic Adapter
Nvidia 1070
32 Ram
Additional USB PCI - Express Card to have a separate USB

Later there should be at least:
1) Win 10 or Win8 or Win 7 for gaming and doing CAD work. Only this VM should use the Nvidia adapter
2) NAS, i think im want to use OpenMediaVault
3) e.g. Ubuntu with Kodi using the integrated Intel graphic
4) a Web Server, Maybe a dedicated SVN Server,... Cloud (maybe integrated into OMV, dont know till yet)

<!--
WARNING: THIS IS AN AUTO-GENERATED FILE. CHANGES TO IT ARE LIKELY TO BE
OVERWRITTEN AND LOST. Changes to this xml configuration should be made using:
  virsh edit Windows10
or other application using the libvirt API.
-->

<domain type='kvm'>
  <name>Windows10</name>
  <uuid>037f2d00-67d2-401a-ae2f-eaa75404fd67</uuid>
  <memory unit='KiB'>16384000</memory>
  <currentMemory unit='KiB'>16384000</currentMemory>
  <vcpu placement='static'>6</vcpu>
  <os>
    <type arch='x86_64' machine='pc-i440fx-2.6'>hvm</type>
    <loader readonly='yes' type='pflash'>usr/share/ovmf/x64/ovmf_x64.bin</loader>
    <nvram>/var/lib/libvirt/qemu/nvram/Windows10_VARS.fd</nvram>
    <boot dev='hd'/>
  </os>
  <features>
    <acpi/>
    <apic/>
    <vmport state='off'/>
    <kvm>
      <hidden state='on'/>
    </kvm>
  </features>
  <cpu mode='host-model'>
    <model fallback='allow'/>
  </cpu>
  <clock offset='utc'>
    <timer name='rtc' tickpolicy='catchup'/>
    <timer name='pit' tickpolicy='delay'/>
    <timer name='hpet' present='no'/>
  </clock>
  <on_poweroff>destroy</on_poweroff>
  <on_reboot>restart</on_reboot>
  <on_crash>restart</on_crash>
  <pm>
    <suspend-to-mem enabled='no'/>
    <suspend-to-disk enabled='no'/>
  </pm>
  <devices>
    <emulator>/usr/sbin/qemu-system-x86_64</emulator>
    <disk type='file' device='disk'>
      <driver name='qemu' type='qcow2'/>
      <source file='/var/lib/libvirt/images/Windows10.qcow2'/>
      <target dev='sda' bus='scsi'/>
      <address type='drive' controller='0' bus='0' target='0' unit='0'/>
    </disk>
    <disk type='file' device='cdrom'>
      <driver name='qemu' type='raw'/>
      <source file='/home/malte/Dokumente/ISOs/de_windows_10_multiple_editions_x64_dvd_6846954.iso'/>
      <target dev='hdb' bus='ide'/>
      <readonly/>
      <address type='drive' controller='0' bus='0' target='0' unit='1'/>
    </disk>
    <disk type='file' device='cdrom'>
      <driver name='qemu' type='raw'/>
      <source file='/home/malte/Downloads/virtio-win-0.1.102.iso'/>
      <target dev='hdc' bus='ide'/>
      <readonly/>
      <address type='drive' controller='0' bus='1' target='0' unit='0'/>
    </disk>
    <disk type='file' device='cdrom'>
      <driver name='qemu' type='raw'/>
      <source file='/home/malte/Downloads/virtio-win-0.1.118.iso'/>
      <target dev='hdd' bus='ide'/>
      <readonly/>
      <address type='drive' controller='0' bus='1' target='0' unit='1'/>
    </disk>
    <controller type='usb' index='0' model='ich9-ehci1'>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x05' function='0x7'/>
    </controller>
    <controller type='usb' index='0' model='ich9-uhci1'>
      <master startport='0'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x05' function='0x0' multifunction='on'/>
    </controller>
    <controller type='usb' index='0' model='ich9-uhci2'>
      <master startport='2'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x05' function='0x1'/>
    </controller>
    <controller type='usb' index='0' model='ich9-uhci3'>
      <master startport='4'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x05' function='0x2'/>
    </controller>
    <controller type='scsi' index='0' model='virtio-scsi'>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x04' function='0x0'/>
    </controller>
    <controller type='pci' index='0' model='pci-root'/>
    <controller type='ide' index='0'>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x1'/>
    </controller>
    <controller type='virtio-serial' index='0'>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x06' function='0x0'/>
    </controller>
    <interface type='direct'>
      <mac address='52:54:00:a8:91:64'/>
      <source dev='eth0' mode='bridge'/>
      <model type='rtl8139'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x0'/>
    </interface>
    <serial type='pty'>
      <target port='0'/>
    </serial>
    <console type='pty'>
      <target type='serial' port='0'/>
    </console>
    <channel type='spicevmc'>
      <target type='virtio' name='com.redhat.spice.0'/>
      <address type='virtio-serial' controller='0' bus='0' port='1'/>
    </channel>
    <input type='mouse' bus='ps2'/>
    <input type='keyboard' bus='ps2'/>
    <sound model='ich6'>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x0'/>
    </sound>
    <hostdev mode='subsystem' type='pci' managed='yes'>
      <source>
        <address domain='0x0000' bus='0x03' slot='0x00' function='0x0'/>
      </source>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x09' function='0x0'/>
    </hostdev>
    <hostdev mode='subsystem' type='pci' managed='yes'>
      <source>
        <address domain='0x0000' bus='0x01' slot='0x00' function='0x0'/>
      </source>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x07' function='0x0'/>
    </hostdev>
    <hostdev mode='subsystem' type='pci' managed='yes'>
      <source>
        <address domain='0x0000' bus='0x01' slot='0x00' function='0x1'/>
      </source>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x08' function='0x0'/>
    </hostdev>
    <redirdev bus='usb' type='spicevmc'>
    </redirdev>
    <redirdev bus='usb' type='spicevmc'>
    </redirdev>
    <memballoon model='virtio'>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x0a' function='0x0'/>
    </memballoon>
  </devices>
  <qemu:commandline>
    <qemu:arg value='-device'/>
    <qemu:arg value='vfio-pci,host=01:00.0,multifunction=on,x-vga=on'/>
    <qemu:arg value='-device'/>
    <qemu:arg value='vfio-pci,host=01:00.1'/>
  </qemu:commandline>
</domain>

Offline

#4 2016-07-11 23:34:57

Omar007
Member
Registered: 2015-04-09
Posts: 357

Re: Setting up KVM VGA Passthrough

OVMF or SeaBios is determined by your loader, which in your case is OVMF as far as I can see (<loader readonly='yes' type='pflash'>usr/share/ovmf/x64/ovmf_x64.bin</loader>).
I have personally always used OVMF with my VMs running any Windows >7 or when doing anything with device passthrough.

Since you're getting output of the VM on the GPU it seems everything on the passthrough side is working fine. You'd only need a script to bind/unbind if you aren't binding your GPU to vfio-pci directly, which you are so you don't need one.
Atm I'm assuming it's something within the VM (since it shows the Microsoft driver instead of the nVidia driver). I'd almost say try it again just in case or instead of using the nVidia download, see if you can let windows search for the driver through windows update. Maybe that will result in different behaviour or maybe some different output.

From a look at your last config I can't really see anything out of the ordinary (although, iirc, multifuction=on and x-vga=on haven't been needed since quite a while either).
I'll check tomorrow if I can pull up one of my VGA passthrough configs to check for any discrepancies. The only thing I know for sure is that back then I had to make sure the drivers would not be able to tell it was running in a VM by setting the hidden state and disabling some hypervisor features (which is now replaced by basically a one-liner as stated in the WIKI under the Error 43 section).

In the meantime, there is also apparently this now: https://github.com/sk1080/nvidia-kvm-patcher
Then you would not need to use the hidden state flag anymore either.

Last edited by Omar007 (2016-07-11 23:43:43)

Offline

#5 2016-07-12 17:31:44

maltejahn
Member
Registered: 2016-07-10
Posts: 13

Re: Setting up KVM VGA Passthrough

Hi,

thanks for you support. I was able to to this Patch to the Nvidia driver mentioned. I have also seen "Nvidia 1070" in my Device Manager. But maybe thats not my real problem at the moment. Windows 10 tries to update, but fails. And in the end Win10 reboots again and again. Im going to install Windows 7 to see if it change things

I will be back in a while. Thans for your support!

Offline

#6 2016-07-12 22:38:24

Omar007
Member
Registered: 2015-04-09
Posts: 357

Re: Setting up KVM VGA Passthrough

I've checked my VM config and the main differences I can see with yours are:
1) I'm disabling certain hypervisor functionality due to Error 43.
2) I'm using 'host-passthrough' for the CPU.
3) I'm handling PCIe devices through libvirt as well instead of through QEMU args (GPU and GPU Audio @ 01:00.0 and 01:00.1):

...
    <hostdev mode='subsystem' type='pci' managed='yes'>
      <source>
        <address domain='0x0000' bus='0x01' slot='0x00' function='0x0'/>
      </source>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x0'/>
    </hostdev>
    <hostdev mode='subsystem' type='pci' managed='yes'>
      <source>
        <address domain='0x0000' bus='0x01' slot='0x00' function='0x1'/>
      </source>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x09' function='0x0'/>
    </hostdev>
...

As I said before though this was made quite a while ago and there are now other solutions for 1) that result in better VM performance. 2) and 3) are changes you could make to your config if you want but neither should be impacting whether or not driver installation in Windows is working or not; from what I can see the VM config itself should really be fine.

Last edited by Omar007 (2016-07-12 22:43:59)

Offline

#7 2016-07-14 17:19:19

maltejahn
Member
Registered: 2016-07-10
Posts: 13

Re: Setting up KVM VGA Passthrough

Hi,

so after a while i saw a little progress...
I am now able to see a second "Microsoft Basic Display Adapter" and if i try to install a driver, it recognizes it as a GTX1070. But it was hard to get there (for me)
The Main issue with Win10 was the automatic upgrade. It tried to download automatically the needed drivers. I disabled the passthrough of my GTX1070 for my VM, did all upgrades and disabled automatic driver download/installation.

But, strange, Win10 ignored it and installed a driver for my 1070 which ended in a bluescreen which says there is a problem with "nvlddmkm.sys". I detached the graphic card from the VM and then i was able to start Win10 again.

Then i deleted the nvlddmkm.sys and tried to install the patched NVidia Driver. I selected a file to install instead to allow Win10 to find a driver automatically. The patched Driver then shows up a "compatible", but without signature. And I am not able to install it with a "strange" error saying something like "error .. wasnt able to add driver to memory

So, then i disabled signature test of Win10 "Disable driver signature enforcement”
Now, i got the error during driver installation "The required line was not found in the INF"

I tried to be very smart - without success
i looked for an entry "1b81..." which i found, but with an different SUBSYS. So i changed that entry. It tried to install, but failed with the error again
http://forums.laptopvideo2go.com/topic/ … inf-files/

Also strange. Im looking for Subsys 3302-1462, but cant find this text anywhere in the NVidia driver folder
Strange, i cant find a 1b81 in the MSI Driver File which should be NVMIi.inf


So i have choosen a inf file with an 1b81 Device an edited two lines. I repaced the Subsys numbers with my one. I looked like it start to install (red window which says dont install a driver with a bad signature), but still "The required line was not found in the INF". One time i got a "not supported by this windows version"

So, maybe someone has an idea...

Heres is my current xml file. I added a partition instead a file. That improved alot. Now i get something

<!--
WARNING: THIS IS AN AUTO-GENERATED FILE. CHANGES TO IT ARE LIKELY TO BE
OVERWRITTEN AND LOST. Changes to this xml configuration should be made using:
  virsh edit Win10VM
or other application using the libvirt API.
-->

<domain type='kvm'>
  <name>Win10VM</name>
  <uuid>700b27b9-2d95-4402-8e46-e72aae8a4c26</uuid>
  <memory unit='KiB'>16384000</memory>
  <currentMemory unit='KiB'>16384000</currentMemory>
  <vcpu placement='static'>6</vcpu>
  <os>
    <type arch='x86_64' machine='pc-i440fx-2.6'>hvm</type>
    <loader readonly='yes' type='pflash'>/usr/share/ovmf/x64/ovmf_x64.bin</loader>
    <nvram>/var/lib/libvirt/qemu/nvram/Win10VM_VARS.fd</nvram>
    <boot dev='hd'/>
  </os>
  <features>
    <acpi/>
    <apic/>
    <hyperv>
      <relaxed state='on'/>
      <vapic state='on'/>
      <spinlocks state='on' retries='8191'/>
    </hyperv>
    <kvm>
      <hidden state='on'/>
    </kvm>
    <vmport state='off'/>
  </features>
  <cpu mode='custom' match='exact'>
    <model fallback='allow'>Skylake-Client</model>
  </cpu>
  <clock offset='localtime'>
    <timer name='rtc' tickpolicy='catchup'/>
    <timer name='pit' tickpolicy='delay'/>
    <timer name='hpet' present='no'/>
    <timer name='hypervclock' present='yes'/>
  </clock>
  <on_poweroff>destroy</on_poweroff>
  <on_reboot>restart</on_reboot>
  <on_crash>restart</on_crash>
  <pm>
    <suspend-to-mem enabled='no'/>
    <suspend-to-disk enabled='no'/>
  </pm>
  <devices>
    <emulator>/usr/sbin/qemu-system-x86_64</emulator>
    <disk type='block' device='disk'>
      <driver name='qemu' type='raw'/>
      <source dev='/dev/nvme0n1p3'/>
      <target dev='vda' bus='virtio'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x09' function='0x0'/>
    </disk>
    <disk type='file' device='cdrom'>
      <driver name='qemu' type='raw'/>
      <target dev='hdb' bus='ide'/>
      <readonly/>
      <address type='drive' controller='0' bus='0' target='0' unit='1'/>
    </disk>
    <disk type='file' device='cdrom'>
      <driver name='qemu' type='raw'/>
      <source file='/home/malte/Dokumente/virtio-win-0.1.102.iso'/>
      <target dev='hdc' bus='ide'/>
      <readonly/>
      <address type='drive' controller='0' bus='1' target='0' unit='0'/>
    </disk>
    <disk type='file' device='cdrom'>
      <driver name='qemu' type='raw'/>
      <source file='/home/malte/Dokumente/virtio-win-0.1.118.iso'/>
      <target dev='hdd' bus='ide'/>
      <readonly/>
      <address type='drive' controller='0' bus='1' target='0' unit='1'/>
    </disk>
    <controller type='usb' index='0' model='ich9-ehci1'>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x06' function='0x7'/>
    </controller>
    <controller type='usb' index='0' model='ich9-uhci1'>
      <master startport='0'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x06' function='0x0' multifunction='on'/>
    </controller>
    <controller type='usb' index='0' model='ich9-uhci2'>
      <master startport='2'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x06' function='0x1'/>
    </controller>
    <controller type='usb' index='0' model='ich9-uhci3'>
      <master startport='4'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x06' function='0x2'/>
    </controller>
    <controller type='scsi' index='0' model='virtio-scsi'>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x05' function='0x0'/>
    </controller>
    <controller type='pci' index='0' model='pci-root'/>
    <controller type='ide' index='0'>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x1'/>
    </controller>
    <controller type='virtio-serial' index='0'>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x07' function='0x0'/>
    </controller>
    <interface type='direct'>
      <mac address='52:54:00:0c:83:7d'/>
      <source dev='enp0s31f6' mode='bridge'/>
      <model type='rtl8139'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x0'/>
    </interface>
    <serial type='pty'>
      <target port='0'/>
    </serial>
    <console type='pty'>
      <target type='serial' port='0'/>
    </console>
    <channel type='spicevmc'>
      <target type='virtio' name='com.redhat.spice.0'/>
      <address type='virtio-serial' controller='0' bus='0' port='1'/>
    </channel>
    <input type='tablet' bus='usb'/>
    <input type='mouse' bus='ps2'/>
    <input type='keyboard' bus='ps2'/>
    <graphics type='spice' autoport='yes'>
      <listen type='address'/>
      <image compression='off'/>
    </graphics>
    <sound model='ich6'>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x04' function='0x0'/>
    </sound>
    <video>
      <model type='qxl' ram='65536' vram='65536' vgamem='16384' heads='1' primary='yes'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x0'/>
    </video>
    <hostdev mode='subsystem' type='pci' managed='yes'>
      <source>
        <address domain='0x0000' bus='0x03' slot='0x00' function='0x0'/>
      </source>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x0c' function='0x0'/>
    </hostdev>
    <hostdev mode='subsystem' type='pci' managed='yes'>
      <source>
        <address domain='0x0000' bus='0x01' slot='0x00' function='0x0'/>
      </source>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x0a' function='0x0'/>
    </hostdev>
    <redirdev bus='usb' type='spicevmc'>
    </redirdev>
    <redirdev bus='usb' type='spicevmc'>
    </redirdev>
    <memballoon model='virtio'>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x08' function='0x0'/>
    </memballoon>
  </devices>
</domain>

Sorry for that post, i tried the things during i wrote it down.

Offline

#8 2016-07-18 07:49:31

sk1080
Member
Registered: 2016-07-18
Posts: 4

Re: Setting up KVM VGA Passthrough

Seeing as people are starting to try to use it, I should add some bulletproofing to my patcher script. The testsigning crap is really iffy right now.
I would recommend blacklisting the nvidia updates from windows update.

As far as "The required line was not found in the INF", thats kinda interesting. Wonder what I screwed up. Going to probably give my script a re-run on a clean box and make sure im not forgetting something...
Suppose I should set it up for other OS also. I will add to my todo list for the week.

Also, on my system, after a few days, another windows update struck and replaced my modified driver causing bluescreen. Eventually it rolled it back after a few reboots though...

EDIT: I should note that before even bothering to try my script, you should be able to get it working by hiding kvm, that is, using hidden state=on, supposing your packages are up to date (latest normal packages should be fine, I don't have libvirt or qemu pulled from the AUR)

EDIT2: I did go ahead and update the readme on github a bit to hopefully cover some common problems, short of actually fixing them right now

Last edited by sk1080 (2016-07-18 08:07:50)

Offline

#9 2016-07-18 15:43:34

maltejahn
Member
Registered: 2016-07-10
Posts: 13

Re: Setting up KVM VGA Passthrough

Hi,

thanks again for your help.

Today i started again to install the driver.

I edited your patch file to use WDK 10 (since he needed files arent in the folder 8.0 and the update folder 8.1). There was now error during using your patch (with admin rights)

I tried to install it again. And for a few minutes it worked! I installed the patched driver with this special "ignore driver signature" using F8 key on startup. After rebooting and using F8 key again - there my NVidia worked well. For about 5 Minutes. Then it was freezing and rebooted. Back in Win 10 normal mode all was gone. Now i have a "Code 3. Driver is damaged or not enough memory". A quick "uninstall" and "remove driver" doenst helped that much.


1. I thought using your patch gives me a driver "package" with signatured driver. But it seems that Win10 still uses old drivers (driver details - there is a list with a bunch of files which should be emty - in my opinion if Win10 would have deleted the driver completely
2. I copied the file nvlddmkm.sys and replaced the one in windows/system32/driver. No i got the 43 error
3. Is there a really way to get rid of the driver?

Offline

#10 2016-07-18 22:49:44

sk1080
Member
Registered: 2016-07-18
Posts: 4

Re: Setting up KVM VGA Passthrough

So, first, id boot without the graphics card added, turn on "show hidden devices" in device manager, and uninstall all other instances of your graphics card checking that checkbox that says like "uninstall driver software" or whatever it says.

Second, what was the error when running the script?

EDIT: Argh, looks like no certmgr with the windows 10 WDK by itself.

Last edited by sk1080 (2016-07-18 22:53:13)

Offline

#11 2016-07-18 23:01:06

alphaniner
Member
From: Ancapistan
Registered: 2010-07-12
Posts: 2,810

Re: Setting up KVM VGA Passthrough

I think it's necessary to set the environment variable 'devmgr_show_nonpresent_devices' to 1 and select "show hidden devices" in order to show removed devices.

https://msdn.microsoft.com/en-us/librar … s.85).aspx


But whether the Constitution really be one thing, or another, this much is certain - that it has either authorized such a government as we have had, or has been powerless to prevent it. In either case, it is unfit to exist.
-Lysander Spooner

Offline

#12 2016-07-19 00:38:08

sk1080
Member
Registered: 2016-07-18
Posts: 4

Re: Setting up KVM VGA Passthrough

Just updated my repo with better testsigning logic by moving the cert generation/stuff to powershell, removed accidental dependency on VS tools in the process.

Offline

#13 2016-07-19 14:57:01

maltejahn
Member
Registered: 2016-07-10
Posts: 13

Re: Setting up KVM VGA Passthrough

Hi,

the patch doesnt work for me. The Signtool gives me:

Vertificate Path c:\Testsign.cer
Certificate Found in Root Store
+ Signing Catalog File
SignTool Error: No Certificates were found that met all the given criteria

Again it was possible to use the changed Driver using this "dont check for Driver signature" enviroment (F8).
When im back in normal mode the Driver is gone.

Offline

#14 2016-07-19 23:50:55

sk1080
Member
Registered: 2016-07-18
Posts: 4

Re: Setting up KVM VGA Passthrough

maltejahn wrote:

Hi,

the patch doesnt work for me. The Signtool gives me:

Vertificate Path c:\Testsign.cer
Certificate Found in Root Store
+ Signing Catalog File
SignTool Error: No Certificates were found that met all the given criteria

Again it was possible to use the changed Driver using this "dont check for Driver signature" enviroment (F8).
When im back in normal mode the Driver is gone.

Could you open up certmgr.msc, go to action->find certificates, and find and delete all instances of "SKSoftware"? Additionally, if there are instances of the cert, it should show up in three places (Trusted Root Certification Authorities, Personal, and Intermediate Certification Authorities). Additionally, if it is in root, check the details to see if Key Usage is "Digital Signature (80)". My guess is somehow the cert isn't getting added, but im not sure. I got a similar error when I had forgot "-Type CodeSigningCert" in New-SelfSignedCertificate. I did go ahead and run things through on a fresh windows 10 install with just python/the wdk and things went fine. I am however considering porting the rest over to powershell, as I really don't need to be making python a dep to this process...

Offline

#15 2016-07-20 15:17:47

maltejahn
Member
Registered: 2016-07-10
Posts: 13

Re: Setting up KVM VGA Passthrough

Hi,

after removing old certificates I was able to run the complete patch. Finally i was able to install 369.39. The driver signature was now correct.

Thanks alot! Later i want to update to install the latest NVidia driver. Have you tried 381.81?

Just for anyone else who is interessted in this. Using 3D Mark Time Spy Demo i got:

Native
Overall 5755 (GPU6019; CPU 4611)

Arch/KVM/Win10
Overall 4209 (GPU 5906; CPU 1602)
15 Gb attached using hugepages
200 Gb partition on a NVME Disk (block device, virtio)
6 CPU (Host has 4+ 4HT) added using Virt Manager

So i think there is alot to improve. First i will try my luck with cputune.
But also the hard disk ist horrible slow (SSD Benchmark says 1mb/s writing 4k sectors)

Offline

#16 2016-07-20 19:56:37

Omar007
Member
Registered: 2015-04-09
Posts: 357

Re: Setting up KVM VGA Passthrough

You can probably get some more performance out of the virtual CPU by setting its mode to 'host-passthrough' and bind it to specific cores.

Offline

Board footer

Powered by FluxBB