[SOLVED] Libvirt -- Unable to Completely Pass NVIDIA GPU to Windows VM

heywoodlh · 2020-05-01 16:20:51

I am running my home server setup on Arch Linux on my System76 Oryx Pro. I am a Linux sysadmin of over five years but haven't plunged into GPU passthrough in QEMU/KVM until now, so forgive my inexperience.

I have followed this Arch Wiki article to get GPU passthrough setup for the Nvidia 2070 in my Oryx Pro: https://wiki.archlinux.org/index.php/PC … h_via_OVMF. I believe I have followed the instructions as far as I could and have gotten the PCI devices added to my VM in libvirt. However, the GPU isn't being fully passed through and isn't recognized by Windows.

I have primarily been using virt-manager to manage the VM but am relatively comfortable with virsh.
I have passed the following kernel parameters to my host (passed using systemd-boot) and they seem to be working:

intel_iommu=on vfio-pci.ids=10de:1f10,10de:10f9,10de:1ada,10de:1adb

Here is the dmesg verifying that the device is enabled:

[    5.763332] vfio-pci 0000:01:00.0: optimus capabilities: enabled, status dynamic power, hda bios codec supported
[    5.766336] vfio-pci 0000:01:00.0: vgaarb: changed VGA decodes: olddecodes=io+mem,decodes=io+mem:owns=none
[    9.073229] vfio-pci 0000:01:00.0: enabling device (0002 -> 0003)

I can see that VFIO is being assigned as the driver to the PCI devices:

$ lspci -nnk
...
01:00.0 VGA compatible controller [0300]: NVIDIA Corporation TU106M [GeForce RTX 2070 Mobile] [10de:1f10] (rev a1)
	Subsystem: NVIDIA Corporation TU106M [GeForce RTX 2070 Mobile] [10de:0000]
	Kernel driver in use: vfio-pci
	Kernel modules: nouveau
01:00.1 Audio device [0403]: NVIDIA Corporation TU106 High Definition Audio Controller [10de:10f9] (rev a1)
	Subsystem: NVIDIA Corporation TU106 High Definition Audio Controller [10de:0000]
	Kernel driver in use: vfio-pci
	Kernel modules: snd_hda_intel
01:00.2 USB controller [0c03]: NVIDIA Corporation TU106 USB 3.1 Host Controller [10de:1ada] (rev a1)
	Subsystem: NVIDIA Corporation TU106 USB 3.1 Host Controller [10de:0000]
	Kernel driver in use: vfio-pci
	Kernel modules: xhci_pci
01:00.3 Serial bus controller [0c80]: NVIDIA Corporation TU106 USB Type-C UCSI Controller [10de:1adb] (rev a1)
	Subsystem: NVIDIA Corporation TU106 USB Type-C UCSI Controller [10de:0000]
	Kernel driver in use: vfio-pci
	Kernel modules: i2c_nvidia_gpu
...

In virt-manager I have added the PCI devices to my VM and didn't have any noticeable errors from libvirt once I passed through the relevant PCI devices and added them to the VM via virt-manager.

Anyway, the problem is that in the Windows VM the GPU isn't being recognized. After installing VFIO guest tools these PCI devices are the only Nvidia devices detected:

Name: NVIDIA Virtual Audio Device (Wave Extensible) (WDM)                                                               
Name: NVIDIA USB 3.10 eXtensible Host Controller - 1.10 (Microsoft)

What else do I need to configure in libvirt to get the GPU passed through to the VM?

Here is my full XML file:

<domain type="kvm">
  <name>windows</name>
  <uuid>1a150a84-4c41-48d9-a378-222d34e687dc</uuid>
  <metadata>
    <libosinfo:libosinfo xmlns:libosinfo="http://libosinfo.org/xmlns/libvirt/domain/1.0">
      <libosinfo:os id="http://microsoft.com/win/10"/>
    </libosinfo:libosinfo>
  </metadata>
  <memory unit="KiB">16775168</memory>
  <currentMemory unit="KiB">16775168</currentMemory>
  <vcpu placement="static">6</vcpu>
  <os>
    <type arch="x86_64" machine="pc-q35-4.2">hvm</type>
    <loader readonly="yes" type="pflash">/usr/share/ovmf/x64/OVMF_CODE.fd</loader>
    <nvram>/var/lib/libvirt/qemu/nvram/windows_VARS.fd</nvram>
  </os>
  <features>
    <acpi/>
    <apic/>
    <hyperv>
      <relaxed state="on"/>
      <vapic state="on"/>
      <spinlocks state="on" retries="8191"/>
    </hyperv>
    <vmport state="off"/>
    <ioapic driver="kvm"/>
  </features>
  <cpu mode="host-passthrough" check="none"/>
  <clock offset="localtime">
    <timer name="rtc" tickpolicy="catchup"/>
    <timer name="pit" tickpolicy="delay"/>
    <timer name="hpet" present="no"/>
    <timer name="hypervclock" present="yes"/>
  </clock>
  <on_poweroff>destroy</on_poweroff>
  <on_reboot>restart</on_reboot>
  <on_crash>destroy</on_crash>
  <pm>
    <suspend-to-mem enabled="no"/>
    <suspend-to-disk enabled="no"/>
  </pm>
  <devices>
    <emulator>/usr/bin/qemu-system-x86_64</emulator>
    <disk type="file" device="disk">
      <driver name="qemu" type="qcow2"/>
      <source file="/var/lib/libvirt/images/windows.qcow2"/>
      <target dev="sda" bus="sata"/>
      <boot order="1"/>
      <address type="drive" controller="0" bus="0" target="0" unit="0"/>
    </disk>
    <controller type="usb" index="0" model="qemu-xhci" ports="15">
      <address type="pci" domain="0x0000" bus="0x02" slot="0x00" function="0x0"/>
    </controller>
    <controller type="sata" index="0">
      <address type="pci" domain="0x0000" bus="0x00" slot="0x1f" function="0x2"/>
    </controller>
    <controller type="pci" index="0" model="pcie-root"/>
    <controller type="pci" index="1" model="pcie-root-port">
      <model name="pcie-root-port"/>
      <target chassis="1" port="0x10"/>
      <address type="pci" domain="0x0000" bus="0x00" slot="0x02" function="0x0" multifunction="on"/>
    </controller>
    <controller type="pci" index="2" model="pcie-root-port">
      <model name="pcie-root-port"/>
      <target chassis="2" port="0x11"/>
      <address type="pci" domain="0x0000" bus="0x00" slot="0x02" function="0x1"/>
    </controller>
    <controller type="pci" index="3" model="pcie-root-port">
      <model name="pcie-root-port"/>
      <target chassis="3" port="0x12"/>
      <address type="pci" domain="0x0000" bus="0x00" slot="0x02" function="0x2"/>
    </controller>
    <controller type="pci" index="4" model="pcie-root-port">
      <model name="pcie-root-port"/>
      <target chassis="4" port="0x13"/>
      <address type="pci" domain="0x0000" bus="0x00" slot="0x02" function="0x3"/>
    </controller>
    <controller type="pci" index="5" model="pcie-root-port">
      <model name="pcie-root-port"/>
      <target chassis="5" port="0x14"/>
      <address type="pci" domain="0x0000" bus="0x00" slot="0x02" function="0x4"/>
    </controller>
    <controller type="pci" index="6" model="pcie-root-port">
      <model name="pcie-root-port"/>
      <target chassis="6" port="0x15"/>
      <address type="pci" domain="0x0000" bus="0x00" slot="0x02" function="0x5"/>
    </controller>
    <controller type="pci" index="7" model="pcie-root-port">
      <model name="pcie-root-port"/>
      <target chassis="7" port="0x16"/>
      <address type="pci" domain="0x0000" bus="0x00" slot="0x02" function="0x6"/>
    </controller>
    <controller type="pci" index="8" model="pcie-root-port">
      <model name="pcie-root-port"/>
      <target chassis="8" port="0x17"/>
      <address type="pci" domain="0x0000" bus="0x00" slot="0x02" function="0x7"/>
    </controller>
    <controller type="pci" index="9" model="pcie-root-port">
      <model name="pcie-root-port"/>
      <target chassis="9" port="0x18"/>
      <address type="pci" domain="0x0000" bus="0x00" slot="0x03" function="0x0"/>
    </controller>
    <controller type="virtio-serial" index="0">
      <address type="pci" domain="0x0000" bus="0x03" slot="0x00" function="0x0"/>
    </controller>
    <interface type="direct">
      <mac address="52:54:00:37:9e:45"/>
      <source dev="eth0" mode="passthrough"/>
      <model type="e1000e"/>
      <address type="pci" domain="0x0000" bus="0x01" slot="0x00" function="0x0"/>
    </interface>
    <serial type="pty">
      <target type="isa-serial" port="0">
        <model name="isa-serial"/>
      </target>
    </serial>
    <console type="pty">
      <target type="serial" port="0"/>
    </console>
    <channel type="spicevmc">
      <target type="virtio" name="com.redhat.spice.0"/>
      <address type="virtio-serial" controller="0" bus="0" port="1"/>
    </channel>
    <input type="tablet" bus="usb">
      <address type="usb" bus="0" port="2"/>
    </input>
    <input type="mouse" bus="ps2"/>
    <input type="keyboard" bus="ps2"/>
    <graphics type="spice" autoport="yes">
      <listen type="address"/>
    </graphics>
    <sound model="ich9">
      <address type="pci" domain="0x0000" bus="0x00" slot="0x1b" function="0x0"/>
    </sound>
    <video>
      <model type="qxl" ram="65536" vram="65536" vgamem="16384" heads="1" primary="yes"/>
      <address type="pci" domain="0x0000" bus="0x00" slot="0x01" function="0x0"/>
    </video>
    <hostdev mode="subsystem" type="usb" managed="yes">
      <source>
        <vendor id="0x1058"/>
        <product id="0x25e1"/>
      </source>
      <address type="usb" bus="0" port="1"/>
    </hostdev>
    <hostdev mode="subsystem" type="pci" managed="yes">
      <source>
        <address domain="0x0000" bus="0x01" slot="0x00" function="0x0"/>
      </source>
      <address type="pci" domain="0x0000" bus="0x05" slot="0x00" function="0x0" multifunction="on"/>
    </hostdev>
    <hostdev mode="subsystem" type="pci" managed="yes">
      <source>
        <address domain="0x0000" bus="0x01" slot="0x00" function="0x1"/>
      </source>
      <address type="pci" domain="0x0000" bus="0x05" slot="0x00" function="0x1"/>
    </hostdev>
    <hostdev mode="subsystem" type="pci" managed="yes">
      <source>
        <address domain="0x0000" bus="0x01" slot="0x00" function="0x2"/>
      </source>
      <address type="pci" domain="0x0000" bus="0x05" slot="0x00" function="0x2"/>
    </hostdev>
    <hostdev mode="subsystem" type="pci" managed="yes">
      <source>
        <address domain="0x0000" bus="0x01" slot="0x00" function="0x3"/>
      </source>
      <address type="pci" domain="0x0000" bus="0x05" slot="0x00" function="0x3"/>
    </hostdev>
    <redirdev bus="usb" type="spicevmc">
      <address type="usb" bus="0" port="3"/>
    </redirdev>
    <redirdev bus="usb" type="spicevmc">
      <address type="usb" bus="0" port="4"/>
    </redirdev>
    <memballoon model="virtio">
      <address type="pci" domain="0x0000" bus="0x08" slot="0x00" function="0x0"/>
    </memballoon>
  </devices>
</domain>

Any help would be GREATLY appreciated. Thanks in advance for any suggestions.

(For reference, I posted on the VFIO subreddit as well: https://www.reddit.com/r/VFIO/comments/ … nstaller/)

Last edited by heywoodlh (2020-06-15 18:49:10)

heywoodlh · 2020-06-15 18:48:40

Found out from System76 support that the laptop does not support full passthrough for that GPU.

Arch Linux

#1 2020-05-01 16:20:51

[SOLVED] Libvirt -- Unable to Completely Pass NVIDIA GPU to Windows VM

#2 2020-06-15 18:48:40

Re: [SOLVED] Libvirt -- Unable to Completely Pass NVIDIA GPU to Windows VM

Board footer