You are not logged in.

#1 2018-02-26 02:29:09

rmanne
Member
Registered: 2015-12-08
Posts: 11

[SOLVED] GPU Passthrough doesn't work with NVIDIA GPU

I'm trying to passthrough an NVIDIA gpu to windows 10, while it's the only gpu. Any help would be appreciated.

Desktop: Ryzen 7 1700 + Taichi X370.
I have 2 GPUs, a GTX 1050, and a really old AMD HD 7770, but I'd like to only use the GTX 1050 for this build (mostly to save pci lanes).

I use EFISTUB to boot. The efibootmgr command: `sudo efibootmgr --disk /dev/nvme0n1 --part 1 --create --gpt --label "Arch Linux PT" --loader /vmlinuz-linux-lts-custom --unicode "root=/dev/nvme0n1p2 rw initrd=/initramfs-linux-lts-custom.img rcu_nocbs=0-15 iommu=1 amd_iommu=on video=efifb:off rd.driver.pre=vfio-pci"`. My modprobe.d entry for vfio-pci: `options vfio-pci ids=10de:1c81,10de:0fb9,1002:683d,1002:aab0 disable_vga=1`.

Initially, I tried with just the 1050. I constant got Error code 43, which seemed to be connected to NVIDIA driver nonsense, but even after making the appropriate changes, it would still refuse to work. When I disabled the virtualized video driver (QXL/Cirrus for spice), Windows just went black, while Ubuntu got a bit further in that the greeter showed up, but it never finished booting. In dmesg, I saw `vfio-pci 0000:0e:00.0: Invalid PCI ROM header signature: expecting 0xaa55, got 0xffff`.

Then, I tried with both the 7770 and 1050, with 7770 as the boot gpu. When I had both, I was able to passthrough both to windows, at the same time. 7770 was completely effortless to get started. Windows recognized it and installed the drivers, and started outputting to the screen even while spice/qxl was still connected, and it allowed me to freely move my mouse between the two displays (extended). A couple of reboots were needed for the 1050 driver, but finally it also started working. There were no errors/fatals in the dmesg log this time. I disconnected the 7770 from the vm and after a reboot, the 1050 was still functional. So there were no problems with the 1050 while the 7770 was the boot gpu.

Then, I tried just using the 1050 again, but again it started throwing those PCI ROM exceptions. Using nvflash, I extracted the rom file and ensured that it supports EFI:

Valid ROM signature found @a00h, PCIR offset 1a0h
	PCIR: type 0 (x86 PC-AT), vendor: 10de, device: 1c81, class: 030000
	PCIR: revision 0, vendor revision: 1
Valid ROM signature found @fa00h, PCIR offset 1ch
	PCIR: type 3 (EFI), vendor: 10de, device: 1c81, class: 030000
	PCIR: revision 3, vendor revision: 0
		EFI: Signature Valid, Subsystem: Boot, Machine: X64
	Last image

Aside, passmark crashes the system while collecting "cpu info", and 3dmark occasionally refuses to start up, stuck at "collecting system info" (takes a couple of reboots and a couple of restarts). But cinemark and furmark run just fine. I suspect it's because kvm with a ryzen cpu appears to treat it as an EPYC cpu and the model just isn't recognized. A successful 3dmark score in question (with the 7770): https://www.3dmark.com/3dm/25336610

My win10 xmldump:

<domain type='kvm' id='7'>
  <name>win10</name>
  <uuid>9fd47c99-7020-47f5-bf1e-f026f6ede7d6</uuid>
  <memory unit='KiB'>8388608</memory>
  <currentMemory unit='KiB'>8388608</currentMemory>
  <vcpu placement='static'>4</vcpu>
  <resource>
    <partition>/machine</partition>
  </resource>
  <os>
    <type arch='x86_64' machine='pc-i440fx-2.11'>hvm</type>
    <loader readonly='yes' type='pflash'>/usr/share/ovmf/ovmf_code_x64.bin</loader>
    <nvram>/var/lib/libvirt/qemu/nvram/win10_VARS.fd</nvram>
    <boot dev='hd'/>
  </os>
  <features>
    <acpi/>
    <apic/>
    <hyperv>
      <relaxed state='on'/>
      <vapic state='on'/>
      <spinlocks state='on' retries='8191'/>
      <vendor_id state='on' value='whatever'/>
    </hyperv>
    <kvm>
      <hidden state='on'/>
    </kvm>
    <vmport state='off'/>
  </features>
  <cpu mode='custom' match='exact' check='full'>
    <model fallback='forbid'>EPYC</model>
    <vendor>AMD</vendor>
    <topology sockets='1' cores='4' threads='1'/>
    <feature policy='require' name='x2apic'/>
    <feature policy='require' name='tsc-deadline'/>
    <feature policy='require' name='hypervisor'/>
    <feature policy='require' name='tsc_adjust'/>
    <feature policy='require' name='cmp_legacy'/>
    <feature policy='disable' name='monitor'/>
    <feature policy='disable' name='svm'/>
  </cpu>
  <clock offset='localtime'>
    <timer name='rtc' tickpolicy='catchup'/>
    <timer name='pit' tickpolicy='delay'/>
    <timer name='hpet' present='no'/>
    <timer name='hypervclock' present='yes'/>
  </clock>
  <on_poweroff>destroy</on_poweroff>
  <on_reboot>restart</on_reboot>
  <on_crash>destroy</on_crash>
  <pm>
    <suspend-to-mem enabled='no'/>
    <suspend-to-disk enabled='no'/>
  </pm>
  <devices>
    <emulator>/usr/bin/qemu-system-x86_64</emulator>
    <disk type='file' device='disk'>
      <driver name='qemu' type='raw'/>
      <source file='/var/lib/libvirt/images/win10.img'/>
      <backingStore/>
      <target dev='sda' bus='sata'/>
      <alias name='sata0-0-0'/>
      <address type='drive' controller='0' bus='0' target='0' unit='0'/>
    </disk>
    <disk type='file' device='cdrom'>
      <target dev='sdb' bus='sata'/>
      <readonly/>
      <alias name='sata0-0-1'/>
      <address type='drive' controller='0' bus='0' target='0' unit='1'/>
    </disk>
    <controller type='usb' index='0' model='ich9-ehci1'>
      <alias name='usb'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x05' function='0x7'/>
    </controller>
    <controller type='usb' index='0' model='ich9-uhci1'>
      <alias name='usb'/>
      <master startport='0'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x05' function='0x0' multifunction='on'/>
    </controller>
    <controller type='usb' index='0' model='ich9-uhci2'>
      <alias name='usb'/>
      <master startport='2'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x05' function='0x1'/>
    </controller>
    <controller type='usb' index='0' model='ich9-uhci3'>
      <alias name='usb'/>
      <master startport='4'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x05' function='0x2'/>
    </controller>
    <controller type='pci' index='0' model='pci-root'>
      <alias name='pci.0'/>
    </controller>
    <controller type='sata' index='0'>
      <alias name='sata0'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x06' function='0x0'/>
    </controller>
    <interface type='network'>
      <mac address='52:54:00:e6:d5:98'/>
      <source network='default' bridge='virbr0'/>
      <target dev='vnet0'/>
      <model type='rtl8139'/>
      <alias name='net0'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x0'/>
    </interface>
    <input type='mouse' bus='ps2'>
      <alias name='input0'/>
    </input>
    <input type='keyboard' bus='ps2'>
      <alias name='input1'/>
    </input>
    <hostdev mode='subsystem' type='usb' managed='yes'>
      <source>
        <vendor id='0xfeed'/>
        <product id='0x1307'/>
        <address bus='3' device='4'/>
      </source>
      <alias name='hostdev0'/>
      <address type='usb' bus='0' port='1'/>
    </hostdev>
    <hostdev mode='subsystem' type='usb' managed='yes'>
      <source>
        <vendor id='0x046d'/>
        <product id='0xc332'/>
        <address bus='3' device='3'/>
      </source>
      <alias name='hostdev1'/>
      <address type='usb' bus='0' port='2'/>
    </hostdev>
    <hostdev mode='subsystem' type='pci' managed='yes'>
      <driver name='vfio'/>
      <source>
        <address domain='0x0000' bus='0x0e' slot='0x00' function='0x0'/>
      </source>
      <alias name='hostdev2'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x08' function='0x0'/>
    </hostdev>
    <hostdev mode='subsystem' type='pci' managed='yes'>
      <driver name='vfio'/>
      <source>
        <address domain='0x0000' bus='0x0e' slot='0x00' function='0x1'/>
      </source>
      <alias name='hostdev3'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x09' function='0x0'/>
    </hostdev>
    <memballoon model='virtio'>
      <alias name='balloon0'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x0a' function='0x0'/>
    </memballoon>
  </devices>
  <seclabel type='dynamic' model='dac' relabel='yes'>
    <label>+99:+999</label>
    <imagelabel>+99:+999</imagelabel>
  </seclabel>
</domain>

I also tried this with q35 instead of i440fx, but there was no difference.


SOLVED: https://lime-technology.com/forums/topi … in-unraid/
It seems that the results from nvflash ARE NOT THE SAME AS A MANUAL DUMP. This is not clear in the arch wiki. Doing this solved the problem.

Last edited by rmanne (2018-02-28 05:49:03)

Offline

Board footer

Powered by FluxBB