You are not logged in.
Hello.
I have been trying to setup GPU passthrogh to my VM, but I am not sure what I have done wrong.
So I have setup VM and configured host with using this and the problem is that NVIDIA driver inside VM doesn't detect NVIDIA card.
So here is all of the modification and hardware I am trying to do it on:
-- Hardware --
Laptop: Precision 7740
CPU: Intel i7-9750H
GPU: NVIDIA Quadro RTX 3000 Mobile
RAM: 32GB
IOMMU in BIOS: Bundled with "virtualization" option
-- HOST Config --
GRUB Kernel Arguments: vmlinuz-linux cryptdevice=PARTUUID=151d2c84-949d-c544-896e-a30e74b9beab:root root=/dev/vg0/root rw add_efi_memmap loglevel=3 intel_iommu=on iommu=pt vfio-pci.ids=10de:1f36,10de:10f9,10de:1ada,10de:1adb
Check if IOMMU is enabled:
# dmesg | grep -i "iommu enabled"
[ 0.050821] DMAR: IOMMU enabled/etc/mkinitcpio.conf (I have added some HOOKS because I have LUKS+LVM drive setup)
...
MODULES=(vfio_pci vfio vfio_iommu_type1 vfio_virqfd)
...
HOOKS=(base udev keyboard autodetect modconf kms keymap consolefont encrypt block lvm2 filesystems fsck)
...IOMMU Group containing my NVIDIA card and it's components: (Yes, I am aware that CPU is included in this group, but in a case of CPU only it can not be passed)
IOMMU Group 1:
00:01.0 PCI bridge [0604]: Intel Corporation 6th-10th Gen Core Processor PCIe Controller (x16) [8086:1901] (rev 07)
01:00.0 VGA compatible controller [0300]: NVIDIA Corporation TU106GLM [Quadro RTX 3000 Mobile / Max-Q] [10de:1f36] (rev a1)
01:00.1 Audio device [0403]: NVIDIA Corporation TU106 High Definition Audio Controller [10de:10f9] (rev a1)
01:00.2 USB controller [0c03]: NVIDIA Corporation TU106 USB 3.1 Host Controller [10de:1ada] (rev a1)
01:00.3 Serial bus controller [0c80]: NVIDIA Corporation TU106 USB Type-C UCSI Controller [10de:1adb] (rev a1)Output of lspci -nnk (nvidia card is bind to vfio-pci as it supposed to be):
$ lspci -nnk
...
01:00.0 VGA compatible controller [0300]: NVIDIA Corporation TU106GLM [Quadro RTX 3000 Mobile / Max-Q] [10de:1f36] (rev a1)
Subsystem: Dell Device [1028:0927]
Kernel driver in use: vfio-pci
Kernel modules: nouveau, nvidia_drm, nvidia
01:00.1 Audio device [0403]: NVIDIA Corporation TU106 High Definition Audio Controller [10de:10f9] (rev a1)
Subsystem: Dell Device [1028:0927]
Kernel driver in use: vfio-pci
Kernel modules: snd_hda_intel
01:00.2 USB controller [0c03]: NVIDIA Corporation TU106 USB 3.1 Host Controller [10de:1ada] (rev a1)
Subsystem: NVIDIA Corporation Device [10de:0000]
Kernel driver in use: vfio-pci
Kernel modules: xhci_pci
01:00.3 Serial bus controller [0c80]: NVIDIA Corporation TU106 USB Type-C UCSI Controller [10de:1adb] (rev a1)
Subsystem: NVIDIA Corporation Device [10de:0000]
Kernel driver in use: vfio-pci
Kernel modules: i2c_nvidia_gpu
...-- Guest / libvirt / virt-manager config --
virt-manager XML config:
<domain type="kvm">
<name>win10p</name>
<uuid>67d8af7a-ab3e-4dce-8fa3-2906fcfc8bb1</uuid>
<metadata>
<libosinfo:libosinfo xmlns:libosinfo="http://libosinfo.org/xmlns/libvirt/domain/1.0">
<libosinfo:os id="http://microsoft.com/win/10"/>
</libosinfo:libosinfo>
</metadata>
<memory unit="KiB">25165824</memory>
<currentMemory unit="KiB">25165824</currentMemory>
<vcpu placement="static">10</vcpu>
<os firmware="efi">
<type arch="x86_64" machine="pc-i440fx-7.1">hvm</type>
</os>
<features>
<acpi/>
<apic/>
<hyperv mode="custom">
<relaxed state="on"/>
<vapic state="on"/>
<spinlocks state="on" retries="8191"/>
</hyperv>
<vmport state="off"/>
</features>
<cpu mode="host-passthrough" check="none" migratable="on">
<topology sockets="1" dies="1" cores="5" threads="2"/>
</cpu>
<clock offset="localtime">
<timer name="rtc" tickpolicy="catchup"/>
<timer name="pit" tickpolicy="delay"/>
<timer name="hpet" present="no"/>
<timer name="hypervclock" present="yes"/>
</clock>
<on_poweroff>destroy</on_poweroff>
<on_reboot>restart</on_reboot>
<on_crash>destroy</on_crash>
<pm>
<suspend-to-mem enabled="no"/>
<suspend-to-disk enabled="no"/>
</pm>
<devices>
<emulator>/usr/bin/qemu-system-x86_64</emulator>
<disk type="file" device="disk">
<driver name="qemu" type="qcow2" discard="unmap"/>
<source file="/run/media/yesnt/main/vms/win10p.qcow2"/>
<target dev="vda" bus="virtio"/>
<boot order="2"/>
<address type="pci" domain="0x0000" bus="0x00" slot="0x08" function="0x0"/>
</disk>
<controller type="usb" index="0" model="qemu-xhci" ports="15">
<address type="pci" domain="0x0000" bus="0x00" slot="0x04" function="0x0"/>
</controller>
<controller type="pci" index="0" model="pci-root"/>
<controller type="sata" index="0">
<address type="pci" domain="0x0000" bus="0x00" slot="0x05" function="0x0"/>
</controller>
<controller type="virtio-serial" index="0">
<address type="pci" domain="0x0000" bus="0x00" slot="0x06" function="0x0"/>
</controller>
<interface type="network">
<mac address="52:54:00:c4:18:a8"/>
<source network="default"/>
<model type="e1000"/>
<address type="pci" domain="0x0000" bus="0x00" slot="0x09" function="0x0"/>
</interface>
<serial type="pty">
<target type="isa-serial" port="0">
<model name="isa-serial"/>
</target>
</serial>
<console type="pty">
<target type="serial" port="0"/>
</console>
<channel type="spicevmc">
<target type="virtio" name="com.redhat.spice.0"/>
<address type="virtio-serial" controller="0" bus="0" port="1"/>
</channel>
<input type="tablet" bus="usb">
<address type="usb" bus="0" port="1"/>
</input>
<input type="mouse" bus="ps2"/>
<input type="keyboard" bus="ps2"/>
<graphics type="spice" autoport="yes">
<listen type="address"/>
<image compression="off"/>
</graphics>
<sound model="ich9">
<address type="pci" domain="0x0000" bus="0x00" slot="0x03" function="0x0"/>
</sound>
<audio id="1" type="spice"/>
<video>
<model type="qxl" ram="65536" vram="65536" vgamem="16384" heads="1" primary="yes"/>
<address type="pci" domain="0x0000" bus="0x00" slot="0x02" function="0x0"/>
</video>
<hostdev mode="subsystem" type="pci" managed="yes">
<source>
<address domain="0x0000" bus="0x01" slot="0x00" function="0x0"/>
</source>
<address type="pci" domain="0x0000" bus="0x00" slot="0x0a" function="0x0"/>
</hostdev>
<hostdev mode="subsystem" type="pci" managed="yes">
<source>
<address domain="0x0000" bus="0x01" slot="0x00" function="0x1"/>
</source>
<address type="pci" domain="0x0000" bus="0x00" slot="0x0b" function="0x0"/>
</hostdev>
<hostdev mode="subsystem" type="pci" managed="yes">
<source>
<address domain="0x0000" bus="0x01" slot="0x00" function="0x2"/>
</source>
<address type="pci" domain="0x0000" bus="0x00" slot="0x0c" function="0x0"/>
</hostdev>
<hostdev mode="subsystem" type="pci" managed="yes">
<source>
<address domain="0x0000" bus="0x01" slot="0x00" function="0x3"/>
</source>
<address type="pci" domain="0x0000" bus="0x00" slot="0x0d" function="0x0"/>
</hostdev>
<redirdev bus="usb" type="spicevmc">
<address type="usb" bus="0" port="2"/>
</redirdev>
<redirdev bus="usb" type="spicevmc">
<address type="usb" bus="0" port="3"/>
</redirdev>
<memballoon model="virtio">
<address type="pci" domain="0x0000" bus="0x00" slot="0x07" function="0x0"/>
</memballoon>
</devices>
</domain>Libvirt Config:
(I wasn't able to find where it is supposed to be located but I haven't it modified so it's default as in libvirt package.)
Output of lsmod | grep kvm (check if KVM module is loaded):
$ lsmod | grep kvm
kvm_intel 389120 0
kvm 1138688 1 kvm_intel
irqbypass 16384 2 vfio_pci_core,kvmNVIDIA Driver File:
527.27-quadro-rtx-desktop-notebook-win10-win11-64bit-international-dch-whql.exe
"Other Devices" in WIndows Device Manager:
PCI Device
PCI Device
PCI Simple Communication Controller
Note: Fourth NVIDIA device is listed under "Universal Serial Bus controllers" as "NVIDIA USB 3.10 eXtensible Host Controller - 1.10 (Microsoft)"
EDIT: I have tried this fix, but it didn't fix anything.
Also in Windows Device Manager I have 3 PCI devices but also there is additional "Microsoft Basic Display Adapter" which fails to start.
If I missed some configs, please mention them and I am going to provide them.
Thanks very much in advice!
Last edited by someReallyRandomName (2022-12-12 13:39:47)
Offline
Okay I figured it out.
To fix this issue for my case was to:
1. Make sure that pci domain, bus, slot and function that was presented to VM was the same as on host.
2. Add this after </device> and before </domain>:
<qemu:override>
<qemu:device alias="hostdev0">
<qemu:frontend>
<qemu:property name="x-pci-sub-vendor-id" type="unsigned" value="4136"/>
<qemu:property name="x-pci-sub-device-id" type="unsigned" value="2343"/>
</qemu:frontend>
</qemu:device>
</qemu:override>where vender and device ids' are integer representation of hex value "Subsystem" after typing this command:
$ lspci -nks 01:00.0
01:00.0 0300: 10de:1f36 (rev a1)
Subsystem: 1028:0927
Kernel driver in use: vfio-pci
Kernel modules: nouveau, nvidia_drm, nvidiaAlso remember to replace first line with:
<domain xmlns:qemu="http://libvirt.org/schemas/domain/qemu/1.0" type="kvm">so libvirt can properly validate XML.
Thanks to guy-92 for helping me out!
Last edited by someReallyRandomName (2022-12-12 13:41:37)
Offline