You are not logged in.

Pages: 1

#1 2024-08-10 21:23:46

rodlema
Member
Registered: 2024-08-10
Posts: 1

PCI passthrough via OVMF: vfio-pci failed with error -22

Hello everyone! I'm trying to set up a VM with GPU passthrough. I'm following this guide. I'm stuck with: 

vfio-pci 0000:01:00.0: probe with driver vfio-pci failed with error -22

I tried passing my NVIDIA GPU first, and since I couldn't, I tried with my AMD GPU afterward. Neither worked, same error code for both.

My system

OS: Arch Linux x86_64
Kernel: Linux 6.10.3-arch1-2
DE: Xfce4 4.18
WM: Xfwm4 (X11)
CPU: Intel(R) Core(TM) i5-10400 (12) @ 4.30 GHz
GPU 1: NVIDIA GeForce RTX 3080 [Discrete]
GPU 2: AMD Radeon RX 6700 XT [Discrete]
GPU 3: Intel UHD Graphics 630 @ 1.10 GHz [Integrated]
Memory: 23.30 GiB
Swap: 4.00 GiB

VT-d is enabled in BIOS
I've already updated my BIOS

Is IOMMU enabled?

sudo dmesg | grep -i -e DMAR -e IOMMU
[    0.000000] Command line: BOOT_IMAGE=/boot/vmlinuz-linux root=UUID=e8e348c8-c65b-4535-8786-19274a2a2531 rw loglevel=3 quiet intel_iommu=on iommu=pt vfio-pci.ids=10de:2206,10de:1aef module_blacklist=nouveau,nvidia,nvidia_drm,nvidia_modeset,nvidia_uvm
[    0.064751] Kernel command line: BOOT_IMAGE=/boot/vmlinuz-linux root=UUID=e8e348c8-c65b-4535-8786-19274a2a2531 rw loglevel=3 quiet intel_iommu=on iommu=pt vfio-pci.ids=10de:2206,10de:1aef module_blacklist=nouveau,nvidia,nvidia_drm,nvidia_modeset,nvidia_uvm
[    0.064832] DMAR: IOMMU enabled
[    0.343633] iommu: Default domain type: Passthrough (set via kernel command line)

IOMMU Groups

Script I'm using:

#!/bin/bash
for d in /sys/kernel/iommu_groups/*/devices/*; do
  n=${d#*/iommu_groups/*}; n=${n%%/*}
  printf 'IOMMU Group %s ' "$n"
  lspci -nns "${d##*/}"
done

Output (relevant lines, full output below, GPU is isolated):

01:00.0 VGA compatible controller [0300]: NVIDIA Corporation GA102 [GeForce RTX 3080] [10de:2206] (rev a1)
01:00.1 Audio device [0403]: NVIDIA Corporation GA102 High Definition Audio Controller [10de:1aef] (rev a1)
07:00.0 VGA compatible controller [0300]: Advanced Micro Devices, Inc. [AMD/ATI] Navi 22 [Radeon RX 6700/6700 XT/6750 XT / 6800M/6850M XT] [1002:73df] (rev c1)
07:00.1 Audio device [0403]: Advanced Micro Devices, Inc. [AMD/ATI] Navi 21/23 HDMI/DP Audio Controller [1002:ab28]

What I did

Files modified:

  • /etc/default/grub

  • /etc/mkinitcpio.conf

  • /etc/modprobe.d/vfio.conf

  • /etc/modprobe.d/blacklist.conf

After I modified them, I ran:

grub-mkconfig -o /boot/grub/grub.cfg
mkinitcpio -P

  • I added 

    intel_iommu=on iommu=pt

    module_blacklist=nouveau,nvidia,nvidia_drm,nvidia_modeset,nvidia_uvm

    vfio-pci.ids=10de:2206,10de:1aef

    to /etc/default/grub's 

    GRUB_CMDLINE_LINUX_DEFAULT

    and updated with 

    grub-mkconfig -o /boot/grub/grub.cfg

  • I added 

    vfio_pci vfio vfio_iommu_type1

    to /etc/mkinitcpio.conf's 

    MODULES=()

  • I added 

    softdep nvidia pre: vfio-pci

    options vfio-pci ids=10de:2206,10de:1aef

    to /etc/modprobe.d/vfio.conf

  • I added 

    blacklist nvidia

    blacklist nvidia_drm

    blacklist nvidia_modeset

    blacklist nvidia_uvm

    blacklist nouveau

    to /etc/modprobe.d/blacklist.conf

  • I regenerate the initramfs with 

    mkinitcpio -P

  • I rebooted

Before the modifications

lsmod | grep nvidia

nvidia_drm            122880  2
nvidia_uvm           6684672  0
nvidia_modeset       1617920  3 nvidia_drm
nvidia              61005824  36 nvidia_uvm,nvidia_modeset
video                  77824  4 asus_wmi,amdgpu,i915,nvidia_modeset

lspci -nnk -s 0000:01:00.0

01:00.0 VGA compatible controller [0300]: NVIDIA Corporation GA102 [GeForce RTX 3080] [10de:2206] (rev a1)
	Subsystem: Micro-Star International Co., Ltd. [MSI] RTX 3080 10GB GAMING X TRIO [1462:3892]
	Kernel driver in use: nvidia
	Kernel modules: nouveau, nvidia_drm, nvidia

lspci -nnk -s 0000:01:00.1

01:00.1 Audio device [0403]: NVIDIA Corporation GA102 High Definition Audio Controller [10de:1aef] (rev a1)
	Subsystem: Micro-Star International Co., Ltd. [MSI] Device [1462:3892]
	Kernel driver in use: snd_hda_intel
	Kernel modules: snd_hda_intel

After the modifications

lsmod | grep vfio                 

vfio_pci               16384  0
vfio_pci_core          98304  1 vfio_pci
vfio_iommu_type1       49152  0
vfio                   77824  3 vfio_pci_core,vfio_iommu_type1,vfio_pci
iommufd               110592  1 vfio


lsmod | grep nvidia

(nothing)

lspci -nnk -s 0000:01:00.0

01:00.0 VGA compatible controller [0300]: NVIDIA Corporation GA102 [GeForce RTX 3080] [10de:2206] (rev a1)
	Subsystem: Micro-Star International Co., Ltd. [MSI] RTX 3080 10GB GAMING X TRIO [1462:3892]
	Kernel modules: nouveau, nvidia_drm, nvidia
    (kernel driver in use disappeared, no vfio)

lspci -nnk -s 0000:01:00.1

01:00.1 Audio device [0403]: NVIDIA Corporation GA102 High Definition Audio Controller [10de:1aef] (rev a1)
	Subsystem: Micro-Star International Co., Ltd. [MSI] Device [1462:3892]
	Kernel driver in use: snd_hda_intel
	Kernel modules: snd_hda_intel

DMESG

dmseg | grep vfio
[    0.000000] Command line: BOOT_IMAGE=/boot/vmlinuz-linux root=UUID=e8e348c8-c65b-4535-8786-19274a2a2531 rw loglevel=3 quiet intel_iommu=on iommu=pt vfio-pci.ids=10de:2206,10de:1aef module_blacklist=nouveau,nvidia,nvidia_drm,nvidia_modeset,nvidia_uvm
[    0.064751] Kernel command line: BOOT_IMAGE=/boot/vmlinuz-linux root=UUID=e8e348c8-c65b-4535-8786-19274a2a2531 rw loglevel=3 quiet intel_iommu=on iommu=pt vfio-pci.ids=10de:2206,10de:1aef module_blacklist=nouveau,nvidia,nvidia_drm,nvidia_modeset,nvidia_uvm
[    1.143050] VFIO - User Level meta-driver version: 0.3
[    1.151943] vfio-pci 0000:01:00.0: vgaarb: VGA decodes changed: olddecodes=io+mem,decodes=io+mem:owns=none
[    1.151964] vfio-pci 0000:01:00.0: probe with driver vfio-pci failed with error -22
[    1.151968] vfio_pci: add [10de:2206[ffffffff:ffffffff]] class 0x000000/00000000
[    1.151980] vfio-pci 0000:01:00.0: vgaarb: VGA decodes changed: olddecodes=io+mem,decodes=io+mem:owns=none
[    1.151987] vfio-pci 0000:01:00.0: probe with driver vfio-pci failed with error -22
[    1.152025] vfio-pci 0000:01:00.1: probe with driver vfio-pci failed with error -22
[    1.152027] vfio_pci: add [10de:1aef[ffffffff:ffffffff]] class 0x000000/00000000

I've also run

#!/bin/bash

gpu="0000:01:00.0"
aud="0000:01:00.1"
gpu_vd="$(cat /sys/bus/pci/devices/$gpu/vendor) $(cat /sys/bus/pci/devices/$gpu/device)"
aud_vd="$(cat /sys/bus/pci/devices/$aud/vendor) $(cat /sys/bus/pci/devices/$aud/device)"

function bind_vfio {
  echo "$gpu" > "/sys/bus/pci/devices/$gpu/driver/unbind"
  echo "$aud" > "/sys/bus/pci/devices/$aud/driver/unbind"
  echo "$gpu_vd" > /sys/bus/pci/drivers/vfio-pci/new_id
  echo "$aud_vd" > /sys/bus/pci/drivers/vfio-pci/new_id
}
 
function unbind_vfio {
  echo "$gpu_vd" > "/sys/bus/pci/drivers/vfio-pci/remove_id"
  echo "$aud_vd" > "/sys/bus/pci/drivers/vfio-pci/remove_id"
  echo 1 > "/sys/bus/pci/devices/$gpu/remove"
  echo 1 > "/sys/bus/pci/devices/$aud/remove"
  echo 1 > "/sys/bus/pci/rescan"
}

I've also tried with my AMD GPU

Current config:

// /etc/modprobe.d/vfio.conf

softdep drm pre: vfio-pci
options vfio-pci ids=1002:73df,1002:ab28
// /etc/modprobe.conf/blacklist.conf

blacklist amdgpu
// /etc/mkinitcpio.conf

MODULES=(vfio_pci vfio vfio_iommu_type1)
BINARIES=()
FILES=()
HOOKS=(base udev autodetect microcode modconf kms keyboard keymap consolefont block filesystems fsck)
// /etc/default/grub

GRUB_DEFAULT=0
GRUB_TIMEOUT=5
GRUB_DISTRIBUTOR="Arch"
GRUB_CMDLINE_LINUX_DEFAULT="loglevel=3 quiet intel_iommu=on iommu=pt module_blacklist=amdgpu vfio-pci.ids=1002:73df,1002:ab28"
GRUB_CMDLINE_LINUX=""
GRUB_PRELOAD_MODULES="part_gpt part_msdos"
GRUB_TIMEOUT_STYLE=menu
GRUB_TERMINAL_INPUT=console
GRUB_GFXMODE=auto
GRUB_GFXPAYLOAD_LINUX=keep
GRUB_DISABLE_RECOVERY=true

FULL DMESG

lspci:

00:00.0 Host bridge: Intel Corporation Comet Lake-S 6c Host Bridge/DRAM Controller (rev 03)
00:01.0 PCI bridge: Intel Corporation 6th-10th Gen Core Processor PCIe Controller (x16) (rev 03)
00:02.0 VGA compatible controller: Intel Corporation CometLake-S GT2 [UHD Graphics 630] (rev 03)
00:14.0 USB controller: Intel Corporation Comet Lake PCH-V USB Controller
00:15.0 Signal processing controller: Intel Corporation Device a3e0
00:15.1 Signal processing controller: Intel Corporation Device a3e1
00:16.0 Communication controller: Intel Corporation Comet Lake PCH-V HECI Controller
00:17.0 SATA controller: Intel Corporation 400 Series Chipset Family SATA AHCI Controller
00:1b.0 PCI bridge: Intel Corporation Device a3e9 (rev f0)
00:1b.4 PCI bridge: Intel Corporation Comet Lake PCI Express Root Port #21 (rev f0)
00:1c.0 PCI bridge: Intel Corporation Device a392 (rev f0)
00:1c.4 PCI bridge: Intel Corporation Comet Lake PCI Express Root Port #05 (rev f0)
00:1d.0 PCI bridge: Intel Corporation Comet Lake PCI Express Root Port 9 (rev f0)
00:1f.0 ISA bridge: Intel Corporation B460 Chipset LPC/eSPI Controller
00:1f.2 Memory controller: Intel Corporation Cannon Lake PCH Power Management Controller
00:1f.3 Audio device: Intel Corporation Comet Lake PCH-V cAVS
00:1f.4 SMBus: Intel Corporation Comet Lake PCH-V SMBus Host Controller
00:1f.6 Ethernet controller: Intel Corporation Ethernet Connection (12) I219-V
01:00.0 VGA compatible controller: NVIDIA Corporation GA102 [GeForce RTX 3080] (rev a1)
01:00.1 Audio device: NVIDIA Corporation GA102 High Definition Audio Controller (rev a1)
02:00.0 USB controller: ASMedia Technology Inc. ASM3241 USB 3.2 Gen 2 Host Controller
03:00.0 Non-Volatile memory controller: Realtek Semiconductor Co., Ltd. RTS5763DL NVMe SSD Controller (DRAM-less) (rev 01)
05:00.0 PCI bridge: Advanced Micro Devices, Inc. [AMD/ATI] Navi 10 XL Upstream Port of PCI Express Switch (rev c1)
06:00.0 PCI bridge: Advanced Micro Devices, Inc. [AMD/ATI] Navi 10 XL Downstream Port of PCI Express Switch
07:00.0 VGA compatible controller: Advanced Micro Devices, Inc. [AMD/ATI] Navi 22 [Radeon RX 6700/6700 XT/6750 XT / 6800M/6850M XT] (rev c1)
07:00.1 Audio device: Advanced Micro Devices, Inc. [AMD/ATI] Navi 21/23 HDMI/DP Audio Controller

virt-manager error when I try to run the VM with PCI passthrough

Error starting domain: unsupported configuration: host doesn't support passthrough of host PCI devices

Traceback (most recent call last):
  File "/usr/share/virt-manager/virtManager/asyncjob.py", line 72, in cb_wrapper
    callback(asyncjob, *args, **kwargs)
  File "/usr/share/virt-manager/virtManager/asyncjob.py", line 108, in tmpcb
    callback(*args, **kwargs)
  File "/usr/share/virt-manager/virtManager/object/libvirtobject.py", line 57, in newfn
    ret = fn(self, *args, **kwargs)
          ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/share/virt-manager/virtManager/object/domain.py", line 1402, in startup
    self._backend.create()
  File "/usr/lib/python3.12/site-packages/libvirt.py", line 1379, in create
    raise libvirtError('virDomainCreate() failed')
libvirt.libvirtError: unsupported configuration: host doesn't support passthrough of host PCI devices

Last edited by rodlema (2024-08-11 02:56:07)

Offline

#2 2024-08-11 15:14:05

cryptearth
Member
Registered: 2024-02-03
Posts: 1,809

Re: PCI passthrough via OVMF: vfio-pci failed with error -22

have you tried https://aur.archlinux.org/packages/gpu- … gh-manager already?

Offline

#3 2024-08-11 15:22:56

Lone_Wolf
Administrator
From: Netherlands, Europe
Registered: 2005-10-04
Posts: 14,418

Re: PCI passthrough via OVMF: vfio-pci failed with error -22

remove iommu=pt , nowadays it's rarely needed.

Post the output of 

zgrep HSA_AMD_SVM /proc/config.gz

Links that may be relevant (especially the last one)  :

https://askubuntu.com/questions/1509972 … out-reboot
https://www.reddit.com/r/VFIO/comments/ … utm_term=1
https://gitlab.freedesktop.org/drm/amd/-/issues/2794

Disliking systemd intensely, but not satisfied with alternatives so focusing on taming systemd.

clean chroot building not flexible enough ?
Try clean chroot manager by graysky

Offline

#4 2024-08-11 21:57:14

cloverskull
Member
Registered: 2018-09-30
Posts: 259

Re: PCI passthrough via OVMF: vfio-pci failed with error -22

Can you paste the entirety of the output when you run the script to identify IOMMU groups?

Offline

Pages: 1

Board footer

Powered by FluxBB