You are not logged in.

Pages: 1

#1 2021-06-12 04:07:17

fulbokoy
Member
Registered: 2021-06-12
Posts: 3

VFIO PCI passthrough with two AMD GPUs

Hello guys,
I have a fresh Arch installation and was trying to passthrough one of my GPUs to a Windows 10 VM but with no success.
What happens is that after isolating the GPU and trying to use it on the VM, the VM "boots" but it gets stuck and the CPU usage keep constant.

Hardware info:
ASUS ROG Crosshair VIII Dark Hero. AMD X570 ATX.
AMD Ryzen 5 5600X 6-Core Processor

What I have done so far:
I installed Windows on the VM without passing the GPU and passing a SATA disk directly and it works, also I'm able to dual boot.
Once I pass the GPU the VM doesn't display anything (no black screen, just no signal) and seems like doing "nothing".

OS info:

> uname -r
5.12.9-arch1-1

Some IOMMU groups:

IOMMU Group 23:
        04:00.0 Ethernet controller [0200]: Realtek Semiconductor Co., Ltd. RTL8125 2.5GbE Controller [10ec:8125]
IOMMU Group 26:
        0a:00.0 VGA compatible controller [0300]: Advanced Micro Devices, Inc. [AMD/ATI] Baffin [Radeon RX 550 640SP / RX 560/560X] [1002:67ff] (rev cf)
        0a:00.1 Audio device [0403]: Advanced Micro Devices, Inc. [AMD/ATI] Baffin HDMI/DP Audio [Radeon RX 550 640SP / RX 560/560X] [1002:aae0]
IOMMU Group 27:
        0b:00.0 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Vega 10 PCIe Bridge [1022:1470] (rev c3)
IOMMU Group 28:
        0c:00.0 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Vega 10 PCIe Bridge [1022:1471]
IOMMU Group 29:
        0d:00.0 VGA compatible controller [0300]: Advanced Micro Devices, Inc. [AMD/ATI] Vega 10 XL/XT [Radeon RX Vega 56/64] [1002:687f] (rev c3)
IOMMU Group 30:
        0d:00.1 Audio device [0403]: Advanced Micro Devices, Inc. [AMD/ATI] Vega 10 HDMI Audio [Radeon Vega 56/64] [1002:aaf8]

Output for dmesg | grep -i vfio (also shows the grub kernel parameters)

[    0.000000] Command line: BOOT_IMAGE=/vmlinuz-linux root=UUID=b4e601d5-a7f8-45a2-a5a0-7f5d8d9fc28f rw loglevel=3 quiet amd_iommu=on iommu=pt vfio_iommu_type1.allow_unsafe_interrupts=1
[    0.067670] Kernel command line: BOOT_IMAGE=/vmlinuz-linux root=UUID=b4e601d5-a7f8-45a2-a5a0-7f5d8d9fc28f rw loglevel=3 quiet amd_iommu=on iommu=pt vfio_iommu_type1.allow_unsafe_interrupts=1
[    1.325379] VFIO - User Level meta-driver version: 0.3
[    1.328041] vfio-pci 0000:0d:00.0: vgaarb: changed VGA decodes: olddecodes=io+mem,decodes=io+mem:owns=none
[    1.346623] vfio_pci: add [1002:687f[ffffffff:ffffffff]] class 0x000000/00000000
[    1.363282] vfio_pci: add [1002:aaf8[ffffffff:ffffffff]] class 0x000000/00000000
[    1.380052] vfio_pci: add [10ec:8125[ffffffff:ffffffff]] class 0x000000/00000000
[    1.380056] vfio_pci: add [1022:7901[ffffffff:ffffffff]] class 0x000000/00000000
[    3.163304] vfio-pci 0000:0d:00.0: vgaarb: changed VGA decodes: olddecodes=io+mem,decodes=io+mem:owns=none

Output for lspci -nnk -d 1002:687f

0d:00.0 VGA compatible controller [0300]: Advanced Micro Devices, Inc. [AMD/ATI] Vega 10 XL/XT [Radeon RX Vega 56/64] [1002:687f] (rev c3)
        Subsystem: Advanced Micro Devices, Inc. [AMD/ATI] RX Vega64 [1002:6b76]
        Kernel driver in use: vfio-pci
        Kernel modules: amdgpu

vfio configuration cat /etc/modprobe.d/vfio.conf

options vfio-pci ids=1002:687f,1002:aaf8,10ec:8125,1022:7901

Modules in /etc/mkinitcpio.conf

MODULES=(vfio_pci vfio vfio_iommu_type1 vfio_virqfd amdgpu radeon)

Some other useful information:

  • I used to have a ASUS ROG Strix X470-I and it worked passing those cards and using the host with the integrated Ryzen 5 2400G GPU.

  • I've tried changing the cards to different slots or passing RX560 instead of Vega56 but the same result.

  • I created the VM using virt-manager but edited the xml to pass the SATA disk and specify Vega56 ROM.

  • I'm using OVMF and tried with pc-i440fx and pc-q35 BIOS.

I had no problem with my previous board but this time I have no clue what's happening. Any help is appreciated!
Thanks!

Last edited by fulbokoy (2021-06-12 04:10:05)

Offline

#2 2021-06-12 11:07:46

Lone_Wolf
Administrator
From: Netherlands, Europe
Registered: 2005-10-04
Posts: 15,348

Re: VFIO PCI passthrough with two AMD GPUs

[    1.346623] vfio_pci: add [1002:687f[ffffffff:ffffffff]] class 0x000000/00000000
[    1.363282] vfio_pci: add [1002:aaf8[ffffffff:ffffffff]] class 0x000000/00000000
[    1.380052] vfio_pci: add [10ec:8125[ffffffff:ffffffff]] class 0x000000/00000000
[    1.380056] vfio_pci: add [1022:7901[ffffffff:ffffffff]] class 0x000000/00000000

1002:687f is your vega card
1002:aaf8 is the audio of the vega card
10ec:8125 is a network card

1022:7901 isn't listed in the partial iommu group , but https://pci-ids.ucw.cz/read/PC/1022/7901 indicates it's  a sata controller .

It's also not clear to me if you have 2 discrete videocards or 1 integrated and another discrete one.

please post full lspci -knn and SYSTEMD_LESS=FRXMK journalctl -b .


I created the VM using virt-manager but edited the xml to pass the SATA disk and specify Vega56 ROM.

complete contents of the xml file please.

Are you using latest firmware for your motherboard ?

Welcome to archlinux forums.

Disliking systemd intensely, but not satisfied with alternatives so focusing on taming systemd.

clean chroot building not flexible enough ?
Try clean chroot manager by graysky

Offline

#3 2021-06-12 16:47:05

fulbokoy
Member
Registered: 2021-06-12
Posts: 3

Re: VFIO PCI passthrough with two AMD GPUs

Thanks for your reply!

I have two discrete video cards, a RX560 and a Vega 56. I want to use the RX560 for the host and pass the Vega 56 to the guest.

As requeted:
The full result for lspci -knn:

00:00.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse Root Complex [1022:1480]
	Subsystem: ASUSTeK Computer Inc. Device [1043:8808]
00:00.2 IOMMU [0806]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse IOMMU [1022:1481]
	Subsystem: ASUSTeK Computer Inc. Device [1043:8808]
00:01.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge [1022:1482]
00:01.2 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse GPP Bridge [1022:1483]
	Kernel driver in use: pcieport
00:02.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge [1022:1482]
	DeviceName:  Onboard IGD
00:03.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge [1022:1482]
00:03.1 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse GPP Bridge [1022:1483]
	Kernel driver in use: pcieport
00:03.2 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse GPP Bridge [1022:1483]
	Kernel driver in use: pcieport
00:04.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge [1022:1482]
00:05.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge [1022:1482]
00:07.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge [1022:1482]
00:07.1 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse Internal PCIe GPP Bridge 0 to bus[E:B] [1022:1484]
	Kernel driver in use: pcieport
00:08.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge [1022:1482]
00:08.1 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse Internal PCIe GPP Bridge 0 to bus[E:B] [1022:1484]
	Kernel driver in use: pcieport
00:14.0 SMBus [0c05]: Advanced Micro Devices, Inc. [AMD] FCH SMBus Controller [1022:790b] (rev 61)
	Subsystem: ASUSTeK Computer Inc. Device [1043:87c0]
	Kernel driver in use: piix4_smbus
	Kernel modules: i2c_piix4, sp5100_tco
00:14.3 ISA bridge [0601]: Advanced Micro Devices, Inc. [AMD] FCH LPC Bridge [1022:790e] (rev 51)
	Subsystem: ASUSTeK Computer Inc. Device [1043:87c0]
00:18.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Matisse Device 24: Function 0 [1022:1440]
00:18.1 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Matisse Device 24: Function 1 [1022:1441]
00:18.2 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Matisse Device 24: Function 2 [1022:1442]
00:18.3 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Matisse Device 24: Function 3 [1022:1443]
	Kernel driver in use: k10temp
	Kernel modules: k10temp
00:18.4 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Matisse Device 24: Function 4 [1022:1444]
00:18.5 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Matisse Device 24: Function 5 [1022:1445]
00:18.6 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Matisse Device 24: Function 6 [1022:1446]
00:18.7 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Matisse Device 24: Function 7 [1022:1447]
01:00.0 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Matisse Switch Upstream [1022:57ad]
	Kernel driver in use: pcieport
02:01.0 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Matisse PCIe GPP Bridge [1022:57a3]
	Kernel driver in use: pcieport
02:03.0 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Matisse PCIe GPP Bridge [1022:57a3]
	Kernel driver in use: pcieport
02:05.0 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Matisse PCIe GPP Bridge [1022:57a3]
	Kernel driver in use: pcieport
02:06.0 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Matisse PCIe GPP Bridge [1022:57a3]
	Kernel driver in use: pcieport
02:08.0 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Matisse PCIe GPP Bridge [1022:57a4]
	Kernel driver in use: pcieport
02:09.0 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Matisse PCIe GPP Bridge [1022:57a4]
	Kernel driver in use: pcieport
02:0a.0 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Matisse PCIe GPP Bridge [1022:57a4]
	Kernel driver in use: pcieport
03:00.0 Non-Volatile memory controller [0108]: Sandisk Corp WD Black NVMe SSD [15b7:5001]
	Subsystem: Marvell Technology Group Ltd. Device [1b4b:1093]
	Kernel driver in use: nvme
04:00.0 Ethernet controller [0200]: Realtek Semiconductor Co., Ltd. RTL8125 2.5GbE Controller [10ec:8125]
	Subsystem: ASUSTeK Computer Inc. Device [1043:879b]
	Kernel driver in use: vfio-pci
	Kernel modules: r8169
05:00.0 Ethernet controller [0200]: Intel Corporation I211 Gigabit Network Connection [8086:1539] (rev 03)
	Subsystem: ASUSTeK Computer Inc. Device [1043:85f0]
	Kernel driver in use: igb
	Kernel modules: igb
06:00.0 Network controller [0280]: Intel Corporation Wi-Fi 6 AX200 [8086:2723] (rev 1a)
	Subsystem: Intel Corporation Device [8086:0084]
	Kernel driver in use: iwlwifi
	Kernel modules: iwlwifi
07:00.0 Non-Essential Instrumentation [1300]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse Reserved SPP [1022:1485]
	Subsystem: ASUSTeK Computer Inc. Device [1043:8808]
07:00.1 USB controller [0c03]: Advanced Micro Devices, Inc. [AMD] Matisse USB 3.0 Host Controller [1022:149c]
	Subsystem: ASUSTeK Computer Inc. Device [1043:8808]
	Kernel driver in use: xhci_hcd
	Kernel modules: xhci_pci
07:00.3 USB controller [0c03]: Advanced Micro Devices, Inc. [AMD] Matisse USB 3.0 Host Controller [1022:149c]
	Subsystem: Advanced Micro Devices, Inc. [AMD] Device [1022:148c]
	Kernel driver in use: xhci_hcd
	Kernel modules: xhci_pci
08:00.0 SATA controller [0106]: Advanced Micro Devices, Inc. [AMD] FCH SATA Controller [AHCI mode] [1022:7901] (rev 51)
	Subsystem: Advanced Micro Devices, Inc. [AMD] FCH SATA Controller [AHCI mode] [1022:7901]
	Kernel driver in use: ahci
09:00.0 SATA controller [0106]: Advanced Micro Devices, Inc. [AMD] FCH SATA Controller [AHCI mode] [1022:7901] (rev 51)
	Subsystem: Advanced Micro Devices, Inc. [AMD] FCH SATA Controller [AHCI mode] [1022:7901]
	Kernel driver in use: ahci
0a:00.0 VGA compatible controller [0300]: Advanced Micro Devices, Inc. [AMD/ATI] Baffin [Radeon RX 550 640SP / RX 560/560X] [1002:67ff] (rev cf)
	Subsystem: Micro-Star International Co., Ltd. [MSI] Device [1462:8a91]
	Kernel driver in use: amdgpu
	Kernel modules: amdgpu
0a:00.1 Audio device [0403]: Advanced Micro Devices, Inc. [AMD/ATI] Baffin HDMI/DP Audio [Radeon RX 550 640SP / RX 560/560X] [1002:aae0]
	Subsystem: Micro-Star International Co., Ltd. [MSI] Device [1462:aae0]
	Kernel driver in use: snd_hda_intel
	Kernel modules: snd_hda_intel
0b:00.0 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Vega 10 PCIe Bridge [1022:1470] (rev c3)
	Kernel driver in use: pcieport
0c:00.0 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Vega 10 PCIe Bridge [1022:1471]
	Kernel driver in use: pcieport
0d:00.0 VGA compatible controller [0300]: Advanced Micro Devices, Inc. [AMD/ATI] Vega 10 XL/XT [Radeon RX Vega 56/64] [1002:687f] (rev c3)
	Subsystem: Advanced Micro Devices, Inc. [AMD/ATI] RX Vega64 [1002:6b76]
	Kernel driver in use: vfio-pci
	Kernel modules: amdgpu
0d:00.1 Audio device [0403]: Advanced Micro Devices, Inc. [AMD/ATI] Vega 10 HDMI Audio [Radeon Vega 56/64] [1002:aaf8]
	Subsystem: Advanced Micro Devices, Inc. [AMD/ATI] Vega 10 HDMI Audio [Radeon Vega 56/64] [1002:aaf8]
	Kernel driver in use: vfio-pci
	Kernel modules: snd_hda_intel
0e:00.0 Non-Essential Instrumentation [1300]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Function [1022:148a]
	Subsystem: ASUSTeK Computer Inc. Device [1043:8808]
0f:00.0 Non-Essential Instrumentation [1300]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse Reserved SPP [1022:1485]
	Subsystem: ASUSTeK Computer Inc. Device [1043:8808]
0f:00.1 Encryption controller [1080]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse Cryptographic Coprocessor PSPCPP [1022:1486]
	Subsystem: ASUSTeK Computer Inc. Device [1043:8808]
	Kernel driver in use: ccp
	Kernel modules: ccp
0f:00.3 USB controller [0c03]: Advanced Micro Devices, Inc. [AMD] Matisse USB 3.0 Host Controller [1022:149c]
	Subsystem: ASUSTeK Computer Inc. Device [1043:87c0]
	Kernel driver in use: xhci_hcd
	Kernel modules: xhci_pci
0f:00.4 Audio device [0403]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse HD Audio Controller [1022:1487]
	Subsystem: ASUSTeK Computer Inc. Device [1043:87c4]
	Kernel driver in use: snd_hda_intel
	Kernel modules: snd_hda_intel

SYSTEMD_LESS=FRXMK journalctl -b content is in this gist journalctl output

The xml for my VM is:

<domain type="kvm">
  <name>win10</name>
  <uuid>5d4e7bec-5de9-45b3-b4c1-dcc9ff042e1f</uuid>
  <metadata>
    <libosinfo:libosinfo xmlns:libosinfo="http://libosinfo.org/xmlns/libvirt/domain/1.0">
      <libosinfo:os id="http://microsoft.com/win/10"/>
    </libosinfo:libosinfo>
  </metadata>
  <memory unit="KiB">16777216</memory>
  <currentMemory unit="KiB">16777216</currentMemory>
  <vcpu placement="static">6</vcpu>
  <os>
    <type arch="x86_64" machine="pc-i440fx-6.0">hvm</type>
    <loader readonly="yes" type="pflash">/usr/share/edk2-ovmf/x64/OVMF_CODE.fd</loader>
    <nvram>/var/lib/libvirt/qemu/nvram/win10_VARS.fd</nvram>
    <bootmenu enable="no"/>
  </os>
  <features>
    <acpi/>
    <apic/>
    <hyperv>
      <relaxed state="on"/>
      <vapic state="on"/>
      <spinlocks state="on" retries="8191"/>
      <vendor_id state="on" value="randomid"/>
    </hyperv>
    <vmport state="off"/>
  </features>
  <cpu mode="host-passthrough" check="none" migratable="on"/>
  <clock offset="localtime">
    <timer name="rtc" tickpolicy="catchup"/>
    <timer name="pit" tickpolicy="delay"/>
    <timer name="hpet" present="no"/>
    <timer name="hypervclock" present="yes"/>
  </clock>
  <on_poweroff>destroy</on_poweroff>
  <on_reboot>restart</on_reboot>
  <on_crash>destroy</on_crash>
  <pm>
    <suspend-to-mem enabled="no"/>
    <suspend-to-disk enabled="no"/>
  </pm>
  <devices>
    <emulator>/usr/bin/qemu-system-x86_64</emulator>
    <disk type="file" device="cdrom">
      <driver name="qemu" type="raw"/>
      <source file="/home/fulbokoy/Downloads/Win10_21H1_EnglishInternational_x64.iso"/>
      <target dev="sda" bus="sata"/>
      <readonly/>
      <address type="drive" controller="0" bus="0" target="0" unit="0"/>
    </disk>
    <disk type="file" device="cdrom">
      <driver name="qemu" type="raw"/>
      <source file="/home/fulbokoy/Downloads/virtio-win-0.1.190.iso"/>
      <target dev="sdb" bus="sata"/>
      <readonly/>
      <address type="drive" controller="0" bus="0" target="0" unit="1"/>
    </disk>
    <disk type="block" device="disk">
      <driver name="qemu" type="raw" cache="unsafe"/>
      <source dev="/dev/disk/by-id/ata-INTEL_SSDSC2KW512G8_BTLA74160GCK512DGN"/>
      <target dev="vdb" bus="virtio"/>
      <boot order="1"/>
      <address type="pci" domain="0x0000" bus="0x00" slot="0x06" function="0x0"/>
    </disk>
    <controller type="usb" index="0" model="qemu-xhci" ports="15">
      <address type="pci" domain="0x0000" bus="0x00" slot="0x03" function="0x0"/>
    </controller>
    <controller type="pci" index="0" model="pci-root"/>
    <controller type="sata" index="0">
      <address type="pci" domain="0x0000" bus="0x00" slot="0x04" function="0x0"/>
    </controller>
    <input type="mouse" bus="ps2"/>
    <input type="keyboard" bus="ps2"/>
    <sound model="ich9">
      <address type="pci" domain="0x0000" bus="0x00" slot="0x02" function="0x0"/>
    </sound>
    <audio id="1" type="none"/>
    <hostdev mode="subsystem" type="pci" managed="yes">
      <driver name="vfio"/>
      <source>
        <address domain="0x0000" bus="0x04" slot="0x00" function="0x0"/>
      </source>
      <address type="pci" domain="0x0000" bus="0x00" slot="0x05" function="0x0"/>
    </hostdev>
    <hostdev mode="subsystem" type="usb" managed="yes">
      <source>
        <vendor id="0x174c"/>
        <product id="0x3074"/>
      </source>
      <address type="usb" bus="0" port="1"/>
    </hostdev>
    <hostdev mode="subsystem" type="pci" managed="yes">
      <source>
        <address domain="0x0000" bus="0x0d" slot="0x00" function="0x0"/>
      </source>
      <rom bar="on" file="/etc/firmware/Vega56.rom"/>
      <address type="pci" domain="0x0000" bus="0x00" slot="0x07" function="0x0"/>
    </hostdev>
    <hostdev mode="subsystem" type="pci" managed="yes">
      <source>
        <address domain="0x0000" bus="0x0d" slot="0x00" function="0x1"/>
      </source>
      <address type="pci" domain="0x0000" bus="0x00" slot="0x08" function="0x0"/>
    </hostdev>
    <redirdev bus="usb" type="spicevmc">
      <address type="usb" bus="0" port="2"/>
    </redirdev>
    <redirdev bus="usb" type="spicevmc">
      <address type="usb" bus="0" port="3"/>
    </redirdev>
    <memballoon model="virtio">
      <address type="pci" domain="0x0000" bus="0x00" slot="0x09" function="0x0"/>
    </memballoon>
  </devices>
</domain>

As for the motherboard version, I'm using Version 3401 that is not the latest, but latest is Version 3501 Beta Version so I avoid it. Should I give it a try?

Thanks again!

Offline

#4 2021-06-13 10:51:03

Lone_Wolf
Administrator
From: Netherlands, Europe
Registered: 2005-10-04
Posts: 15,348

Re: VFIO PCI passthrough with two AMD GPUs

Update AMD AM4 AGESA V2 PI 1.2.0.2

Newer agesa often solves low level issues, does this board ofter an easy method to revert a firmware update ?

   <hostdev mode="subsystem" type="pci" managed="yes">
      <driver name="vfio"/>
      <source>
        <address domain="0x0000" bus="0x04" slot="0x00" function="0x0"/>
      </source>
      <address type="pci" domain="0x0000" bus="0x00" slot="0x05" function="0x0"/>
    </hostdev>

That's the only reference to vfio in that xml file , are you sure that's the correct file ?

journal looks good overall with maybe some sound related issues, but there is one related to libvirt :

Jun 12 18:25:24 tabibito libvirtd[464]: Cannot find 'dmidecode' in path: No such file or directory
Jun 12 18:25:24 tabibito systemd[545]: Reached target Sound Card.
Jun 12 18:25:24 tabibito libvirtd[464]: Cannot find 'dmidecode' in path: No such file or directory

dmidecode is an optional dep for libvirtd, better install it (and check other optional dependencies) .


Jun 12 18:25:22 tabibito kernel: Command line: BOOT_IMAGE=/vmlinuz-linux root=UUID=b4e601d5-a7f8-45a2-a5a0-7f5d8d9fc28f rw loglevel=3 quiet amd_iommu=on iommu=pt vfio_iommu_type1.allow_unsafe_interrupts=1

amd_iommu=on is not mentioned in kernel paramaters* ,
iommu=pt does exist but I've seen plenty of logs from amd systems where pci-passthrough works fine without iommu=pt .

Are you sure you need the vfio_iommu_type1.allow_unsafe_interrupts=1 parameter ?



* AMD processors have used IOMMU much longer then intel, it's normal for amd system to enable iommu through firmware, not OS .

Disliking systemd intensely, but not satisfied with alternatives so focusing on taming systemd.

clean chroot building not flexible enough ?
Try clean chroot manager by graysky

Offline

#5 2021-06-13 16:12:35

fulbokoy
Member
Registered: 2021-06-12
Posts: 3

Re: VFIO PCI passthrough with two AMD GPUs

Hi!
Based on you suggestions I've done the following changes:

  • Updated the BIOS to Version 3501.

  • Installed dmidecode. Other missing optional dependencies were storage related.

  • Left only the iommu=pt parameter.

  • Corrected the xml file sections to use  <driver name="vfio"/>.

Corrected xml file section:

<hostdev mode="subsystem" type="pci" managed="yes">
      <driver name="vfio"/>
      <source>
        <address domain="0x0000" bus="0x0d" slot="0x00" function="0x0"/>
      </source>
      <rom bar="on" file="/etc/firmware/Vega56.rom"/>
      <address type="pci" domain="0x0000" bus="0x00" slot="0x07" function="0x0"/>
    </hostdev>
    <hostdev mode="subsystem" type="pci" managed="yes">
      <driver name="vfio"/>
      <source>
        <address domain="0x0000" bus="0x0d" slot="0x00" function="0x1"/>
      </source>
      <address type="pci" domain="0x0000" bus="0x00" slot="0x08" function="0x0"/>
    </hostdev>

Still it's the same behavior.
Any other information you may want to look at? Or this looks like a motherboard related problem? sad
Or maybe should I try ACS override? I think I have good groups though.
Thanks!

Last edited by fulbokoy (2021-06-13 16:14:37)

Offline

Pages: 1

Board footer

Powered by FluxBB