RX 6800 XT GPU passthrough KVM

archiba · 2021-06-19 21:33:32

Hello
I have done a search on the forums and couldn't find a solution to my problem and was hoping someone could help me and catch what I'm missing.
Here's my current problem, my gaming guest VM goes black-screen during AMD driver installer and if i wait long enough my host system eventually crash reboots.

When my host system boots back up it uses the guest GPU rather than the host GPU ignoring the virto-pci ids. To fix it back to the correct order a hard shutdown and removing the display cable is needed.
This reminded me of a behavior of the earlier generation with the AMD reset bug but from everything I've read on the internet it should be fixed for the 6800 series.

I followed this guide: https://wiki.archlinux.org/title/PCI_pa … h_via_OVMF right after a fresh install of archlinux and i3 but couldn't figure out where to put the iommu=pt

Im running: Linux archlinux 5.12.11-arch1-1 #1 SMP PREEMPT Wed, 16 Jun 2021 15:25:28 +0000 x86_64 GNU/Linux
Here is my lspci -k output:

 04:00.0 VGA compatible controller: NVIDIA Corporation GK208B [GeForce GT 710] (rev a1)
	Subsystem: Micro-Star International Co., Ltd. [MSI] Device 8c93
	Kernel driver in use: nvidia
	Kernel modules: nouveau, nvidia_drm, nvidia
04:00.1 Audio device: NVIDIA Corporation GK208 HDMI/DP Audio Controller (rev a1)
	Subsystem: Micro-Star International Co., Ltd. [MSI] Device 8c93
	Kernel driver in use: snd_hda_intel
	Kernel modules: snd_hda_intel
05:00.0 Non-Volatile memory controller: Samsung Electronics Co Ltd NVMe SSD Controller SM981/PM981/PM983
	Subsystem: Samsung Electronics Co Ltd Device a801
	Kernel driver in use: nvme
07:00.0 Ethernet controller: Intel Corporation Ethernet Controller I225-V (rev 02)
	Subsystem: ASUSTeK Computer Inc. Device 87d2
	Kernel driver in use: igc
	Kernel modules: igc
08:00.0 PCI bridge: Advanced Micro Devices, Inc. [AMD/ATI] Navi 10 XL Upstream Port of PCI Express Switch (rev c1)
	Kernel driver in use: pcieport
09:00.0 PCI bridge: Advanced Micro Devices, Inc. [AMD/ATI] Navi 10 XL Downstream Port of PCI Express Switch
	Kernel driver in use: pcieport
0a:00.0 VGA compatible controller: Advanced Micro Devices, Inc. [AMD/ATI] Navi 21 [Radeon RX 6800/6800 XT / 6900 XT] (rev c1)
	Subsystem: Tul Corporation / PowerColor Device 2406
	Kernel driver in use: vfio-pci
	Kernel modules: amdgpu
0a:00.1 Audio device: Advanced Micro Devices, Inc. [AMD/ATI] Navi 21 HDMI Audio [Radeon RX 6800/6800 XT / 6900 XT]
	Subsystem: Advanced Micro Devices, Inc. [AMD/ATI] Navi 21 HDMI Audio [Radeon RX 6800/6800 XT / 6900 XT]
	Kernel driver in use: vfio-pci
	Kernel modules: snd_hda_intel
0b:00.0 Non-Essential Instrumentation [1300]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Function
	Subsystem: ASUSTeK Computer Inc. Device 8808
0c:00.0 Non-Essential Instrumentation [1300]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse Reserved SPP
	Subsystem: ASUSTeK Computer Inc. Device 8808
0c:00.1 Encryption controller: Advanced Micro Devices, Inc. [AMD] Starship/Matisse Cryptographic Coprocessor PSPCPP
	Subsystem: ASUSTeK Computer Inc. Device 8808
	Kernel driver in use: ccp
	Kernel modules: ccp
0c:00.3 USB controller: Advanced Micro Devices, Inc. [AMD] Matisse USB 3.0 Host Controller
	Subsystem: ASUSTeK Computer Inc. Device 87c0
	Kernel driver in use: xhci_hcd
	Kernel modules: xhci_pci
0c:00.4 Audio device: Advanced Micro Devices, Inc. [AMD] Starship/Matisse HD Audio Controller
	Subsystem: ASUSTeK Computer Inc. Device 87c5
	Kernel driver in use: snd_hda_intel
	Kernel modules: snd_hda_intel

Hardware:
AMD Ryzen 7 5800X
PowerColor Radeon RX 6800 XT 16GB Red Dragon

My iommu group script output from the guide:

 
IOMMU Group 0:
	00:01.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge [1022:1482]
IOMMU Group 1:
	00:01.1 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse GPP Bridge [1022:1483]
IOMMU Group 2:
	00:01.2 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse GPP Bridge [1022:1483]
IOMMU Group 3:
	00:02.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge [1022:1482]
IOMMU Group 4:
	00:03.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge [1022:1482]
IOMMU Group 5:
	00:03.1 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse GPP Bridge [1022:1483]
IOMMU Group 6:
	00:04.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge [1022:1482]
IOMMU Group 7:
	00:05.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge [1022:1482]
IOMMU Group 8:
	00:07.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge [1022:1482]
IOMMU Group 9:
	00:07.1 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse Internal PCIe GPP Bridge 0 to bus[E:B] [1022:1484]
IOMMU Group 10:
	00:08.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge [1022:1482]
IOMMU Group 11:
	00:08.1 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse Internal PCIe GPP Bridge 0 to bus[E:B] [1022:1484]
IOMMU Group 12:
	00:14.0 SMBus [0c05]: Advanced Micro Devices, Inc. [AMD] FCH SMBus Controller [1022:790b] (rev 61)
	00:14.3 ISA bridge [0601]: Advanced Micro Devices, Inc. [AMD] FCH LPC Bridge [1022:790e] (rev 51)
IOMMU Group 13:
	00:18.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Matisse Device 24: Function 0 [1022:1440]
	00:18.1 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Matisse Device 24: Function 1 [1022:1441]
	00:18.2 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Matisse Device 24: Function 2 [1022:1442]
	00:18.3 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Matisse Device 24: Function 3 [1022:1443]
	00:18.4 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Matisse Device 24: Function 4 [1022:1444]
	00:18.5 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Matisse Device 24: Function 5 [1022:1445]
	00:18.6 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Matisse Device 24: Function 6 [1022:1446]
	00:18.7 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Matisse Device 24: Function 7 [1022:1447]
IOMMU Group 14:
	01:00.0 Non-Volatile memory controller [0108]: Samsung Electronics Co Ltd NVMe SSD Controller PM9A1/PM9A3/980PRO [144d:a80a]
IOMMU Group 15:
	02:00.0 USB controller [0c03]: Advanced Micro Devices, Inc. [AMD] Device [1022:43ee]
	02:00.1 SATA controller [0106]: Advanced Micro Devices, Inc. [AMD] Device [1022:43eb]
	02:00.2 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Device [1022:43e9]
	03:00.0 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Device [1022:43ea]
	03:04.0 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Device [1022:43ea]
	03:08.0 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Device [1022:43ea]
	03:09.0 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Device [1022:43ea]
	04:00.0 VGA compatible controller [0300]: NVIDIA Corporation GK208B [GeForce GT 710] [10de:128b] (rev a1)
	04:00.1 Audio device [0403]: NVIDIA Corporation GK208 HDMI/DP Audio Controller [10de:0e0f] (rev a1)
	05:00.0 Non-Volatile memory controller [0108]: Samsung Electronics Co Ltd NVMe SSD Controller SM981/PM981/PM983 [144d:a808]
	07:00.0 Ethernet controller [0200]: Intel Corporation Ethernet Controller I225-V [8086:15f3] (rev 02)
IOMMU Group 16:
	08:00.0 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD/ATI] Navi 10 XL Upstream Port of PCI Express Switch [1002:1478] (rev c1)
IOMMU Group 17:
	09:00.0 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD/ATI] Navi 10 XL Downstream Port of PCI Express Switch [1002:1479]
IOMMU Group 18:
	0a:00.0 VGA compatible controller [0300]: Advanced Micro Devices, Inc. [AMD/ATI] Navi 21 [Radeon RX 6800/6800 XT / 6900 XT] [1002:73bf] (rev c1)
IOMMU Group 19:
	0a:00.1 Audio device [0403]: Advanced Micro Devices, Inc. [AMD/ATI] Navi 21 HDMI Audio [Radeon RX 6800/6800 XT / 6900 XT] [1002:ab28]
IOMMU Group 20:
	0b:00.0 Non-Essential Instrumentation [1300]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Function [1022:148a]
IOMMU Group 21:
	0c:00.0 Non-Essential Instrumentation [1300]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse Reserved SPP [1022:1485]
IOMMU Group 22:
	0c:00.1 Encryption controller [1080]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse Cryptographic Coprocessor PSPCPP [1022:1486]
IOMMU Group 23:
	0c:00.3 USB controller [0c03]: Advanced Micro Devices, Inc. [AMD] Matisse USB 3.0 Host Controller [1022:149c]
IOMMU Group 24:
	0c:00.4 Audio device [0403]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse HD Audio Controller [1022:1487]

my /etc/mkinitcpio.conf:

 
MODULES=(vfio_pci vfio vfio_iommu_type1 vfio_virqfd nvidia nvidia_modeset nvidia_uvm nvidia_drm)
BINARIES=()
FILES=()
HOOKS=(base udev autodetect modconf vfio keyboard keymap block encrypt filesystems fsck)

my modprobe.d/virtio.conf:

options vfio-pci ids=1002:73bf,1002:ab28

jasonwryan · 2021-06-19 21:42:59

archiba wrote:

but couldn't figure out where to put the iommu=pt

There's a link to the kernel parameters page right in the relevant section of the wiki page you quoted.

archiba · 2021-06-19 22:23:44

jasonwryan wrote:

archiba wrote:
but couldn't figure out where to put the iommu=pt
There's a link to the kernel parameters page right in the relevant section of the wiki page you quoted.

Yes im using the default that came with archlinux systemd-boot. Does that mean i put iommu=pt in /boot/loader/loader.conf ?

V1del · 2021-06-19 22:26:43

No, you add it to the entry definition, read the links you are pointed to.

jasonwryan · 2021-06-19 22:28:54

Did you not read that page either? It's spelled out for you in excruciating detail:

wiki page wrote:

To make the change persistent after reboot, edit /boot/loader/entries/arch.conf (assuming you set up your EFI system partition) and add them to the options line

If you can't, or won't, read the documentation, Arch is not for you.

archiba · 2021-06-19 22:53:59

My apologies like i said I'm new to arch and i was confused by the opening statement "press e" so had to research what systemdboot was since i have no experience with arch previously. cat /proc/cmdline now shows:

initrd=\initramfs-linux.img cryptdevice=PARTUUID=3d2820b9-844c-4555-a250-8fe526fc5f6e:luksdev root=/dev/mapper/luksdev rw intel_pstate=no_hwp random.trust_cpu=on amd_iommu=on iommu=pt vfio-pci.ids=1002:73bf,1002:ab28

It didnt improve my situation.

Last edited by archiba (2021-06-19 23:48:16)

Lone_Wolf · 2021-06-20 12:38:32

intel_pstate=no_hwp random.trust_cpu=on amd_iommu=on iommu=pt vfio-pci.ids=1002:73bf,1002:ab28

Don't copy things blindly, intel_pstate=no_hwp is unnecessary & useless on amd processor systems.

Is early microcode loading configured ?

Brand and model of your motherboard ?
Which firmware version ?

In which physical slots are your 2 videocards located ?
Have you setup the firmware to use the nvidiacard as primary videocard* ?

Does it make a difference if you move the nvidia modules to the start of the modules= line in mkinitcpio.conf ?

* not all firmware allows choosing between discrete cards for that purpose, but if they do it should be set correctly.

archiba · 2021-06-22 14:53:14

Lone_Wolf wrote:

intel_pstate=no_hwp random.trust_cpu=on amd_iommu=on iommu=pt vfio-pci.ids=1002:73bf,1002:ab28
Don't copy things blindly, intel_pstate=no_hwp is unnecessary & useless on amd processor systems.
Is early microcode loading configured ?
Brand and model of your motherboard ?
Which firmware version ?
In which physical slots are your 2 videocards located ?
Have you setup the firmware to use the nvidiacard as primary videocard* ?
Does it make a difference if you move the nvidia modules to the start of the modules= line in mkinitcpio.conf ?
* not all firmware allows choosing between discrete cards for that purpose, but if they do it should be set correctly.

Thank you for the reply. My host GPU is in the second PCIE slot i think since the RX6800XT 16GB is 2.5 slots and in the primary PCIE on the motherboard.
I have a ASUS ROG STRIX B550-F GAMING and in the BIOS i can see the GPUs but im unsure if i can choose which one to boot from, ill have to check. Im out of town for a few days so cant test moving nvidia modules before virtio but ill get back with the results.

Doing some more searching online ive found some arch users use a script and have a path to it under files=() in the mkinitcpio. The guide has no mentioning of this. Anyone able to provide the reasoning for this ?

Arch Linux

#1 2021-06-19 21:33:32

RX 6800 XT GPU passthrough KVM

#2 2021-06-19 21:42:59

Re: RX 6800 XT GPU passthrough KVM

#3 2021-06-19 22:23:44

Re: RX 6800 XT GPU passthrough KVM

#4 2021-06-19 22:26:43

Re: RX 6800 XT GPU passthrough KVM

#5 2021-06-19 22:28:54

Re: RX 6800 XT GPU passthrough KVM

#6 2021-06-19 22:53:59

Re: RX 6800 XT GPU passthrough KVM

#7 2021-06-20 12:38:32

Re: RX 6800 XT GPU passthrough KVM

#8 2021-06-22 14:53:14

Re: RX 6800 XT GPU passthrough KVM

Board footer