PCI Passthrough with identical GPU's not working.

jddoxtator · 2022-01-02 23:21:19

I have followed the guide to a tee in order to set up PCI passthrough with two RX 6800's (One being host while one being guest) but I keep getting permission denied on boot when it tried to override the driver of the guest card to vfio-pci.

It passes by too quick for me to read the whole thing but it gives me a directory with the vfio-pci driver then - Override Driver - Permission denied.

I have tried all the scripts for identical graphics cards and the one in the notes from 2017

#!/bin/sh

for i in /sys/devices/pci*/*/*/boot_vga; do
if [ $(cat "$i") -eq 0 ]; then
GPU="${i%/boot_vga}"
AUDIO="$(echo "$GPU" | sed -e "s/0$/1/")"
echo "vfio-pci" > "$GPU/driver_override"
if [ -d "$AUDIO" ]; then
echo "vfio-pci" > "$AUDIO/driver_override"
fi
fi
done

modprobe -i vfio-pci

still I get permission denied and the system loads the AMD driver to both GPU's making PCI passthrough impossible.

Lone_Wolf · 2022-01-05 11:17:12

Please post the output of the script at https://wiki.archlinux.org/title/PCI_pa … _are_valid

Also /etc/mkinitcpio.conf and a full journal (run as root)

SYSTEMD_LESS=FRXMK journalctl -b |& curl -F 'f:1=<-' ix.io

(it will show an url when finished uploading, post that )

jddoxtator · 2022-01-05 14:35:57

Lone_Wolf wrote:

Please post the output of the script at https://wiki.archlinux.org/title/PCI_pa … _are_valid
Also /etc/mkinitcpio.conf and a full journal (run as root)
SYSTEMD_LESS=FRXMK journalctl -b |& curl -F 'f:1=<-' ix.io
(it will show an url when finished uploading, post that )

Thanks for the reply. Here is the info for script:

IOMMU Group 0:
00:01.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge [1022:1482]
IOMMU Group 1:
00:01.1 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse GPP Bridge [1022:1483]
IOMMU Group 2:
00:01.2 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse GPP Bridge [1022:1483]
IOMMU Group 3:
00:02.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge [1022:1482]
IOMMU Group 4:
00:03.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge [1022:1482]
IOMMU Group 5:
00:04.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge [1022:1482]
IOMMU Group 6:
00:05.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge [1022:1482]
IOMMU Group 7:
00:07.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge [1022:1482]
IOMMU Group 8:
00:07.1 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse Internal PCIe GPP Bridge 0 to bus[E:B] [1022:1484]
IOMMU Group 9:
00:08.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge [1022:1482]
IOMMU Group 10:
00:08.1 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse Internal PCIe GPP Bridge 0 to bus[E:B] [1022:1484]
IOMMU Group 11:
00:14.0 SMBus [0c05]: Advanced Micro Devices, Inc. [AMD] FCH SMBus Controller [1022:790b] (rev 61)
00:14.3 ISA bridge [0601]: Advanced Micro Devices, Inc. [AMD] FCH LPC Bridge [1022:790e] (rev 51)
IOMMU Group 12:
00:18.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Starship Device 24; Function 0 [1022:1490]
00:18.1 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Starship Device 24; Function 1 [1022:1491]
00:18.2 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Starship Device 24; Function 2 [1022:1492]
00:18.3 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Starship Device 24; Function 3 [1022:1493]
00:18.4 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Starship Device 24; Function 4 [1022:1494]
00:18.5 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Starship Device 24; Function 5 [1022:1495]
00:18.6 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Starship Device 24; Function 6 [1022:1496]
00:18.7 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Starship Device 24; Function 7 [1022:1497]
IOMMU Group 13:
01:00.0 Non-Volatile memory controller [0108]: Phison Electronics Corporation E16 PCIe4 NVMe Controller [1987:5016] (rev 01)
IOMMU Group 14:
02:00.0 Non-Volatile memory controller [0108]: Samsung Electronics Co Ltd NVMe SSD Controller PM9A1/PM9A3/980PRO [144d:a80a]
IOMMU Group 15:
03:00.0 Non-Essential Instrumentation [1300]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Function [1022:148a]
IOMMU Group 16:
04:00.0 Non-Essential Instrumentation [1300]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse Reserved SPP [1022:1485]
IOMMU Group 17:
04:00.3 USB controller [0c03]: Advanced Micro Devices, Inc. [AMD] Starship USB 3.0 Host Controller [1022:148c]
IOMMU Group 18:
20:01.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge [1022:1482]
IOMMU Group 19:
20:02.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge [1022:1482]
IOMMU Group 20:
20:03.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge [1022:1482]
IOMMU Group 21:
20:03.1 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse GPP Bridge [1022:1483]
IOMMU Group 22:
20:04.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge [1022:1482]
IOMMU Group 23:
20:05.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge [1022:1482]
IOMMU Group 24:
20:07.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge [1022:1482]
IOMMU Group 25:
20:07.1 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse Internal PCIe GPP Bridge 0 to bus[E:B] [1022:1484]
IOMMU Group 26:
20:08.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge [1022:1482]
IOMMU Group 27:
20:08.1 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse Internal PCIe GPP Bridge 0 to bus[E:B] [1022:1484]
IOMMU Group 28:
21:00.0 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD/ATI] Navi 10 XL Upstream Port of PCI Express Switch [1002:1478] (rev c1)
IOMMU Group 29:
22:00.0 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD/ATI] Navi 10 XL Downstream Port of PCI Express Switch [1002:1479]
IOMMU Group 30:
23:00.0 VGA compatible controller [0300]: Advanced Micro Devices, Inc. [AMD/ATI] Navi 21 [Radeon RX 6800/6800 XT / 6900 XT] [1002:73bf] (rev c1)
IOMMU Group 31:
23:00.1 Audio device [0403]: Advanced Micro Devices, Inc. [AMD/ATI] Navi 21 HDMI Audio [Radeon RX 6800/6800 XT / 6900 XT] [1002:ab28]
IOMMU Group 32:
24:00.0 Non-Essential Instrumentation [1300]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Function [1022:148a]
IOMMU Group 33:
25:00.0 Non-Essential Instrumentation [1300]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse Reserved SPP [1022:1485]
IOMMU Group 34:
25:00.1 Encryption controller [1080]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse Cryptographic Coprocessor PSPCPP [1022:1486]
IOMMU Group 35:
25:00.3 USB controller [0c03]: Advanced Micro Devices, Inc. [AMD] Starship USB 3.0 Host Controller [1022:148c]
IOMMU Group 36:
25:00.4 Audio device [0403]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse HD Audio Controller [1022:1487]
IOMMU Group 37:
40:01.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge [1022:1482]
IOMMU Group 38:
40:01.1 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse GPP Bridge [1022:1483]
IOMMU Group 39:
40:02.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge [1022:1482]
IOMMU Group 40:
40:03.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge [1022:1482]
IOMMU Group 41:
40:03.1 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse GPP Bridge [1022:1483]
IOMMU Group 42:
40:04.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge [1022:1482]
IOMMU Group 43:
40:05.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge [1022:1482]
IOMMU Group 44:
40:07.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge [1022:1482]
IOMMU Group 45:
40:07.1 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse Internal PCIe GPP Bridge 0 to bus[E:B] [1022:1484]
IOMMU Group 46:
40:08.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge [1022:1482]
IOMMU Group 47:
40:08.1 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse Internal PCIe GPP Bridge 0 to bus[E:B] [1022:1484]
IOMMU Group 48:
41:00.0 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Matisse Switch Upstream [1022:57ad]
IOMMU Group 49:
42:03.0 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Matisse PCIe GPP Bridge [1022:57a3]
IOMMU Group 50:
42:05.0 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Matisse PCIe GPP Bridge [1022:57a3]
IOMMU Group 51:
42:06.0 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Matisse PCIe GPP Bridge [1022:57a3]
IOMMU Group 52:
42:08.0 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Matisse PCIe GPP Bridge [1022:57a4]
46:00.0 Non-Essential Instrumentation [1300]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse Reserved SPP [1022:1485]
46:00.1 USB controller [0c03]: Advanced Micro Devices, Inc. [AMD] Matisse USB 3.0 Host Controller [1022:149c]
46:00.3 USB controller [0c03]: Advanced Micro Devices, Inc. [AMD] Matisse USB 3.0 Host Controller [1022:149c]
IOMMU Group 53:
42:09.0 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Matisse PCIe GPP Bridge [1022:57a4]
47:00.0 SATA controller [0106]: Advanced Micro Devices, Inc. [AMD] FCH SATA Controller [AHCI mode] [1022:7901] (rev 51)
IOMMU Group 54:
42:0a.0 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Matisse PCIe GPP Bridge [1022:57a4]
48:00.0 SATA controller [0106]: Advanced Micro Devices, Inc. [AMD] FCH SATA Controller [AHCI mode] [1022:7901] (rev 51)
IOMMU Group 55:
43:00.0 Ethernet controller [0200]: Intel Corporation I210 Gigabit Network Connection [8086:1533] (rev 03)
IOMMU Group 56:
44:00.0 Ethernet controller [0200]: Intel Corporation I210 Gigabit Network Connection [8086:1533] (rev 03)
IOMMU Group 57:
45:00.0 Network controller [0280]: Intel Corporation Wi-Fi 6 AX200 [8086:2723] (rev 1a)
IOMMU Group 58:
49:00.0 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD/ATI] Navi 10 XL Upstream Port of PCI Express Switch [1002:1478] (rev c3)
IOMMU Group 59:
4a:00.0 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD/ATI] Navi 10 XL Downstream Port of PCI Express Switch [1002:1479]
IOMMU Group 60:
4b:00.0 VGA compatible controller [0300]: Advanced Micro Devices, Inc. [AMD/ATI] Navi 21 [Radeon RX 6800/6800 XT / 6900 XT] [1002:73bf] (rev c3)
IOMMU Group 61:
4b:00.1 Audio device [0403]: Advanced Micro Devices, Inc. [AMD/ATI] Navi 21 HDMI Audio [Radeon RX 6800/6800 XT / 6900 XT] [1002:ab28]
IOMMU Group 62:
4c:00.0 Non-Essential Instrumentation [1300]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Function [1022:148a]
IOMMU Group 63:
4d:00.0 Non-Essential Instrumentation [1300]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse Reserved SPP [1022:1485]
IOMMU Group 64:
60:01.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge [1022:1482]
IOMMU Group 65:
60:02.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge [1022:1482]
IOMMU Group 66:
60:03.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge [1022:1482]
IOMMU Group 67:
60:04.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge [1022:1482]
IOMMU Group 68:
60:05.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge [1022:1482]
IOMMU Group 69:
60:07.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge [1022:1482]
IOMMU Group 70:
60:07.1 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse Internal PCIe GPP Bridge 0 to bus[E:B] [1022:1484]
IOMMU Group 71:
60:08.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge [1022:1482]
IOMMU Group 72:
60:08.1 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse Internal PCIe GPP Bridge 0 to bus[E:B] [1022:1484]
IOMMU Group 73:
61:00.0 Non-Essential Instrumentation [1300]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Function [1022:148a]
IOMMU Group 74:
62:00.0 Non-Essential Instrumentation [1300]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse Reserved SPP [1022:1485]

jddoxtator · 2022-01-05 14:37:55

Lone_Wolf wrote:

Please post the output of the script at https://wiki.archlinux.org/title/PCI_pa … _are_valid
Also /etc/mkinitcpio.conf and a full journal (run as root)
SYSTEMD_LESS=FRXMK journalctl -b |& curl -F 'f:1=<-' ix.io
(it will show an url when finished uploading, post that )

And here is the generated link: http://ix.io/3L6F

jddoxtator · 2022-01-05 14:40:33

Lone_Wolf wrote:

Please post the output of the script at https://wiki.archlinux.org/title/PCI_pa … _are_valid
Also /etc/mkinitcpio.conf and a full journal (run as root)
SYSTEMD_LESS=FRXMK journalctl -b |& curl -F 'f:1=<-' ix.io
(it will show an url when finished uploading, post that )

here is the contents of /etc/mkinitcpio.conf :

# vim:set ft=sh
# MODULES
# The following modules are loaded before any boot hooks are
# run. Advanced users may wish to specify all system modules
# in this array. For instance:
# MODULES=(piix ide_disk reiserfs)
MODULES=(vfio_pci vfio vfio_iommu_type1 vfio_virqfd)

# BINARIES
# This setting includes any additional binaries a given user may
# wish into the CPIO image. This is run last, so it may be used to
# override the actual binaries included by a given hook
# BINARIES are dependency parsed, so you may safely ignore libraries
BINARIES=()

# FILES
# This setting is similar to BINARIES above, however, files are added
# as-is and are not parsed in any way. This is useful for config files.
FILES=(/usr/local/bin/vfio-pci-override.sh)

# HOOKS
# This is the most important setting in this file. The HOOKS control the
# modules and scripts added to the image, and what happens at boot time.
# Order is important, and it is recommended that you do not change the
# order in which HOOKS are added. Run 'mkinitcpio -H <hook name>' for
# help on a given hook.
# 'base' is _required_ unless you know precisely what you are doing.
# 'udev' is _required_ in order to automatically load modules
# 'filesystems' is _required_ unless you specify your fs modules in MODULES
# Examples:
## This setup specifies all modules in the MODULES setting above.
## No raid, lvm2, or encrypted root is needed.
# HOOKS=(base)
#
## This setup will autodetect all modules for your system and should
## work as a sane default
# HOOKS=(base udev autodetect block filesystems)
#
## This setup will generate a 'full' image which supports most systems.
## No autodetection is done.
# HOOKS=(base udev block filesystems)
#
## This setup assembles a pata mdadm array with an encrypted root FS.
## Note: See 'mkinitcpio -H mdadm' for more information on raid devices.
# HOOKS=(base udev block mdadm encrypt filesystems)
#
## This setup loads an lvm2 volume group on a usb device.
# HOOKS=(base udev block lvm2 filesystems)
#
## NOTE: If you have /usr on a separate partition, you MUST include the
# usr, fsck and shutdown hooks.
HOOKS=(base udev autodetect modconf block lvm2 filesystems keyboard fsck)

# COMPRESSION
# Use this to compress the initramfs image. By default, zstd compression
# is used. Use 'cat' to create an uncompressed image.
#COMPRESSION="zstd"
#COMPRESSION="gzip"
#COMPRESSION="bzip2"
#COMPRESSION="lzma"
#COMPRESSION="xz"
#COMPRESSION="lzop"
#COMPRESSION="lz4"

# COMPRESSION_OPTIONS
# Additional options for the compressor
#COMPRESSION_OPTIONS=()

Lone_Wolf · 2022-01-06 11:58:19

Some general things first :

Use [ code ] tags for output like the iommu group list and mkinitcpio.conf content, it makes posts much more readable .
see https://bbs.archlinux.org/help.php#bbcode

Don't make several posts in a few minutes if no one has responded in between , just edit the first post and add the new content ( often people precede added content with edit1 , edit2 etc )

end of general stuff

A. configure early microcode loading if you haven't done that yet, see https://wiki.archlinux.org/title/Microcode

B.

Jan 02 18:28:08 archlinux systemd-modules-load[824]: Inserted module 'crypto_user'
Jan 02 18:28:08 archlinux systemd-random-seed[830]: Kernel entropy pool is not initialized yet, waiting until it is.

lack of entropy can increase boot time by a lot , read https://wiki.archlinux.org/title/Random … generation

C.

Jan 02 18:28:08 archlinux kernel: DMI: Gigabyte Technology Co., Ltd. TRX40 DESIGNARE/TRX40 DESIGNARE, BIOS FA 07/14/2020

That bios version inidcates you have HW revision 1.1 / 1.2 / 1.3 of the TRX40 designare motherboard.
https://www.gigabyte.com/Motherboard/TR … rt-dl-bios shows there are 2 later versions.

One of them includes an agesa update which very often have fixes for low level issues.
please update your firmware .

D.
Once you have taken care of A-B-C , temporarily add audit=0 systemd.unit=multi-user.target to your bootloader kernel line and reboot.

This will bring you to a console login, login as root .

rerun the commands from #2 and upload/post their output .
reboot to your normal gui login , post lspci -k as user (NOT as root) .

I advise to do it like this to reduce the amount of info that's not important for troubleshooting this.

jddoxtator · 2022-01-06 14:39:35

Lone_Wolf wrote:

Some general things first :
Use [ code ] tags for output like the iommu group list and mkinitcpio.conf content, it makes posts much more readable .
see https://bbs.archlinux.org/help.php#bbcode
Don't make several posts in a few minutes if no one has responded in between , just edit the first post and add the new content ( often people precede added content with edit1 , edit2 etc )
end of general stuff

A. configure early microcode loading if you haven't done that yet, see https://wiki.archlinux.org/title/Microcode
B.
Jan 02 18:28:08 archlinux systemd-modules-load[824]: Inserted module 'crypto_user'
Jan 02 18:28:08 archlinux systemd-random-seed[830]: Kernel entropy pool is not initialized yet, waiting until it is.
lack of entropy can increase boot time by a lot , read https://wiki.archlinux.org/title/Random … generation

C.
Jan 02 18:28:08 archlinux kernel: DMI: Gigabyte Technology Co., Ltd. TRX40 DESIGNARE/TRX40 DESIGNARE, BIOS FA 07/14/2020
That bios version inidcates you have HW revision 1.1 / 1.2 / 1.3 of the TRX40 designare motherboard.
https://www.gigabyte.com/Motherboard/TR … rt-dl-bios shows there are 2 later versions.
One of them includes an agesa update which very often have fixes for low level issues.
please update your firmware .

D.
Once you have taken care of A-B-C , temporarily add audit=0 systemd.unit=multi-user.target to your bootloader kernel line and reboot.
This will bring you to a console login, login as root .
rerun the commands from #2 and upload/post their output .
reboot to your normal gui login , post lspci -k as user (NOT as root) .
I advise to do it like this to reduce the amount of info that's not important for troubleshooting this.

Fixed A, that was already installed anyways, but I am having trouble understanding B.
I read the link and understand what entropy is and why linux uses it. However, after running the suggested code to see what is loading before enough entropy is generated, I am unsure of how to proceed.

I ran this code and got this output :

 sudo dmesg | grep -e random:
[    0.000000] random: get_random_u64 called from __kmem_cache_create+0x2a/0x540 with crng_init=0
[    2.011506] random: fast init done
[    2.513613] random: lvm: uninitialized urandom read (4 bytes read)
[    2.518578] random: lvm: uninitialized urandom read (4 bytes read)
[    2.531613] random: lvm: uninitialized urandom read (2 bytes read)
[    4.333555] random: dbus-daemon: uninitialized urandom read (12 bytes read)
[    4.359026] random: dbus-daemon: uninitialized urandom read (12 bytes read)
[    4.376527] random: NetworkManager: uninitialized urandom read (16 bytes read)
[    4.570037] random: crng init done

So I can see that lvm dbus-daemon NetworkManager are all loading before they have enough entropy. Do I need to modify the size of entropy generated by urandom or just some how load it before these module calls?

Edit: It has also come to my attention that lvm could also be an issue. If there is some process that must be loaded before the system can load lvm, then obviously a call to modify something that exists on a lvm image would not be granted access.

Last edited by jddoxtator (2022-01-06 15:07:27)

Lone_Wolf · 2022-01-06 16:11:36

Those numbers actually look good (and better then on my own system).
I read up a bit and realised rng issues haven't been around since kernel 5.6 .
I was wrong, don't think entropy related changes are needed on your system.

jddoxtator · 2022-01-06 17:49:15

Ignoring part B then

Part C is now done. It was a bit of a pain because I was unaware that a BIOS reset would nuke my UEFI boot table. Had to use a rescue disk, remote in and reinstall grub, but everything is back now.

Part D:
I'm assuming command #2 means the generated link. So here it is after modifying bootloader and logging in as root : http://ix.io/3LgH

Here is the output of lspci -k as user:

 lspci -k
00:00.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Starship/Matisse Root Complex
        Subsystem: Advanced Micro Devices, Inc. [AMD] Starship/Matisse Root Complex
00:00.2 IOMMU: Advanced Micro Devices, Inc. [AMD] Starship/Matisse IOMMU
        Subsystem: Advanced Micro Devices, Inc. [AMD] Starship/Matisse IOMMU
00:01.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge
00:01.1 PCI bridge: Advanced Micro Devices, Inc. [AMD] Starship/Matisse GPP Bridge
        Kernel driver in use: pcieport
00:01.2 PCI bridge: Advanced Micro Devices, Inc. [AMD] Starship/Matisse GPP Bridge
        Kernel driver in use: pcieport
00:02.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge
00:03.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge
00:04.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge
00:05.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge
00:07.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge
00:07.1 PCI bridge: Advanced Micro Devices, Inc. [AMD] Starship/Matisse Internal PCIe GPP Bridge 0 to bus[E:B]
        Kernel driver in use: pcieport
00:08.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge
00:08.1 PCI bridge: Advanced Micro Devices, Inc. [AMD] Starship/Matisse Internal PCIe GPP Bridge 0 to bus[E:B]
        Kernel driver in use: pcieport
00:14.0 SMBus: Advanced Micro Devices, Inc. [AMD] FCH SMBus Controller (rev 61)
        Subsystem: Gigabyte Technology Co., Ltd Device 5001
        Kernel modules: i2c_piix4, sp5100_tco
00:14.3 ISA bridge: Advanced Micro Devices, Inc. [AMD] FCH LPC Bridge (rev 51)
        Subsystem: Gigabyte Technology Co., Ltd Device 5001
00:18.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Starship Device 24; Function 0
00:18.1 Host bridge: Advanced Micro Devices, Inc. [AMD] Starship Device 24; Function 1
00:18.2 Host bridge: Advanced Micro Devices, Inc. [AMD] Starship Device 24; Function 2
00:18.3 Host bridge: Advanced Micro Devices, Inc. [AMD] Starship Device 24; Function 3
        Kernel driver in use: k10temp
        Kernel modules: k10temp
00:18.4 Host bridge: Advanced Micro Devices, Inc. [AMD] Starship Device 24; Function 4
00:18.5 Host bridge: Advanced Micro Devices, Inc. [AMD] Starship Device 24; Function 5
00:18.6 Host bridge: Advanced Micro Devices, Inc. [AMD] Starship Device 24; Function 6
00:18.7 Host bridge: Advanced Micro Devices, Inc. [AMD] Starship Device 24; Function 7
01:00.0 Non-Volatile memory controller: Phison Electronics Corporation E16 PCIe4 NVMe Controller (rev 01)
        Subsystem: Phison Electronics Corporation E16 PCIe4 NVMe Controller
        Kernel driver in use: nvme
02:00.0 Non-Volatile memory controller: Samsung Electronics Co Ltd NVMe SSD Controller PM9A1/PM9A3/980PRO
        Subsystem: Samsung Electronics Co Ltd Device a801
        Kernel driver in use: nvme
03:00.0 Non-Essential Instrumentation [1300]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Function
        Subsystem: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Function
04:00.0 Non-Essential Instrumentation [1300]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse Reserved SPP
        Subsystem: Advanced Micro Devices, Inc. [AMD] Starship/Matisse Reserved SPP
04:00.3 USB controller: Advanced Micro Devices, Inc. [AMD] Starship USB 3.0 Host Controller
        Subsystem: Gigabyte Technology Co., Ltd Device 5007
        Kernel driver in use: xhci_hcd
        Kernel modules: xhci_pci
20:00.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Starship/Matisse Root Complex
        Subsystem: Advanced Micro Devices, Inc. [AMD] Starship/Matisse Root Complex
20:00.2 IOMMU: Advanced Micro Devices, Inc. [AMD] Starship/Matisse IOMMU
        Subsystem: Advanced Micro Devices, Inc. [AMD] Starship/Matisse IOMMU
20:01.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge
20:02.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge
20:03.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge
20:03.1 PCI bridge: Advanced Micro Devices, Inc. [AMD] Starship/Matisse GPP Bridge
        Kernel driver in use: pcieport
20:04.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge
20:05.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge
20:07.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge
20:07.1 PCI bridge: Advanced Micro Devices, Inc. [AMD] Starship/Matisse Internal PCIe GPP Bridge 0 to bus[E:B]
        Kernel driver in use: pcieport
20:08.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge
20:08.1 PCI bridge: Advanced Micro Devices, Inc. [AMD] Starship/Matisse Internal PCIe GPP Bridge 0 to bus[E:B]
        Kernel driver in use: pcieport
21:00.0 PCI bridge: Advanced Micro Devices, Inc. [AMD/ATI] Navi 10 XL Upstream Port of PCI Express Switch (rev c1)
        Kernel driver in use: pcieport
22:00.0 PCI bridge: Advanced Micro Devices, Inc. [AMD/ATI] Navi 10 XL Downstream Port of PCI Express Switch
        Kernel driver in use: pcieport
23:00.0 VGA compatible controller: Advanced Micro Devices, Inc. [AMD/ATI] Navi 21 [Radeon RX 6800/6800 XT / 6900 XT] (rev c1)
        Subsystem: Sapphire Technology Limited Device 439e
        Kernel driver in use: amdgpu
        Kernel modules: amdgpu
23:00.1 Audio device: Advanced Micro Devices, Inc. [AMD/ATI] Navi 21 HDMI Audio [Radeon RX 6800/6800 XT / 6900 XT]
        Subsystem: Advanced Micro Devices, Inc. [AMD/ATI] Navi 21 HDMI Audio [Radeon RX 6800/6800 XT / 6900 XT]
        Kernel driver in use: snd_hda_intel
        Kernel modules: snd_hda_intel
24:00.0 Non-Essential Instrumentation [1300]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Function
        Subsystem: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Function
25:00.0 Non-Essential Instrumentation [1300]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse Reserved SPP
        Subsystem: Advanced Micro Devices, Inc. [AMD] Starship/Matisse Reserved SPP
25:00.1 Encryption controller: Advanced Micro Devices, Inc. [AMD] Starship/Matisse Cryptographic Coprocessor PSPCPP
        Subsystem: Advanced Micro Devices, Inc. [AMD] Starship/Matisse Cryptographic Coprocessor PSPCPP
        Kernel driver in use: ccp
        Kernel modules: ccp
25:00.3 USB controller: Advanced Micro Devices, Inc. [AMD] Starship USB 3.0 Host Controller
        Subsystem: Gigabyte Technology Co., Ltd Device 5007
        Kernel driver in use: xhci_hcd
        Kernel modules: xhci_pci
25:00.4 Audio device: Advanced Micro Devices, Inc. [AMD] Starship/Matisse HD Audio Controller
        Subsystem: Advanced Micro Devices, Inc. [AMD] Device d102
        Kernel driver in use: snd_hda_intel
        Kernel modules: snd_hda_intel
40:00.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Starship/Matisse Root Complex
        Subsystem: Advanced Micro Devices, Inc. [AMD] Starship/Matisse Root Complex
40:00.2 IOMMU: Advanced Micro Devices, Inc. [AMD] Starship/Matisse IOMMU
        Subsystem: Advanced Micro Devices, Inc. [AMD] Starship/Matisse IOMMU
40:01.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge
40:01.1 PCI bridge: Advanced Micro Devices, Inc. [AMD] Starship/Matisse GPP Bridge
        Kernel driver in use: pcieport
40:02.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge
40:03.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge
40:03.1 PCI bridge: Advanced Micro Devices, Inc. [AMD] Starship/Matisse GPP Bridge
        Kernel driver in use: pcieport
40:04.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge
40:05.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge
40:07.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge
40:07.1 PCI bridge: Advanced Micro Devices, Inc. [AMD] Starship/Matisse Internal PCIe GPP Bridge 0 to bus[E:B]
        Kernel driver in use: pcieport
40:08.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge
40:08.1 PCI bridge: Advanced Micro Devices, Inc. [AMD] Starship/Matisse Internal PCIe GPP Bridge 0 to bus[E:B]
        Kernel driver in use: pcieport
41:00.0 PCI bridge: Advanced Micro Devices, Inc. [AMD] Matisse Switch Upstream
        Kernel driver in use: pcieport
42:03.0 PCI bridge: Advanced Micro Devices, Inc. [AMD] Matisse PCIe GPP Bridge
        Kernel driver in use: pcieport
42:05.0 PCI bridge: Advanced Micro Devices, Inc. [AMD] Matisse PCIe GPP Bridge
        Kernel driver in use: pcieport
42:06.0 PCI bridge: Advanced Micro Devices, Inc. [AMD] Matisse PCIe GPP Bridge
        Kernel driver in use: pcieport
42:08.0 PCI bridge: Advanced Micro Devices, Inc. [AMD] Matisse PCIe GPP Bridge
        Kernel driver in use: pcieport
42:09.0 PCI bridge: Advanced Micro Devices, Inc. [AMD] Matisse PCIe GPP Bridge
        Kernel driver in use: pcieport
42:0a.0 PCI bridge: Advanced Micro Devices, Inc. [AMD] Matisse PCIe GPP Bridge
        Kernel driver in use: pcieport
43:00.0 Ethernet controller: Intel Corporation I210 Gigabit Network Connection (rev 03)
        Subsystem: Gigabyte Technology Co., Ltd Device e000
        Kernel driver in use: igb
        Kernel modules: igb
44:00.0 Ethernet controller: Intel Corporation I210 Gigabit Network Connection (rev 03)
        Subsystem: Gigabyte Technology Co., Ltd Device e000
        Kernel driver in use: igb
        Kernel modules: igb
45:00.0 Network controller: Intel Corporation Wi-Fi 6 AX200 (rev 1a)
        DeviceName: Onboard LAN RTL8125
        Subsystem: Intel Corporation Wi-Fi 6 AX200NGW
        Kernel driver in use: iwlwifi
        Kernel modules: iwlwifi
46:00.0 Non-Essential Instrumentation [1300]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse Reserved SPP
        Subsystem: Advanced Micro Devices, Inc. [AMD] Starship/Matisse Reserved SPP
46:00.1 USB controller: Advanced Micro Devices, Inc. [AMD] Matisse USB 3.0 Host Controller
        Subsystem: Advanced Micro Devices, Inc. [AMD] Device 1486
        Kernel driver in use: xhci_hcd
        Kernel modules: xhci_pci
46:00.3 USB controller: Advanced Micro Devices, Inc. [AMD] Matisse USB 3.0 Host Controller
        Subsystem: Advanced Micro Devices, Inc. [AMD] Device 148c
        Kernel driver in use: xhci_hcd
        Kernel modules: xhci_pci
47:00.0 SATA controller: Advanced Micro Devices, Inc. [AMD] FCH SATA Controller [AHCI mode] (rev 51)
        DeviceName: X570/590 SATA0
        Subsystem: Advanced Micro Devices, Inc. [AMD] FCH SATA Controller [AHCI mode]
        Kernel driver in use: ahci
48:00.0 SATA controller: Advanced Micro Devices, Inc. [AMD] FCH SATA Controller [AHCI mode] (rev 51)
        DeviceName: X570/590 SATA1
        Subsystem: Advanced Micro Devices, Inc. [AMD] FCH SATA Controller [AHCI mode]
        Kernel driver in use: ahci
49:00.0 PCI bridge: Advanced Micro Devices, Inc. [AMD/ATI] Navi 10 XL Upstream Port of PCI Express Switch (rev c3)
        Kernel driver in use: pcieport
4a:00.0 PCI bridge: Advanced Micro Devices, Inc. [AMD/ATI] Navi 10 XL Downstream Port of PCI Express Switch
        Kernel driver in use: pcieport
4b:00.0 VGA compatible controller: Advanced Micro Devices, Inc. [AMD/ATI] Navi 21 [Radeon RX 6800/6800 XT / 6900 XT] (rev c3)
        Subsystem: Tul Corporation / PowerColor Device 2407
        Kernel driver in use: amdgpu
        Kernel modules: amdgpu
4b:00.1 Audio device: Advanced Micro Devices, Inc. [AMD/ATI] Navi 21 HDMI Audio [Radeon RX 6800/6800 XT / 6900 XT]
        Subsystem: Advanced Micro Devices, Inc. [AMD/ATI] Navi 21 HDMI Audio [Radeon RX 6800/6800 XT / 6900 XT]
        Kernel driver in use: snd_hda_intel
        Kernel modules: snd_hda_intel
4c:00.0 Non-Essential Instrumentation [1300]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Function
        Subsystem: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Function
4d:00.0 Non-Essential Instrumentation [1300]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse Reserved SPP
        Subsystem: Advanced Micro Devices, Inc. [AMD] Starship/Matisse Reserved SPP
60:00.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Starship/Matisse Root Complex
        Subsystem: Advanced Micro Devices, Inc. [AMD] Starship/Matisse Root Complex
60:00.2 IOMMU: Advanced Micro Devices, Inc. [AMD] Starship/Matisse IOMMU
        Subsystem: Advanced Micro Devices, Inc. [AMD] Starship/Matisse IOMMU
60:01.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge
60:02.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge
60:03.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge
60:04.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge
60:05.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge
60:07.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge
60:07.1 PCI bridge: Advanced Micro Devices, Inc. [AMD] Starship/Matisse Internal PCIe GPP Bridge 0 to bus[E:B]
        Kernel driver in use: pcieport
60:08.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge
60:08.1 PCI bridge: Advanced Micro Devices, Inc. [AMD] Starship/Matisse Internal PCIe GPP Bridge 0 to bus[E:B]
        Kernel driver in use: pcieport
61:00.0 Non-Essential Instrumentation [1300]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Function
        Subsystem: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Function
62:00.0 Non-Essential Instrumentation [1300]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse Reserved SPP
        Subsystem: Advanced Micro Devices, Inc. [AMD] Starship/Matisse Reserved SPP

Lone_Wolf · 2022-01-07 11:53:24

It would have been useful to see if the IOMMU groups had changed (seeing videocard and its audo output in different iommu group as in the script output in post #2 feels weird) , but I think the journal + lspci give enough info to continue.

First videocard + audio

23:00.0 VGA compatible controller: Advanced Micro Devices, Inc. [AMD/ATI] Navi 21 [Radeon RX 6800/6800 XT / 6900 XT] (rev c1)
        Subsystem: Sapphire Technology Limited Device 439e
        Kernel driver in use: amdgpu
        Kernel modules: amdgpu
23:00.1 Audio device: Advanced Micro Devices, Inc. [AMD/ATI] Navi 21 HDMI Audio [Radeon RX 6800/6800 XT / 6900 XT]
        Subsystem: Advanced Micro Devices, Inc. [AMD/ATI] Navi 21 HDMI Audio [Radeon RX 6800/6800 XT / 6900 XT]
        Kernel driver in use: snd_hda_intel
        Kernel modules: snd_hda_intel

Your 2nd videocard + audio

4b:00.0 VGA compatible controller: Advanced Micro Devices, Inc. [AMD/ATI] Navi 21 [Radeon RX 6800/6800 XT / 6900 XT] (rev c3)
        Subsystem: Tul Corporation / PowerColor Device 2407
        Kernel driver in use: amdgpu
        Kernel modules: amdgpu
4b:00.1 Audio device: Advanced Micro Devices, Inc. [AMD/ATI] Navi 21 HDMI Audio [Radeon RX 6800/6800 XT / 6900 XT]
        Subsystem: Advanced Micro Devices, Inc. [AMD/ATI] Navi 21 HDMI Audio [Radeon RX 6800/6800 XT / 6900 XT]
        Kernel driver in use: snd_hda_intel
        Kernel modules: snd_hda_intel

Jan 06 12:31:46 archlinux kernel: pci 0000:23:00.0: vgaarb: setting as boot VGA device
Jan 06 12:31:46 archlinux kernel: pci 0000:23:00.0: vgaarb: VGA device added: decodes=io+mem,owns=io+mem,locks=none
Jan 06 12:31:46 archlinux kernel: pci 0000:4b:00.0: vgaarb: VGA device added: decodes=io+mem,owns=none,locks=none
Jan 06 12:31:46 archlinux kernel: pci 0000:23:00.0: vgaarb: bridge control possible
Jan 06 12:31:46 archlinux kernel: pci 0000:4b:00.0: vgaarb: bridge control possible
Jan 06 12:31:46 archlinux kernel: vgaarb: loaded

We can conclude the card at 23:00.0 is your primary card and bootvga, while the one at 4b:00.0 is secondary and the one you could passthrough .

PCI passthrough can be very tricky to get right, to simplify things I'll use the 2nd wiki script example "Passthrough selected GPU" .

NB I have adjusted the values in DEVS to the ones corresponding to your second card.

#!/bin/sh

DEVS="0000:4b:00.0 0000:4b:00.1"

if [ ! -z "$(ls -A /sys/class/iommu)" ]; then
    for DEV in $DEVS; do
        echo "vfio-pci" > /sys/bus/pci/devices/$DEV/driver_override
    done
fi

modprobe -i vfio-pci

doublecheck the steps in https://wiki.archlinux.org/title/PCI_pa … stallation , then reboot .\

If everything works, lspci -k should now show 4b:00.0 and 4b:00.1 are using vfio-pci instead of amdgpu .

Incase it still doesn't , remove loglevel=3 & quiet from the kenrel line, but keep audit=0 systemd.unit=multi-user.target .
reboot and post fresh journal .

jddoxtator · 2022-01-08 02:44:38

Tried the code with the IOMMU group address and still getting amdgpu driver

Here is the new log after applying default grub file changes: http://ix.io/3Ls2

Lone_Wolf · 2022-01-08 12:18:44

There's no sign of the script being run at all, time for a sanity check .

post /etc/mkinitcpio.conf and /etc/modprobe.d/vfio.conf .

jddoxtator · 2022-01-09 01:11:04

/etc/mkinitcpio.conf :

 # vim:set ft=sh
# MODULES
# The following modules are loaded before any boot hooks are
# run.  Advanced users may wish to specify all system modules
# in this array.  For instance:
#     MODULES=(piix ide_disk reiserfs)
MODULES=(vfio_pci vfio vfio_iommu_type1 vfio_virqfd)

# BINARIES
# This setting includes any additional binaries a given user may
# wish into the CPIO image.  This is run last, so it may be used to
# override the actual binaries included by a given hook
# BINARIES are dependency parsed, so you may safely ignore libraries
BINARIES=()

# FILES
# This setting is similar to BINARIES above, however, files are added
# as-is and are not parsed in any way.  This is useful for config files.
FILES=(/usr/local/bin/vfio-pci-override.sh)

# HOOKS
# This is the most important setting in this file.  The HOOKS control the
# modules and scripts added to the image, and what happens at boot time.
# Order is important, and it is recommended that you do not change the
# order in which HOOKS are added.  Run 'mkinitcpio -H <hook name>' for
# help on a given hook.
# 'base' is _required_ unless you know precisely what you are doing.
# 'udev' is _required_ in order to automatically load modules
# 'filesystems' is _required_ unless you specify your fs modules in MODULES
# Examples:
##   This setup specifies all modules in the MODULES setting above.
##   No raid, lvm2, or encrypted root is needed.
#    HOOKS=(base)
#
##   This setup will autodetect all modules for your system and should
##   work as a sane default
#    HOOKS=(base udev autodetect block filesystems)
#
##   This setup will generate a 'full' image which supports most systems.
##   No autodetection is done.
#    HOOKS=(base udev block filesystems)
#
##   This setup assembles a pata mdadm array with an encrypted root FS.
##   Note: See 'mkinitcpio -H mdadm' for more information on raid devices.
#    HOOKS=(base udev block mdadm encrypt filesystems)
#
##   This setup loads an lvm2 volume group on a usb device.
#    HOOKS=(base udev block lvm2 filesystems)
#
##   NOTE: If you have /usr on a separate partition, you MUST include the
#    usr, fsck and shutdown hooks.
HOOKS=(base udev autodetect modconf block lvm2 filesystems keyboard fsck)

# COMPRESSION
# Use this to compress the initramfs image. By default, zstd compression
# is used. Use 'cat' to create an uncompressed image.
#COMPRESSION="zstd"
#COMPRESSION="gzip"
#COMPRESSION="bzip2"
#COMPRESSION="lzma"
#COMPRESSION="xz"
#COMPRESSION="lzop"
#COMPRESSION="lz4"

# COMPRESSION_OPTIONS
# Additional options for the compressor
#COMPRESSION_OPTIONS=()

/etc/modprobe.d/vfio.conf :

 install vfio-pci /usr/local/bin/vfio-pci-override.sh

Lone_Wolf · 2022-01-09 11:12:23

Those look good.

What does

ls -l /usr/local/bin

output ?

Add these 2 lines

echo "Enter something: "
read mytest

just under the #!/bin/sh line in /usr/bin/local/vfio-pci-override.sh

regenerate initramfs, reboot .
When the script is run your system should now display "Enter something: " and wait for your input .

jddoxtator · 2022-01-10 01:55:27

Output for code:

 ls -l /usr/local/bin
total 8
-rwxr-xr-x 1 root root 1193 Sep 26 11:23 ddgtk
-rw-r--r-- 1 root root  255 Jan  9 20:33 vfio-pci-override.sh

Tried the 2 lines added to vfio-pci-override.sh and nothing happened. Just booted directly to desktop.

Maybe a sanity check of my own.

mkinitcpio -P

is for regenerating, correct? I also tried

mkinitcpio --config /etc/mkinitcpio.conf --generate /boot/initramfs.img

but got the same result.

Lone_Wolf · 2022-01-10 11:16:24

I personally prefer mkinitcpio -p dedicatedpreset but -P is fine for regenerating, yes.

Glad I asked for the ls output, because that explains why the script isn't executed : It lacks the executable flag.

As root run

chmod +x /usr/local/bin/vfio-pci-override.sh

Verify the change is correct by re-running "ls -l /usr/local/bin" , the line vfio-pci-override.sh should look the same as that for ddgtk .

No need to regenerate the initramfs, just reboot and report what happens.

Last edited by Lone_Wolf (2022-01-10 11:17:26)

jddoxtator · 2022-01-10 15:39:37

Alright, some interesting things of note happened after that change.

First of all the prompt finally appeared on boot asking for input. Unfortunately, it loaded before the USB drivers so I couldn't actually input anything. Another trip in with the rescue disk to revert that change.

Secondly, that change seems to have fixed a reboot hang issue I was having. Don't know if that is relevant. Before, sometimes when I would reboot, it would hang at a blank screen and never reboot. Now it does.

Thirdly, secondary card is now loading the vfio driver.

Now for a intrigue question. Will that chmod +x command be required for every script that I want to run on boot? Because there are some task i would like to complete on ever boot.

For instance, currently I have to modify the pwm1_enable pwm1 and power1_cap files in the hwmon folder on every boot manually for each card. I would like to run a scrip to do this as it is obviously a waste of time.

Edit: Secondary issue. I have added the card successfully to Virtual Machine Manager but I get a list of errors when trying to install a windows 10 VM.

 Unable to complete install: 'internal error: qemu unexpectedly closed the monitor: 2022-01-10T16:10:39.521338Z qemu-system-x86_64: warning: This feature depends on other features that were not requested: CPUID.8000000AH:EDX.svme-addr-chk [bit 28]
2022-01-10T16:10:39.521489Z qemu-system-x86_64: warning: This feature depends on other features that were not requested: CPUID.8000000AH:EDX.svme-addr-chk [bit 28]
2022-01-10T16:10:39.521757Z qemu-system-x86_64: warning: This feature depends on other features that were not requested: CPUID.8000000AH:EDX.svme-addr-chk [bit 28]
2022-01-10T16:10:39.522001Z qemu-system-x86_64: warning: This feature depends on other features that were not requested: CPUID.8000000AH:EDX.svme-addr-chk [bit 28]
qxl_send_events: spice-server bug: guest stopped, ignoring
2022-01-10T16:10:39.546931Z qemu-system-x86_64: -device {"driver":"vfio-pci","host":"0000:4b:00.0","id":"hostdev0","bus":"pci.4","addr":"0x0"}: vfio 0000:4b:00.0: failed to open /dev/vfio/60: Permission denied'

Traceback (most recent call last):
  File "/usr/share/virt-manager/virtManager/asyncjob.py", line 65, in cb_wrapper
    callback(asyncjob, *args, **kwargs)
  File "/usr/share/virt-manager/virtManager/createvm.py", line 2001, in _do_async_install
    installer.start_install(guest, meter=meter)
  File "/usr/share/virt-manager/virtinst/install/installer.py", line 701, in start_install
    domain = self._create_guest(
  File "/usr/share/virt-manager/virtinst/install/installer.py", line 649, in _create_guest
    domain = self.conn.createXML(install_xml or final_xml, 0)
  File "/usr/lib/python3.10/site-packages/libvirt.py", line 4393, in createXML
    raise libvirtError('virDomainCreateXML() failed')
libvirt.libvirtError: internal error: qemu unexpectedly closed the monitor: 2022-01-10T16:10:39.521338Z qemu-system-x86_64: warning: This feature depends on other features that were not requested: CPUID.8000000AH:EDX.svme-addr-chk [bit 28]
2022-01-10T16:10:39.521489Z qemu-system-x86_64: warning: This feature depends on other features that were not requested: CPUID.8000000AH:EDX.svme-addr-chk [bit 28]
2022-01-10T16:10:39.521757Z qemu-system-x86_64: warning: This feature depends on other features that were not requested: CPUID.8000000AH:EDX.svme-addr-chk [bit 28]
2022-01-10T16:10:39.522001Z qemu-system-x86_64: warning: This feature depends on other features that were not requested: CPUID.8000000AH:EDX.svme-addr-chk [bit 28]
qxl_send_events: spice-server bug: guest stopped, ignoring
2022-01-10T16:10:39.546931Z qemu-system-x86_64: -device {"driver":"vfio-pci","host":"0000:4b:00.0","id":"hostdev0","bus":"pci.4","addr":"0x0"}: vfio 0000:4b:00.0: failed to open /dev/vfio/60: Permission denied

Last edited by jddoxtator (2022-01-10 16:11:09)

Lone_Wolf · 2022-01-11 09:54:44

A.

https://wiki.archlinux.org/title/Help:R … executable
A simple method to determine if a file needs to be executable is to verify if its first line is a shebang .
shebangs always start with #! .

B.
There are multiple ways to have things set at boot, a script is not always needed or the best solution.
your not-passed through card is driven by the amdgpu kernel module, check what options it has.
See https://wiki.archlinux.org/title/Kernel_module

C.
One thing to keep in mind is that you only can set things under /sys if linux is managing them.
The main job of vfio is to prevent linux from managing devices, so you may have to use the guest to set anything for the passed through card.

D.

Unable to complete install: 'internal error: qemu unexpectedly closed the monitor: 2022-01-10T16:10:39.521338Z qemu-system-x86_64: warning: This feature depends on other features that were not requested: CPUID.8000000AH:EDX.svme-addr-chk [bit 28]

This is due to a change in qemu, https://gitlab.com/qemu-project/qemu/-/issues/776

https://wiki.archlinux.org/title/QEMU#virtio shows the parameters you need to add to the qemu command line.

Personally I don't use virt-manager (or other frontends) to run VMs , but it should be possible to add those options using virsh to edit the xml file created by virt-manager.

Arch Linux

#1 2022-01-02 23:21:19

PCI Passthrough with identical GPU's not working.

#2 2022-01-05 11:17:12

Re: PCI Passthrough with identical GPU's not working.

#3 2022-01-05 14:35:57

Re: PCI Passthrough with identical GPU's not working.

#4 2022-01-05 14:37:55

Re: PCI Passthrough with identical GPU's not working.

#5 2022-01-05 14:40:33

Re: PCI Passthrough with identical GPU's not working.

#6 2022-01-06 11:58:19

Re: PCI Passthrough with identical GPU's not working.

#7 2022-01-06 14:39:35

Re: PCI Passthrough with identical GPU's not working.

#8 2022-01-06 16:11:36

Re: PCI Passthrough with identical GPU's not working.

#9 2022-01-06 17:49:15

Re: PCI Passthrough with identical GPU's not working.

#10 2022-01-07 11:53:24

Re: PCI Passthrough with identical GPU's not working.

#11 2022-01-08 02:44:38

Re: PCI Passthrough with identical GPU's not working.

#12 2022-01-08 12:18:44

Re: PCI Passthrough with identical GPU's not working.

#13 2022-01-09 01:11:04

Re: PCI Passthrough with identical GPU's not working.

#14 2022-01-09 11:12:23

Re: PCI Passthrough with identical GPU's not working.

#15 2022-01-10 01:55:27

Re: PCI Passthrough with identical GPU's not working.

#16 2022-01-10 11:16:24

Re: PCI Passthrough with identical GPU's not working.

#17 2022-01-10 15:39:37

Re: PCI Passthrough with identical GPU's not working.

#18 2022-01-11 09:54:44

Re: PCI Passthrough with identical GPU's not working.

Board footer