You are not logged in.

Pages: 1

#1 2018-02-05 18:37:42

itractus
Member
From: Belgium
Registered: 2017-02-04
Posts: 6

GPU passthrough hurdles, VFIO edition

So I am trying to passthrough my second GPU using vfio.
Here is my hardware information:
AMD FX-8350
ASUS crosshair V formula
AMD HD7970 Matrix (I have 2, and I am not using the crossfire bridge.)
What I configured:
IOMMU is enabledin BIOS,

[itractus@titanium ~]$ dmesg | grep -e DMAR -e IOMMU
[    1.663665] AMD-Vi: Found IOMMU at 0000:00:00.2 cap 0x40
[    4.254576] AMD IOMMUv2 driver by Joerg Roedel <jroedel@suse.de>
[    4.254577] AMD IOMMUv2 functionality not available on this system

The second GPU is in an isolated group:

[itractus@titanium ~]$ for d in /sys/kernel/iommu_groups/*/devices/*; do 
>     n=${d#*/iommu_groups/*}; n=${n%%/*}
>     printf 'IOMMU Group %s ' "$n"
>     lspci -nns "${d##*/}"
> done;
...
IOMMU Group 23 07:00.0 VGA compatible controller [0300]: Advanced Micro Devices, Inc. [AMD/ATI] Tahiti XT [Radeon HD 7970/8970 OEM / R9 280X] [1002:6798]
IOMMU Group 23 07:00.1 Audio device [0403]: Advanced Micro Devices, Inc. [AMD/ATI] Tahiti HDMI Audio [Radeon HD 7870 XT / 7950/7970] [1002:aaa0]
...

Next I set up following 2 files as per wiki instruction with 2 identical GPU's:

[itractus@titanium ~]$ cat /etc/modprobe.d/vfio.conf
install vfio-pci /usr/bin/vfio-pci-override.sh

And

[itractus@titanium ~]$ cat /usr/bin/vfio-pci-override.sh
#!/bin/sh

GROUP="0000:07:00.0"
DEVS="0000:07:00.0 0000:07:00.1 ."
if [ ! -z "$(ls -A /sys/class/iommu)" ]; then
	for DEV in $DEVS; do
		echo "vfio-pci" > /sys/bus/pci/devices/$GROUP/$DEV/driver_override
	done
fi

modprobe -i vfio-pci

I also added the correct settings to the following:

[itractus@titanium ~]$ cat /etc/mkinitcpio.conf 
...
MODULES=(vfio-pci vfio_iommu_type1)
...
FILES=(/etc/modprobe.d/vfio.conf /usr/bin/vfio-pci-override.sh)
...
HOOKS=(base udev autodetect modconf block filesystems keyboard fsck)

After all of that, I should be able to regenerate initramfs and reboot for my GPU to be running on vfio-pci, however 

dmesg | grep -i vfio

produces nothing. Even worse, I can see that the second GPU is still running the radeon driver.

[itractus@titanium ~]$ lspci -nnk -d 1002:6798
01:00.0 VGA compatible controller [0300]: Advanced Micro Devices, Inc. [AMD/ATI] Tahiti XT [Radeon HD 7970/8970 OEM / R9 280X] [1002:6798]
	Subsystem: ASUSTeK Computer Inc. Tahiti XT2 [Matrix HD 7970 Platinum] [1043:044c]
	Kernel driver in use: radeon
	Kernel modules: radeon, amdgpu
07:00.0 VGA compatible controller [0300]: Advanced Micro Devices, Inc. [AMD/ATI] Tahiti XT [Radeon HD 7970/8970 OEM / R9 280X] [1002:6798]
	Subsystem: ASUSTeK Computer Inc. Tahiti XT2 [Matrix HD 7970 Platinum] [1043:044c]
	Kernel driver in use: radeon
	Kernel modules: radeon, amdgpu

This is where I'm stuck, I can't seem to find what's happening, or why my second GPU isn't loading in vfio-pci.
What did I do wrong/overlooked?

Thanks in advance

edit: Changed title to be a little less... Confusing, a little more.... Fun?

Last edited by itractus (2018-02-09 13:55:37)

W H Y    Do    We    .  .  .

Offline

#2 2018-02-09 11:13:46

itractus
Member
From: Belgium
Registered: 2017-02-04
Posts: 6

Re: GPU passthrough hurdles, VFIO edition

So after a couple of days of looking into this issue further I came to the following conclusion:
I forgot to set kernel parameters for vfio. Not only that, I noticed that I have a 'broken' BIOS and thus needed to follow this guide to set the ioapic table up correctly. My UEFI boot entry now looks like this:

Arch 	HD(1,GPT,3014e852-3683-41b4-a946-525d74ee8714,0x800,0x100000)/File(\vmlinuz-linux)root=/dev/sdb2 rw initrd=/initramfs-linux.img amd_iommu=on iommu=pt ivrs_ioapic[9]=00:14.0 ivrs_ioapic[10]=00:00.2

This should be enough to get iommu going, but if I check now, I get the following:

titanium% dmesg | grep -e DMAR -e IOMMU
[    1.280117] AMD-Vi: Found IOMMU at 0000:00:00.2 cap 0x40

So not a single entry on DMAR.... Also no more mention of the IOMMU V2, but this might be due to my motherboard being from 2012.

Another thing that changed after boot was the IOMMU group of my GPU. This I did not excpect, since I did not phisically touch the GPU:

[itractus@titanium ~]$ #!/bin/bash
[itractus@titanium ~]$ shopt -s nullglob
[itractus@titanium ~]$ for d in /sys/kernel/iommu_groups/*/devices/*; do 
>     n=${d#*/iommu_groups/*}; n=${n%%/*}
>     printf 'IOMMU Group %s ' "$n"
>     lspci -nns "${d##*/}"
> done;
...
IOMMU Group 21 06:00.0 VGA compatible controller [0300]: Advanced Micro Devices, Inc. [AMD/ATI] Tahiti XT [Radeon HD 7970/8970 OEM / R9 280X] [1002:6798]
IOMMU Group 21 06:00.1 Audio device [0403]: Advanced Micro Devices, Inc. [AMD/ATI] Tahiti HDMI Audio [Radeon HD 7870 XT / 7950/7970] [1002:aaa0]
...

Seeing as this happened for reasons unknown to me, I had to alter the following:

[itractus@titanium ~]$ cat /usr/bin/vfio-pci-override.sh
#!/bin/sh

GROUP="0000:06:00.0"
DEVS="0000:06:00.0 0000:06:00.1 ."
if [ ! -z "$(ls -A /sys/class/iommu)" ]; then
	for DEV in $DEVS; do
		echo "vfio-pci" > /sys/bus/pci/devices/$GROUP/$DEV/driver_override
	done
fi

modprobe -i vfio-pci

But hey, it's not all bad. I can now find a vfio entry in dmesg:

[itractus@titanium ~]$ dmesg | grep -i vfio
[    1.515124] VFIO - User Level meta-driver version: 0.3

Edit:
Totally forgot to mention that my second gpu is still booting in radeon:

[itractus@titanium ~]$ lspci -nnk -d 1002:6798
01:00.0 VGA compatible controller [0300]: Advanced Micro Devices, Inc. [AMD/ATI] Tahiti XT [Radeon HD 7970/8970 OEM / R9 280X] [1002:6798]
	Subsystem: ASUSTeK Computer Inc. Tahiti XT2 [Matrix HD 7970 Platinum] [1043:044c]
	Kernel driver in use: radeon
	Kernel modules: radeon, amdgpu
06:00.0 VGA compatible controller [0300]: Advanced Micro Devices, Inc. [AMD/ATI] Tahiti XT [Radeon HD 7970/8970 OEM / R9 280X] [1002:6798]
	Subsystem: ASUSTeK Computer Inc. Tahiti XT2 [Matrix HD 7970 Platinum] [1043:044c]
	Kernel driver in use: radeon
	Kernel modules: radeon, amdgpu

Last edited by itractus (2018-02-09 11:25:06)

W H Y    Do    We    .  .  .

Offline

#3 2018-02-12 17:22:16

itractus
Member
From: Belgium
Registered: 2017-02-04
Posts: 6

Re: GPU passthrough hurdles, VFIO edition

So, another little step forward, I found out that I misstyped something in /etc/mkinitcpio.conf :
qkix6t.png
This should be an underscore, altough I'm not sure about this, a friend with a running GPU passthrough has it configured this way. This is what the mkinitcpio looks like now:

titanium% cat /etc/mkinitcpio.conf 
...
MODULES=(vfio_pci vfio_iommu_type1 vfio_virqfd)
...
BINARIES=()
...
FILES=(/etc/modprobe.d/vfio.conf /usr/bin/vfio-pci-override.sh)
...
HOOKS=(base udev autodetect modconf block filesystems keyboard fsck)
...

After regeneratin the initramfs,  still no change to my previous post. Any help or tips are apreciated.

W H Y    Do    We    .  .  .

Offline

#4 2018-02-13 12:13:14

itractus
Member
From: Belgium
Registered: 2017-02-04
Posts: 6

Re: GPU passthrough hurdles, VFIO edition

lhoqvso wrote:

...
my setup is different ... I think I can help you.
...

That would be great!

lhoqvso wrote:

I do not know if you setup needs special requirements but mine was far more simple.

It is not really similar, since I am using 2 identical GPU's on a board with a 'broken' bios.

lhoqvso wrote:

...
I needed to uninstall the drivers several times and I think that even know I do not have them installed. I said that in case it helps you just before I send you the files.
...

I have tried this numerous times, I think I need to find a way to force the second gpu to use vfio as a kernel driver, but even with a script specified in /etc/mkinitcpio.conf, see first post.

W H Y    Do    We    .  .  .

Offline

#5 2018-02-13 12:29:13

itractus
Member
From: Belgium
Registered: 2017-02-04
Posts: 6

Re: GPU passthrough hurdles, VFIO edition

I'm actually not home tonight, but tomorrow  would be possible.

W H Y    Do    We    .  .  .

Offline

Pages: 1

Board footer

Powered by FluxBB