You are not logged in.

#1 2019-03-23 20:57:15

Draizer
Member
Registered: 2015-03-17
Posts: 8

[SOLVED] Binding GPU to vfio-pci breaks onboard sound card

Hello all,

I've upgraded my hardware, and while setting up the VFIO passthrough for my new components I've ran into a weird issue with the motherboard's integrated sound card.

Basically, if I don't pass anything through, everything works fine. However, if I bind my GPU (RTX 2060) to the vfio-pci driver, then the following appears on dmesg when doing a grep on my sound card's ID :


$ dmesg | grep 20:00.3
[    0.708709] pci 0000:20:00.3: [1022:1457] type 00 class 0x040300
[    0.708718] pci 0000:20:00.3: reg 0x10: [mem 0xf7900000-0xf7907fff]
[    0.708735] pci 0000:20:00.3: enabling Extended Tags
[    0.708759] pci 0000:20:00.3: PME# supported from D0 D3hot D3cold
[    0.941670] iommu: Adding device 0000:20:00.3 to group 21
[    0.941682] iommu: Using direct mapping for device 0000:20:00.3
[    7.412207] snd_hda_intel: probe of 0000:20:00.3 failed with error -2

The sound card doesn't work at all. It's detected on lspci, but no kernel modules are in use and ALSA doesn't see it. Tried forcing the snd_hda_intel module to load, to no avail. Only way to make it work is to reboot without GPU passthrough.

Here is the output of the same command without passthrough (works as expected) :

$ dmesg | grep 20:00.3
[    0.708401] pci 0000:20:00.3: [1022:1457] type 00 class 0x040300
[    0.708410] pci 0000:20:00.3: reg 0x10: [mem 0xf7900000-0xf7907fff]
[    0.708427] pci 0000:20:00.3: enabling Extended Tags
[    0.708451] pci 0000:20:00.3: PME# supported from D0 D3hot D3cold
[    0.943565] iommu: Adding device 0000:20:00.3 to group 21
[    6.872050] snd_hda_intel 0000:20:00.3: enabling device (0000 -> 0002)
[    6.907632] input: HD-Audio Generic Front Mic as /devices/pci0000:00/0000:00:08.1/0000:20:00.3/sound/card1/input7
[    6.907648] input: HD-Audio Generic Rear Mic as /devices/pci0000:00/0000:00:08.1/0000:20:00.3/sound/card1/input8
[    6.907660] input: HD-Audio Generic Line as /devices/pci0000:00/0000:00:08.1/0000:20:00.3/sound/card1/input9
[    6.907673] input: HD-Audio Generic Line Out Front as /devices/pci0000:00/0000:00:08.1/0000:20:00.3/sound/card1/input10
[    6.907686] input: HD-Audio Generic Line Out Surround as /devices/pci0000:00/0000:00:08.1/0000:20:00.3/sound/card1/input11
[    6.907698] input: HD-Audio Generic Line Out CLFE as /devices/pci0000:00/0000:00:08.1/0000:20:00.3/sound/card1/input12
[    6.907713] input: HD-Audio Generic Front Headphone as /devices/pci0000:00/0000:00:08.1/0000:20:00.3/sound/card1/input13

The integrated sound card has a different IOMMU group from the GPU (21 vs 15), a different vendor ID... so I have no idea why binding the GPU to vfio causes this, but I've confirmed this is the case. I've found absolutely no clues after browsing on quite a few threads both on these forums and elsewhere. I'll admit I am at a loss.

Arch kernel is on version 5.0.3.

Kernel options :

options	root=/dev/sda2 rw amd_iommu=on iommu=pt nvidia-drm.modeset=1 vfio-pci.ids=10de:1f08,10de:10f9,10de:1ada,10de:1adb

All 4 of the IDs belong to the GPU, nothing else is currently passed through. Removing them makes the sound card work once again.

Hardware is as follow :

-MSI x470 Gaming plus (IOMMU & virtualization enabled)
-Integrated sound card : Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) HD Audio Controller (Realtek ALC898)
-AMD Ryzen 2700x
-NVIDIA GT 710 (host GPU)
-NVIDIA RTX 2060 (guest GPU)

I will of course provide more information if necessary, I'm just not sure which logs could be relevant.

Thank you in advance for all replies.

EDIT : I've narrowed down the issue specifically to the GPU's integrated sound card. When I remove it from my kernel parameters my motherboard's sound card works fine. I've confirmed it only works if I specifically remove the GPU's sound card, if I remove any other component from my parameters but leave the sound card in the issue still happens. Of course since the GPU's sound card is part of the same IOMMU group as the actual GPU this doesn't change much.

Last edited by Draizer (2019-04-06 21:54:13)

Offline

#2 2019-03-24 15:47:26

Lone_Wolf
Member
From: Netherlands, Europe
Registered: 2005-10-04
Posts: 11,868

Re: [SOLVED] Binding GPU to vfio-pci breaks onboard sound card

amd_iommu=on iommu=pt

https://www.kernel.org/doc/html/v5.0/ad … eters.html

amd_iommu doesn't have an "on" option, pt appears to be for x86 not x86_64 .
Having said that, those options should not harm the system.

Please post the whole output of the script at https://wiki.archlinux.org/index.php/PC … _are_valid so we can see which iommu groups hold which devices.

Do you have microcode updating configured ?
What firmware version are you running ?

What type of monitors are you using : HDMI , DisplayPort, DVI ?

Last edited by Lone_Wolf (2019-03-24 15:48:44)


Disliking systemd intensely, but not satisfied with alternatives so focusing on taming systemd.


(A works at time B)  && (time C > time B ) ≠  (A works at time C)

Offline

#3 2019-03-24 16:11:13

Draizer
Member
Registered: 2015-03-17
Posts: 8

Re: [SOLVED] Binding GPU to vfio-pci breaks onboard sound card

Hello Lone_Wolf,


Thanks for the answer. I'll correct the parameters right away.

Here are the IOMMU groups :

IOMMU Group 0 00:01.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) PCIe Dummy Host Bridge [1022:1452]
IOMMU Group 10 00:08.1 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) Internal PCIe GPP Bridge 0 to Bus B [1022:1454]
IOMMU Group 11 00:14.0 SMBus [0c05]: Advanced Micro Devices, Inc. [AMD] FCH SMBus Controller [1022:790b] (rev 59)
IOMMU Group 11 00:14.3 ISA bridge [0601]: Advanced Micro Devices, Inc. [AMD] FCH LPC Bridge [1022:790e] (rev 51)
IOMMU Group 12 00:18.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) Data Fabric: Device 18h; Function 0 [1022:1460]
IOMMU Group 12 00:18.1 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) Data Fabric: Device 18h; Function 1 [1022:1461]
IOMMU Group 12 00:18.2 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) Data Fabric: Device 18h; Function 2 [1022:1462]
IOMMU Group 12 00:18.3 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) Data Fabric: Device 18h; Function 3 [1022:1463]
IOMMU Group 12 00:18.4 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) Data Fabric: Device 18h; Function 4 [1022:1464]
IOMMU Group 12 00:18.5 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) Data Fabric: Device 18h; Function 5 [1022:1465]
IOMMU Group 12 00:18.6 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) Data Fabric: Device 18h; Function 6 [1022:1466]
IOMMU Group 12 00:18.7 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) Data Fabric: Device 18h; Function 7 [1022:1467]
IOMMU Group 13 03:00.0 USB controller [0c03]: Advanced Micro Devices, Inc. [AMD] Device [1022:43d0] (rev 01)
IOMMU Group 13 03:00.1 SATA controller [0106]: Advanced Micro Devices, Inc. [AMD] 400 Series Chipset SATA Controller [1022:43c8] (rev 01)
IOMMU Group 13 03:00.2 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] 400 Series Chipset PCIe Bridge [1022:43c6] (rev 01)
IOMMU Group 13 16:00.0 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] 400 Series Chipset PCIe Port [1022:43c7] (rev 01)
IOMMU Group 13 16:01.0 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] 400 Series Chipset PCIe Port [1022:43c7] (rev 01)
IOMMU Group 13 16:02.0 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] 400 Series Chipset PCIe Port [1022:43c7] (rev 01)
IOMMU Group 13 16:03.0 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] 400 Series Chipset PCIe Port [1022:43c7] (rev 01)
IOMMU Group 13 16:04.0 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] 400 Series Chipset PCIe Port [1022:43c7] (rev 01)
IOMMU Group 13 16:08.0 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] 400 Series Chipset PCIe Port [1022:43c7] (rev 01)
IOMMU Group 13 18:00.0 Ethernet controller [0200]: Realtek Semiconductor Co., Ltd. RTL8111/8168/8411 PCI Express Gigabit Ethernet Controller [10ec:8168] (rev 15)
IOMMU Group 13 19:00.0 Ethernet controller [0200]: Realtek Semiconductor Co., Ltd. RTL8111/8168/8411 PCI Express Gigabit Ethernet Controller [10ec:8168] (rev 06)
IOMMU Group 13 1c:00.0 USB controller [0c03]: ASMedia Technology Inc. ASM1142 USB 3.1 Host Controller [1b21:1242]
IOMMU Group 14 1d:00.0 VGA compatible controller [0300]: NVIDIA Corporation GK208B [GeForce GT 710] [10de:128b] (rev a1)
IOMMU Group 14 1d:00.1 Audio device [0403]: NVIDIA Corporation GK208 HDMI/DP Audio Controller [10de:0e0f] (rev a1)
IOMMU Group 15 1e:00.0 VGA compatible controller [0300]: NVIDIA Corporation TU106 [GeForce RTX 2060 Rev. A] [10de:1f08] (rev a1)
IOMMU Group 15 1e:00.1 Audio device [0403]: NVIDIA Corporation TU106 High Definition Audio Controller [10de:10f9] (rev a1)
IOMMU Group 15 1e:00.2 USB controller [0c03]: NVIDIA Corporation TU106 USB 3.1 Host Controller [10de:1ada] (rev a1)
IOMMU Group 15 1e:00.3 Serial bus controller [0c80]: NVIDIA Corporation TU106 USB Type-C Port Policy Controller [10de:1adb] (rev a1)
IOMMU Group 16 1f:00.0 Non-Essential Instrumentation [1300]: Advanced Micro Devices, Inc. [AMD] Zeppelin/Raven/Raven2 PCIe Dummy Function [1022:145a]
IOMMU Group 17 1f:00.2 Encryption controller [1080]: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) Platform Security Processor [1022:1456]
IOMMU Group 18 1f:00.3 USB controller [0c03]: Advanced Micro Devices, Inc. [AMD] USB 3.0 Host controller [1022:145f]
IOMMU Group 19 20:00.0 Non-Essential Instrumentation [1300]: Advanced Micro Devices, Inc. [AMD] Zeppelin/Renoir PCIe Dummy Function [1022:1455]
IOMMU Group 1 00:01.3 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) PCIe GPP Bridge [1022:1453]
IOMMU Group 20 20:00.2 SATA controller [0106]: Advanced Micro Devices, Inc. [AMD] FCH SATA Controller [AHCI mode] [1022:7901] (rev 51)
IOMMU Group 21 20:00.3 Audio device [0403]: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) HD Audio Controller [1022:1457]
IOMMU Group 2 00:02.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) PCIe Dummy Host Bridge [1022:1452]
IOMMU Group 3 00:03.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) PCIe Dummy Host Bridge [1022:1452]
IOMMU Group 4 00:03.1 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) PCIe GPP Bridge [1022:1453]
IOMMU Group 5 00:03.2 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) PCIe GPP Bridge [1022:1453]
IOMMU Group 6 00:04.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) PCIe Dummy Host Bridge [1022:1452]
IOMMU Group 7 00:07.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) PCIe Dummy Host Bridge [1022:1452]
IOMMU Group 8 00:07.1 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) Internal PCIe GPP Bridge 0 to Bus B [1022:1454]
IOMMU Group 9 00:08.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) PCIe Dummy Host Bridge [1022:1452]

No microcode updating configured right now, but it is an interesting thought. I'm going to set it up right now.

I'm not if this is the information you wanted but the microcode version is 0x800820b according to the output of grep 'stepping\|model\|microcode' /proc/cpuinfo

For monitors on the host I'm on HDMI+VGA, HDMI+DVI on the guest.

Offline

#4 2019-03-25 16:22:35

Lone_Wolf
Member
From: Netherlands, Europe
Registered: 2005-10-04
Posts: 11,868

Re: [SOLVED] Binding GPU to vfio-pci breaks onboard sound card

I wanted to know the BIOS/UEFI firmware version you had but didn't make that clear.

Which of the ones listed at https://www.msi.com/Motherboard/support … #down-bios do you have installed ?

Does the bios have a setting to determine which of your videocards boots first / is primary ?


Disliking systemd intensely, but not satisfied with alternatives so focusing on taming systemd.


(A works at time B)  && (time C > time B ) ≠  (A works at time C)

Offline

#5 2019-03-30 08:42:57

Draizer
Member
Registered: 2015-03-17
Posts: 8

Re: [SOLVED] Binding GPU to vfio-pci breaks onboard sound card

Hello Lone_Wolf,

Apologies for the delay in my answer, I've been kept busy during the last few days.

In the BIOS my version is marked as being 7B79AMS.A50, which doesn't look to be on MSI's website but should be the default, starting BIOS (I didn't update it).

The BIOS does not have an option to select the primary video card, so my host's GT710 has been put in the first PCI slot and the guest's RTX2060 in the second one.

Offline

#6 2019-03-31 11:59:56

Lone_Wolf
Member
From: Netherlands, Europe
Registered: 2005-10-04
Posts: 11,868

Re: [SOLVED] Binding GPU to vfio-pci breaks onboard sound card

In the BIOS my version is marked as being 7B79AMS.A50, which doesn't look to be on MSI's website

I downloaded the bios that matches that most closely, 7B79vA5.zip and found it has 7B79AMS.A50 inside.
That suggests MSI uses a simplified version number.

Iommu grouping shows nothing suspicious.
Around page 30 of the (english) user manual, a diagram shows the location & numbering of PCIe slots.
Which card is in PCI_E1 and which in PCI_E4 ?

I do think the issue may be at mobo hardware or bios level, are you up for some experimental testing ?

Kep the videocards in the slots they are now in, test the 4 possible vfio hdmi audio setups, for working mobo sound.

Both hdmi audio assigned to vfio
only GT 710 audio assigned to vfio
only RTX2060 audio assigned to vfio
no hdmi audio assigned to vfio

Once you've done those, switch the cards and repeat the 4 tests.


Disliking systemd intensely, but not satisfied with alternatives so focusing on taming systemd.


(A works at time B)  && (time C > time B ) ≠  (A works at time C)

Offline

#7 2019-04-06 21:53:49

Draizer
Member
Registered: 2015-03-17
Posts: 8

Re: [SOLVED] Binding GPU to vfio-pci breaks onboard sound card

Hello again,

To answer your initial question : GT710 is in slot E1 and RTX2060 in slot E4.

So, I took the time to run all the tests as indicated, the issue appeared when either of the cards were on slot E4 and their audio was assigned to vfio.

In the end I bit the bullet and updated my BIOS to the latest available version (7B79vA9), and now it works as expected on my initial setup... Except that now when I try and boot my VM I have a new issue :

vfio-pci 0000:1e:00.3: Refused to change power state, currently in D3

So in the end I still can't use my VM, but this particular issue has been solved so I'll mark it as such. Thank you for your time Lone_Wolf, as you did put me on the right track.

EDIT : For the record my issue was due to the BIOS version, I downgraded to version 7B79vA7 and now the passthrough works exactly as expected.

Last edited by Draizer (2019-04-08 13:51:36)

Offline

Board footer

Powered by FluxBB