i2c-designware-pci claiming vfio-pci device, breaks kvm

tessaracht · 2022-04-04 17:39:55

I'm using the PCI Passthrough with OVMF guide to run a Windows VM, where I'm dedicating a 6800 XT video card to that VM. the PCI device numbers are 09:00.0-09:00.3, and I'm binding them to the vfio-pci driver for use in the VM via the `vfio-pci.ids=` kernel option.

It's been working fine for a while, but I've been having issues with the recent 5.15 point releases where it'll often fail to start the VM after a fresh reboot. looking at dmesg, it appears as if i2c-designware-pci is for some reason claiming the USB device that the video card presents, even though that's supposed to be held by vfio-pci for the VM to use:

[    5.586022] i2c-designware-pci 0000:09:00.3: enabling device (0000 -> 0002)dsa
[   86.892490] i2c-designware-pci 0000:09:00.3: AMD-Vi: Event logged [IO_PAGE_FAULT domain=0x0014 address=0x0 flags=0x0020]

the first event is this unrelated module claiming the device at boot, and the second is the VM attempting to start and failing, I presume because the designware module is already working with it. after this happens, kvm is completely broken until I reboot, and I can't start any other VMs, even ones that don't do pci-e passthrough.

anyone know why the designware module is misbehaving, and how I get it to leave that device alone?

+DS_DV+ · 2022-08-27 15:42:39

hi @tessaracht

could you solve this problem?

i followed:
https://www.heiko-sieger.info/running-w … figuration

and are stuck at the same problem you discribed:

 sudo lspci -kn | grep -A 2 0e:00
0e:00.0 0300: 1002:73bf (rev c1)
	Subsystem: 1002:0e3a
	Kernel driver in use: vfio-pci
--
0e:00.1 0403: 1002:ab28
	Subsystem: 1002:ab28
	Kernel driver in use: vfio-pci
--
0e:00.2 0c03: 1002:73a6
	Subsystem: 1002:73a6
	Kernel driver in use: vfio-pci
--
0e:00.3 0c80: 1002:73a4
	Subsystem: 1002:0408
	Kernel driver in use: i2c-designware-pci

tessaracht · 2022-08-27 21:52:19

unfortunately I haven't found a workaround. it seems like the i2c-designware-pci driver is built into the kernel on arch, and so there's no way to disable it for specific pci ids. I had to just skip binding the usb devince on my video card, and only bind the audio and video devices. not ideal, especially if you've got a usb-c monitor.

loqs · 2022-08-27 22:03:50

Have you tried preventing the module from initializing? https://unix.stackexchange.com/question … s-a-module

+DS_DV+ · 2022-08-27 22:49:50

@tessaracht: thanks for the reply (:
if i find a solution ill post it.

loqs wrote:

Have you tried preventing the module from initializing? https://unix.stackexchange.com/question … s-a-module

this reads like it disables the driver completely for the host.
i used that method on a vm nas for its lsi controller.
Therefor i am pretty certain it would work. But if you have other devices which require the driver if fear it sadly is not a suitable solution.

Last edited by +DS_DV+ (2022-08-27 22:50:28)

loqs · 2022-08-27 22:54:23

In that case have you tried unbinding the driver from the device if you can not stop it binding https://stackoverflow.com/questions/201 … -in-anothe

+DS_DV+ · 2022-08-28 15:06:58

@loqs thanks for your help <3
since i cant get AMD drivers to work i guess it has to do with the driver.

you last method sadly densest seem to work

$sudo echo -n 0000:0e:00.3 unbind
0000:0e:00.3 unbind%                                                                                                                                                                       $sudo lspci -kn | grep -A 2 0e:00
0e:00.0 0300: 1002:73bf (rev c1)
	Subsystem: 1002:0e3a
	Kernel driver in use: vfio-pci
--
0e:00.1 0403: 1002:ab28
	Subsystem: 1002:ab28
	Kernel driver in use: vfio-pci
--
0e:00.2 0c03: 1002:73a6
	Subsystem: 1002:73a6
	Kernel driver in use: vfio-pci
--
0e:00.3 0c80: 1002:73a4
	Subsystem: 1002:0408
	Kernel driver in use: i2c-designware-pci

+DS_DV+ · 2022-08-29 07:51:48

@loqs
so now i have "blacklist i2c-designware-pci" inside my /etc/modprobe.de/vfio.conf
i have the init block you recommended inside my grub and i tried unbinding it.

no change what so ever

i don't know if that is the problem why my vm cant install its drivers but it is pretty frustrating

Lone_Wolf · 2022-08-29 10:42:28

This sounds like you may need an ACS enabled kernel.

To verify if that would help, please post the output of the script at https://wiki.archlinux.org/title/PCI_pa … _are_valid

+DS_DV+ · 2022-08-29 12:44:52

@Lone_Wolf: i have the ASUS Crosshai VI Hero with its latest BIOS - i can enable and disable ACS there inside the AMD CBS options menu (:

IOMMU Group 0:
	00:01.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge [1022:1482]
IOMMU Group 1:
	00:01.1 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse GPP Bridge [1022:1483]
IOMMU Group 2:
	00:01.3 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse GPP Bridge [1022:1483]
IOMMU Group 3:
	00:02.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge [1022:1482]
IOMMU Group 4:
	00:03.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge [1022:1482]
IOMMU Group 5:
	00:03.1 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse GPP Bridge [1022:1483]
IOMMU Group 6:
	00:03.2 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse GPP Bridge [1022:1483]
IOMMU Group 7:
	00:04.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge [1022:1482]
IOMMU Group 8:
	00:05.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge [1022:1482]
IOMMU Group 9:
	00:07.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge [1022:1482]
IOMMU Group 10:
	00:07.1 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse Internal PCIe GPP Bridge 0 to bus[E:B] [1022:1484]
IOMMU Group 11:
	00:08.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge [1022:1482]
IOMMU Group 12:
	00:08.1 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse Internal PCIe GPP Bridge 0 to bus[E:B] [1022:1484]
IOMMU Group 13:
	00:14.0 SMBus [0c05]: Advanced Micro Devices, Inc. [AMD] FCH SMBus Controller [1022:790b] (rev 61)
	00:14.3 ISA bridge [0601]: Advanced Micro Devices, Inc. [AMD] FCH LPC Bridge [1022:790e] (rev 51)
IOMMU Group 14:
	00:18.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Matisse/Vermeer Data Fabric: Device 18h; Function 0 [1022:1440]
	00:18.1 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Matisse/Vermeer Data Fabric: Device 18h; Function 1 [1022:1441]
	00:18.2 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Matisse/Vermeer Data Fabric: Device 18h; Function 2 [1022:1442]
	00:18.3 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Matisse/Vermeer Data Fabric: Device 18h; Function 3 [1022:1443]
	00:18.4 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Matisse/Vermeer Data Fabric: Device 18h; Function 4 [1022:1444]
	00:18.5 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Matisse/Vermeer Data Fabric: Device 18h; Function 5 [1022:1445]
	00:18.6 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Matisse/Vermeer Data Fabric: Device 18h; Function 6 [1022:1446]
	00:18.7 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Matisse/Vermeer Data Fabric: Device 18h; Function 7 [1022:1447]
IOMMU Group 15:
	01:00.0 Non-Volatile memory controller [0108]: Samsung Electronics Co Ltd NVMe SSD Controller SM961/PM961/SM963 [144d:a804]
IOMMU Group 16:
	02:00.0 USB controller [0c03]: Advanced Micro Devices, Inc. [AMD] X370 Series Chipset USB 3.1 xHCI Controller [1022:43b9] (rev 02)
	02:00.1 SATA controller [0106]: Advanced Micro Devices, Inc. [AMD] X370 Series Chipset SATA Controller [1022:43b5] (rev 02)
	02:00.2 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] X370 Series Chipset PCIe Upstream Port [1022:43b0] (rev 02)
	03:00.0 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] 300 Series Chipset PCIe Port [1022:43b4] (rev 02)
	03:02.0 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] 300 Series Chipset PCIe Port [1022:43b4] (rev 02)
	03:03.0 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] 300 Series Chipset PCIe Port [1022:43b4] (rev 02)
	03:04.0 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] 300 Series Chipset PCIe Port [1022:43b4] (rev 02)
	03:05.0 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] 300 Series Chipset PCIe Port [1022:43b4] (rev 02)
	03:06.0 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] 300 Series Chipset PCIe Port [1022:43b4] (rev 02)
	03:07.0 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] 300 Series Chipset PCIe Port [1022:43b4] (rev 02)
	04:00.0 USB controller [0c03]: ASMedia Technology Inc. ASM1143 USB 3.1 Host Controller [1b21:1343]
	05:00.0 Ethernet controller [0200]: Intel Corporation I211 Gigabit Network Connection [8086:1539] (rev 03)
IOMMU Group 17:
	0b:00.0 VGA compatible controller [0300]: Advanced Micro Devices, Inc. [AMD/ATI] Lexa XT [Radeon PRO WX 3200] [1002:6981] (rev 10)
	0b:00.1 Audio device [0403]: Advanced Micro Devices, Inc. [AMD/ATI] Baffin HDMI/DP Audio [Radeon RX 550 640SP / RX 560/560X] [1002:aae0]
IOMMU Group 18:
	0c:00.0 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD/ATI] Navi 10 XL Upstream Port of PCI Express Switch [1002:1478] (rev c1)
IOMMU Group 19:
	0d:00.0 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD/ATI] Navi 10 XL Downstream Port of PCI Express Switch [1002:1479]
IOMMU Group 20:
	0e:00.0 VGA compatible controller [0300]: Advanced Micro Devices, Inc. [AMD/ATI] Navi 21 [Radeon RX 6800/6800 XT / 6900 XT] [1002:73bf] (rev c1)
IOMMU Group 21:
	0e:00.1 Audio device [0403]: Advanced Micro Devices, Inc. [AMD/ATI] Navi 21/23 HDMI/DP Audio Controller [1002:ab28]
IOMMU Group 22:
	0e:00.2 USB controller [0c03]: Advanced Micro Devices, Inc. [AMD/ATI] Device [1002:73a6]
IOMMU Group 23:
	0e:00.3 Serial bus controller [0c80]: Advanced Micro Devices, Inc. [AMD/ATI] Navi 21 USB [1002:73a4]
IOMMU Group 24:
	0f:00.0 Non-Essential Instrumentation [1300]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Function [1022:148a]
IOMMU Group 25:
	10:00.0 Non-Essential Instrumentation [1300]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse Reserved SPP [1022:1485]
IOMMU Group 26:
	10:00.1 Encryption controller [1080]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse Cryptographic Coprocessor PSPCPP [1022:1486]
IOMMU Group 27:
	10:00.3 USB controller [0c03]: Advanced Micro Devices, Inc. [AMD] Matisse USB 3.0 Host Controller [1022:149c]

apparently the major problem was that linux does not like "above 4G encoding" and "resizeable BAR" support. Since i disabled both i can install a driver.
Sadly i can reboot the VM. I always have to reboot the host. otherwise the VM has a black screen

Lone_Wolf · 2022-08-30 10:54:57

Doesn't look like acs is needed, and t's probably the combo of resizable bar and above 4G that's the issue.

Above 4G decpoding has been around a long time and works flawlessly with linux.
It can't be used when csm is active (which is default for 99.9% of efi systems) and requires the drivers to access graphics memory directly to benefit from it.

On systems where above 4g is disabled drivers have no method to access graphics memory directly and need to fall back to using the GART for that.
GART allows direct access to a part of the graphics memory and typically is limited to 256Mib (1 Gib does occur on some systems) and needs extra steps to access other parts of graphics memory .

Resizable bar expands the funtionality of the gart by allowing it to be dynamically resized so it can access all memory of a graphics card .
Don't know how it deals with multiple videocards,. but switching the gart to another card always required extra steps.

Above 4G decoding does work with the old static method, but conflicts with resizable bar.
It's not limited by number of cards, only by the maximum directly accessible amount of memory .

Since you have atleast 2 videocards, I suggest you re-enable above 4G decoding .

Last edited by Lone_Wolf (2022-08-30 10:55:39)

Arch Linux

#1 2022-04-04 17:39:55

i2c-designware-pci claiming vfio-pci device, breaks kvm

#2 2022-08-27 15:42:39

Re: i2c-designware-pci claiming vfio-pci device, breaks kvm

#3 2022-08-27 21:52:19

Re: i2c-designware-pci claiming vfio-pci device, breaks kvm

#4 2022-08-27 22:03:50

Re: i2c-designware-pci claiming vfio-pci device, breaks kvm

#5 2022-08-27 22:49:50

Re: i2c-designware-pci claiming vfio-pci device, breaks kvm

#6 2022-08-27 22:54:23

Re: i2c-designware-pci claiming vfio-pci device, breaks kvm

#7 2022-08-28 15:06:58

Re: i2c-designware-pci claiming vfio-pci device, breaks kvm

#8 2022-08-29 07:51:48

Re: i2c-designware-pci claiming vfio-pci device, breaks kvm

#9 2022-08-29 10:42:28

Re: i2c-designware-pci claiming vfio-pci device, breaks kvm

#10 2022-08-29 12:44:52

Re: i2c-designware-pci claiming vfio-pci device, breaks kvm

#11 2022-08-30 10:54:57

Re: i2c-designware-pci claiming vfio-pci device, breaks kvm

Board footer