[SOLVED] Fail to read PCI config space; lspci and /dev/mem read fails

sherrellbc · 2016-12-29 20:15:26

Recently I began an effort to take a deep-dive in in various components of the kernel; lately it has been PCI. I am not particularly keen to all the workings of pci, so feel free to correct where necessary.

So, as I was reading some of the technical write-ups on PCI I was concurrently playing around with lspci. So, naturally, I dumped a few of the configuration spaces for various devices and decoded the output using the standard specification. Eventually I decided to take a look at my graphics card.

$ lspci
...
01:00.0 VGA compatible controller: Advanced Micro Devices, Inc. [AMD/ATI] Saturn XT [FirePro M6100] (rev ff)
...

Revision "ff" was strange, so I dumped the configuration space:

$ lspci -s 01:00.0 -xxx
01:00.0 VGA compatible controller: Advanced Micro Devices, Inc. [AMD/ATI] Saturn XT [FirePro M6100] (rev ff)
00: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff
10: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff
20: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff
30: ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff

So, there appears to be a problem reading the configuration space for this device. Obviously during the initial PCI enumeration the kernel cached the vendor/device IDs, because at this point we can't such data from the device directly since the vendor/device ID fields are also reading as F.

$ lspci -s 01:00.0 -n
01:00.0 0300: 1002:6640 (rev ff)

Dumping both functions of the PCI device still return error:

01:00.0 VGA compatible controller: Advanced Micro Devices, Inc. [AMD/ATI] Saturn XT [FirePro M6100] (rev ff) (prog-if ff)
        !!! Unknown header type 7f
        Kernel driver in use: radeon
        Kernel modules: radeon

01:00.1 Audio device: Advanced Micro Devices, Inc. [AMD/ATI] Tobago HDMI Audio [Radeon R7 360 / R9 360 OEM] (rev ff) (prog-if ff)
        !!! Unknown header type 7f
        Kernel driver in use: snd_hda_intel
        Kernel modules: snd_hda_intel

So, perhaps lspci is having issues. I thought to try reading the configuration space directly from memory. /dev/mem to the rescue. The default kernel configuration for Arch exposes /dev/mem to userspace but tightly restricts the address spaces available due to macro STRICT_DEVMEM.

$ zgrep -i devmem /proc/config.gz 
CONFIG_DEVMEM=y
CONFIG_ARCH_HAS_DEVMEM_IS_ALLOWED=y
CONFIG_STRICT_DEVMEM=y
CONFIG_IO_STRICT_DEVMEM=y

From what I could find about STRICT_DEVMEM we have (source):

If this option is switched on, the /dev/mem file only allows
userspace access to PCI space and the BIOS code and data regions.
This is sufficient for dosemu and X and all common users of
/dev/mem.

Also, the source code implementing the logic.

So, we should be able to read PCI configuration space. First, we have to determine where in memory the configuration spaces are mapped. Luckily we can dump the MCFG APCI table exposed at /sys/firmware/acpi/tables/MCFG

$ cat /sys/firmware/acpi/tables/MCFG | xxd
00000000: 4d43 4647 3c00 0000 019c 4445 4c4c 2020  MCFG<.....DELL  
00000010: 4342 5833 2020 2000 0920 0701 4d53 4654  CBX3   .. ..MSFT
00000020: 9700 0000 0000 0000 0000 0000 0000 00f8  ................
00000030: 0000 0000 0000 003f 0000 0000            .......?....

I will spare you the decoding time; the pci base configuration address is 0x0000 0000 f800 0000. I wrote a simple script that uses dd to offset into /dev/mem to the appropriate address for a given bus/device/function configuration space.

#!/bin/bash

BUS=$1
DEV=$2
FUNC=$3
LEN=$4
BASE_PCI_ADDR=0xF8000000
OFFSET=$(($BASE_PCI_ADDR + $BUS*0x100000 + $DEV*0x8000 + $FUNC*0x1000))

printf "\nReading PCI config space for BUS/DEV/FUNC($BUS/$DEV/$FUNC); addr=0x%x\n" $OFFSET
dd if=/dev/mem bs=1 status=none skip=$OFFSET count=$LEN | xxd

Let's give it a try:

$ sudo pci_dump.sh 1 0 0 64
Reading PCI config space for BUS/DEV/FUNC(1/0/0); addr=0xf8100000
dd: error reading '/dev/mem': Operation not permitted

The above script does work to read early memory within the first 1MB (i.e. the first 256 pages of physical memory). However, it does not seem to work as expected to read the PCI config space directly.

In summary: there are two problems here. The first is that my video PCI device seems to have died and we cannot properly use /dev/mem to read the MMIO configuration spaces directly. It seems the latter is either a spectacular failure on my part to understand the Linux PCI implementation, or the associated logic is broken in the kernel.

Last edited by sherrellbc (2017-01-05 17:50:33)

sherrellbc · 2016-12-29 20:45:29

A few years ago someone posted on the mailing list with a similar issue, but error was different (at least at that time). There was a concern that dd uses the read syscall rather than mmap, which could be causing some problems.

I would be more concerned about this if the lspci results were not reading as all F. Further, we should at least see something rather than being denied at the gate.

Last edited by sherrellbc (2016-12-29 20:54:20)

sherrellbc · 2017-01-04 14:34:08

Interestingly, I also noticed that the PCI Express Graphics (PEG) slot is disabled. Recall from above that the graphics card's GPU and audio functions are device 0, bus 1, and functions 0 and 1.

$cat /proc/acpi/wakeup
Device  S-state   Status   Sysfs node
...
PEGP      S4    *disabled  pci:0000:01:00.0
PEGA      S4    *disabled  pci:0000:01:00.1
...

I am not sure if this is related.

Last edited by sherrellbc (2017-01-04 14:34:41)

R00KIE · 2017-01-04 17:29:33

You might want to make sure you have something using the AMD card before you try to read something, I suspect that is a discrete gpu and you have an integrated gpu. If that is the case then the driver automatically turns the card off to save power.

What I find strange is that since a while back lspci does cause the device to wake up and the correct revision is reported, at least that is the behavior I see with my laptop.

sherrellbc · 2017-01-04 18:14:29

R00KIE wrote:

You might want to make sure you have something using the AMD card before you try to read something, I suspect that is a discrete gpu and you have an integrated gpu. If that is the case then the driver automatically turns the card off to save power.

I was wondering the same thing. However, the logs from Xorg seem to indicate the driver is properly loaded. Xorg logs are admittedly difficult to understand unless well versed, so perhaps I am misinterpreting them.

[    21.532] (==) Matched intel as autoconfigured driver 0
[    21.532] (==) Matched ati as autoconfigured driver 1
[    21.532] (==) Matched intel as autoconfigured driver 2
[    21.532] (==) Matched modesetting as autoconfigured driver 3
[    21.532] (==) Matched fbdev as autoconfigured driver 4
[    21.532] (==) Matched vesa as autoconfigured driver 5
[    21.532] (==) Assigned the driver to the xf86ConfigLayout
[    21.532] (II) LoadModule: "intel"
[    21.533] (WW) Warning, couldn't open module intel
[    21.533] (II) UnloadModule: "intel"
[    21.533] (II) Unloading intel
[    21.533] (EE) Failed to load module "intel" (module does not exist, 0)
[    21.533] (II) LoadModule: "ati"
[    21.533] (II) Loading /usr/lib/xorg/modules/drivers/ati_drv.so
[    21.533] (II) Module ati: vendor="X.Org Foundation"
[    21.533]    compiled for 1.18.4, module version = 7.8.0
[    21.533]    Module class: X.Org Video Driver
[    21.533]    ABI class: X.Org Video Driver, version 20.0
[    21.533] (II) LoadModule: "radeon"
[    21.533] (II) Loading /usr/lib/xorg/modules/drivers/radeon_drv.so
[    21.537] (II) Module radeon: vendor="X.Org Foundation"
[    21.537]    compiled for 1.18.4, module version = 7.8.0
[    21.537]    Module class: X.Org Video Driver
[    21.537]    ABI class: X.Org Video Driver, version 20.0
[    21.537] (II) LoadModule: "modesetting"
[    21.537] (II) Loading /usr/lib/xorg/modules/drivers/modesetting_drv.so
[    21.538] (II) Module modesetting: vendor="X.Org Foundation"
[    21.538]    compiled for 1.18.4, module version = 1.18.4
[    21.538]    Module class: X.Org Video Driver
[    21.538]    ABI class: X.Org Video Driver, version 20.0
[    21.538] (II) LoadModule: "fbdev"
[    21.538] (II) Loading /usr/lib/xorg/modules/drivers/fbdev_drv.so
[    21.539] (II) Module fbdev: vendor="X.Org Foundation"
[    21.539]    compiled for 1.18.0, module version = 0.4.4
[    21.539]    Module class: X.Org Video Driver
[    21.539]    ABI class: X.Org Video Driver, version 20.0
[    21.539] (II) LoadModule: "vesa"
[    21.539] (II) Loading /usr/lib/xorg/modules/drivers/vesa_drv.so
[    21.540] (II) Module vesa: vendor="X.Org Foundation"
[    21.540]    compiled for 1.18.0, module version = 2.3.4
[    21.540]    Module class: X.Org Video Driver
[    21.540]    ABI class: X.Org Video Driver, version 20.0
[    21.540] (II) RADEON: Driver for ATI/AMD Radeon chipsets:
        ATI Radeon Mobility X600 (M24), ATI FireMV 2400,
        ATI Radeon Mobility X300 (M24), ATI FireGL M24 GL,
        ATI Radeon X600 (RV380), ATI FireGL V3200 (RV380),
        ATI Radeon IGP320 (A3), ATI Radeon IGP330/340/350 (A4),
        ....

Also, the output lspci reports "radeon" as the loaded driver for the pci device.

In contrary to the above, the output of lshw reports differently ..

$ sudo lshw -c video
  *-display                 
       description: VGA compatible controller
       product: 4th Gen Core Processor Integrated Graphics Controller
       vendor: Intel Corporation
       physical id: 2
       bus info: pci@0000:00:02.0
       version: 06
       width: 64 bits
       clock: 33MHz
       capabilities: msi pm vga_controller bus_master cap_list rom
       configuration: driver=i915 latency=0
       resources: irq:33 memory:f5800000-f5bfffff memory:d0000000-dfffffff ioport:f000(size=64) memory:c0000-dffff

Xorg seems to load the driver but not use it. Last I checked my graphics card is supported by the driver. There appear to be no errors in the Xorg log indicating the radeon driver was not in use. It seems as if the driver was loaded and found the ATI hardware (per output of lspci) but decided against using it.

Last edited by sherrellbc (2017-01-04 18:23:07)

R00KIE · 2017-01-04 21:09:24

It's not the Xorg driver that does the power management, it's the kernel driver. You can run something like glxgears to keep the card awake but if you have more than one card make sure you are running glxgears on the amd card.

Lone_Wolf · 2017-01-05 12:41:25

That xorg log matches a hybrid graphics setup, with X using modesetting to drive the intel card and radeon for the amd card.

I do agree the kernel driver is likely to have powered down everything related to the amd card.

sherrellbc, you have realized lshw shows info about the intel videocard, not the amd one ?

sherrellbc · 2017-01-05 13:04:56

Lone_Wolf wrote:

That xorg log matches a hybrid graphics setup, with X using modesetting to drive the intel card and radeon for the amd card.
I do agree the kernel driver is likely to have powered down everything related to the amd card.

This appears to be a default mode of operation as I have not explicitly configured X for either direction. I was loosely basing my expectation of the system on "if I have a dedicated GPU it should be used by default".

Lone_Wolf wrote:

sherrellbc, you have realized lshw shows info about the intel videocard, not the amd one ?

Yes. I was confused because lspci reported the driver for the GPU was radeon but lshw reported the i915 driver was being used for video. This, of course, implies the AMD graphics card is not being used at all -- which makes a lot of sense if we are entertaining the idea that it is being powered down. How can I ensure the dedicated card has a priority of use over the integrated Intel VGA?

Regardless of the above, what can be said about the null PCI configuration space of the GPU?

Lone_Wolf · 2017-01-05 15:04:31

How can I ensure the dedicated card has a priority of use over the integrated Intel VGA?

A hybrid graphics setup always has a primary and a secondary card. Usually the integrated card is primary, but there are motherboards that allow to change the primary card in firmware.
For running X, the PRIME wiki page outlines the (limited) options available for using both cards.
Wayland has it's own mechanism for hybrid graphics, which appear to depend heavily on the wayland compositor used.

------------------------------------------

what can be said about the null PCI configuration space of the GPU?

first you need the address for the active (intel) videocard . posting full lspci -k output should make that easy.
Then I suggest getting the PCI configuration space for the intel GPU and see if that matches the spec / expectations.

If it does, try to find how to determine whether a gpu is active so the code can output something like "warning, card is powered down, no reliable data available" .

Last edited by Lone_Wolf (2017-01-05 15:05:23)

sherrellbc · 2017-01-05 17:49:16

Lone_Wolf wrote:

That xorg log matches a hybrid graphics setup, with X using modesetting to drive the intel card and radeon for the amd card.

How did you come to this conclusion? Was it just that Xorg decided to first load the i915/modesetting driver and the radeon driver? I ask because you were right. I disabled "AMD PowerXpress" from the UEFI setup and I now all appears to work. I can now read the PCI configuration space for the graphics card. As an added side-effect, my graphics look much better.

$ lspci -xxxs 01:00.0
01:00.0 VGA compatible controller: Advanced Micro Devices, Inc. [AMD/ATI] Saturn XT [FirePro M6100]
00: 02 10 40 66 07 04 10 00 00 00 00 03 10 00 80 00
10: 0c 00 00 e0 00 00 00 00 0c 00 00 f0 00 00 00 00
20: 01 e0 00 00 00 00 e0 f7 00 00 00 00 28 10 cd 15
30: 00 00 e4 f7 48 00 00 00 00 00 00 00 0b 01 00 00

    26.392] (II) LoadModule: "glx"
[    26.393] (II) Loading /usr/lib/xorg/modules/extensions/libglx.so
[    26.404] (II) Module glx: vendor="X.Org Foundation"
[    26.404]    compiled for 1.18.4, module version = 1.0.0
[    26.404]    ABI class: X.Org Server Extension, version 9.0
[    26.404] (==) AIGLX enabled
[    26.404] (==) Matched ati as autoconfigured driver 0
[    26.404] (==) Matched ati as autoconfigured driver 1
[    26.404] (==) Matched modesetting as autoconfigured driver 2
[    26.404] (==) Matched fbdev as autoconfigured driver 3
[    26.404] (==) Matched vesa as autoconfigured driver 4
[    26.404] (==) Assigned the driver to the xf86ConfigLayout
[    26.404] (II) LoadModule: "ati"
[    26.406] (II) Loading /usr/lib/xorg/modules/drivers/ati_drv.so
[    26.406] (II) Module ati: vendor="X.Org Foundation"
[    26.406]    compiled for 1.18.4, module version = 7.8.0
[    26.406]    Module class: X.Org Video Driver
[    26.406]    ABI class: X.Org Video Driver, version 20.0
[    26.406] (II) LoadModule: "radeon"
[    26.407] (II) Loading /usr/lib/xorg/modules/drivers/radeon_drv.so
[    26.410] (II) Module radeon: vendor="X.Org Foundation"
[    26.410]    compiled for 1.18.4, module version = 7.8.0
[    26.410]    Module class: X.Org Video Driver
[    26.410]    ABI class: X.Org Video Driver, version 20.0
[    26.410] (II) LoadModule: "modesetting"
[    26.410] (II) Loading /usr/lib/xorg/modules/drivers/modesetting_drv.so
[    26.411] (II) Module modesetting: vendor="X.Org Foundation"
[    26.411]    compiled for 1.18.4, module version = 1.18.4
[    26.411]    Module class: X.Org Video Driver
[    26.411]    ABI class: X.Org Video Driver, version 20.0
[    26.411] (II) LoadModule: "fbdev"
[    26.411] (II) Loading /usr/lib/xorg/modules/drivers/fbdev_drv.so
[    26.412] (II) Module fbdev: vendor="X.Org Foundation"
[    26.412]    compiled for 1.18.0, module version = 0.4.4
[    26.412]    Module class: X.Org Video Driver
[    26.412]    ABI class: X.Org Video Driver, version 20.0
[    26.412] (II) LoadModule: "vesa"
[    26.413] (II) Loading /usr/lib/xorg/modules/drivers/vesa_drv.so
[    26.414] (II) Module vesa: vendor="X.Org Foundation"
[    26.414]    compiled for 1.18.0, module version = 2.3.4
[    26.414]    Module class: X.Org Video Driver
[    26.414]    ABI class: X.Org Video Driver, version 20.0
[    26.414] (II) RADEON: Driver for ATI/AMD Radeon chipsets:
        ATI Radeon Mobility X600 (M24), ATI FireMV 2400,

From the X log above I noticed that it no longer even attempts to load the i915 driver for the integrated graphics. Actually, as far as I can tell, the bios never even told Linux that integrated graphics exists on the system. The integrated graphics device is not shown in lspci any longer.

So, does X just enumerate all graphics hardware it finds after probing the PCI device list?

Last edited by sherrellbc (2017-01-05 18:44:53)

Lone_Wolf · 2017-01-06 11:58:48

Hybrid graphics is basically just a special case of having two gpus in 1 system.
X relies on udev to detect such things, udev gets lots of info from kernel, kernel gets info from firmware/hardware

[    21.532] (==) Matched intel as autoconfigured driver 0
[    21.532] (==) Matched ati as autoconfigured driver 1

Since intel doesn't make discrete videocards, there's only one possible setup where both intel and ati are autodetected by xorg :
hybrid graphics with an intel integrated gpu and a discrete amd gpu .
Combining the lshw output showing intel gpu with the lspci output showing amd gpu confirmed that setup.
(Full lspci output would also have done that).

Arch Linux

#1 2016-12-29 20:15:26

[SOLVED] Fail to read PCI config space; lspci and /dev/mem read fails

#2 2016-12-29 20:45:29

Re: [SOLVED] Fail to read PCI config space; lspci and /dev/mem read fails

#3 2017-01-04 14:34:08

Re: [SOLVED] Fail to read PCI config space; lspci and /dev/mem read fails

#4 2017-01-04 17:29:33

Re: [SOLVED] Fail to read PCI config space; lspci and /dev/mem read fails

#5 2017-01-04 18:14:29

Re: [SOLVED] Fail to read PCI config space; lspci and /dev/mem read fails

#6 2017-01-04 21:09:24

Re: [SOLVED] Fail to read PCI config space; lspci and /dev/mem read fails

#7 2017-01-05 12:41:25

Re: [SOLVED] Fail to read PCI config space; lspci and /dev/mem read fails

#8 2017-01-05 13:04:56

Re: [SOLVED] Fail to read PCI config space; lspci and /dev/mem read fails

#9 2017-01-05 15:04:31

Re: [SOLVED] Fail to read PCI config space; lspci and /dev/mem read fails

#10 2017-01-05 17:49:16

Re: [SOLVED] Fail to read PCI config space; lspci and /dev/mem read fails

#11 2017-01-06 11:58:48

Re: [SOLVED] Fail to read PCI config space; lspci and /dev/mem read fails

Board footer