dGPU not turning off when not used

OrakMoya · 2023-06-04 12:39:48

Hello,

I have a ThinkPad P52 with an Nvidia Quadro P1000 GPU. I have the laptop run in hybrid mode to use the iGPU for the desktop and like and the dGPU only for apps that really need it. I installed official Nvidia drivers, the nvidia-prime package to be able to run any app with the dGPU, as well as switcheroo because it was recommended for Gnome Integration here .

However, the dGPU never seems to turn off. Gnome-shell does seem to have a bug making it use the dGPU for the shell despite it not being the default, but I seem to have fixed this by not loading the Nvidia modules before a wayland compositors as instructed here. Running `nvidia-smi` does show that no program is using the dGPU:

+---------------------------------------------------------------------------------------+
| NVIDIA-SMI 530.41.03              Driver Version: 530.41.03    CUDA Version: 12.1     |
|-----------------------------------------+----------------------+----------------------+
| GPU  Name                  Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf            Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|                                         |                      |               MIG M. |
|=========================================+======================+======================|
|   0  Quadro P1000                    Off| 00000000:01:00.0 Off |                  N/A |
| N/A   51C    P0               N/A /  N/A|      0MiB /  4096MiB |      0%      Default |
|                                         |                      |                  N/A |
+-----------------------------------------+----------------------+----------------------+
                                                                                         
+---------------------------------------------------------------------------------------+
| Processes:                                                                            |
|  GPU   GI   CI        PID   Type   Process name                            GPU Memory |
|        ID   ID                                                             Usage      |
|=======================================================================================|
|  No running processes found                                                           |
+---------------------------------------------------------------------------------------+

, but running `cat /sys/class/drm/card1/device/power_state` always shows "D0" and never anything else, meaning the card is always on.

Loading bbswitch with modprobe confirms this, as running "cat /proc/acpi/bbswitch" always returns ON, even when i echo "OFF" into it.

seth · 2023-06-04 13:44:05

Runtime D3 is only supported for turing and newer chips, https://wiki.archlinux.org/title/PRIME# … Management

Can you bbswitch off the GPU before gnome (or GDM) starts?
Does it work w/ gnome on X11?
Is there a monitor attached to the GPU?

OrakMoya · 2023-06-04 14:28:29

On X, "nvidia-smi" reports that xorg is running on the dGPU so it keeps it on. I can bbswitch the GPU off if I install bumblebee and have it disable the nvidia module, but I don't want to use bumblebee because using the dGPU with it seems janky. There is no monitor attached to it.

seth · 2023-06-04 14:30:27

"nvidia-smi" reports that xorg is running on the dGPU so it keeps it on

No, that's not what that means, the thing keepign the GPU busy at that time is btw. nvidia-smi…

Check the power state and demand w/ eg. powertop, but you're not eligible for rtd3, so you cannot *fully* power down the GPU.

OrakMoya · 2023-06-04 14:49:34

Not exactly sure where in powertop I can see the powerstate and demand. The "Idle Stats" page has some info on the gpu:

                    |             GPU     |
                    |                     |
                    | Powered On  7.6%    |
                    | RC6        92.4%    |
                    | RC6p        0.0%    |
                    | RC6pp       0.0%    |

, but the "Device stats" tab shows " PCI Device: NVIDIA Corporation GP107GLM [Quadro P1000 Mobile]" at 100.0% usage. When running "prime-run glxgears", powertop shows 0.0 GPU ops/seconds, and the "Powered on" percentage above doesnt change much.

seth · 2023-06-04 15:02:23

When running "prime-run glxgears", powertop shows 0.0 GPU ops/seconds

Because that's gonna be the intel GPU.

Look at the overall power drain and compare "glxgears" and "prime-run glxgears"

OrakMoya · 2023-06-04 15:34:17

I did some testing by monitoring the power draw of the battery. With bumblebee off and the GPU in D0 power state, on X the laptop when completely idle pulled about 9.3W. Running "glxgears" saw an increase to about 12W. I couldnt exactly measure how much it would consume running on the GPU because "prime-run glxgears" insisted on rendering at 6000+FPS, even when passing "__GL_SYNC_TO_VBLANK=1 vblank_mode=1" to it.

Enabling bumblebee and rebooting, bbswitch reported the card is "OFF". Power draw then was actually about 10W when idle, so now I'm convinced that bbswitch "OFF" isnt actually off.

It isnt a really big deal if the GPU cant turn off, the laptop does have a 90Wh battery so even with it on it would last 10 hours idling.

seth · 2023-06-04 15:45:43

so now I'm convinced that bbswitch "OFF" isnt actually off

Can you disable the GPU in the BIOS and check the power draw w/ that config?

insisted on rendering at 6000+FPS, even when passing "__GL_SYNC_TO_VBLANK=1 vblank_mode=1" to it

https://wiki.archlinux.org/title/PRIME# … ronization - or you could test the unsync'd runs against each other

Also the switcheroo-stuff seems to interfere w/ at least bumblebee and I'd not be surprised if it collides w/ regular prime as well, so maybe delay the gnome integration a bit and focus on the principal functionality first.

OrakMoya · 2023-06-04 16:27:29

The BIOS only lets me choose between Discrete and Hybrid graphics, so I cant disable the dGPU entirely through the BIOS.

Trying to enable PRIME synchronization returns this on X:

X Error of failed request:  BadName (named color or font does not exist)
  Major opcode of failed request:  140 (RANDR)
  Minor opcode of failed request:  11 (RRQueryOutputProperty)
  Serial number of failed request:  28
  Current serial number in output stream:  28

Though I did notice that I forgot to remove bumblebee when testing the power draw with the card "off" (this resulted in the external monitor ports not working and the card not being visible in "/sys/class/drm" even with bumblebeed.service being disabled, but I could still run prime-run and blender could still render on it). I removed it and managed to get glxgears to sync to vblank with prime-run. Power draw seems to be the same between both the dGPU and the iGPU (they are running on seperate GPUs, checked with glxinfo).

Disabling and removing switcheroo doesn't seem to have made a difference.

Last edited by OrakMoya (2023-06-04 16:31:20)

seth · 2023-06-05 21:00:04

The BIOS only lets me choose between Discrete and Hybrid graphics, so I cant disable the dGPU entirely through the BIOS.

Activate discrete (only) and compare its power draw w/ your current hybrid setup.

OrakMoya · 2023-06-07 14:33:32

On discrete only, the laptop pulls a bit more than 9 Watts while displaying on the ThinkPad LCD. Disabling the LCD in BIOS (its hardwired to the iGPU) and making it display on the HDMI (external ports are hardwired to the dGPU), GDM fails to start but the laptop still consumes a bit more than 9 Watts.

All of this leads me to believe that the dGPU is on at all times, despite the description in BIOS inferring otherwise:

https://i.imgur.com/bmatbxLl.jpg

moderator edit -- replaced oversized image with link.
Pasting pictures and code

Last edited by 2ManyDogs (2023-06-07 22:26:00)

seth · 2023-06-07 17:57:00

Is a BIOS update available for the system?

OrakMoya · 2023-06-07 22:06:20

Already newest BIOS version. I updated it through Windows immediately after receiving it beacuse the Arch Wiki page for the laptop has a note for BIOS bricking at an old enough BIOS version.

seth · 2023-06-08 06:10:22

I updated it through Windows

3rd link below. Mandatory.
Disable it (it's NOT the BIOS setting!) and reboot windows and linux twice for voodo reasons.

OrakMoya · 2023-06-08 06:57:43

I cant seem to find the link you're referring to?

seth · 2023-06-08 07:02:58

In my signature.
http://wiki.archlinux.org/index.php/Dua … ibernation

OrakMoya · 2023-06-08 07:57:28

I completely wiped windows as I was installing arch after updating the BIOS.

Arch Linux

#1 2023-06-04 12:39:48

dGPU not turning off when not used

#2 2023-06-04 13:44:05

Re: dGPU not turning off when not used

#3 2023-06-04 14:28:29

Re: dGPU not turning off when not used

#4 2023-06-04 14:30:27

Re: dGPU not turning off when not used

#5 2023-06-04 14:49:34

Re: dGPU not turning off when not used

#6 2023-06-04 15:02:23

Re: dGPU not turning off when not used

#7 2023-06-04 15:34:17

Re: dGPU not turning off when not used

#8 2023-06-04 15:45:43

Re: dGPU not turning off when not used

#9 2023-06-04 16:27:29

Re: dGPU not turning off when not used

#10 2023-06-05 21:00:04

Re: dGPU not turning off when not used

#11 2023-06-07 14:33:32

Re: dGPU not turning off when not used

#12 2023-06-07 17:57:00

Re: dGPU not turning off when not used

#13 2023-06-07 22:06:20

Re: dGPU not turning off when not used

#14 2023-06-08 06:10:22

Re: dGPU not turning off when not used

#15 2023-06-08 06:57:43

Re: dGPU not turning off when not used

#16 2023-06-08 07:02:58

Re: dGPU not turning off when not used

#17 2023-06-08 07:57:28

Re: dGPU not turning off when not used

Board footer