[SOLVED] NVIDIA: peculiar bbswitch issue with runtime PM

tactless500 · 2025-01-15 16:44:15

Hello all, my first post, hopefully in the correct category.

TLDR: bbswitch switches nvidia nicely on and off. Enable runtime PM for nvida. Disable runtime PM for nvidia. now bbswitch cannot switch the card on, even though runtime PM is disabled.

Current state: Thinkpad T480 laptop with discrete nvidia MX150 card. I use it with nvidia prop. driver. I have bbswitch installed (and autoloaded on boot), which allows me to completely disable the DGPU and enjoy the sweet powertop output of 3.6W on idle. I have nvidia_prime installed, so when i want some program (freecad) to use the DGPU, I do (with a script):

#!/bin/sh
[ ! -f /proc/acpi/bbswitch ] && echo "bbswitch module not found \n attempting to load bbswitch module" && sudo modprobe bbswitch
echo "sending ON to bbswitch" && echo ON | sudo tee /proc/acpi/bbswitch && cat /proc/acpi/bbswitch | grep "ON" && echo "loading nvidia kernel modules" && sudo modprobe nvidia nvidia_modeset nvidia_drm nvidia_uvm || echo "failed to enable gpu in bbswitch"

which enables the gpu with bbswich and loads the kernel drivers for the card. To check:

$ cat /proc/acpi/bbswitch && lsmod | grep nvidia
0000:01:00.0 ON
...

I did not capture the response on the lsmod, but it correctly lists the nvidia driver stack as loaded.
Then i run

$ prime-run freecad

and monitor the DGPU usage with nvtop.
To disable DGPU, the reverse process works well - close all processes that use the card which can be found with:

$ lsof | grep nvidia

and use another script to unload the nvidia driver stack and write OFF to /proc/acpi/bbswitch. All in all, this has been a very useful setup for me, as power is saved when dpgu not needed, but i still have quick access to the card when i want it.

I also use TLP, where the config options:

RUNTIME_PM_DENYLIST="01:00.0" # NVIDIA PCI address
RUNTIME_PM_DRIVER_DENYLIST="nouveau nvidia"

are necessary to prevent the runtime PM from interfering with the bbswitch. If runtime PM is active for the card, i am unable to write ON to /proc/acpi/bbswitch. If nvidia driver is attempted to be loaded while bbswitch is in OFF state, that messes up the PCI space and requires reboot to be able to load the nvidia driver again, as per bbswitch docs.

Issue: after enabling the card using the sript above, I went into powertop, which suggested to me to enable runtime PM for the now up-and-running nvidia card. I did this (changing the appropriate options to GOOD in powertop) thinking I can easily reverse this later. I later switched off the card, as per process above. now i am unable to reenable the card (again, process above) unless i reboot the machine. Even when i change the runtime PM options in powertop back to BAD. Here is the powetop options:

    Bad           Runtime PM for PCI Device NVIDIA Corporation GP108M [GeForce MX150]

and another one, but i cannot find it now. i think it was Runtime PM for I2C for the nvidia

I believe that the runtime PM changes something else, somewhere else in the system that prevents from wiritng to bbswitch to enable it and get ON in /proc/acpi/bbswitch (by the way, the echo ON | sudo tee /proc/acpi/bbswitch does not throw an error, i do have write rights, etc. it is just something else overwriting the config). Persists after PM disabled

Attempting to echo ON to bbswitch will yield in dmesg and journald:

Jan 15 09:54:18 t480-arch kernel: bbswitch: enabling discrete graphics

However, the bbswitch still has OFF.
In addition I checked that the following happens:

cat /sys/bus/pci/devices/0000:01:00.0/power/runtime_status
active

so the card is in a weird regime, where bbswitch is OFF, but /sys says it is active.

To be clear, the above only happens when runtime PM is somehow enabled (by tlp or powertop, or manually), and persists until reboot even if runtime pm is manually disabled later for the PCI of the DGPU.

So does anyone have any idea what is happening there? i would like to learn about what happens in the background when the runtime pm is enabled for the dgpu that prevents the bbswitch from re-enabling the card later, even with PM disabled. Any replies welcome and let me know if something is unlcear in the post and should be edited!

Last edited by tactless500 (2025-01-17 00:01:10)

Lone_Wolf · 2025-01-16 09:21:18

Look at https://wiki.archlinux.org/title/Bumble … kernel_4.8 , especially the text in the colored box seems relevant .

tactless500 · 2025-01-17 00:00:41

Lone_Wolf wrote:

Look at https://wiki.archlinux.org/title/Bumble … kernel_4.8 , especially the text in the colored box seems relevant .

Wow, I can't believe I missed that, that explains it. Thank you! So the wiki says that since 4.8 kernel the PM method changed but bbswitch project never adapted to this. The solution suggested at arch wiki is to globally disable power management for all PCI devices in a kernel parameter - not very power friendly. It seems there is no possibility to disable PM of specific devices using that way.

Maybe it is the time to check out PRIME using nvidia_prime, just curious if it can completely power down the card - like bbswitch can.

Last edited by tactless500 (2025-01-17 00:04:08)

tactless500 · 2025-01-17 00:09:29

I am actually thinking to try to make the power management file unwriteable as a quick solution, so that no software, like powertop, would be able to enable this. As enabling PM for the card will make it impossible for bbswitch to enable the card later.

sudo chmod 444 /sys/bus/pci/devices/0000:01:00.0/power/control

Any thoughts?

Last edited by tactless500 (2025-01-17 00:10:29)

Arch Linux

#1 2025-01-15 16:44:15

[SOLVED] NVIDIA: peculiar bbswitch issue with runtime PM

#2 2025-01-16 09:21:18

Re: [SOLVED] NVIDIA: peculiar bbswitch issue with runtime PM

#3 2025-01-17 00:00:41

Re: [SOLVED] NVIDIA: peculiar bbswitch issue with runtime PM

#4 2025-01-17 00:09:29

Re: [SOLVED] NVIDIA: peculiar bbswitch issue with runtime PM

Board footer