NVIDIA GPU not staying/returning to D3cold after resume from suspend

Sparronator9999 · Yesterday 03:05:09

Hi all, new Arch Linux user (and recent Windows refugee) here. Not sure if this should be posted in Kernel & Hardware instead, but I am posting about a laptop, so here goes nothing.

I've been following the wiki to set up my laptop with hybrid graphics. After following the wiki's section on enabling RTD3 power management (I used the nvidia-prime-rtd3pm AUR package in the end, but I did try manually configuring the udev and module parameters according to the wiki too), everything works file except for one issue: the GPU doesn't re-enter (or stay in?) D3cold after system resume, which is indicated by my laptop's power LED turning orange (along with the standard high power draw on battery).

I found the following workarounds, but there are probably more (such as running a game on the dGPU, but I didn't test this yet):

- Toggle runtime PM off and on again, which worked until I tried to automate this with systemd (see the end of this post).
- Run `lspci` as root (running it as a normal user does nothing - probably because lspci doesn't poll the physical devices, to my understanding).

Just from this brief testing, it seems to me like the GPU driver doesn't properly restore the D3PM state until something interacts with it.

Any suggestions on how I can fix or work around this issue?

Laptop specs

MSI GF63 Thin 11SC
CPU: Intel Core i7-11800H
GPU (integrated): Intel UHD Graphics
GPU (dedicated): NVIDIA GeForce GTX 1650 with Max-Q design

Relevant software

Linux 7.1.4-arch1-1 kernel
mesa 1:26.1.5-1 for (userspace?) Intel GPU drivers
nvidia-open-dkms + nvidia-utils 610.43.03-3 for NVIDIA GPU drivers
KDE Plasma 6.7.3 desktop environment (using Wayland sessions)

Systemd workaround I tried

Based on this Stack Exchange answer.

[Unit]
Description=Fix NVIDIA RTD3 power management after suspend
After=suspend.target

[Service]
ExecStart=/usr/bin/bash /usr/local/sbin/nvidia_rtd3_pm_fix.sh

[Install]
WantedBy=suspend.target

The script pointed to by the above systemd unit has the following contents:

#!/bin/bash
echo on > /sys/bus/pci/devices/0000\:01\:00.0/power/control
echo auto > /sys/bus/pci/devices/0000\:01\:00.0/power/control

Feel free to ask for more information if I missed anything.

Edit: Fixed the `nvidia_rtd3_pm_fix.sh` script not being executed correctly (I forgot to add `/usr/bin/bash` to the `ExecStart` section of the systemd unit, to run the bash script with... well, bash). This didn't fix my problem, though, so leaving it open.

Last edited by Sparronator9999 (Yesterday 04:17:21)

seth · Yesterday 12:04:12

Any suggestions on how I can fix or work around this issue?

https://wiki.archlinux.org/title/PRIME#NVIDIA - tried a udev rule?

Otherwise/alternatively

- Toggle runtime PM off and on again, which worked until I tried to automate this with systemd (see the end of this post).

Make /usr/local/sbin/nvidia_rtd3_pm_fix.sh log its actions, https://man.archlinux.org/man/logger.1 and post the complete system journal for a boot after a wakeup causing this:

sudo journalctl -b | curl -s -H "Accept: application/json, */*" --upload-file - 'https://paste.c-net.org/'

Given "it works manually but not via script" there's a good chance that this is just a race condition and a strategic "ExecStartPre=sleep 3" circumvents that

Sparronator9999 · Today 10:38:41

seth wrote:

https://wiki.archlinux.org/title/PRIME#NVIDIA - tried a udev rule?

Yes, I tried both the NVIDIA recommended ones as well as the community suggested rules (which fixed my dGPU not suspending on system start). The NVIDIA rules don't seem to have an effect on my system.

seth wrote:

Given "it works manually but not via script" there's a good chance that this is just a race condition and a strategic "ExecStartPre=sleep 3" circumvents that

I did try that (even going as high as 10 seconds on the `sleep` command), but the solution ended up being to toggle runtime PM off and on, wait for the GPU to "suspend" (it isn't actually properly in the D3cold state at this point), then toggle runtime PM off and on again. The next time the GPU is suspended, it'll properly enter D3cold. The whole process took upwards of half a minute from wake to proper GPU suspend. Here's the script I ended up using for that (saved to `/usr/local/sbin/nvidia-rtd3-pm-fix.sh`):

#!/bin/bash
if [ $EUID -ne 0 ]; then
    echo Please run as root
    exit 1
fi

control_path='/sys/bus/pci/devices/0000:01:00.0/power/control'                                                       
status_path='/sys/bus/pci/devices/0000:01:00.0/power/runtime_status'

echo Toggling runtime PM on/off after wake from suspend...
echo on > $control_path
echo auto > $control_path

echo Waiting for GPU to suspend...
while :; do
    status_val=`cat $status_path`
    echo $status_val
    if [ "$status_val" = "suspended" ]; then
        break
    fi
    sleep 1
done

echo Toggling runtime PM on/off again to work around NVIDIA driver bug preventing GPU from actually suspending...
echo on > $control_path
echo auto > $control_path
echo Done!

I also updated the systemd unit to run after the `hibernate` and `hybrid-sleep` targets, as I found this bug happens when resuming from hibernation as well. I'm not sure if running after `suspend-then-hibernate.target` is also necessary. Here's that again, for anyone reading this in the future (save it to `/etc/systemd/system/nvidia-rtd3-pm-fix.service`):

[Unit]
Description=NVIDIA Runtime D3 Power Management Fix
After=suspend.target hibernate.target hybrid-sleep.target

[Service]
Type=simple
ExecStart=/bin/bash /usr/local/sbin/nvidia-rtd3-pm-fix.sh

[Install]
WantedBy=suspend.target hibernate.target hybrid-sleep.target

If you have any improvements to make to my script (e.g. some other method to watch for updates to '/sys/bus/pci/devices/0000:01:00.0/power/runtime_status' instead of polling it every second), feel free to reply with them. Otherwise, I suppose I'll mark this as solved, even though I think this solution is more of a hack and that NVIDIA should properly fix this before I reconsider using nouveau + NVK as my GPU driver.

PS: please ignore the slightly different file/unit names - I ended up scrapping my initial script and unit when I thought they didn't work initially.

Arch Linux

#1 Yesterday 03:05:09