Unable to set NVIDIA Card to D3 state on hybrid laptop

qwerty812 · 2023-07-05 18:32:48

Hi, I just installed Arch (KDE PLASMA on wayland) on my Acer Nitro 5 Laptop. It has an integrated GPU (from i5-10300H CPU) and a dedicated one (GTX1650Ti)
As from my previous Ubuntu installation I managed to set the dedicated GPU to D3 state when not in use.
Trying to do the same on Arch doesn't work (It keeps on D0 state).
I'm running Arch with KDE on Wayland

I installed nvidia closed source drivers (nvidia package)
I followed PRIME render offload section

That's the content of my /etc/udev/rules.d/80-nvidia-pm.rules file:

# Enable runtime PM for NVIDIA VGA/3D controller devices on driver bind
ACTION=="bind", SUBSYSTEM=="pci", ATTR{vendor}=="0x10de", ATTR{class}=="0x030000", TEST=="power/control", ATTR{power/control}="auto"
ACTION=="bind", SUBSYSTEM=="pci", ATTR{vendor}=="0x10de", ATTR{class}=="0x030200", TEST=="power/control", ATTR{power/control}="auto"

# Disable runtime PM for NVIDIA VGA/3D controller devices on driver unbind
ACTION=="unbind", SUBSYSTEM=="pci", ATTR{vendor}=="0x10de", ATTR{class}=="0x030000", TEST=="power/control", ATTR{power/control}="on"
ACTION=="unbind", SUBSYSTEM=="pci", ATTR{vendor}=="0x10de", ATTR{class}=="0x030200", TEST=="power/control", ATTR{power/control}="on"

Content of /etc/modprobe.d/nvidia-pm.conf file:

options nvidia "NVreg_DynamicPowerManagement=0x02"

I enabled nvidia-persistenced service with

sudo systemctl enable nvidia-persistenced.service

sudo systemctl start nvidia-persistenced.service

Checking power states gives:

❯ cat /sys/class/drm/card*/device/power_state
D0
D0

Checking card capabilities:

❯ cat /proc/driver/nvidia/gpus/0000\:01\:00.0/power
Runtime D3 status:          Enabled (fine-grained)
Video Memory:               Active

GPU Hardware Support:
 Video Memory Self Refresh: Supported
 Video Memory Off:          Supported

❯ nvidia-smi
Wed Jul  5 20:18:37 2023       
+---------------------------------------------------------------------------------------+
| NVIDIA-SMI 535.54.03              Driver Version: 535.54.03    CUDA Version: 12.2     |
|-----------------------------------------+----------------------+----------------------+
| GPU  Name                 Persistence-M | Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp   Perf          Pwr:Usage/Cap |         Memory-Usage | GPU-Util  Compute M. |
|                                         |                      |               MIG M. |
|=========================================+======================+======================|
|   0  NVIDIA GeForce GTX 1650 Ti     On  | 00000000:01:00.0 Off |                  N/A |
| N/A   40C    P8               1W /  50W |      1MiB /  4096MiB |      0%      Default |
|                                         |                      |                  N/A |
+-----------------------------------------+----------------------+----------------------+
                                                                                         
+---------------------------------------------------------------------------------------+
| Processes:                                                                            |
|  GPU   GI   CI        PID   Type   Process name                            GPU Memory |
|        ID   ID                                                             Usage      |
|=======================================================================================|
|  No running processes found                                                           |
+---------------------------------------------------------------------------------------+

One strange thing I noticed is that cat /sys/bus/pci/devices/0000\:01\:00.0/power/control always gives 'on' after reboot (That should be changed to auto by the first rule, right?)

Thanks in advance

seth · 2023-07-05 20:43:30

Do you have the nvidia kernel modules in the initramfs?
Did you rebuild the latter?
Can you manually move the power/control to auto?

qwerty812 · 2023-07-06 12:41:06

Hi, yes I added the module with the line: MODULES=(btrfs nvidia nvidia_modeset nvidia_uvm nvidia_drm) in /etc/mkinitcpio.conf
Also I saw it gets loaded here:

lsinitcpio /boot/initramfs-linux.img | grep nvidia
etc/modprobe.d/nvidia.conf
etc/modprobe.d/nvidia-pm.conf
usr/lib/firmware/nvidia/
usr/lib/firmware/nvidia/535.54.03/
usr/lib/firmware/nvidia/535.54.03/gsp_ga10x.bin
usr/lib/firmware/nvidia/535.54.03/gsp_tu10x.bin
usr/lib/modprobe.d/nvidia-pm.conf
usr/lib/modprobe.d/nvidia-utils.conf
usr/lib/modules/6.4.1-arch2-1/extramodules/nvidia-drm.ko
usr/lib/modules/6.4.1-arch2-1/extramodules/nvidia.ko
usr/lib/modules/6.4.1-arch2-1/extramodules/nvidia-modeset.ko
usr/lib/modules/6.4.1-arch2-1/extramodules/nvidia-uvm.ko

Yes I can successfully move the power to auto manually but at every reboot I find it set to 'on'

seth · 2023-07-06 14:53:37

Does the GPU (after a moment) enter D3 when setting the power/control to auto?

What happens if you udevadm trigger? udevadm control --reload; udevadm trigger?
https://wiki.archlinux.org/title/Udev#Loading_new_rules

What if you skip/remove the unbinding rule?

qwerty812 · 2023-07-06 16:29:45

Hi, thanks in advance.
I don't know what I did but now when I issue `echo auto | sudo tee -a /sys/bus/pci/devices/0000\:01\:00.0/power/control` the card correctly goes to D3 state!

Now I just need to make it permanent with udev rules
if I issue `sudo udevadm test /sys/bus/pci/devices/0000\:01\:00.0` I can see my rule being read but running `sudo udevadm control --reload` and 'sudo udevadm trigger' doesn't make any change to `/sys/bus/pci/devices/0000\:01\:00.0/power/control` value (which keeps being 'on'). I also checked the values in the rule file being right (for vendor and so on)

Removing the rule doesn't make any change.

Another issue is the external display connected through HDMI doesn't receive any signal (but that's minor for now)

Still thanks

EDIT: removing " nvidia nvidia_modeset nvidia_uvm nvidia_drm" from MODULES=(btrfs nvidia nvidia_modeset nvidia_uvm nvidia_drm) in /etc/mkinitcpio.conf made it possible to boot with gpu power in auto mode.
No clue what happened and what's the issue with adding those modules, maybe adding some of them overrides the udev rule?

Last edited by qwerty812 (2023-07-06 16:45:20)

seth · 2023-07-06 19:43:00

No clue what happened and what's the issue with adding those modules, maybe adding some of them overrides the udev rule?

The udev rule didn't make it into the initramfs and wasn't applied when the GPU was bound (ie. the ruled event occurred)
Add it manually to the FILES array.

qwerty812 · 2023-07-06 20:48:53

Ok, that's working now, Thanks a lot!

Now I'm trying to get the external display working through HDMI on wayland (on xorg works).
I read I have to add nvidia-drm.modeset=1 to kernel module lines.
When I set it to 1 I can't get to the login screen and I can only access the terminal with ctrl+alt+f2.
Here's the output from "sudo journalctl -b" when I can't get to the login screen.

Tried scraping through those lines but I can't spot any meaningful errors... Don't know what to do now

seth · 2023-07-06 20:53:48

Don't know what to do now

1. open a new thread (one issue per thread, https://wiki.archlinux.org/title/General_guidelines )
2. post an xorg log there (and name your actual wayland compositor) - but chances are the HDMI output is wired to the nvidia chip and accessed through reverse prime; what also will make it impossible to have it enter D3

qwerty812 · 2023-07-06 20:59:22

Thanks, I'll do it.

My idea was to keep the GPU in D3 state when not in use (when not using prime-run or external HDMI connected) and then power on when needed (of course leaving D3 state)

Still thanks for the troubleshoot

r0ng · 2023-08-03 11:11:44

Same issues here except there are no nvidia modules in my initramfs.

The udev rule also does work as shown as below, but never work on boot.
https://www.reddit.com/r/linuxquestions … t_on_boot/

seth · 2023-08-03 11:46:16

seth wrote:

No clue what happened and what's the issue with adding those modules, maybe adding some of them overrides the udev rule?
The udev rule didn't make it into the initramfs and wasn't applied when the GPU was bound (ie. the ruled event occurred)
Add it manually to the FILES array.

?

r0ng · 2023-08-03 12:10:34

/etc/mkinitcpio.conf

MODULES=(i915)
BINARIES=()
FILES=(/etc/udev/rules.d/80-nvidia-pm.rules)
HOOKS=(base udev autodetect modconf kms keyboard keymap consolefont block filesystems fsck)

Output of lsinitcpio /boot/initramfs-linux.img | grep nvidia

etc/modprobe.d/nvidia.conf
etc/udev/rules.d/80-nvidia-pm.rules
usr/lib/firmware/nvidia/
usr/lib/firmware/nvidia/ga102/
...

seth · 2023-08-03 19:48:11

Boot only the rescue.target to make sure it doesn't get reset by some userspace tool.

The udev rule also does work as shown as below

The reddit post actually only displays the rule, for clarification the explicit "udevadm trigger" applies the rule?

r0ng · 2023-08-04 05:32:26

Thanks you are right, in rescue.target does successfully apply the udev rule,
But after switch back to graphical.target, it fail again, even only in tty
Any idea how to pinpoint such user space tool?

Btw after some investigation, with nvidia in the module of initramfs, the "bind" actions just completely disappear

kernel: pci 0000:01:00.0: [10de:1f97] type 00 class 0x030200
kernel: pci 0000:01:00.0: reg 0x10: [mem 0x81000000-0x81ffffff]
kernel: pci 0000:01:00.0: reg 0x14: [mem 0x6000000000-0x600fffffff 64bit pref]
kernel: pci 0000:01:00.0: reg 0x1c: [mem 0x6010000000-0x6011ffffff 64bit pref]
kernel: pci 0000:01:00.0: reg 0x24: [io  0x3000-0x307f]
kernel: pci 0000:01:00.0: reg 0x30: [mem 0xfff80000-0xffffffff pref]
kernel: pci 0000:01:00.0: Enabling HDA controller
kernel: pci 0000:01:00.0: PME# supported from D0 D3hot D3cold
kernel: pci 0000:01:00.0: 63.012 Gb/s available PCIe bandwidth, limited by 16.0 GT/s PCIe x4 link at 0000:00:06.0 (capable of 252.048 Gb/s with 16.0 GT/s PCIe x16 link)
kernel: pci 0000:01:00.0: can't claim BAR 6 [mem 0xfff80000-0xffffffff pref]: no compatible bridge window
kernel: pci 0000:01:00.0: BAR 6: no space for [mem size 0x00080000 pref]
kernel: pci 0000:01:00.0: BAR 6: failed to assign [mem size 0x00080000 pref]
kernel: nvidia 0000:01:00.0: enabling device (0006 -> 0007)
kernel: nvidia 0000:01:00.0: Enabling HDA controller
kernel: [drm] Initialized nvidia-drm 0.0.0 20160202 for 0000:01:00.0 on minor 0
udevadm[224]: 0000:01:00.0: Triggered device with action 'add'.
systemd-udevd[263]: 0000:01:00.0: Device is queued (SEQNUM=3727, ACTION=add)
systemd-udevd[263]: 0000:01:00.0: SEQNUM=3727 blocked by SEQNUM=3724
systemd-udevd[263]: 0000:01:00.0: Device ready for processing (SEQNUM=3727, ACTION=add)
systemd-udevd[263]: 0000:01:00.0: sd-device-monitor(manager): Passed 354 byte to netlink monitor.
(udev-worker)[316]: 0000:01:00.0: Processing device (SEQNUM=3727, ACTION=add)
(udev-worker)[316]: 0000:01:00.0: /usr/lib/udev/rules.d/50-udev-default.rules:18 Importing properties from results of builtin command 'hwdb --subsystem=pci'
(udev-worker)[316]: 0000:01:00.0: hwdb modalias key: "pci:v000010DEd00001F97sv000017AAsd00003F87bc03sc02i00"
(udev-worker)[316]: 0000:01:00.0: /usr/lib/udev/rules.d/50-udev-default.rules:27 Importing properties from results of builtin command 'path_id'
(udev-worker)[316]: 0000:01:00.0: /usr/lib/udev/rules.d/80-drivers.rules:5 RUN 'kmod load'
(udev-worker)[316]: 0000:01:00.0: sd-device: Created db file '/run/udev/data/+pci:0000:01:00.0' for '/devices/pci0000:00/0000:00:06.0/0000:01:00.0'
(udev-worker)[316]: 0000:01:00.0: Running built-in command "kmod load"
(udev-worker)[316]: 0000:01:00.0: Device processed (SEQNUM=3727, ACTION=add)
(udev-worker)[316]: 0000:01:00.0: sd-device-monitor(worker): Passed 587 byte to netlink monitor.
(udev-worker)[316]: card0: /usr/lib/udev/rules.d/60-drm.rules:6 Added SYMLINK 'dri/by-path/pci-0000:01:00.0-card'
(udev-worker)[316]: card0: Successfully created symlink '/dev/dri/by-path/pci-0000:01:00.0-card' to '/dev/dri/card0'
(udev-worker)[316]: card0: sd-device: Created db file '/run/udev/data/c226:0' for '/devices/pci0000:00/0000:00:06.0/0000:01:00.0/drm/card0'
(udev-worker)[311]: renderD128: /usr/lib/udev/rules.d/60-drm.rules:10 Added SYMLINK 'dri/by-path/pci-0000:01:00.0-render'
(udev-worker)[311]: renderD128: Successfully created symlink '/dev/dri/by-path/pci-0000:01:00.0-render' to '/dev/dri/renderD128'
(udev-worker)[311]: renderD128: sd-device: Created db file '/run/udev/data/c226:128' for '/devices/pci0000:00/0000:00:06.0/0000:01:00.0/drm/renderD128'

Without such modules

kernel: pci 0000:01:00.0: [10de:1f97] type 00 class 0x030200
kernel: pci 0000:01:00.0: reg 0x10: [mem 0x81000000-0x81ffffff]
kernel: pci 0000:01:00.0: reg 0x14: [mem 0x6000000000-0x600fffffff 64bit pref]
kernel: pci 0000:01:00.0: reg 0x1c: [mem 0x6010000000-0x6011ffffff 64bit pref]
kernel: pci 0000:01:00.0: reg 0x24: [io  0x3000-0x307f]
kernel: pci 0000:01:00.0: reg 0x30: [mem 0xfff80000-0xffffffff pref]
kernel: pci 0000:01:00.0: Enabling HDA controller
kernel: pci 0000:01:00.0: PME# supported from D0 D3hot D3cold
kernel: pci 0000:01:00.0: 63.012 Gb/s available PCIe bandwidth, limited by 16.0 GT/s PCIe x4 link at 0000:00:06.0 (capable of 252.048 Gb/s with 16.0 GT/s PCIe x16 link)
kernel: pci 0000:01:00.0: can't claim BAR 6 [mem 0xfff80000-0xffffffff pref]: no compatible bridge window
kernel: pci 0000:01:00.0: BAR 6: no space for [mem size 0x00080000 pref]
kernel: pci 0000:01:00.0: BAR 6: failed to assign [mem size 0x00080000 pref]
udevadm[219]: 0000:01:00.0: Triggered device with action 'add'.
systemd-udevd[254]: 0000:01:00.0: Device is queued (SEQNUM=3659, ACTION=add)
systemd-udevd[254]: 0000:01:00.0: SEQNUM=3659 blocked by SEQNUM=3656
systemd-udevd[254]: 0000:01:00.0: Device ready for processing (SEQNUM=3659, ACTION=add)
systemd-udevd[254]: 0000:01:00.0: sd-device-monitor(manager): Passed 340 byte to netlink monitor.
(udev-worker)[285]: 0000:01:00.0: Processing device (SEQNUM=3659, ACTION=add)
(udev-worker)[285]: 0000:01:00.0: /usr/lib/udev/rules.d/50-udev-default.rules:18 Importing properties from results of builtin command 'hwdb --subsystem=pci'
(udev-worker)[285]: 0000:01:00.0: hwdb modalias key: "pci:v000010DEd00001F97sv000017AAsd00003F87bc03sc02i00"
(udev-worker)[285]: 0000:01:00.0: /usr/lib/udev/rules.d/50-udev-default.rules:27 Importing properties from results of builtin command 'path_id'
(udev-worker)[285]: 0000:01:00.0: /usr/lib/udev/rules.d/80-drivers.rules:5 RUN 'kmod load'
(udev-worker)[285]: 0000:01:00.0: sd-device: Created db file '/run/udev/data/+pci:0000:01:00.0' for '/devices/pci0000:00/0000:00:06.0/0000:01:00.0'
(udev-worker)[285]: 0000:01:00.0: Running built-in command "kmod load"
kernel: nvidia 0000:01:00.0: enabling device (0006 -> 0007)
systemd-udevd[254]: 0000:01:00.0: Device is queued (SEQNUM=4362, ACTION=bind)
systemd-udevd[254]: 0000:01:00.0: SEQNUM=4362 blocked by SEQNUM=3659
kernel: [drm] Initialized nvidia-drm 0.0.0 20160202 for 0000:01:00.0 on minor 1
(udev-worker)[285]: 0000:01:00.0: Device processed (SEQNUM=3659, ACTION=add)
(udev-worker)[285]: 0000:01:00.0: sd-device-monitor(worker): Passed 573 byte to netlink monitor.
systemd-udevd[254]: 0000:01:00.0: Device ready for processing (SEQNUM=4362, ACTION=bind)
systemd-udevd[254]: 0000:01:00.0: sd-device-monitor(manager): Passed 342 byte to netlink monitor.
(udev-worker)[277]: 0000:01:00.0: Processing device (SEQNUM=4362, ACTION=bind)
(udev-worker)[277]: 0000:01:00.0: /usr/lib/udev/rules.d/50-udev-default.rules:18 Importing properties from results of builtin command 'hwdb --subsystem=pci'
(udev-worker)[277]: 0000:01:00.0: hwdb modalias key: "pci:v000010DEd00001F97sv000017AAsd00003F87bc03sc02i00"
(udev-worker)[277]: 0000:01:00.0: /usr/lib/udev/rules.d/50-udev-default.rules:27 Importing properties from results of builtin command 'path_id'
(udev-worker)[277]: 0000:01:00.0: /etc/udev/rules.d/80-nvidia-pm.rules:3 ATTR '/sys/devices/pci0000:00/0000:00:06.0/0000:01:00.0/power/control' writing 'auto'
(udev-worker)[277]: 0000:01:00.0: sd-device: Created db file '/run/udev/data/+pci:0000:01:00.0' for '/devices/pci0000:00/0000:00:06.0/0000:01:00.0'
(udev-worker)[277]: 0000:01:00.0: Device processed (SEQNUM=4362, ACTION=bind)
(udev-worker)[277]: 0000:01:00.0: sd-device-monitor(worker): Passed 575 byte to netlink monitor.
(udev-worker)[284]: card1: /usr/lib/udev/rules.d/60-drm.rules:6 Added SYMLINK 'dri/by-path/pci-0000:01:00.0-card'
(udev-worker)[257]: renderD129: /usr/lib/udev/rules.d/60-drm.rules:10 Added SYMLINK 'dri/by-path/pci-0000:01:00.0-render'
(udev-worker)[284]: card1: Successfully created symlink '/dev/dri/by-path/pci-0000:01:00.0-card' to '/dev/dri/card1'
(udev-worker)[284]: card1: sd-device: Created db file '/run/udev/data/c226:1' for '/devices/pci0000:00/0000:00:06.0/0000:01:00.0/drm/card1'
(udev-worker)[257]: renderD129: Successfully created symlink '/dev/dri/by-path/pci-0000:01:00.0-render' to '/dev/dri/renderD129'
(udev-worker)[257]: renderD129: sd-device: Created db file '/run/udev/data/c226:129' for '/devices/pci0000:00/0000:00:06.0/0000:01:00.0/drm/renderD129'

I also try using changing the udev rule "ACTION=add" but it does not work as well.

seth · 2023-08-04 07:30:35

Any idea how to pinpoint such user space tool?

Please open a new thread, subject it "nvidia enters D0 after boot" or sth. like that and post a complete system journal.
Pretty much everything that talks to the nvidia chip is suspect, tlp, bbswitch, acpi_call, laptop_mode_tools, nvidia-smi, …

r0ng · 2024-04-30 12:58:00

For reference I found tlp is the reason and disable it solve the problem

Arch Linux

#1 2023-07-05 18:32:48

Unable to set NVIDIA Card to D3 state on hybrid laptop

#2 2023-07-05 20:43:30

Re: Unable to set NVIDIA Card to D3 state on hybrid laptop

#3 2023-07-06 12:41:06

Re: Unable to set NVIDIA Card to D3 state on hybrid laptop

#4 2023-07-06 14:53:37

Re: Unable to set NVIDIA Card to D3 state on hybrid laptop

#5 2023-07-06 16:29:45

Re: Unable to set NVIDIA Card to D3 state on hybrid laptop

#6 2023-07-06 19:43:00

Re: Unable to set NVIDIA Card to D3 state on hybrid laptop

#7 2023-07-06 20:48:53

Re: Unable to set NVIDIA Card to D3 state on hybrid laptop

#8 2023-07-06 20:53:48

Re: Unable to set NVIDIA Card to D3 state on hybrid laptop

#9 2023-07-06 20:59:22

Re: Unable to set NVIDIA Card to D3 state on hybrid laptop

#10 2023-08-03 11:11:44

Re: Unable to set NVIDIA Card to D3 state on hybrid laptop

#11 2023-08-03 11:46:16

Re: Unable to set NVIDIA Card to D3 state on hybrid laptop

#12 2023-08-03 12:10:34

Re: Unable to set NVIDIA Card to D3 state on hybrid laptop

#13 2023-08-03 19:48:11

Re: Unable to set NVIDIA Card to D3 state on hybrid laptop

#14 2023-08-04 05:32:26

Re: Unable to set NVIDIA Card to D3 state on hybrid laptop

#15 2023-08-04 07:30:35

Re: Unable to set NVIDIA Card to D3 state on hybrid laptop

#16 2024-04-30 12:58:00

Re: Unable to set NVIDIA Card to D3 state on hybrid laptop

Board footer