Power consumption differences Arch vs OpenWrt

graysky · 2023-07-06 20:43:38

If I boot into Arch on a mini PC (AMD Ryzen 7) power consumption while idle is 5 watts (linux-lts currently 6.1.38). If I boot the same box into OpenWrt running the same 6.1.38 kernel, idle power consumption is 9 watts. I post this not asking for OpenWrt support, but to seek some thoughts around which kernel settings I might key on that could be driving the lower power consumption in Arch with the goal of using them on the OpenWrt image in an attempt to lower the power consumption there.

Both systems are using the schedutil governor and amd_pstate=passive.

There did notice some differences in the two kernel's tick settings but I already switched Arch's options over to OpenWrt's and found no change (still 9 watts).

For reference, here is Arch's lts config and here is my Openwrt config. Thanks for the ideas.

Trilby · 2023-07-06 20:49:25

Are you just assuming this is due to kernel differences? The two systems use different inits and process supervisors, different user space tools, and different default configurations of which processes start at boot.

graysky · 2023-07-06 20:52:00

You are right, Trilby. Plenty of other differences as well.

Trilby · 2023-07-06 20:53:25

You could test - just boot the arch system with the openwrt kernel (or vice versa). Then you'll know whether adjust kernel configs is even the right place to put your efforts.

Last edited by Trilby (2023-07-06 20:53:49)

graysky · 2023-07-06 21:06:59

I might not be doing it correctly. Is the kernel just /boot/vmlinuz-linux-lts? I adjusted OpenWrt's entry to use that rather than it's native one but nothing happened upon attempting to boot. Just a black screen/dead keyboard.

The original line is commented out.

EDIT: perhaps that file is compressed and that is the issue (I believe Arch is using zstd). Need to check to see what OpenWrt is using.

serial --unit=0 --speed=115200 --word=8 --parity=no --stop=1 --rtscts=off
terminal_input console serial; terminal_output console serial

set default="0"
set timeout="1"
search -l kernel -s root

menuentry "OpenWrt passive" {
  #linux /boot/vmlinuz root=PARTUUID=7fe1d9bc-6bf4-44dc-b869-b213c3af5102 rootwait amd-pstate=passive console=tty0 console=ttyS0,115200n8 noinitrd
  linux /boot/vmlinuz-linux-lts root=PARTUUID=7fe1d9bc-6bf4-44dc-b869-b213c3af5102 rootwait amd-pstate=passive console=tty0 console=ttyS0,115200n8 noinitrd
}

Last edited by graysky (2023-07-06 21:10:01)

Trilby · 2023-07-07 01:25:56

I focused on booting arch with the openwrt kernel as that has a better chance to work "out of the box". There are a handful of deal-breakers that might prevent such a kernel swap from working seemlessly ... especially if - as suggested by that exerpt - one or both of the OSs runs without an initrd.

graysky · 2023-07-07 06:33:49

Good idea, it partially boots but just seems to hang (no console). I see what might be a show stopper in the output:

...
Failed to mount devtmpfs at /dev/: No such device
Freezing execution.

graysky · 2023-07-07 06:51:03

After a little trial and error, I built into the OpenWrt kernel:

CONFIG_KERNEL_DEVTMPFS=y
CONFIG_KERNEL_DEVTMPFS_MOUNT=y
CONFIG_KERNEL_FS_POSIX_ACL=y

That did boot Arch and the idle power consumption on Arch with that kernel was 9 watts matching OpenWrt and I think pointing to something in the kernel...

Last edited by graysky (2023-07-07 06:55:21)

Trilby · 2023-07-07 12:58:45

Agreed - now you know it's the kernel. You can start testing config variants. There are a 7522 differences between the configs which means a fully blindly naive approach could take up to a maximum of 87 rebuilds to identify relevant settings but a average / predicted number of builts about half that (assuming complete independence of settings). However starting with a minimally educated guess, there are just 33 config settings that differ and include the word "POWER":

CONFIG_AXP20X_POWER
CONFIG_CPU_FREQ_GOV_POWERSAVE
CONFIG_DEVFREQ_GOV_POWERSAVE
CONFIG_DPTF_POWER
CONFIG_HID_LCPOWER
CONFIG_INPUT_POWERMATE
CONFIG_INTEL_POWERCLAMP
CONFIG_IP5XXX_POWER
CONFIG_IPMI_POWEROFF
CONFIG_MAX8925_POWER
CONFIG_PDA_POWER
CONFIG_PERF_EVENTS_AMD_POWER
CONFIG_POWERCAP
CONFIG_POWER_RESET
CONFIG_POWER_RESET_ATC260X
CONFIG_POWER_RESET_MT6323
CONFIG_POWER_RESET_RESTART
CONFIG_POWER_RESET_TPS65086
CONFIG_POWER_SUPPLY_HWMON
CONFIG_SENSORS_ACPI_POWER
CONFIG_SENSORS_FAM15H_POWER
CONFIG_SND_AC97_POWER_SAVE
CONFIG_SND_AC97_POWER_SAVE_DEFAULT
CONFIG_SND_HDA_POWER_SAVE_DEFAULT
CONFIG_SURFACE_3_POWER_OPREGION
CONFIG_TEST_POWER
CONFIG_THERMAL_EMERGENCY_POWEROFF_DELAY_MS
CONFIG_THERMAL_GOV_POWER_ALLOCATOR
CONFIG_WM831X_POWER
CONFIG_WM8350_POWER
CONFIG_X86_POWERNOW_K8
CONFIG_XPOWER_PMIC_OPREGION
CONFIG_XZ_DEC_POWERPC

I'd suspect that any difference would be captured in this set. So test setting all of these in the openwrt config to match the arch config. If that doesn't reduce the power usage, then were back to finding a next semi-educated guess, but if it does reduce power usage, you may have a sufficient solution, or - at worst - you'd need to do 6 more rounds of compiling to pinpoint exactly which of these 33 settings are (most) important.

graysky · 2023-07-07 15:03:56

Thanks Trilby ... just curiously, how did you compute the diffs and pick out those containing POWER and strip off all the extraneous text?

Trilby · 2023-07-07 15:14:04

Any text processing tool would do. I initially just grepped to config files to get actual settings (`grep '^C' <file>`). I also sorted this grep output for good measure as I knew I was going to use `comm` on the results which I did to get what was different between the two configs. This (or a `wc -l` of this) gave an estimate of how much work a totally naive approach would take and it seemed unwise. So then from that list I just grepped for POWER, and this gave a much more manageable set of options to work with as a first shot.

As an aside, the set of options uniq to openwrt is quite manageable (n=222). However, the relevant settings are likely in the arch kernel as that's where you are claiming less power is used, right? There are n=7300 settings unique to the arch config, and just splitting/checking all of these could take up to 86 rebuilds (ceil(sqrt(7300)) to narrow down the important candidates ... which seems unreasonable to me.

If the set of options including POWER turn out not to be relevant, my next semi-educated-stab-in-the-dark would be these:

$ grep FREQ arch.uniq
CONFIG_CPU_FREQ_GOV_CONSERVATIVE=y
CONFIG_CPU_FREQ_GOV_POWERSAVE=y
CONFIG_CPU_FREQ_GOV_USERSPACE=y
CONFIG_DEVFREQ_GOV_PASSIVE=m
CONFIG_DEVFREQ_GOV_PERFORMANCE=m
CONFIG_DEVFREQ_GOV_POWERSAVE=m
CONFIG_DEVFREQ_GOV_SIMPLE_ONDEMAND=m
CONFIG_DEVFREQ_GOV_USERSPACE=m
CONFIG_DEVFREQ_THERMAL=y
CONFIG_INTEL_UNCORE_FREQ_CONTROL=m
CONFIG_PM_DEVFREQ=y
CONFIG_PM_DEVFREQ_EVENT=y
CONFIG_X86_ACPI_CPUFREQ=m
CONFIG_X86_ACPI_CPUFREQ_CPB=y
CONFIG_X86_AMD_FREQ_SENSITIVITY=m
CONFIG_X86_PCC_CPUFREQ=m

Though you can drop the next two from that list as they are set in both configs, just as modules for arch and builtin for openwrt:

$ grep FREQ openwrt.uniq 
CONFIG_X86_ACPI_CPUFREQ=y
CONFIG_X86_AMD_FREQ_SENSITIVITY=y

Last edited by Trilby (2023-07-07 18:39:44)

graysky · 2023-07-07 23:15:54

Thanks for all your suggestions, Trilby. In building kernels and testing, I did am important control experiment, I disabled some of the interfaces I setup on OpenWrt which were not present on Arch. One of my interfaces is for my iot devices, many of which are cameras. They flow about 15 GB/hour to another interface which hosts a camera monitoring box. So that's 30 GB/hour of traffic. If I stop the camera monitoring software, the CPU usage under OpenWrt dropped to 5 watts, matching Arch's.

I cannot explain why booting Arch with OpenWrt's kernel raised the power consumption in Arch though...

Last edited by graysky (2023-07-08 11:23:22)

seth · 2023-07-08 06:14:18

Maybe powertop can help to narrow down the consumer (and thus relevant config) - how do you measure the power draw atm?
Let's say your ethernet chip has a power saving mode.
Let's also say that it cannot enter it w/ vlans present.
4W for one ethernet chip seems excessive.
Do they have impact on the cpu frequency?
https://wiki.archlinux.org/title/CPU_fr … _governors

watch cat /sys/devices/system/cpu/cpu[0-9]*/cpufreq/scaling_cur_freq

graysky · 2023-07-08 11:21:27

I ran powertop and switched on all it's suggestions with no change in the power draw. Measuring with a kill a watt meter.

The VLAN statement I made above (now edited) was incorrect. Higher power consumption was due to security cameras on one interface flowing to another interface. Stopping the monitoring box dropped power consumption.

Arch Linux

#1 2023-07-06 20:43:38

Power consumption differences Arch vs OpenWrt

#2 2023-07-06 20:49:25

Re: Power consumption differences Arch vs OpenWrt

#3 2023-07-06 20:52:00

Re: Power consumption differences Arch vs OpenWrt

#4 2023-07-06 20:53:25

Re: Power consumption differences Arch vs OpenWrt

#5 2023-07-06 21:06:59

Re: Power consumption differences Arch vs OpenWrt

#6 2023-07-07 01:25:56

Re: Power consumption differences Arch vs OpenWrt

#7 2023-07-07 06:33:49

Re: Power consumption differences Arch vs OpenWrt

#8 2023-07-07 06:51:03

Re: Power consumption differences Arch vs OpenWrt

#9 2023-07-07 12:58:45

Re: Power consumption differences Arch vs OpenWrt

#10 2023-07-07 15:03:56

Re: Power consumption differences Arch vs OpenWrt

#11 2023-07-07 15:14:04

Re: Power consumption differences Arch vs OpenWrt

#12 2023-07-07 23:15:54

Re: Power consumption differences Arch vs OpenWrt

#13 2023-07-08 06:14:18

Re: Power consumption differences Arch vs OpenWrt

#14 2023-07-08 11:21:27

Re: Power consumption differences Arch vs OpenWrt

Board footer