Arch Linux

TE

Member

Registered: 2014-06-21

Posts: 78

[SOLVED] Kernel 4.8.7-1 intel_pstate powersave cpufreq slowdown

I'm still debugging but throwing this out there to see if anyone else is noticing -- I upgraded my laptop here (Dell M4600) this morning from kernel 4.8.6 to 4.8.7, rebooted and immediately noticed everything was "slow" (loading Firefox/chromium). My mission this morning was to compile a bunch of c++ code with gcc and clang and it was crawling.

I found this: https://git.kernel.org/cgit/linux/kerne … id2=v4.8.6

...so they definitely changed things. I don't normally mess with any of this (run at pure defaults), so I installed cpupower to easily set the governor to performance and it's "back to normal". This laptop is more like a desktop and stays plugged in to A/C all the time, but does hibernate on inactivity. Here's what my CPU looks like while it was struggling to compile (the lspcu was run over and over during my compiles, it was always ~650 ish!).

$ lscpu | grep MHz
CPU MHz:               627.685
CPU max MHz:           3500.0000
CPU min MHz:           800.0000

$ lscpu
Architecture:          x86_64
CPU op-mode(s):        32-bit, 64-bit
Byte Order:            Little Endian
CPU(s):                8
On-line CPU(s) list:   0-7
Thread(s) per core:    2
Core(s) per socket:    4
Socket(s):             1
NUMA node(s):          1
Vendor ID:             GenuineIntel
CPU family:            6
Model:                 42
Model name:            Intel(R) Core(TM) i7-2760QM CPU @ 2.40GHz
Stepping:              7
CPU MHz:               625.195
CPU max MHz:           3500.0000
CPU min MHz:           800.0000
BogoMIPS:              4791.68
Virtualization:        VT-x
L1d cache:             32K
L1i cache:             32K
L2 cache:              256K
L3 cache:              6144K
NUMA node0 CPU(s):     0-7
Flags:                 fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx rdtscp lm constant_tsc arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc aperfmperf eagerfpu pni pclmulqdq dtes64 monitor ds_cpl vmx smx est tm2 ssse3 cx16 xtpr pdcm pcid sse4_1 sse4_2 x2apic popcnt tsc_deadline_timer aes xsave avx lahf_lm epb tpr_shadow vnmi flexpriority ept vpid xsaveopt dtherm ida arat pln pts

$ cat /sys/devices/system/cpu/cpu*/cpufreq/scaling_governor
powersave
powersave
powersave
powersave
powersave
powersave
powersave
powersave

Holler at me if you've already gone down this road and figured out what broke, I can't downgrade/reboot to test 4.8.6 again for awhile so just going to run in performance mode to get my work done.

Last edited by TE (2017-03-24 13:50:38)

graysky

Wiki Maintainer

From: :wq

Registered: 2008-12-01

Posts: 10,597

Website

Re: [SOLVED] Kernel 4.8.7-1 intel_pstate powersave cpufreq slowdown

Haven't noticed a difference but difference hardware:

% lscpu | grep MHz
CPU MHz:               4399.902
CPU max MHz:           4400.0000
CPU min MHz:           800.0000

% lscpu
Architecture:          x86_64
CPU op-mode(s):        32-bit, 64-bit
Byte Order:            Little Endian
CPU(s):                8
On-line CPU(s) list:   0-7
Thread(s) per core:    2
Core(s) per socket:    4
Socket(s):             1
Vendor ID:             GenuineIntel
CPU family:            6
Model:                 60
Model name:            Intel(R) Core(TM) i7-4790K CPU @ 4.00GHz
Stepping:              3
CPU MHz:               4399.902
CPU max MHz:           4400.0000
CPU min MHz:           800.0000
BogoMIPS:              7999.96
Virtualization:        VT-x
L1d cache:             32K
L1i cache:             32K
L2 cache:              256K
L3 cache:              8192K
Flags:                 fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx pdpe1gb rdtscp lm constant_tsc arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc aperfmperf eagerfpu pni pclmulqdq dtes64 monitor ds_cpl vmx est tm2 ssse3 sdbg fma cx16 xtpr pdcm pcid sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand lahf_lm abm tpr_shadow vnmi flexpriority ept vpid fsgsbase tsc_adjust bmi1 avx2 smep bmi2 erms invpcid xsaveopt dtherm ida arat pln pts

% cat /sys/devices/system/cpu/cpu*/cpufreq/scaling_governor
powersave
powersave
powersave
powersave
powersave
powersave
powersave
powersave

---

CPU-optimized Linux-ck packages @ Repo-ck  • AUR packages • Zsh and other configs

graysky

Wiki Maintainer

From: :wq

Registered: 2008-12-01

Posts: 10,597

Website

Re: [SOLVED] Kernel 4.8.7-1 intel_pstate powersave cpufreq slowdown

@OP - Here is a simple script that will monitor your frequency and plot some statistics.  Call it like so:

% stats.sh 30          
Collecting data for 30 seconds...

All values in MHz except count which is just the number of measurements...
median : 2745.65
mean   : 2906.65
max    : 4400.1
min    : 856.6
count  : 30

---

CPU-optimized Linux-ck packages @ Repo-ck  • AUR packages • Zsh and other configs

TE

Member

Registered: 2014-06-21

Posts: 78

Re: [SOLVED] Kernel 4.8.7-1 intel_pstate powersave cpufreq slowdown

Cool thanks - I'll be able to dig in a bit later, Saturday stuff right now (just mowed the lawn) and see how it goes. I just woke this laptop up from suspend and it's still set to performance, so at least that's working as expected (it didn't revert to powersave). I plan on testing first with intel_pstate=disable and see what the older ACPI ondemand thinks, then downgrade and test it all again. I dread having to recompile the kernel with that commit removed, but it's probably one of the steps I'll take if it "seems like" this is it. Much appreciated.

R00KIE

Forum Fellow

From: Between a computer and a chair

Registered: 2008-09-14

Posts: 4,734

Re: [SOLVED] Kernel 4.8.7-1 intel_pstate powersave cpufreq slowdown

I haven't noticed any difference either, but then again I'm also on different hardware.

> lscpu
Architecture:          x86_64
CPU op-mode(s):        32-bit, 64-bit
Byte Order:            Little Endian
CPU(s):                4
On-line CPU(s) list:   0-3
Thread(s) per core:    2
Core(s) per socket:    2
Socket(s):             1
NUMA node(s):          1
Vendor ID:             GenuineIntel
CPU family:            6
Model:                 78
Model name:            Intel(R) Core(TM) i7-6500U CPU @ 2.50GHz
Stepping:              3
CPU MHz:               799.963
CPU max MHz:           3100.0000
CPU min MHz:           400.0000
BogoMIPS:              5186.00
Virtualization:        VT-x
L1d cache:             32K
L1i cache:             32K
L2 cache:              256K
L3 cache:              4096K
NUMA node0 CPU(s):     0-3
Flags:                 fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx pdpe1gb rdtscp lm constant_tsc art arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc aperfmperf eagerfpu pni pclmulqdq dtes64 monitor ds_cpl vmx est tm2 ssse3 sdbg fma cx16 xtpr pdcm pcid sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand lahf_lm abm 3dnowprefetch epb intel_pt tpr_shadow vnmi flexpriority ept vpid fsgsbase tsc_adjust bmi1 avx2 smep bmi2 erms invpcid mpx rdseed adx smap clflushopt xsaveopt xsavec xgetbv1 xsaves dtherm ida arat pln pts hwp hwp_notify hwp_act_window hwp_epp

> cat /sys/devices/system/cpu/cpu*/cpufreq/scaling_governor
powersave
powersave
powersave
powersave

---

R00KIE
Tm90aGluZyB0byBzZWUgaGVyZSwgbW92ZSBhbG9uZy4K

TE

Member

Registered: 2014-06-21

Posts: 78

Re: [SOLVED] Kernel 4.8.7-1 intel_pstate powersave cpufreq slowdown

Well it's something with intel_pstate for sure - I don't want to say it's exactly that commit (have not specifically tested), but I used your stats script slightly modified to trap a ctrl+c and ran it under different settings. The test: reboot to X setup, start Firefox once (my laptop drive is slow) and cancel the Lastpass login dialog, exit. Then start the stats gathering script and redo the same start/exit Firefox test.

$ cat data.txt 
4.8.7-1 default (powersave)
median : 638.2
mean   : 643.146
max    : 705.1
min    : 627.2
count  : 39

4.8.7-1 intel_pstate=disable
median : 800
mean   : 983.333
max    : 2200.0
min    : 800.0
count  : 12

4.8.7-1 performance
median : 2135.4
mean   : 2000.51
max    : 2599.9
min    : 1100.0
count  : 12

Strangely, I downgraded real quick to 4.8.6-1 and it seemed just as bad, but not sure how or why as it was fine before? It was only a quick test as I need to go do other things right now - but if I just set the governor to performance (via cpupower config/service) it brings me back to expected performance again on 4.8.7-1. I'll keep troubleshooting it later when I get time.. (or not if I don't and just leave cpupower active). *shrug*

graysky

Wiki Maintainer

From: :wq

Registered: 2008-12-01

Posts: 10,597

Website

Re: [SOLVED] Kernel 4.8.7-1 intel_pstate powersave cpufreq slowdown

@TE - I would caution against using only 12 measurements... seems like a larger sample set would be safer, no?

---

CPU-optimized Linux-ck packages @ Repo-ck  • AUR packages • Zsh and other configs

TE

Member

Registered: 2014-06-21

Posts: 78

Re: [SOLVED] Kernel 4.8.7-1 intel_pstate powersave cpufreq slowdown

The slowdown is *immediately* noticeable - I've been using this laptop for a long time and with the same desktop (MATE) and rarely change anything, so I know it's "feel" if you will. I can tell as soon as I get the lightdm dialog and just get my desktop it's super slow, and watching the lag as Firefox starts is quite easy to spot. Note that that first test of 39 counts is telltale -- I literally start gathering data, do the firefox test, and stop gathering data. So it takes 39 cycles to do the same test as it takes 12 cycles (give or take a cycle or two for my human slowness in hitting ctrl+c) when at the intel_pstate default of powersave. $0.02.

Edit: Oh sorry - I failed to mention that I start the test with 'bash stats.sh 120' and hit ctrl+c to cancel it. Here is the modified version if you'd like to save it, thanks for sharing. https://gist.github.com/troyengel/01609 … 97d5a9e173

Last edited by TE (2016-11-12 20:54:19)

graysky

Wiki Maintainer

From: :wq

Registered: 2008-12-01

Posts: 10,597

Website

Re: [SOLVED] Kernel 4.8.7-1 intel_pstate powersave cpufreq slowdown

@TE - I just pushed a commit that allows gnuplot to make a nice histogram with your dataset:

EDIT: I also took your ctrl+c trap (nice by the way).

Last edited by graysky (2016-11-12 21:26:11)

---

CPU-optimized Linux-ck packages @ Repo-ck  • AUR packages • Zsh and other configs

TE

Member

Registered: 2014-06-21

Posts: 78

Re: [SOLVED] Kernel 4.8.7-1 intel_pstate powersave cpufreq slowdown

Interesting data point for the morning - I tried re-setting the governor to powersave *after* setting it to performance... and it works! So there's a new avenue to test when I get a chance to reboot - do the same test as above, then back to back I'll run the 'cpupower frequency-set -g' to performance then powersave. Perhaps it's only during boot/init that it's failing...

$ bash stats.sh 600
Collecting data for 600 seconds...

All values in MHz except count which is just the number of measurements...
median : 1094.65
mean   : 1318.73
max    : 3372.0
min    : 799.9
count  : 600

That's more like normal, so lis definitely another clue.

graysky

Wiki Maintainer

From: :wq

Registered: 2008-12-01

Posts: 10,597

Website

Re: [SOLVED] Kernel 4.8.7-1 intel_pstate powersave cpufreq slowdown

If interested, I created a PKGBUILD for a pretty cool python script (data_hacks) that makes ASCII historgrams. Example on my dataset:

cat /run/user/1000/stats-20161113-113230|histogram.py -p
# NumSamples = 120; Min = 799.80; Max = 4401.10
# Mean = 3197.669167; Variance = 1911154.428466; SD = 1382.445091; Median 4137.300000
# each ∎ represents a count of 1
  799.8000 -  1159.9300 [    18]: ∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎ (15.00%)
 1159.9300 -  1520.0600 [     7]: ∎∎∎∎∎∎∎ (5.83%)
 1520.0600 -  1880.1900 [     7]: ∎∎∎∎∎∎∎ (5.83%)
 1880.1900 -  2240.3200 [     4]: ∎∎∎∎ (3.33%)
 2240.3200 -  2600.4500 [     5]: ∎∎∎∎∎ (4.17%)
 2600.4500 -  2960.5800 [     8]: ∎∎∎∎∎∎∎∎ (6.67%)
 2960.5800 -  3320.7100 [     5]: ∎∎∎∎∎ (4.17%)
 3320.7100 -  3680.8400 [     1]: ∎ (0.83%)
 3680.8400 -  4040.9700 [     3]: ∎∎∎ (2.50%)
 4040.9700 -  4401.1000 [    62]: ∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎ (51.67%)

Here is the script that uses histogram.py from that AUR package: https://github.com/graysky2/bin/blob/ma … stogram.sh

Last edited by graysky (2016-11-13 16:59:51)

---

CPU-optimized Linux-ck packages @ Repo-ck  • AUR packages • Zsh and other configs

newbie1962

Member

From: italy

Registered: 2012-07-24

Posts: 137

Re: [SOLVED] Kernel 4.8.7-1 intel_pstate powersave cpufreq slowdown

hi,
thank for the script,i have this error

me ~ $ cat /run/user/1000/stats-20161113-113230|histogram.py -p
cat: /run/user/1000/stats-20161113-113230: File o directory non esistente
Traceback (most recent call last):
  File "/usr/bin/histogram.py", line 250, in <module>
    histogram(load_stream(sys.stdin, options.agg_value_key, options.agg_key_value), options)
  File "/usr/bin/histogram.py", line 134, in histogram
    min_v = min(data, key=lambda x: x.value)
ValueError: min() arg is an empty se

but if I start your terminal script to me from this:

me ~ $ '/home/me/cpufreq_histogram.sh' 120
Collecting data for 120 seconds...
# NumSamples = 120; Min = 1199.70; Max = 3301.40
# Mean = 1440.380000; Variance = 160066.251600; SD = 400.082806; Median 1300.100000
# each ∎ represents a count of 1
 1199.7000 -  1409.8700 [    87]: ∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎ (72.50%)
 1409.8700 -  1620.0400 [    13]: ∎∎∎∎∎∎∎∎∎∎∎∎∎ (10.83%)
 1620.0400 -  1830.2100 [     9]: ∎∎∎∎∎∎∎∎∎ (7.50%)
 1830.2100 -  2040.3800 [     3]: ∎∎∎ (2.50%)
 2040.3800 -  2250.5500 [     1]: ∎ (0.83%)
 2250.5500 -  2460.7200 [     3]: ∎∎∎ (2.50%)
 2460.7200 -  2670.8900 [     0]:  (0.00%)
 2670.8900 -  2881.0600 [     1]: ∎ (0.83%)
 2881.0600 -  3091.2300 [     0]:  (0.00%)
 3091.2300 -  3301.4000 [     3]: ∎∎∎ (2.50%)
me ~ $ 

where am I wrong.
thank

Last edited by newbie1962 (2016-11-14 11:08:44)

---

hp-envy dv7

graysky

Wiki Maintainer

From: :wq

Registered: 2008-12-01

Posts: 10,597

Website

Re: [SOLVED] Kernel 4.8.7-1 intel_pstate powersave cpufreq slowdown

@OP - I have noticed a difference with 4.9-rc5 vs 4.8.7... under 4.9-rc5 the median is 2.6 GHz compared to 4.1 GHz.

% cpufreq_histogram.sh 4.9-rc5 
# NumSamples = 120; Min = 799.80; Max = 4400.60
# Mean = 2677.014167; Variance = 1759971.617383; SD = 1326.639219; Median 2597.350000
# each ∎ represents a count of 1
  799.8000 -  1159.8800 [    26]: ∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎ (21.67%)
 1159.8800 -  1519.9600 [     7]: ∎∎∎∎∎∎∎ (5.83%)
 1519.9600 -  1880.0400 [     6]: ∎∎∎∎∎∎ (5.00%)
 1880.0400 -  2240.1200 [     7]: ∎∎∎∎∎∎∎ (5.83%)
 2240.1200 -  2600.2000 [    16]: ∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎ (13.33%)
 2600.2000 -  2960.2800 [    11]: ∎∎∎∎∎∎∎∎∎∎∎ (9.17%)
 2960.2800 -  3320.3600 [     2]: ∎∎ (1.67%)
 3320.3600 -  3680.4400 [     6]: ∎∎∎∎∎∎ (5.00%)
 3680.4400 -  4040.5200 [     6]: ∎∎∎∎∎∎ (5.00%)
 4040.5200 -  4400.6000 [    33]: ∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎ (27.50%)

% cpufreq_histogram.sh 4.8.7                               
# NumSamples = 120; Min = 799.80; Max = 4401.10
# Mean = 3197.669167; Variance = 1911154.428466; SD = 1382.445091; Median 4137.300000
# each ∎ represents a count of 1
  799.8000 -  1159.9300 [    18]: ∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎ (15.00%)
 1159.9300 -  1520.0600 [     7]: ∎∎∎∎∎∎∎ (5.83%)
 1520.0600 -  1880.1900 [     7]: ∎∎∎∎∎∎∎ (5.83%)
 1880.1900 -  2240.3200 [     4]: ∎∎∎∎ (3.33%)
 2240.3200 -  2600.4500 [     5]: ∎∎∎∎∎ (4.17%)
 2600.4500 -  2960.5800 [     8]: ∎∎∎∎∎∎∎∎ (6.67%)
 2960.5800 -  3320.7100 [     5]: ∎∎∎∎∎ (4.17%)
 3320.7100 -  3680.8400 [     1]: ∎ (0.83%)
 3680.8400 -  4040.9700 [     3]: ∎∎∎ (2.50%)
 4040.9700 -  4401.1000 [    62]: ∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎∎ (51.67%)

Last edited by graysky (2016-11-14 22:20:19)

---

CPU-optimized Linux-ck packages @ Repo-ck  • AUR packages • Zsh and other configs

TE

Member

Registered: 2014-06-21

Posts: 78

Re: [SOLVED] Kernel 4.8.7-1 intel_pstate powersave cpufreq slowdown

(just getting back to this) It has to be something with this CPU or backplane - two other machines running the identical setup do not exhibit the problem, and when I simply run the cpupower to toggle it things are "back to normal" and it's operationally correct. It's possible this has always been an issue on this gear and maybe I just never noticed? Or maybe "some further action" after it's running triggers it to be fixed, perhaps suspending and waking up? (that would be pretty logical)

So, not really going to worry about it - on powersave governor it ramps up correctly, here's a snapshot where I was just preparing the new google-cloud-sdk package and it was finishing the compile and (rather hefty) compression xzball:

$ bash ./stats.sh 600
Collecting data for 600 seconds...
^C
All values in MHz except count which is just the number of measurements...
median : 3375.55
mean   : 3289.44
max    : 3495.7
min    : 972.3
count  : 68

That works for me, it ramps up and gives me all the MHz when demanded. (I'm on 4.8.8-2 now, and I see 4.8.9 was released last night upstream)

TE

Member

Registered: 2014-06-21

Posts: 78

Re: [SOLVED] Kernel 4.8.7-1 intel_pstate powersave cpufreq slowdown

A final update to this, one of these various commits since last update fixed the problem for me, I honestly don't know which one it is. I upgraded the the 4.10.5 just released and started poking around with cpupower a bit, it looks "back to normal" on a fresh boot. Hurray for bug fixes.

https://git.kernel.org/pub/scm/linux/ke … l_pstate.c

I see at least 5-6 commits since last November that look interesting/relevant, but not really going to bisect my way down further - just happy it's working again. It was probably fixed a few kernel releases ago and I just hadn't noticed.