You are not logged in.

#1 2011-01-03 00:16:47

Telkkar
Member
From: United States
Registered: 2010-12-26
Posts: 38

High load averages and interrupts with 2.6.36/2.6.37

Since I've installed Arch (a week or two ago) on my Macbook Pro, I've had high load averages -- often 0.50+ while idling. Completely idle - just having a terminal open and watching top for 5-10 minutes. Needless to say, I searched the forums and came around to this: https://bbs.archlinux.org/viewtopic.php?id=103346.

That's great and all, but I'm running 2.6.36, not 35.

Then I found another problem intertwined with the high load averages -- kworker and [kernel scheduler] load balancing tick.

Those two things (processes?) are the cause for 87-92% of my wakeups, even while sitting idle watching powertop. Again, I searched the forums, and I came up with this: https://bbs.archlinux.org/viewtopic.php?id=109371  . Similar to my issue, but my CPU load isn't that high -- only 2-3% here on each core.

From the first link -- I tried nohz=off and highres=off. It fixed the load averages and wakeups, but my power usage shot through the roof (someone mentioned in that thread that it was a horrible option for laptops).

Then I tried compiling kernel26-git from the AUR, which is 2.6.37-rc8 (or 2.6.37-rc8-git1, not sure). That kernel fixed my load averages -- at least dropping them to sub 0.10 levels -- but the wakeup issues still remain. Only, instead of just kworker and [kernel scheduler] Load balancing tick, I have [extra timer interrupt] as well.

So, I'm all out of ideas -- I've not dealt much with the kernel before Arch, so I tried what I thought of. I thought of compiling 2.6.35.9 and seeing how that worked, but I wanted to get some input before I spent another hour or two compiling.

Thanks in advance!

Offline

#2 2011-01-04 21:32:32

pogeymanz
Member
Registered: 2008-03-11
Posts: 1,020

Re: High load averages and interrupts with 2.6.36/2.6.37

Unfortunately, I've been trying to wait patiently for a solution to this as well. It's all over the web and affects every distro so I'm hoping that the kernel guys are planning a fix soon.

I guess I'll have to see if I can find the ck patches for 2.6.35...

Offline

#3 2011-01-04 21:46:35

karol
Archivist
Registered: 2009-05-06
Posts: 25,440

Re: High load averages and interrupts with 2.6.36/2.6.37

I'm using the stock kernel26 2.6.36.2-1 on my desktop and my load avg are OK. They were pretty bad with .35 but got well in .36.

Looking at my powertop:

Wakeups-from-idle per second : 373,2    interval: 10,0s

Top causes for wakeups:
  96,4% (300,3)   [kernel module] hrtimer_start_range_ns (tick_sched_timer)

Offline

#4 2011-01-04 22:34:00

Telkkar
Member
From: United States
Registered: 2010-12-26
Posts: 38

Re: High load averages and interrupts with 2.6.36/2.6.37

pogeymanz wrote:

Unfortunately, I've been trying to wait patiently for a solution to this as well. It's all over the web and affects every distro so I'm hoping that the kernel guys are planning a fix soon.

I guess I'll have to see if I can find the ck patches for 2.6.35...

Yeah, I'm really hoping that there's a fix as well. I'd really like to be able to unplug my laptop for more than 2-3 hours at a time; the battery is rated for 6-7 hours.

I'm using the stock kernel26 2.6.36.2-1 on my desktop and my load avg are OK. They were pretty bad with .35 but got well in .36.
Looking at my powertop: <snip>

That's interesting. My load averages are lower than the ones in the thread about 2.6.35 that I linked, but they are still way higher than my 2.6.37-rc8-git1 compile when idle.

-----------------------

The latest snapshot is 2.6.37-rc8-git5, so I think I may try that and see if anything changes.

Last edited by Telkkar (2011-01-04 22:34:19)

Offline

#5 2011-01-05 22:30:07

Telkkar
Member
From: United States
Registered: 2010-12-26
Posts: 38

Re: High load averages and interrupts with 2.6.36/2.6.37

Last night, I had a couple errors compiling the latest and greatest kernel, so I stopped for the night. As that turns out, it was a good thing...

2.6.37 was released today, so I'm expecting it to roll into arch in a couple days, hopefully that will fix something. At least if it doesn't, my broadcom drivers are now in the kernel smile

Anyway, in the meantime, I'm going to test the -git5 kernel. (At least that's what I think it is)

Offline

#6 2011-01-08 19:04:10

Telkkar
Member
From: United States
Registered: 2010-12-26
Posts: 38

Re: High load averages and interrupts with 2.6.36/2.6.37

Every kernel I try to compile fails at the same spot in the compile -- probably a problem with my config. I'll have to look over it later.

2.6.37 was thrown into [testing] today, so if anyone feels adventurous, they could try that.

I'm still having the high interrupt problem. Anyone else reading out there, have you came across a solution? (other than my solutions at the beginning).

Offline

#7 2011-01-08 19:09:20

KlavKalashj
Member
Registered: 2008-10-09
Posts: 376

Re: High load averages and interrupts with 2.6.36/2.6.37

yesterday I had like 0.50+ while idling (according to uptime). Today I installed 2.6.37 from {testing], and removed nohz=off highres=off from grub, and voila:

oscar@arch-laptop: [~] >>> uptime
 20:06:41 up  2:48,  2 users,  load average: 0.02, 0.05, 0.05

Offline

#8 2011-01-08 19:11:35

Telkkar
Member
From: United States
Registered: 2010-12-26
Posts: 38

Re: High load averages and interrupts with 2.6.36/2.6.37

KlavKalashj wrote:

yesterday I had like 0.50+ while idling (according to uptime). Today I installed 2.6.37 from {testing], and removed nohz=off highres=off from grub, and voila:

oscar@arch-laptop: [~] >>> uptime
 20:06:41 up  2:48,  2 users,  load average: 0.02, 0.05, 0.05

Interesting. What about wakeups in powertop?

I'm not *too* concerned with the load as much as I am with the wake ups  -- although they are probably connected at some point.

I think i'll have to try 2.6.37 from testing.


EDIT: I installed 2.6.37 from testing, and my load averages dropped -- finally.

My wakeups also dropped -- they're about half of what they were. However, that's still way more than I'd hope to find on an idle laptop.

Last edited by Telkkar (2011-01-08 19:34:06)

Offline

#9 2011-01-09 01:24:01

KlavKalashj
Member
Registered: 2008-10-09
Posts: 376

Re: High load averages and interrupts with 2.6.36/2.6.37

Telkkar wrote:
KlavKalashj wrote:

yesterday I had like 0.50+ while idling (according to uptime). Today I installed 2.6.37 from {testing], and removed nohz=off highres=off from grub, and voila:

oscar@arch-laptop: [~] >>> uptime
 20:06:41 up  2:48,  2 users,  load average: 0.02, 0.05, 0.05

Interesting. What about wakeups in powertop?

I'm not *too* concerned with the load as much as I am with the wake ups  -- although they are probably connected at some point.

I think i'll have to try 2.6.37 from testing.


EDIT: I installed 2.6.37 from testing, and my load averages dropped -- finally.

My wakeups also dropped -- they're about half of what they were. However, that's still way more than I'd hope to find on an idle laptop.

Wakeups vary greatly - but I would say [extra timer interrupt} is the top cause. Dunno what that is. smile

Edit: I don't really understand Powertop so I don't know what you want to know tongue Also, it completely changes every refresh, so it's impossible to get a good "screenshot" from it.

Last edited by KlavKalashj (2011-01-09 01:25:43)

Offline

#10 2011-01-09 02:04:59

Telkkar
Member
From: United States
Registered: 2010-12-26
Posts: 38

Re: High load averages and interrupts with 2.6.36/2.6.37

KlavKalashj wrote:

Wakeups vary greatly - but I would say [extra timer interrupt} is the top cause. Dunno what that is. smile

Edit: I don't really understand Powertop so I don't know what you want to know tongue Also, it completely changes every refresh, so it's impossible to get a good "screenshot" from it.


Under powertop you'll see something like this:

Top causes for wakeups:
  63.4% ( 44.0)   [kernel scheduler] Load balancing tick
  15.3% ( 10.6)   kworker/0:0
   7.2% (  5.0)   syndaemon
   7.2% (  5.0)   pommed 

That's mine, and that's when I'm idle. >.<

Offline

#11 2011-01-09 21:34:12

helloworld1
Member
Registered: 2010-12-26
Posts: 72

Re: High load averages and interrupts with 2.6.36/2.6.37

My load average is like this when idling (Core i7, 4 thread 2 core)

uptime
 16:33:25 up 2 days, 21:54, 11 users,  load average: 1.58, 1.40, 1.59

Powertop

Wakeups-from-idle per second : 279.5    interval: 10.0s
no ACPI power usage estimate available

Top causes for wakeups:
  38.9% (324.3)   kworker/0:0
  36.1% (300.3)   [kernel scheduler] Load balancing tick
   4.8% ( 39.6)   firefox
   3.2% ( 26.5)   [iwlagn] <interrupt>

BTW: Is kworker the kernel's Worker Queue, a buttom half mechanism? I have heard it was event/0, 1 and etc.

Last edited by helloworld1 (2011-01-09 21:40:01)

Offline

#12 2011-01-10 13:56:53

kokoko3k
Member
Registered: 2008-11-14
Posts: 2,390

Re: High load averages and interrupts with 2.6.36/2.6.37

If some of you uses hibernation, i solved my issue:
https://bbs.archlinux.org/viewtopic.php?id=110949


Help me to improve ssh-rdp !
Retroarch User? Try my koko-aio shader !

Offline

#13 2011-01-11 19:03:16

tjbp
Member
Registered: 2010-11-06
Posts: 43

Re: High load averages and interrupts with 2.6.36/2.6.37

Finally solved for me on a Core 2 Duo T8300 with 2.6.37. big_smile Had this problem for months.

Scratch that, spoke to soon - problem came back seconds after posting. sad

Last edited by tjbp (2011-01-11 19:07:09)

Offline

#14 2011-01-15 06:17:34

Telkkar
Member
From: United States
Registered: 2010-12-26
Posts: 38

Re: High load averages and interrupts with 2.6.36/2.6.37

helloworld1 wrote:

My load average is like this when idling (Core i7, 4 thread 2 core)

uptime
 16:33:25 up 2 days, 21:54, 11 users,  load average: 1.58, 1.40, 1.59

The new 2.6.37 kernel in [testing] makes these drop.

tjbp wrote:

Finally solved for me on a Core 2 Duo T8300 with 2.6.37.  Had this problem for months.
Scratch that, spoke to soon - problem came back seconds after posting.

Darn. sad I really want to get this fixed, but I don't know where to start. sad

Offline

#15 2011-01-17 16:08:13

rubenvb
Member
Registered: 2011-01-14
Posts: 99

Re: High load averages and interrupts with 2.6.36/2.6.37

Hi,

Is it "safe" to install the kernel from testing? Will it solve this issue  (there are conflicting reports here...)
lts
I also thought about temporarily installing kernel26-lts and using th at, but that would require me to install nvidia-lts from AUR. What's the best option?
If installing the kernel(+nvidia) from testing is quite reversible, I can try and see if it fixes the problem for me.

Thanks

EDIT: currently on kernel26-lts with AUR's nvidia-lts and it's working fine (as in low CPU usage), but this is not very informative as the problem started in 2.6.35 somewhere...

Last edited by rubenvb (2011-01-17 16:29:57)

Offline

#16 2011-01-17 21:02:17

pogeymanz
Member
Registered: 2008-03-11
Posts: 1,020

Re: High load averages and interrupts with 2.6.36/2.6.37

Kernel 2.6.37 does not fix the issue for me.

Offline

#17 2011-01-22 03:34:17

Telkkar
Member
From: United States
Registered: 2010-12-26
Posts: 38

Re: High load averages and interrupts with 2.6.36/2.6.37

rubenvb wrote:

EDIT: currently on kernel26-lts with AUR's nvidia-lts and it's working fine (as in low CPU usage), but this is not very informative as the problem started in 2.6.35 somewhere...

Indeed. I'd like to figure out the problem, but I don't know how. Frustrating.

pogeymanz wrote:

Kernel 2.6.37 does not fix the issue for me.

Which issue? Load averages or wakeups?



If ANYONE out there has any idea, throw it our way.

Offline

#18 2011-01-24 11:54:28

karol
Archivist
Registered: 2009-05-06
Posts: 25,440

Re: High load averages and interrupts with 2.6.36/2.6.37

[karol@black ~]$ pacman -Q kernel26
kernel26 2.6.36.3-1
[karol@black ~]$ uptime
 12:56:46 up 5 min,  1 user,  load average: 0.07, 0.12, 0.12
Top causes for wakeups:
  28,3% (  2,6)   [kernel core] hrtimer_start (tick_sched_timer)
  21,7% (  2,0)   [kernel core] clocksource_watchdog (clocksource_watchdog)
  12,0% (  1,1)   [kernel scheduler] Load balancing tick
...

Stock Arch i686 kernel, no kernel line modifications, around 10 wakeups per second on idle, so no problems here.

What's your kernel line?

Last edited by karol (2011-01-24 11:58:47)

Offline

#19 2011-01-24 13:05:07

pogeymanz
Member
Registered: 2008-03-11
Posts: 1,020

Re: High load averages and interrupts with 2.6.36/2.6.37

Telkkar wrote:
pogeymanz wrote:

Kernel 2.6.37 does not fix the issue for me.

Which issue? Load averages or wakeups?

Both issues. I still have high wakeups and load averages.

Offline

#20 2011-01-25 23:40:58

Telkkar
Member
From: United States
Registered: 2010-12-26
Posts: 38

Re: High load averages and interrupts with 2.6.36/2.6.37

linux   /vmlinuz26 root=/dev/sda5 ro  quiet

There's the kernel line, or so I believe. Default boot options that came with grub 2. I may have added in the quiet, not sure.

2.6.37 gives me lower load averages, but still draining my battery like crazy with high wakeups.

I'm going to give 2.6.36.3 a try from [core].


EDIT: The 2.6.36.3-1 kernel fixes the load average (strange, might I add), but I'm still getting high wake-ups.

Last edited by Telkkar (2011-01-26 02:28:40)

Offline

#21 2011-01-26 07:50:30

kokoko3k
Member
Registered: 2008-11-14
Posts: 2,390

Re: High load averages and interrupts with 2.6.36/2.6.37

Did you tried to kill apps one by one and see if wakeups goes down?
Even if i had that kind of problem only on resuming from hibernation, killing X worked for me.


Help me to improve ssh-rdp !
Retroarch User? Try my koko-aio shader !

Offline

#22 2011-01-26 07:53:53

karol
Archivist
Registered: 2009-05-06
Posts: 25,440

Re: High load averages and interrupts with 2.6.36/2.6.37

kokoko3k wrote:

Did you tried to kill apps one by one and see if wakeups goes down?
Even if i had that kind of problem only on resuming from hibernation, killing X worked for me.

Actually, I have more wakeups outside X - 20/sec - compared to 10/sec when I'm using X.

Offline

#23 2011-01-26 08:53:06

kokoko3k
Member
Registered: 2008-11-14
Posts: 2,390

Re: High load averages and interrupts with 2.6.36/2.6.37

...but you're not triggering the bug


Help me to improve ssh-rdp !
Retroarch User? Try my koko-aio shader !

Offline

#24 2011-01-26 14:29:13

Telkkar
Member
From: United States
Registered: 2010-12-26
Posts: 38

Re: High load averages and interrupts with 2.6.36/2.6.37

kokoko3k wrote:

Did you tried to kill apps one by one and see if wakeups goes down?
Even if i had that kind of problem only on resuming from hibernation, killing X worked for me.

I booted into console and still had high wakeups. I decided to kill off some of the daemons I had running at boot -- pommed was a big one. After that, my wakeups dropped to around 7-15 per second. Still a bit higher than I'd like to see.

Pommed seems to be the culprit for me -- although kworker is still showing 10 or so wakeups per second. I can probably get by without using pommed if I can find how to incorporate the keys pommed uses into my awesome config.


kokoko3k, thanks for the tip. For some reason, it completely slipped my mind. Now the question is -- are all these kworker wakeups and ticks from the kernel normal for everyone else as well?

Offline

#25 2011-01-26 15:37:25

kokoko3k
Member
Registered: 2008-11-14
Posts: 2,390

Re: High load averages and interrupts with 2.6.36/2.6.37

10wkps are normal to me, as far as i understood kworker has to do with kernel scheduling tasks.
Powertop-git is very useful because it tells you what it is doing in detail by showing the called function.


Help me to improve ssh-rdp !
Retroarch User? Try my koko-aio shader !

Offline

Board footer

Powered by FluxBB