(Solved) Kernel 2.6.35 I/O congestion

Schnouki · 2010-09-17 13:20:54

Hi,

"nohz=off" and "highress=off" really really help here.

Specs:
- CPU: Intel(R) Xeon(R) CPU E5520 @ 2.27GHz (2 quad-core CPUs + hyper-threading)
- RAM: 16 GB
- kernel 2.6.35.4, x86_64
- GPUs: NVIDIA Quadro FX 3800 + NVIDIA Tesla C2050, running the binary blob (260.24: prerelease for registered CUDA developers)

Before using these options, the idle load was more than 0.5, and from time to time it jumped to 5 and more (once it was over 15) while doing nothing special (single-screen X with Firefox, Thunderbird, emacs, a few terminals and Gajim).
With "nohz=off" only, the load is much lower (0.00 when idle with just X, awesome and one terminal) and everything is responsive again. But according to powertop, there are still have a lot of wakeups due tue the kernel scheduler load balancing tick (approx. 4800/sec!).
With "nohz=off highres=off", everything is back to normal: low load, and almost nothing in powertop.

Thanks everybody for these tips, they are greatly appreciated!

Zom · 2010-09-17 14:13:20

Just a word of warning, pulseaudio has some... interesting issues with highres=off.

As long as you keep the system volume either 100 or 50, and the stream volume either 100 or 50, sound works. Change either of those though and all you'll be hearing is noise until you change it back to either 100 or 50.

Really weird, and removing the highres option works.

R00KIE · 2010-09-17 22:19:09

With both nohz=off and highres=off things seem to be a little better and pulseaudio still works fine here

Some details about the machine/software:

> uname -a
Linux arch64 2.6.35-ARCH #1 SMP PREEMPT Fri Aug 27 17:14:28 CEST 2010 x86_64 AMD Turion(tm) 64 X2 Mobile Technology TL-64 AuthenticAMD GNU/Linux

> lspci -nn | egrep "VGA|Bridge"
00:00.0 Host bridge [0600]: ATI Technologies Inc RS690 Host Bridge [1002:7910]
00:02.0 PCI bridge [0604]: ATI Technologies Inc RS690 PCI to PCI Bridge (PCI Express Graphics Port 0) [1002:7913]
00:06.0 PCI bridge [0604]: ATI Technologies Inc RS690 PCI to PCI Bridge (PCI Express Port 2) [1002:7916]
00:07.0 PCI bridge [0604]: ATI Technologies Inc RS690 PCI to PCI Bridge (PCI Express Port 3) [1002:7917]
00:14.3 ISA bridge [0601]: ATI Technologies Inc SB600 PCI to LPC Bridge [1002:438d]
00:14.4 PCI bridge [0604]: ATI Technologies Inc SBx00 PCI to PCI Bridge [1002:4384]
01:00.0 VGA compatible controller [0300]: ATI Technologies Inc Mobility Radeon HD 2400 [1002:94c9]

4GB RAM

I'm using KMS and xf86-video-ati if that matters.

litemotiv · 2010-09-18 13:55:51

Well it's definitely not working for me, running with nohz=off and highres=off for two days now but as i'm typing this my load wobbles between 0.50 and 1.25. Nothing running but Firefox.

kgas · 2010-09-18 16:34:39

I tried this option in Samsung N210 and Lenovo N500, there is an improvement in uptime (load) but cpu temperature shoots up by ~4 Deg.C (actual, I can feel the temperature raise)

Zom · 2010-09-19 10:38:05

R00KIE wrote:

With both nohz=off and highres=off things seem to be a little better and pulseaudio still works fine here

Oh? Did you try changing the volume on the individual streams? Like, say, have mplayer running and then play music or just stuff with paplay?

I'm using KDE and the *-pulse patches from AUR.

Also, anybody give 2.6.36 a try?

R00KIE · 2010-09-19 20:16:07

Zom wrote:

R00KIE wrote:
With both nohz=off and highres=off things seem to be a little better and pulseaudio still works fine here
Oh? Did you try changing the volume on the individual streams? Like, say, have mplayer running and then play music or just stuff with paplay?

I'm using mplayer (and vlc) recompiled with pulse support and outputting directly to pulse. I did try changing both the stream volume and the device volume (to values different from 50% and 100%) and I didn't notice any artifacts or noise (I did change a couple of things though, I have resample-method = ffmpeg and flat-volumes = no in /etc/pulse/daemon.conf, I have no idea if that matters).

I guess it may be very dependent on the audio controller and software driver combination.

Zom · 2010-09-19 21:00:43

R00KIE wrote:

Zom wrote:
R00KIE wrote:
With both nohz=off and highres=off things seem to be a little better and pulseaudio still works fine here
Oh? Did you try changing the volume on the individual streams? Like, say, have mplayer running and then play music or just stuff with paplay?
I'm using mplayer (and vlc) recompiled with pulse support and outputting directly to pulse. I did try changing both the stream volume and the device volume (to values different from 50% and 100%) and I didn't notice any artifacts or noise (I did change a couple of things though, I have resample-method = ffmpeg and flat-volumes = no in /etc/pulse/daemon.conf, I have no idea if that matters).
I guess it may be very dependent on the audio controller and software driver combination.

Do you run it as a daemon or as a user though?

'Cause I run it as a user.

Leonid.I · 2010-09-19 21:17:35

I think that you, guys, are taking a detour here
Has anyone tried to measure load over ssh? For example, I have a P4+nvidia box, which is idle now with only blank xscreensaver, and the load is 0.00 (ssh). However, in X it is around 0.4-0.2 almost always...

R00KIE · 2010-09-20 00:29:59

Leonid.I wrote:

I think that you, guys, are taking a detour here
Has anyone tried to measure load over ssh? For example, I have a P4+nvidia box, which is idle now with only blank xscreensaver, and the load is 0.00 (ssh). However, in X it is around 0.4-0.2 almost always...

Haven't tried that, but on the console without X running I still see a load higher than 0 when idle, I haven't tried without KMS though.

@Zom
I run it as a user but that file will control how pulse behaves (and more on this will start to derail the thread as Leonid.I says).

Leonid.I · 2010-09-20 13:18:08

OK, I removed acpid from the intel desktop (don't really need it anyway), and am getting load ~0.05 in idle... can't really say if this is specific to my machine.

I too use KMS everywhere, and disabling it is not an option, because for nvidia I use nouveau and on my ati laptop it solves certain mtrr issues, which led to screen corruption. Although, I am almost willing to bet that disabling KMS will help...

I don't understand, however, if this is just an incorrect display of info, or a deeper problem. So far, I haven't noticed any significant increase in heat output or decrease in battery life.

Last edited by Leonid.I (2010-09-20 13:18:51)

mcmillan · 2010-09-27 23:39:01

Just saw a new kernel 2.6.35.6 is in the repos so I tried booting without nohz=off and highres=off (which only partially improved things for me). I've only been up for about 1/2 hour since then but it seems to be running about as well as it was with the 2.6.34 kernel and with less load than with 2.6.35.5 with the options. Anybody else able to confirm this?

STM · 2010-09-27 23:56:49

mcmillan wrote:

Just saw a new kernel 2.6.35.6 is in the repos so I tried booting without nohz=off and highres=off (which only partially improved things for me). I've only been up for about 1/2 hour since then but it seems to be running about as well as it was with the 2.6.34 kernel and with less load than with 2.6.35.5 with the options. Anybody else able to confirm this?

Also just upgraded to .6 but the problem remains when I switch no_hz back on (Core 2 Duo T8300, x86_64).

bfo · 2010-09-28 00:05:39

Try appending clocksource=acpi_pm to grub kernel line and tell us the results then.

STM · 2010-09-28 02:49:50

bfo wrote:

Try appending clocksource=acpi_pm to grub kernel line and tell us the results then.

Gave it a go - interestingly load drops right down to where it should be, but the random latency during idle remains.

bfo · 2010-09-28 08:45:45

There are lots of similar threads as far as this topic is concerned, for example here: http://forums.gentoo.org/viewtopic-p-64 … a6f5554b36. All of these slowdowns seem to be connected. I have your problem too, and am compiling kernel26-mainline just now ( version 2.6.36-rc5), maybe it's worth trying? One thing I am completely sure of, is that .34 series were free of this nasty bug.

Have you tried

dd if=/dev/zero of=test bs=1M count=5024 && rm test -f

on the faulty kernel? Do you get unresponsive desktop then?

Last edited by bfo (2010-09-28 08:59:05)

litemotiv · 2010-09-28 08:53:56

bfo wrote:

am compiling kernel26-mainline just now ( version 2.6.36-rc5), maybe it's worth trying?

It isn't, i already tried the current git-head, the bug is still present there.

I've bisected the problem and have brought it back to 10 commits possibly causing it, results on the bugzilla page: https://bugzilla.kernel.org/show_bug.cgi?id=16525

Zom · 2010-09-28 20:07:40

2.6.35.6-1 seems to have fixed the problem for me.

Anybody else confirm?

skunktrader · 2010-09-28 20:10:11

2.6.35.6-1 still same problem for me

archman-cro · 2010-09-28 21:19:42

I don't really care about "uptime" stuff, but after rebuilding my custom kernel with a 2.6.35.6 patch, I get somewhat lower average load:

~/> uptime
 23:16:48 up  7:40,  1 user,  load average: 0.33, 0.19, 0.11

The kernel is also built with bfs and toi patches.

broch · 2010-09-29 02:45:46

idle 2.6.35.6-bfs
load average: 0.00, 0.00, 0.00
idle 2.6.35.6-cfs
load average: 0.70, 0.52, 0.50

I don't know if cfs is directly to blame but definitely is affected by other changes. Switching to bfs fixes the issue (at least for me)... or running powertop alleviates the problemis .

funny thing is that running powertop in another window will decrease load average for kernel 2.6.35.6-cfs
load average: 0.36, 0.28, 0.20

unfortunately two latest bfs aren't great (latest 350 for obvious reasons because this is major scheduler overhaul), so to kkep in nice either run bfs on 2.6.34.x or 323 on 2.6.35.x

Last edited by broch (2010-09-29 02:48:40)

karol · 2010-09-29 02:49:00

As was already noted, load average depends on what you do, not only on the software, so you need to pay attention on how you measure.

broch · 2010-09-29 02:54:10

karol wrote:

As was already noted, load average depends on what you do, not only on the software, so you need to pay attention on how you measure.

fantastic. I listed kernels as idle.

Last edited by broch (2010-09-29 02:54:28)

karol · 2010-09-29 03:01:28

broch wrote:

karol wrote:
As was already noted, load average depends on what you do, not only on the software, so you need to pay attention on how you measure.
fantastic. I listed kernels as idle.

Define 'idle':
- I've just stopped doing some intense work so I might as well check the load,
--- or ---
- the system was idle for the past 15 minutes so I'll check the load and get back to work.
Also, some people have scheduled some jobs that start when you leave the system to idle and sometimes forget to disable them.

If -bfs works for you, that's great, but many people use the stock one.

broch · 2010-09-29 03:43:32

4thousands posts in one year. This might b overhelming.
idle (english) means - not in use.

If -bfs works for you, that's great, but many people use the stock one.

recently cfs was re-designed to be more competitive for desktop users. In the previous post I poinded out that while kernel with bfs does not have any issues, cfs has them. Either there is a bug in new cfs or new cfs reveals previously hidden bug somewhere else.
I never suggested switching to bfs.

Arch Linux

#76 2010-09-17 13:20:54

Re: (Solved) Kernel 2.6.35 I/O congestion

#77 2010-09-17 14:13:20

Re: (Solved) Kernel 2.6.35 I/O congestion

#78 2010-09-17 22:19:09

Re: (Solved) Kernel 2.6.35 I/O congestion

#79 2010-09-18 13:55:51

Re: (Solved) Kernel 2.6.35 I/O congestion

#80 2010-09-18 16:34:39

Re: (Solved) Kernel 2.6.35 I/O congestion

#81 2010-09-19 10:38:05

Re: (Solved) Kernel 2.6.35 I/O congestion

#82 2010-09-19 20:16:07

Re: (Solved) Kernel 2.6.35 I/O congestion

#83 2010-09-19 21:00:43

Re: (Solved) Kernel 2.6.35 I/O congestion

#84 2010-09-19 21:17:35

Re: (Solved) Kernel 2.6.35 I/O congestion

#85 2010-09-20 00:29:59

Re: (Solved) Kernel 2.6.35 I/O congestion

#86 2010-09-20 13:18:08

Re: (Solved) Kernel 2.6.35 I/O congestion

#87 2010-09-27 23:39:01

Re: (Solved) Kernel 2.6.35 I/O congestion

#88 2010-09-27 23:56:49

Re: (Solved) Kernel 2.6.35 I/O congestion

#89 2010-09-28 00:05:39

Re: (Solved) Kernel 2.6.35 I/O congestion

#90 2010-09-28 02:49:50

Re: (Solved) Kernel 2.6.35 I/O congestion

#91 2010-09-28 08:45:45

Re: (Solved) Kernel 2.6.35 I/O congestion

#92 2010-09-28 08:53:56

Re: (Solved) Kernel 2.6.35 I/O congestion

#93 2010-09-28 20:07:40

Re: (Solved) Kernel 2.6.35 I/O congestion

#94 2010-09-28 20:10:11

Re: (Solved) Kernel 2.6.35 I/O congestion

#95 2010-09-28 21:19:42

Re: (Solved) Kernel 2.6.35 I/O congestion

#96 2010-09-29 02:45:46

Re: (Solved) Kernel 2.6.35 I/O congestion

#97 2010-09-29 02:49:00

Re: (Solved) Kernel 2.6.35 I/O congestion

#98 2010-09-29 02:54:10

Re: (Solved) Kernel 2.6.35 I/O congestion

#99 2010-09-29 03:01:28

Re: (Solved) Kernel 2.6.35 I/O congestion

#100 2010-09-29 03:43:32

Re: (Solved) Kernel 2.6.35 I/O congestion

Board footer