(Solved) Kernel 2.6.35 I/O congestion

karol · 2010-08-30 13:02:33

architech wrote:

Just upgraded to kernel26 2.6.35.4-1 and now the problem is even is worse. On boot and login, the load was 1.42 0.48. 0.30.
Here's the load when idle for 5 minutes.
 00:51:40 up 7 min,  2 users,  load average: 0.42, 0.38, 0.20
Linux 2.6.35-ARCH #1 SMP PREEMPT Fri Aug 27 17:14:28 CEST 2010 x86_64 AMD Athlon(tm) X2 Dual-Core QL-62 AuthenticAMD GNU/Linux

The boot and login load is always high. The weird thing is, I don't see any load-related problems on my Sempron with up-to-date system.

litemotiv · 2010-08-30 13:34:45

archman-cro wrote:

Is all of this somehow related to the vanilla-anti-io-stalling patch in kernel26-ice (AUR)? Should I build the kernel with that option?

I don't know what that patch does or where it came from, but if you want to apply it and report back if it helps anything that'd be great. :-)

SleepyFloyd · 2010-08-30 14:56:18

Just wanted to second this problem. 2.6.35.4 was so bad that even the input from the keyboard/touchpad was laggy. Downgraded to 2.6.34, Load averages are back to:

16:51:30 up 2:11, 2 users, load average: 0.06, 0.02, 0.00

uname -a:
Linux Laptop 2.6.34-ARCH #1 SMP PREEMPT Wed Aug 4 14:31:56 CEST 2010 x86_64 Pentium(R) Dual-Core CPU T4300 @ 2.10GHz GenuineIntel GNU/Linux

EDIT: Installed 2.6.35.4 again to compare.

17:10:51 up 4 min, 2 users, load average: 2.15, 1.32, 0.56

Touchpad stutters so much is practically unusuable.

17:17:04 up 10 min, 2 users, load average: 2.70, 1.93, 1.04

Typing while writing this post is also badly lagging. There is nothing runing but a Xterm with ncmpcpp and Vimprobable.

Last edited by SleepyFloyd (2010-08-30 15:18:49)

Onyros · 2010-08-30 15:33:26

Whoa, I tried 2.6.35.4 as well, just for testing... Completely unusable, even worse than before.

OT: "Sleepy Floyd is superman!" (Warriors' fan?)

Ogion · 2010-08-30 15:35:36

I don't seem to have any trouble.
Kernelver 2.6.35.4-1
Eeepc 1000h

Right now, irc, browser, mplayer with a radio stream (and also streamripper running), some other idle stuff:
% uptime
17:35:28 up 7:28, 0 users, load average: 0.04, 0.16, 0.23

Ogion

SleepyFloyd · 2010-08-30 15:47:24

Onyros wrote:

Whoa, I tried 2.6.35.4 as well, just for testing... Completely unusable, even worse than before.
OT: "Sleepy Floyd is superman!" (Warriors' fan?)

Yeah... I know you from the realgm forums. Remember the GSW-Dallas series?

To not go fully OT:

Back to 2.6.34. Load averages:
17:40:54 up 9 min, 2 users, load average: 0.04, 0.17, 0.14

Downgrading seems to be the only solution at the moment.

I also have a server running here. MPD is playing music (process constantly uses 12-14% CPU), transmission is running, several gvim instances, firefox, thunderbird, pidgin, xchat and some more:

17:43:04 up 9:41, 3 users, load average: 0.73, 0.99, 0.85

Linux server 2.6.35-ARCH #1 SMP PREEMPT Sat Aug 21 07:23:08 UTC 2010 i686 Intel(R) Atom(TM) CPU 230 @ 1.60GHz GenuineIntel GNU/Linux

Not great either, but I guess that's normal for a that CPU. At least neither the mouse nor the keyboard are stuttering. Watching videos (mplayer) or TV (xine with a DVB-S card) also works.

inode · 2010-08-30 17:23:35

Could be a problem with cfs or cfq in .35 ...

I'm using a custom 2.6.35.4 kernel with the CK and BFQ patchsets applied, and using BFS as scheduler + BFQ for i/o. My sysload is:

load average: 0.00, 0.00, 0.00

MadCat_X · 2010-08-31 15:52:49

I'm not sure if this is related, but I've been experiencing some weird behavior on my laptop. I currently run custom 2.6.35.4 with ck patches and the "io_stress" patch. My sysload hardly exceeds 0.5, but every few minutes or so the average 1 minute load would increase from 0.0x to ~0.4 where it would stay for some time and then gradually fall. This is on a system where nothing but Kopete, Opera and Okular is running and none of these applications are really doing anything, CPU load doesn't exceed 15%.
All in all the system seems somewhat slower than usual. I originally started this thread, any chance anybody finds that familiar? Or can I be actually having two separate problems?:)

Last edited by MadCat_X (2010-08-31 15:55:14)

Jodell · 2010-08-31 18:41:50

The problem most likely lies in the CFQ. I'm currently using the 2.6.25.4 kernel with the BFS and problem remains.

litemotiv · 2010-08-31 20:13:20

Jodell wrote:

The problem most likely lies in the CFQ. I'm currently using the 2.6.25.4 kernel with the BFS and problem remains.

If you're using a different scheduler and the problem remains then CFQ obviously is not the cause. I also assume you mean 2.6.35.4.

Jodell · 2010-09-01 05:25:15

litemotiv wrote:

Jodell wrote:
The problem most likely lies in the CFQ. I'm currently using the 2.6.25.4 kernel with the BFS and problem remains.
If you're using a different scheduler and the problem remains then CFQ obviously is not the cause. I also assume you mean 2.6.35.4.

Yes, I meant 2.6.35.4. Sorry about that.

I may have this wrong so please correct me

The BFS and the CFS deals with task scheduling and the CFQ deals with I/O scheduling. Shouldn't the CFQ still be in use because only the task scheduler was changed and not the I/O scheduler?

ngoonee · 2010-09-01 05:29:28

Jodell wrote:

litemotiv wrote:
Jodell wrote:
The problem most likely lies in the CFQ. I'm currently using the 2.6.25.4 kernel with the BFS and problem remains.
If you're using a different scheduler and the problem remains then CFQ obviously is not the cause. I also assume you mean 2.6.35.4.
Yes, I meant 2.6.35.4. Sorry about that.
I may have this wrong so please correct me
The BFS and the CFS deals with task scheduling and the CFQ deals with I/O scheduling. Shouldn't the CFQ still be in use because only the task scheduler was changed and not the I/O scheduler?

Yep, that sounds right. BFS doesn't (afaik) replace CFQ.

inode · 2010-09-01 05:48:45

Jodell wrote:

The BFS and the CFS deals with task scheduling and the CFQ deals with I/O scheduling. Shouldn't the CFQ still be in use because only the task scheduler was changed and not the I/O scheduler?

Yep, CFQ is an I/O allocator not a task scheduler. Btw for those who want to try BFQ, the patches are quite straightforward: http://algo.ing.unimo.it/people/paolo/d … d/patches/

Last edited by inode (2010-09-01 05:49:40)

Jodell · 2010-09-01 06:24:45

inode wrote:

Yep, CFQ is an I/O allocator not a task scheduler. Btw for those who want to try BFQ, the patches are quite straightforward: http://algo.ing.unimo.it/people/paolo/d … d/patches/

Thanks for the link! Unfortunately I'm a noob at patching stuff. However, after taking a look at the PKGBUILD for the BFS kernel in the AUR, I'm sure that I can patch it for BFQ.

I'll report my findings (if any).

litemotiv · 2010-09-01 07:42:28

Jodell wrote:

The BFS and the CFS deals with task scheduling and the CFQ deals with I/O scheduling. Shouldn't the CFQ still be in use because only the task scheduler was changed and not the I/O scheduler?

I was a bit cryptic before, i meant: if you swap CFQ for another scheduler then you know it isn't the culprit. Many people here use something else than CFQ and are having the same problem. Also since you have swapped CFS for BFS and still have the same problem, most likely that isn't the cause either.

Jodell · 2010-09-01 08:37:03

Yeah, it isn't the schedulers that are the problem. Whatever it is, I hope it's fixed in version 2.6.36

Last edited by Jodell (2010-09-01 08:37:16)

entropy · 2010-09-01 09:21:28

$ uname -a
Linux entropy-laptop 2.6.35-fbcondecor #1 SMP PREEMPT Wed Aug 25 02:56:38 EEST 2010 i686 Intel(R) Core(TM)2 Duo CPU T9400 @ 2.53GHz GenuineIntel GNU/Linux
$ uptime
12:20:08 up 1:49, 4 users, load average: 4.19, 4.18, 4.19
$

This is doing mostly nothing, just browsing and gajim. The system feels fine, no stuttering, high power usage or any such symptoms of a high load and CPU usage hovers around 2%.
Very weird numbers.

litemotiv · 2010-09-01 16:06:59

I think we have enough people now to conclude that there is a problem, from now on please only post here if you have a working solution or useful information from upstream.

hiciu · 2010-09-01 17:06:03

Output from powertop may be useful:

Top causes for wakeups:
  63.7% (497.5)   [kernel scheduler] Load balancing tick
  13.3% (103.5)   floyd
   6.0% ( 47.0)   [hda_intel] <interrupt>
   3.0% ( 23.5)   [Rescheduling interrupts] <kernel IPI>
   2.9% ( 22.3)   USB device  2-2 : 2.4GHz 2way RF Mouse Receiver ()
   2.7% ( 21.0)   [ohci_hcd:usb2] <interrupt>

"[kernel scheduler] Load balancing tick" and "[Rescheduling interrupts] <kernel IPI>" looks suspicious, they are new (i.e. I do not recall them from previous kernels).

@karol: I thought that this may be somehow related to high load.

Last edited by hiciu (2010-09-01 17:24:31)

karol · 2010-09-01 17:07:54

hiciu wrote:

Output from powertop may be useful:
Top causes for wakeups:
  63.7% (497.5)   [kernel scheduler] Load balancing tick
  13.3% (103.5)   floyd
   6.0% ( 47.0)   [hda_intel] <interrupt>
   3.0% ( 23.5)   [Rescheduling interrupts] <kernel IPI>
   2.9% ( 22.3)   USB device  2-2 : 2.4GHz 2way RF Mouse Receiver ()
   2.7% ( 21.0)   [ohci_hcd:usb2] <interrupt>
"[kernel scheduler] Load balancing tick" and "[Rescheduling interrupts] <kernel IPI>" looks suspicious, they are new (i.e. I do not recall them from previous kernels).

No, they're not new, but the ticks were not that frequent.
Please, read litemotiv's post.

TigTex · 2010-09-01 20:24:08

Just for debugging, I have no problems to report here. Using 2.6.35.4 my system shows an average load near 0.00 when idle at tty1.

System specs:
Pentium4 3.0ghz (2 threads)
512mb ram
ati 9250se
VIA chipset motherboard (msi pm8m-vh)

testing repo on and up-to-date.

Did anyone tried disabling ACPI? It could be some kind of malfunction powersaving feature. I have no problems but I have older hardware...

andrews · 2010-09-13 19:10:15

Update on https://bugzilla.kernel.org/show_bug.cgi?id=16525
Show that disabling NO_HZ seems to fix it.

litemotiv · 2010-09-13 22:00:33

andrews wrote:

Update on https://bugzilla.kernel.org/show_bug.cgi?id=16525
Show that disabling NO_HZ seems to fix it.

edit: nohz=off boot parameter should be enough to test

Last edited by litemotiv (2010-09-13 22:36:30)

andrews · 2010-09-13 22:21:50

Tested the 2.6.35.4 arch i686 kernel , with the nohz=off kernel line, and this resolve the problem but in powertop I see more than 600 wakeups in kernel scheduler load balancing tick, temperature and load average normal for my system.

Edit : After more deep tests the nohz=off without highres=off is the main cause of high wakeups from idle.
Anyway the 2.6.35 never go to 0.00 load when idle, but "nohz=off highres=off" helps a lot.

Last edited by andrews (2010-09-14 18:06:36)

litemotiv · 2010-09-14 18:37:23

It helps a bit for me as well, though i'm still seeing occasional spikes to 0.50 or even 1.00 load without much cpu activity. Overall the load is better, but still not what it should be.

Arch Linux

#51 2010-08-30 13:02:33

Re: (Solved) Kernel 2.6.35 I/O congestion

#52 2010-08-30 13:34:45

Re: (Solved) Kernel 2.6.35 I/O congestion

#53 2010-08-30 14:56:18

Re: (Solved) Kernel 2.6.35 I/O congestion

#54 2010-08-30 15:33:26

Re: (Solved) Kernel 2.6.35 I/O congestion

#55 2010-08-30 15:35:36

Re: (Solved) Kernel 2.6.35 I/O congestion

#56 2010-08-30 15:47:24

Re: (Solved) Kernel 2.6.35 I/O congestion

#57 2010-08-30 17:23:35

Re: (Solved) Kernel 2.6.35 I/O congestion

#58 2010-08-31 15:52:49

Re: (Solved) Kernel 2.6.35 I/O congestion

#59 2010-08-31 18:41:50

Re: (Solved) Kernel 2.6.35 I/O congestion

#60 2010-08-31 20:13:20

Re: (Solved) Kernel 2.6.35 I/O congestion

#61 2010-09-01 05:25:15

Re: (Solved) Kernel 2.6.35 I/O congestion

#62 2010-09-01 05:29:28

Re: (Solved) Kernel 2.6.35 I/O congestion

#63 2010-09-01 05:48:45

Re: (Solved) Kernel 2.6.35 I/O congestion

#64 2010-09-01 06:24:45

Re: (Solved) Kernel 2.6.35 I/O congestion

#65 2010-09-01 07:42:28

Re: (Solved) Kernel 2.6.35 I/O congestion

#66 2010-09-01 08:37:03

Re: (Solved) Kernel 2.6.35 I/O congestion

#67 2010-09-01 09:21:28

Re: (Solved) Kernel 2.6.35 I/O congestion

#68 2010-09-01 16:06:59

Re: (Solved) Kernel 2.6.35 I/O congestion

#69 2010-09-01 17:06:03

Re: (Solved) Kernel 2.6.35 I/O congestion

#70 2010-09-01 17:07:54

Re: (Solved) Kernel 2.6.35 I/O congestion

#71 2010-09-01 20:24:08

Re: (Solved) Kernel 2.6.35 I/O congestion

#72 2010-09-13 19:10:15

Re: (Solved) Kernel 2.6.35 I/O congestion

#73 2010-09-13 22:00:33

Re: (Solved) Kernel 2.6.35 I/O congestion

#74 2010-09-13 22:21:50

Re: (Solved) Kernel 2.6.35 I/O congestion

#75 2010-09-14 18:37:23

Re: (Solved) Kernel 2.6.35 I/O congestion

Board footer