(Solved) Kernel 2.6.35 I/O congestion

litemotiv · 2010-08-22 11:27:23

Contrary to some reports here, cpu temperature seems alright for me on 2.6.35. I'm seeing strange spikes in the sysload however that i can't relate to anything happening on the system. Even with only a single terminal running, sysload periodically jumps to 0.30 / 0.50, then slowly falls back but after a while jumps up again to at least 0.30.

Cpu usage is close to zero so it must be some I/O congestion somewhere. I've looked at iotop output but the only actual disk writes i'm seeing is jdb2 (journal) which writes between 30kb-50kb every ~5 seconds. I don't see anything interesting being written to the logs though.

While i'm typing this my sysload hangs around 0.40, so for a C2D system with an Intel SSD something is not working right.

Any other ideas to debug I/O congestion and find out which process is causing this?
---

edit: sysload is 1.00 now without any process doing heavy diskwrites, and with X as highest cpu user with around 5%.

Last edited by litemotiv (2011-01-31 17:50:39)

karol · 2010-08-22 11:40:17

What is the sysload? Is

load average: 0.59, 0.56, 0.47

from the top program what you're complaining about? If so, +1.

litemotiv · 2010-08-22 11:44:18

Yes uptime or top. The sysload is the current load on the system (queue): 1.00 is 100% for a single core, 2.00 is full load for a dual core.

Last edited by litemotiv (2010-08-22 11:45:27)

karol · 2010-08-22 11:52:49

litemotiv wrote:

Yes uptime or top. The sysload is the current load on the system (queue): 1.00 is 100% for a single core, 2.00 is full load for a dual core.

I have a P4 and an old harddrive. sysload was often flatlining on 2.6.34 and earlier versions, now it doesn't fall below 0.35.

litemotiv · 2010-08-22 12:14:11

Hmm, well i guess it could very well be caused by the scheduler commit that is linked to in the other thread.

tkdfighter · 2010-08-22 21:27:51

litemotiv wrote:

Yes uptime or top. The sysload is the current load on the system (queue): 1.00 is 100% for a single core, 2.00 is full load for a dual core.

That's not quite correct. Here's a quote from the book "UNIX Power Tools":

The load average tries to measure the number of active processes at any time. As a measure of CPU utilization, the load average is simplistic, poorly defined, but far from useless.

So the system load can go well above 1.00 on a single core, like when you're running a threading webserver and have more requests than you're system can handle. In such a case, processes would be waiting in the system queue response time would go way up. However, I don't think this should happen on a desktop system.

Now that you've mentioned it, mine seems a bit higher than usual, too:

load average: 0.20, 0.40, 0.43

This is on an idle Phenom II quad core.

Last edited by tkdfighter (2010-08-22 21:29:24)

karol · 2010-08-22 21:31:53

tkdfighter wrote:

litemotiv wrote:
Yes uptime or top. The sysload is the current load on the system (queue): 1.00 is 100% for a single core, 2.00 is full load for a dual core.
That's not quite correct. Here's a quote from the book "UNIX Power Tools":
The load average tries to measure the number of active processes at any time. As a measure of CPU utilization, the load average is simplistic, poorly defined, but far from useless.
So the system load can go well above 1.00, like when you're running a threading webserver and have more requests than you're system can handle. In such a case, processes would be waiting in the system queue response time would go way up. However, I don't think this should happen on a desktop system.
Now that you've mentioned it, mine seems a bit higher than usual, too:
load average: 0.20, 0.40, 0.43
This is on an idle Phenom II quad core.

http://en.wikipedia.org/wiki/Load_(computing) :-)

At the moment I have

load average: 1.03, 0.97, 0.89

which is unheard of. I'm just browsing the forum and the web interface to our package db.

Spacenick · 2010-08-22 21:40:47

Sounds like this problem Linus seems to be having so the man might already be on it ;-)
http://lkml.org/lkml/2010/8/15/111

litemotiv · 2010-08-22 21:50:53

Interesting information Spacenick, i don't have Intel graphics though (Nvidia) so i'm not sure if it applies to me.

tkdfighter · 2010-08-23 07:49:11

The sources that Wikipedia article references are quite in-depth. The one on Linux Journal touches queueing theory. Nice

tkdfighter · 2010-08-23 08:00:33

litemotiv wrote:

Interesting information Spacenick, i don't have Intel graphics though (Nvidia) so i'm not sure if it applies to me.

Read the second post, Markus Trippelsdorf is using AMD/Radeon, so it's definitely not limited to Intel. Maybe it's just the processors feeling lonely and the system queue is providing them some warmth and occupation?

karol · 2010-08-23 09:58:34

tkdfighter wrote:

litemotiv wrote:
Interesting information Spacenick, i don't have Intel graphics though (Nvidia) so i'm not sure if it applies to me.
Read the second post, Markus Trippelsdorf is using AMD/Radeon, so it's definitely not limited to Intel. Maybe it's just the processors feeling lonely and the system queue is providing them some warmth and occupation?

My processor has been a good processor, so I can buy him a beer. Wait - he's only seven! I'll buy him a teddy bear, just please tell him to behave.

litemotiv · 2010-08-23 09:59:48

2.6.35.3 didn't change anything in this respect.

karol · 2010-08-23 10:02:08

litemotiv wrote:

2.6.35.3 didn't change anything in this respect.

I'm not yet sure. How can I conduct more scientific tests other than periodically checking the load?

Edit:

 load average: 1.18, 0.89, 0.72

while light web browsing

Last edited by karol (2010-08-23 10:29:31)

combuster · 2010-08-23 10:05:44

load average: 0.43, 0.46, 0.39

But lastnight I had a load of 2.57 in the last minute and other two values were above 1 even though machine was pretty much idling all the time (transmission, twitux and evolution in tray). C2D here also with intel igp.

Onyros · 2010-08-23 10:27:47

Acer 1410 (intel chips SU2300 and 4500MHD), dumbass stuttering, nothing getting logged anywhere. I was getting crazy with this, the easy solution was downgrading, but I wanted to get to the bottom of it. Only found out about this today, and it's been like this ever since the kernel got out of testing.

litemotiv · 2010-08-23 12:11:25

karol wrote:

I'm not yet sure. How can I conduct more scientific tests other than periodically checking the load?
Edit:
 load average: 1.18, 0.89, 0.72
while light web browsing

Since there are no signs of specific processes using much cpu or i/o, i think the only way for now is to check the load. Any load above 0.50 0.30 while (almost) idling seems problematic.

Someone more knowledgeable could cherrypick the kernel and remove that journaling commit, i'd be very interested in the results. Another option is to compile the kernel with more debug output, but that probably produces a lot of incomprehensible data..

Last edited by litemotiv (2010-08-23 12:43:12)

thegritche · 2010-08-23 12:50:55

I think my issue is related to yours (I/O congestion).
Since I upgraded to the 2.6.35.2 (and 2.6.35.3) kernel (from core), my system is unusable.
During the boot, every step takes longer, particularly mounting local filesystems which takes couple minutes.
I can see in the KDE task manager that each process is "waiting for disk" (well the french equivalent) and there is no CPU load.
Moreover, even without being logged into KDE, downgrading to 2.6.34 from the terminal takes more than 10 minutes.
After having downgraded the kernel, everything goes as smoothly as it was. So it is completely related to 2.6.35.

karol · 2010-08-23 12:53:12

I was AFK for half an hour but left the computer on. When I came back it showed

$ uptime
 14:59:38 up  3:02,  1 user,  load average: 0.59, 0.40, 0.31

I don't have any problems, everything is smooth, but I can't say I'm stressing my system much.

Beelzebud · 2010-08-23 21:17:53

I'm not sure if it's related, but last night on 2 PCs running 2.6.35 (latest release from core) I had an I/O error while copying a 66MB file from one partition to another partition on one machine, and then an I/O error when I tried copying it to an NTFS partition on the other machine. The disturbing thing is that when the I/O error occurred on the NTFS drive it caused errors which left orphan files on the disk. Luckily chkdsk in Win7 corrected the volume when I booted in. For now I'm not touching that NTFS drive from Linux.

Last edited by Beelzebud (2010-08-23 21:18:42)

karol · 2010-08-23 22:54:45

I've installed kernel26-i915-2.6.35-1-i686 from [archlinuxfr] - it doesn't seem to suffer from the I/O problems *that* much:

load average: 0.27, 0.28, 0.27

while browsing the forums and

load average: 0.10, 0.08, 0.09

while idling with just a bunch of terminals open (no web browser).

Last edited by karol (2010-08-23 22:55:31)

Procyon · 2010-08-24 14:53:23

I switched back to 2.6.34.

For me the problem was Virtualbox and mplayer having stuttering audio (including really annoying pops) for up to a few minutes at a time with intervals of about half an hour to two/three hours. mpd was somehow unaffected.

loadavg is about a tenth of what it was under 2.6.35. The temperature is the same.

Bl@ster · 2010-08-24 16:36:34

Here is the same problem. Installing kernel26 from [archlinuxfr] helped a bit but not too much; After a while, my desktop keeps freezing for a minute. Now it seems okay, but I'm afraid for another "attack" of machine-hyper-loading

litemotiv · 2010-08-24 16:55:17

I'm back to 2.6.34 for now too, current load is 0.02 0.01 0.01 after 15 minutes of browsing and typing.

Bl@ster · 2010-08-24 17:39:47

litemotiv wrote:

I'm back to 2.6.34 for now too, current load is 0.02 0.01 0.01 after 15 minutes of browsing and typing.

Me too, but I'm lazy and keep trying
I hope Linus will save us all -.-'

Arch Linux

#1 2010-08-22 11:27:23

(Solved) Kernel 2.6.35 I/O congestion

#2 2010-08-22 11:40:17

Re: (Solved) Kernel 2.6.35 I/O congestion

#3 2010-08-22 11:44:18

Re: (Solved) Kernel 2.6.35 I/O congestion

#4 2010-08-22 11:52:49

Re: (Solved) Kernel 2.6.35 I/O congestion

#5 2010-08-22 12:14:11

Re: (Solved) Kernel 2.6.35 I/O congestion

#6 2010-08-22 21:27:51

Re: (Solved) Kernel 2.6.35 I/O congestion

#7 2010-08-22 21:31:53

Re: (Solved) Kernel 2.6.35 I/O congestion

#8 2010-08-22 21:40:47

Re: (Solved) Kernel 2.6.35 I/O congestion

#9 2010-08-22 21:50:53

Re: (Solved) Kernel 2.6.35 I/O congestion

#10 2010-08-23 07:49:11

Re: (Solved) Kernel 2.6.35 I/O congestion

#11 2010-08-23 08:00:33

Re: (Solved) Kernel 2.6.35 I/O congestion

#12 2010-08-23 09:58:34

Re: (Solved) Kernel 2.6.35 I/O congestion

#13 2010-08-23 09:59:48

Re: (Solved) Kernel 2.6.35 I/O congestion

#14 2010-08-23 10:02:08

Re: (Solved) Kernel 2.6.35 I/O congestion

#15 2010-08-23 10:05:44

Re: (Solved) Kernel 2.6.35 I/O congestion

#16 2010-08-23 10:27:47

Re: (Solved) Kernel 2.6.35 I/O congestion

#17 2010-08-23 12:11:25

Re: (Solved) Kernel 2.6.35 I/O congestion

#18 2010-08-23 12:50:55

Re: (Solved) Kernel 2.6.35 I/O congestion

#19 2010-08-23 12:53:12

Re: (Solved) Kernel 2.6.35 I/O congestion

#20 2010-08-23 21:17:53

Re: (Solved) Kernel 2.6.35 I/O congestion

#21 2010-08-23 22:54:45

Re: (Solved) Kernel 2.6.35 I/O congestion

#22 2010-08-24 14:53:23

Re: (Solved) Kernel 2.6.35 I/O congestion

#23 2010-08-24 16:36:34

Re: (Solved) Kernel 2.6.35 I/O congestion

#24 2010-08-24 16:55:17

Re: (Solved) Kernel 2.6.35 I/O congestion

#25 2010-08-24 17:39:47

Re: (Solved) Kernel 2.6.35 I/O congestion

Board footer