TCP, concurrency and kernel version

student975 · 2011-10-27 04:54:59

Hi!

To benchmark some http server I use ApacheBench with such call:

$ ab -c 20000 -n 2000000 -k 127.0.0.1:8910/ok

I had got about 28K rps. Today I have got about 4K rps. I have tried to reboot to kernel26-lts and have got again ~28K rps. So there were some changing in the kernel. Unfortunately I don't remember when I have started benchmarking last time, and can not say exactly which kernel version results in the degradation. I think it was close to kernel numbering changing period (I mean switching from 2.6.x to 3.x).

With humble concurrency level (say, -c 100) I don't see any degradation. I.e. the problem is related to indeed large concurrent clients amount.

Has anybody ideas where to dig in?

Last edited by student975 (2011-10-27 04:57:00)

nullvoid · 2011-10-28 17:26:06

I think I have this problem too. When having a large number of connections to my httpd (lighttpd in this case) the kernel starts sending replies from source IP 0.0.0.0 to ip 0.0.0.0.

I captured some traffic with Wireshark: http://devmoon.se/~peter/pub/lighttpd.pcap

There are two requests made, the first one is made when iptables is blocking external traffic towards port 80, and the second request is made when the HTTPd is under heavy load from external traffic. In the first request the server replies as it should with both source-ip and destination-ip set. In the second request both source and destination IP is 0.0.0.0.

This is a problem which I've noticed just recently.

student975 · 2011-10-28 17:41:40

Both the server [1] and ApacheBench run on the same system - without iptables at all. And ApacheBench doesn't report any errors - it is just performance degradation.

The only changing I have made to the system are: increased nofile limits (to be able to open many connections) and disabled syn cookies.

[1] http://gaydenko.com/scala/tiscaf/httpd/

nullvoid · 2011-10-28 18:03:39

Could you try tcpdumping your lo interface when performing the benchmark? I just downgraded my kernel back to 3.0 and so far I've seen none of the problems I mentioned above.

student975 · 2011-10-29 05:11:57

I guess I can [1], but can not interpret as far as am not educated in the internals.

[1] http://gaydenko.com/tmp/c20000n200000k.01.dump.tar.bz2

student975 · 2011-10-31 02:54:41

The issue exists under the current ck-patched kernel also (linux-ck-corei7 3.0.7-1).

Arch Linux

#1 2011-10-27 04:54:59

TCP, concurrency and kernel version

#2 2011-10-28 17:26:06

Re: TCP, concurrency and kernel version

#3 2011-10-28 17:41:40

Re: TCP, concurrency and kernel version

#4 2011-10-28 18:03:39

Re: TCP, concurrency and kernel version

#5 2011-10-29 05:11:57

Re: TCP, concurrency and kernel version

#6 2011-10-31 02:54:41

Re: TCP, concurrency and kernel version

Board footer