[Benchmarking] Let's organise and standardize

stefanwilkens · 2009-10-26 02:37:55

Update:
Due to lacking enthusiasm, this has been discontinued.
---

First, have a look here:
http://wiki.archlinux.org/index.php/Benchmarking

I propose that we build a list of benchmarking tools and define a set of standards, at the very least guidelines, so that archers may bechmark and compare their systems in a way that is usefull and comparable.

With different drivers being available for graphics chipsets and the endless testing and tweaking to find the optimal performance combination, I propose that we organise ourselves to provide a proper wiki article on benchmarking linux systems. Not just graphics performance, but complete system performance including CPU, Storage, RAM, networking, IRDA and whatever else we may find, including a documented set of standards for users to follow, resulting in a benchmark that they can usefully compare with other archers.

If anybody is willing to help: don't hesitate to add to the wiki article!

Last edited by stefanwilkens (2011-12-24 16:25:36)

skottish · 2009-10-26 03:12:04

I think that this is a very cool idea! I'm going to move this to Community Contributions so that it doesn't get buried too quickly.

stefanwilkens · 2009-10-26 10:50:36

thanks!

I need some assistance here. To be able to compare properly, system information should be added. I propose that we choose a script or app that provides a humanly readable overview of system information. Hardware / software details and such things.

Does anybody know of a script or application that does this? Perhaps anybody that feel up to the task of creating it ?

In the long run, we may even try to expand this system information script into a more complete sollution that invokes benchmarking applications and compiles a list of results along with the system information. This, however, will be somewhat extensive and reqruires that we determine a set of tests. arch-bench or something, wouldn't that be fancy?

For now, system information

Last edited by stefanwilkens (2009-10-26 10:56:45)

Mikko777 · 2009-10-26 10:57:49

stefanwilkens wrote:

thanks!
I need some assistance here. To be able to compare properly, system information should be added. I propose that we choose a script or app that provides a humanly readable overview of system information. Hardware / software details and such things.
Does anybody know of a script or application that does this? Perhaps anybody that feel up to the task of creating it ?

If we use phoronix-suite. its already there.

[~/apps/phoronix]: phoronix-test-suite system-info

=====================================
Phoronix Test Suite v2.0.0 (SANDTORG)
System Information
=====================================

Hardware:
Processor: Intel Core 2 Quad CPU Q9550 @ 2.83GHz (Total Cores: 4), Motherboard: ASUSTeK P5Q-E, Chipset: Intel 4 Series Chipset + ICH10R, System Memory: 3959MB, Disk: 2 x 500GB ST3500320AS, Graphics: ATI Radeon HD 4800 Series, Monitor: SyncMaster

PHP Warning:  file_get_contents(): Filename cannot be empty in /usr/share/phoronix-test-suite/pts-core/objects/phodevi/phodevi_system.php on line 518
Software:
OS: Arch, Kernel: 2.6.31-ARCH (x86_64), Desktop: KDE 4.3.2, Display Server: X.Org Server 1.6.3.901 (1.6.4 RC 1), OpenGL: 2.1.9026, Compiler: GCC 4.4.1, File-System: ext4, Screen Resolution: 1680x10

Oh and qt and gtk tests with some simple opengl test are a must. (kiss!)

##########################################
QGears2:
Rendering: XRender Extension - Test: Gears

63.9869 Frames Per Second
64.164075 Frames Per Second
63.912375 Frames Per Second

Average: 64.02 Frames Per Second
##########################################

stefanwilkens · 2009-10-26 11:03:06

ohhh I hadn't even thought about GUI benchmarking, that's a great idea!

phoronix system info looks good, but it's a fairly huge dependency if it's only used for the system information overview. ideally I'd like to see something smaller, a bash / python / whateverlanguage script we can expand and adjust without redistribution limits.

Last edited by stefanwilkens (2009-10-26 11:05:55)

grey · 2009-10-26 12:53:05

Why do you want to use phoronix only for the system information overview and not for the actual tests? That's what phoronix is supposed to be: a relatively reliable system benchmark.
I used it about a year ago to silence some nagging doubts (yes, we are faster than Debian). It wasn't a painless user interface experience back then, but maybe things have improved, and/or we could provide a nicer front end.

stefanwilkens · 2009-10-26 13:00:34

grey wrote:

Why do you want to use phoronix only for the system information overview and not for the actual tests? That's what phoronix is supposed to be: a relatively reliable system benchmark.
I used it about a year ago to silence some nagging doubts (yes, we are faster than Debian). It wasn't a painless user interface experience back then, but maybe things have improved, and/or we could provide a nicer front end.

I think you misunderstood us here, we certainly want to use phoronix for its tests

What we want to do here is accumulate a scope of linux benchmarking tools and set a standard that users can use to compare system performance. To do that, we need something that shows system information.

Phoronix has the feature, but say you only want to benchmark hard drive performance or RAM bandwidth. It wouldn't be very KISS to install the entire suite just to test RAM and provide system information.

To solve that, I'm looking for a small shell script that provides system info. So that users that don't want to use the complete suite can use a stand-alone tool and add system information to that through this small shell script

Last edited by stefanwilkens (2009-10-26 13:04:17)

Mikko777 · 2009-10-26 13:02:57

stefanwilkens wrote:

ohhh I hadn't even thought about GUI benchmarking, that's a great idea!
phoronix system info looks good, but it's a fairly huge dependency if it's only used for the system information overview. ideally I'd like to see something smaller, a bash / python / whateverlanguage script we can expand and adjust without redistribution limits.

Well thats afaik exactly what pts is?

php script to benchmark using external apps which it installs with pacman. gtk gui is optional and pretty useless

stefanwilkens · 2009-10-26 13:12:25

Mikko777 wrote:

stefanwilkens wrote:
ohhh I hadn't even thought about GUI benchmarking, that's a great idea!
phoronix system info looks good, but it's a fairly huge dependency if it's only used for the system information overview. ideally I'd like to see something smaller, a bash / python / whateverlanguage script we can expand and adjust without redistribution limits.
Well thats afaik exactly what pts is?
php script to benchmark using external apps which it installs with pacman. gtk gui is optional and pretty useless

you're absolutely right, but I would like to adjust the system information output to a more humanly readable style. I would also like users who do not want to use the suite to be able to accumulate complete system information without having to install the entire suite. A small shell script seemed a good sollution to me, and it gives us the option to expand it.

I have thought about simply using phoronix itself, as it seems to include a large pile of tests with various goals. But parts of the phoronix suite seem broken due to incompatabillity between php-gtk / php 5.3 / phoronix?

What do you think about this?

gog · 2009-10-26 13:30:55

include ck's benchmark thing

dont include glgears

stefanwilkens · 2009-10-26 13:35:06

gog wrote:

include ck's benchmark thing
dont include glgears

I'm assuming you mean Interbench? (http://users.on.net/~ckolivas/interbench/)

Fackamato · 2009-10-26 18:25:05

Nice initiative! I'm interested in benchmarking filesystems with bonnie++. What should our parameters be?

stefanwilkens · 2009-10-26 19:14:04

Fackamato wrote:

Nice initiative! I'm interested in benchmarking filesystems with bonnie++. What should our parameters be?

Investigate and suggest something I'm as new to bonnie++ as you are.

the manpages suggest some things to look at:
http://linux.die.net/man/8/bonnie++

Last edited by stefanwilkens (2009-10-26 19:15:06)

Fackamato · 2009-10-26 19:42:18

stefanwilkens wrote:

Fackamato wrote:
Nice initiative! I'm interested in benchmarking filesystems with bonnie++. What should our parameters be?
Investigate and suggest something I'm as new to bonnie++ as you are.
the manpages suggest some things to look at:
http://linux.die.net/man/8/bonnie++

This is an example I came up with:

[fackamato@fackamato-laptop ~]$ bonnie++ -d ./temp/ -s 11944 -n 4 -m Inspiron1520 -x 2

name,file_size,putc,putc_cpu,put_block,put_block_cpu,rewrite,rewrite_cpu,getc,getc_cpu,get_block,get_block_cpu,seeks,seeks_cpu,num_files,seq_create,seq_create_cpu,seq_stat,seq_stat_cpu,seq_del,seq_del_cpu,ran_create,ran_create_cpu,ran_stat,ran_stat_cpu,ran_del,ran_del_cpu
Writing with putc()...done
Writing intelligently...done
Rewriting...done
Reading with getc()...done
Reading intelligently...done
start 'em...done...done...done...
Create files in sequential order...done.
Stat files in sequential order...done.
Delete files in sequential order...done.
Create files in random order...done.
Stat files in random order...done.
Delete files in random order...done.
Inspiron1520,11944M,80396,98,58611,7,30306,4,54864,67,181251,9,8119.3,13,4,+++++,+++,+++++,+++,+++++,+++,+++++,+++,+++++,+++,+++++,+++

I'm not sure how to interpret that. Anyway, the filesystem used is ext4 (relatime, nobarrier, writeback) on a SSD. bfs is used.

It really takes a long time (15 mins?) to bench, if someone can find out how to benchmark with -s 500 or -s 1024, let me know. bonnie++ really wants twice the RAM (in my case, 12 gigs) when it does its thing. (although, I guess I could boot with the kernel param mem=1024 or similar...,)

Fackamato · 2009-10-28 00:40:26

I just ran an interbench on 2.6.31-ARCH vs 2.6.31-bfs. I booted the kernels with this command line:

kernel /vmlinuz26 root=/dev/disk/by-uuid/ef15c6e6-953e-471a-9059-9f756b4b89c9 ro quiet mem=1024m single

I rmmod:ed all the modules I didn't need (sound, usb, lan/wlan etc) and killed all unecessary processes. By the way, when booting into single mode, shouldn't /etc/rc.local not be executed?

These are the results (only one run each):

Using 2368176 loops per ms, running every load for 30 seconds
Benchmarking kernel 2.6.31-ARCH at datestamp 200910272324

--- Benchmarking simulated cpu of Audio real time in the presence of simulated ---
Load    Latency +/- SD (us)  Max Latency   % Desired CPU  % Deadlines Met
None         66 +/- 66.3          75         100            100
Video         31 +/- 32.1         181         100            100
X         47 +/- 48.4          60         100            100
Burn          7 +/- 7.52           9         100            100
Write         63 +/- 68.3         218         100            100
Read         53 +/- 54.2          87         100            100
Compile         16 +/- 17.5          41         100            100

--- Benchmarking simulated cpu of Video real time in the presence of simulated ---
Load    Latency +/- SD (us)  Max Latency   % Desired CPU  % Deadlines Met
None         56 +/- 56.7         222         100            100
X         42 +/- 44.4          85         100            100
Burn         12 +/- 15.5          30         100            100
Write         61 +/- 66.6         512         100            100
Read         51 +/- 52.8          82         100            100
Compile         21 +/- 28.8         426         100            100

Using 2368176 loops per ms, running every load for 30 seconds
Benchmarking kernel 2.6.31-bfs at datestamp 200910272240

--- Benchmarking simulated cpu of Audio real time in the presence of simulated ---
Load    Latency +/- SD (us)  Max Latency   % Desired CPU  % Deadlines Met
None          7 +/- 7.89          10         100            100
Video          6 +/- 9.14         155         100            100
X          7 +/- 7.24          12         100            100
Burn          6 +/- 6.47          13         100            100
Write         16 +/- 16.7          31         100            100
Read         13 +/- 13.6          30         100            100
Compile         15 +/- 16.5         108         100            100

--- Benchmarking simulated cpu of Video real time in the presence of simulated ---
Load    Latency +/- SD (us)  Max Latency   % Desired CPU  % Deadlines Met
None          7 +/- 7.48          20         100            100
X          6 +/- 7.12          24         100            100
Burn          6 +/- 6.55          16         100            100
Write         14 +/- 17.8         214         100            100
Read         12 +/- 13.3          49         100            100
Compile         12 +/- 13.9         122         100            100

So a quick interbench shows -bfs is faster on my system. I can't confirm anything from a user perspective though.

Fackamato · 2010-08-18 16:58:09

Shameless bump, I'm going to run the same benchmark on 2.6.35-ck now, on the same hardware.

Edit: results:

Using 2505074 loops per ms, running every load for 30 seconds
Benchmarking kernel 2.6.35-ck at datestamp 201008181800

--- Benchmarking simulated cpu of Audio in the presence of simulated ---
Load    Latency +/- SD (ms)  Max Latency   % Desired CPU  % Deadlines Met
None       0.0 +/- 0.0        0.0         100            100
Video       0.0 +/- 0.0        0.0         100            100
X       0.0 +/- 0.0        0.0         100            100
Burn       0.0 +/- 0.0        0.0         100            100
Write       0.0 +/- 0.0        0.0         100            100
Read       0.0 +/- 0.0        0.0         100            100
Compile       0.0 +/- 0.0        0.0         100            100

--- Benchmarking simulated cpu of Video in the presence of simulated ---
Load    Latency +/- SD (ms)  Max Latency   % Desired CPU  % Deadlines Met
None       0.0 +/- 0.0        0.0         100            100
X       0.0 +/- 0.0        0.0         100            100
Burn      16.4 +/- 16.6      30.8         100           1.87
Write       0.0 +/- 0.1        1.9         100            100
Read       0.0 +/- 0.0        0.1         100            100
Compile      16.4 +/- 16.9      33.5        96.3           4.94

--- Benchmarking simulated cpu of X in the presence of simulated ---
Load    Latency +/- SD (ms)  Max Latency   % Desired CPU  % Deadlines Met
None       0.0 +/- 0.0        0.0         100            100
Video       0.0 +/- 0.1        2.0         100             99
Burn      56.1 +/- 77.7     171.0        19.2           8.51
Write       0.1 +/- 1.4       23.0        98.7           97.7
Read       0.3 +/- 1.3        7.0          92             89
Compile      65.4 +/- 90.1     210.0        17.2           6.76

--- Benchmarking simulated cpu of Gaming in the presence of simulated ---
Load    Latency +/- SD (ms)  Max Latency   % Desired CPU
None       0.0 +/- 0.0        0.0         100
Video       0.5 +/- 0.6        1.2        99.5
X       1.8 +/- 2.5        5.0        98.3
Burn     161.4 +/- 162.5    173.6        38.3
Write       1.2 +/- 2.8       28.7        98.8
Read       6.1 +/- 6.1        6.4        94.3
Compile     192.1 +/- 194.7    247.7        34.2

Last edited by Fackamato (2010-08-18 17:21:42)

MutantTurkey · 2010-08-20 02:40:33

I don't suppose anyone has suggested making an Arch Linux Benchmarking Utility?

If that idea takes off it would be great because:

A) standardized
B) simple to get, well documented (in the repos)
C) designed in a KISS manner
1) having a plugin system, it would have a TON of optional depends, but no required depends (that way people looking for like, server benchmarks don't need openGL)
2) one unified package, not 20 different ones
3) take the best benchmarking ideas from other projects and unite them into one suite or program.
4) obviously keeping it minimal is would be cool
D) would give everyone something to be proud of (running out of things to say )
E) could obviously help everyone using arch easily find out what performance they are getting, and where they are lacking. so that people will better understand the limitations of their specific computer

I would be interested in see/writing this.

MutantTurkey

Arch Linux

#1 2009-10-26 02:37:55

[Benchmarking] Let's organise and standardize

#2 2009-10-26 03:12:04

Re: [Benchmarking] Let's organise and standardize

#3 2009-10-26 10:50:36

Re: [Benchmarking] Let's organise and standardize

#4 2009-10-26 10:57:49

Re: [Benchmarking] Let's organise and standardize

#5 2009-10-26 11:03:06

Re: [Benchmarking] Let's organise and standardize

#6 2009-10-26 12:53:05

Re: [Benchmarking] Let's organise and standardize

#7 2009-10-26 13:00:34

Re: [Benchmarking] Let's organise and standardize

#8 2009-10-26 13:02:57

Re: [Benchmarking] Let's organise and standardize

#9 2009-10-26 13:12:25

Re: [Benchmarking] Let's organise and standardize

#10 2009-10-26 13:30:55

Re: [Benchmarking] Let's organise and standardize

#11 2009-10-26 13:35:06

Re: [Benchmarking] Let's organise and standardize

#12 2009-10-26 18:25:05

Re: [Benchmarking] Let's organise and standardize

#13 2009-10-26 19:14:04

Re: [Benchmarking] Let's organise and standardize

#14 2009-10-26 19:42:18

Re: [Benchmarking] Let's organise and standardize

#15 2009-10-28 00:40:26

Re: [Benchmarking] Let's organise and standardize

#16 2010-08-18 16:58:09

Re: [Benchmarking] Let's organise and standardize

#17 2010-08-20 02:40:33

Re: [Benchmarking] Let's organise and standardize

Board footer