( FIXED ) Linux CPU Scheduler Optimized for Nehalem i7?

hunterthomson · 2010-02-01 10:29:27

It seems Intel has been working with the OpenSolaris guys to get there OS optimized for the Nehalem Microarchitecture. What is Linux doing to take advantage of the new features available on these chips?

It seems that Intel has helped OpenSolaris take full advantage of their new Nehalem Microarchitecture. They have setup the dispatcher to allocate memory closest to the CPU core that will compute it. Solaris will do it's best to consolidate process threads so as to get the CPU into Turbo Mode more often and put the other cores to sleep so as to give the extra power two the working cores by enabling them to go into Turbo Mode. Solaris will also make use of the new information the Nehalem chips reports about errors and incorporate them in it's Fault Management Architecture (FMA). A quick Google search will yield much information on Solaris optimizations but nothing about Linux optimizations for Nehalem.
overwith
Are these sort of improvements taking place in Linux or dose Linux already have these features?

Do you think Solaris makes better use of the Nehalem architecture or dose Linux?

--------------
I have to say all the media hype has gotten to me and I am seriously thinking about running OpenSolaris as my Host OS on my new i7 computer.

Last edited by hunterthomson (2010-02-18 11:46:47)

Acecero · 2010-02-01 10:44:51

It would be easier to tell the difference by benchmarking with the same apps and same optimization on both systems, if Linux supports the Nehalem architecture. Try it and see for yourself.

hunterthomson · 2010-02-01 11:16:22

Ya, it supports it. The main things that interest me is the Solaris Dispatcher being aware of the Nehalem architecture. So it works to optimize the location of data in memory and what cores to processes threads so as to keep a core or two in a C state. Giving the working a nice Turbo Boost.

If Linux were to put the data needed by a thread in a memory location that is not directly mapped to the core processing the thread it could slow down that thread to a measurable degree. Also, if Linux doesn't try to put say the only 4 threads it is working on, onto just 2 Hyper threaded cores then it would not be getting the full advantage of the Turbo Boost.... Like if it decided to put 2 Threads on One core then 1 Thread on Two more cores. Only one core would be left idle and put in a C state. However, if it put 2 Threads on One core and the other 2 on a second core then that would leave 2 cores idle and put in a C state. This would enable the two working cores to get a nice Turbo Boost.

Last edited by hunterthomson (2010-02-01 11:27:17)

Acecero · 2010-02-01 11:43:43

It sounds like you're hoping for Linux to have better memory allocation. From your example you would probably hope to divide those 4 threads evenly between 4 separate cores within the i7 to get a boost in speed for running separate tasks. Does the Dispatcher actually let you control the threads and where they go in memory? If I have 8 threads, would I be able to split two threads for each core to work on? How does that work?

hunterthomson · 2010-02-01 12:19:27

GOOD 2 cores left idle so the 2 working cores are in Turbo Boost like 2.60Ghz
T T T T
| | | | | | | |
| | | |
C C C C

BAD 1 core left idel smaller Tubo Boost like 1.87Ghz..... These #Ghz are not correct I am lazy but they are close
T T T T
| | | | | | | |
| | | |
C C C C

Ya I don't know what the Linux dispatcher dose. I just know that the Solaris one knows to put the threads on as few cores as posable. Because this was never an issue before now I don't think the Linux Dispatcher cares at all.

Last edited by hunterthomson (2010-02-01 23:33:11)

hunterthomson · 2010-02-04 11:49:26

Well, It seems that Linux has had optimization for NUMA for a long time.

From what I have read the Linux Kernel is extremely multi threaded and uses all the power of Multi-core CUPs. Optimized for SMP.

However, that is NOT how you get all the power out of a Nehalem CPU. It is critical you run all the treads on as few cores as posable. If you have 8 thread and logical 8 cores... No there would be no differences between OpenSolaris & Linux. However, if you don't and your still working all 4 physical cores your losing out on some major Ghz.

I still don't know if it has fewer threads then Cores, if it tries to get the threads on as few cores as it can though.
It may just do the naturaly. Like it numbers all the 8 Logical cores starting at on 12345678. Then sends the fist thread to 1, the next to 2, the next, to 3,... But computers don't run like that in the real world though do they?
Like thread on core 1 could be done in like 1ms, 2 in 10ms, 3 in 4miniuts, 4 in 1ms, bla bla.... Then it start back at core 1 and sends it the next thread. Then core 2 is still buissy so it send the tread to core 3. Then core 2 is done but the Kernel sends the next thread to core 4 instead of core 2...
Then the core in real life could end up in a situation like this most of the time....

CPU = Intel Core i7-720QM Quad Nehalem micro-architecture (Bace CPU Clock 1.6Ghz Max Turbo Bost 2.8Ghz)

T = thread
C = Core

This is fine for old school multi-core CPUs but honorable for a Nehalem
In this case the i7-720QM would still Tubo Bost just not much at all.. It would run at like 1.7Ghz
T T T T
| | | | | | | |
| | | |
C C C C

OpenSolaris would detect this Nehalem architecture and make sure to schedule the threads on as few Physical Cores as it can. So, with the same 4 thread, the thread to Core load would look like this....

With running like this the CPU would be Turbo Boosted up to about 2.6Ghz. So the 4 theads would be done much faster.
T T T T
| | | | | | | |
| | | |
C C C C

--------------
Some relavant Docs I found so far...
http://software.intel.com/en-us/article … -clusters/

http://aplawrence.com/Linux/linux26_features.html

http://www.cs.sfu.ca/~fedorova/papers/T … uation.pdf

Last edited by hunterthomson (2010-02-04 12:34:20)

axel668 · 2010-02-04 12:14:29

http://www.phoronix.com/scan.php?page=a … rnel&num=1
(there might be other benchmarks at Phoronix, or you could run your own benchmarks in similar VMs)

Otherwise, of course Linux (Arch especially !!!) is better than Oracle Solaris

Last edited by axel668 (2010-02-04 12:15:18)

hunterthomson · 2010-02-04 12:42:35

Ya, aside form this.... Owe and ZFS... MAN I sure wish we had ZFS.
OpenSolaris just seems like a pain in the ass. It is so strict about C and C++ making is hard to get applications installed on it. They don't have nearly as many applications as Linux. There are like no Cracking tools... Well some but bu tnot much and no wireless cracking tools. Come on you just have to have aircrack-ng. + I really want to learn more Linux and get RHCE. I know that if I switch to OpenSolaris I will basically stop learning Linux. I also just really like using Archlinux. It is sooooo nice. It just dose what I want it to do. I don't have to "Learn" Archlinux. I can just set-up Archlinux the way I like it and "It Just Works."

So, ya I am probably not going to switch but I'd still like to find out if Linux dose what it needs to do for Nehalem or if anyone is working on it. I guess once these i7 CPU's are more common Linux will quickly adapt and problay do it better then the OpenSolaris guys. Simply because Linux has more Dev's.

-------
Owe about the benchmarking. My main concern is knowing that the CPU is being used to it's full potential not if "In then end Linux is still faster." Though I guess that should be the Real concern.

Last edited by hunterthomson (2010-02-04 12:58:59)

hunterthomson · 2010-02-04 12:56:42

Owe, wow, I just looked at thouse benchmarks !

Ya, forget Solaris
Linux is WAY faster in most tests. On the tests that it did slower, it looked like it was really the File-system that was being tested and not the Kernel. Simply using the right File-system for the right directories would solve that.

Also, I don't know why people who have a 64bit CPU (which is like everyone now) use the 32bit OS. That is just dumb IMHO. Linux64 is way stable. I've never had a problem with 64bit.

Last edited by hunterthomson (2010-02-04 13:00:51)

rusty99 · 2010-02-04 15:27:36

Be wary of basing any sort of conclusion on Phoronix benchmarks...

hunterthomson · 2010-02-05 03:09:29

rusty99 wrote:

Be wary of basing any sort of conclusion on Phoronix benchmarks...

Owe for sure. Like most of the test were not a test of the Kernel.

Also, with the way a Nehalem CPU works with Turbo Mode, all the same tests could yield much different results.

I still have not found anything showing that the Linux Kernel will make the best use of the CPU and Turbo Mode. RedHat and SUSE are using some hacking stuff to control Turbo Mode. However, nether of the methods deal with the CPU scheduling that is vary important in a Nehalem CPU.

I am still tempted to use OpenSolaris as a Host OS. Then do most work in a VM of RedHat (to study for RHCE). The main reason I got a i7-720QM is so I have 8 Logical cores to dedicate to several Guest OS's running at the same time. So, doing work on a Guest RedHat doesn't bother me that much.

However, even the visualization tec that Solaris uses... XEN ... from what I gather is not nearly as good as KVM.

Last edited by hunterthomson (2010-02-05 03:14:52)

wuischke · 2010-02-05 09:25:30

Pardon me, but why don't you just install Solaris and try it for your particular use case? It takes at most a day to set up and will give you better answers than all hearsay about performance.

If it seems too slow to you, install Linux and see if it runs better.

btw: Someone should correct the spelling of Solaris in the title. I though you were trolling upon reading the title.

hunterthomson · 2010-02-05 10:06:17

wuischke wrote:

Pardon me, but why don't you just install Solaris and try it for your particular use case? It takes at most a day to set up and will give you better answers than all hearsay about performance.
If it seems too slow to you, install Linux and see if it runs better.
btw: Someone should correct the spelling of Solaris in the title. I though you were trolling upon reading the title.

Nope, really what you just posted is trolling.

I am not going by hearsay. I am reading document written by Intel and OpenSolaris. This thread is because I can't find any documents about this issue with Linux. As I have said in the post that you aparintly didn't read. I don't really care what OS is faster I want to know if Linux handles CPU Scheduling on Nehalem like OpenSolaris dose.

Do You know how Linux CPU Scheduling works? Do you know if it knows how to best handle a Nehalem CPU?
If not don't post.

Last edited by hunterthomson (2010-02-05 10:46:45)

hunterthomson · 2010-02-05 10:45:45

Ok changed the name of the thread so I don't sound like I am trolling.
I really just want to know if Linux dose what the OpenSolaris scheduler is doing.

flamelab · 2010-02-05 10:59:40

I really don't know what that dispatcher issue is, but the performance and system responsiveness in Linux, using i5 750, is optimal/excellent (x86_64 of course, EXT-4 fs)

Last edited by flamelab (2010-02-05 11:00:05)

kaaposc · 2010-02-05 11:03:26

s/dose/does/g
7 times on this page..

hunterthomson · 2010-02-05 11:57:12

Well, I give up.

In conclusion.

Linux is optimized for NUMA
Turbo Mode dose work in Linux
Linux CPU scheduling is not optimized for Nehalem

When more people are using i7 CPUs then this features will probably be added in.

flamelab · 2010-02-05 12:42:20

It IS fine and VERY good. The performance is VERY good. I can't understand what you want to do.

And it is "does" not "dose".

hunterthomson · 2010-02-16 01:13:17

Just to end this with the facts.

Yep, the Linux kernel does not take full advantage of the Nehalem CPU's.
The CPU scheduler is constantly loading more physical CPU cores then it needs to.
This is resulting in a substantial loss of computing power. If I knew how to code in C I'd get to work fixing it but I don't.

If you have a Dule Core i3 i5 or i7 then this doesn't matter that much to you. However, if you have a Quad Core it makes a vary big differences.

i.e. The loading of the CPU is constantly is a state like this

T= Thread
C= Physical Core
| | = The two Logical Cores for each Physical Core

T T T
| | | | | | | |
| | | |
C C C C

Last edited by hunterthomson (2010-02-16 01:21:54)

Themaister · 2010-02-16 11:10:17

So, you mean that the scheduler is not optimized for the turbo mode because what could have been done with e.g. 2 cores (to trigger turbo mode) is offloaded to 4 cores? I guess I've never noticed since I overclock and have turned turbo mode off. For stuff that is actually pushing the i7, the scheduler is mighty fine though.

thestinger · 2010-02-16 21:21:24

I can't think of any speed critical programs I have that only use one or two cores... firefox benchmarks? (I use chromium now though)

hunterthomson · 2010-02-17 04:18:08

{I am using i7z to watch the speed of the CPU}

Yes, Nehalem CPU's use Tubo Mode (self over-clock) based on how many cores are inactive. This is what the Linux Kernel needs to adapt to.

They "self over-clock" even with all 4 cores active but not by much. This level of over-clocking is not trivial at all. On my i7-720QM. The base speed is 1.6Ghz and the max over-clock is 2.8Ghz.

One major problem I have is that I run VM's all the time. I give 2 cores to each VM. What is happeing almost all the time is that the host OS "Arch" is useing 1-2 cores for it self. Whle the VM is Maxing out 2 cores. My CPU should be running the 2 cores the VM is on at about 2.6Ghz. However, because Linux is spliting the thread accross 3-4 physical cores the 2 cores that the VM has is only running at about 1.8G-2.6Ghz. This is a major loss of power for the VM. Instead of running at 2.6Ghz all the time it is doing work "at load". It is running at an average speed of 1.9Ghz "at load".

What is even worse then that. Is that the Host Linux OS switches the loaded cores form say a "2 in use" to "4 in use" to "3 in use" causing the VM to have it's CPU clock speed going form 2.6Ghz to 1.7Ghz to 2Ghz..... All the while the VM is just doing one thing say compiling a program. This doesn't cause any "Problems" but it is an unnecessary loss of power. it should just be loading One Physical core and staying a 2.6Ghz the whole time. {not 2.8Ghz because the host OS is still using a core}

----------------------------

Aside form that the other point of Turbo Mode (self over-clock) is to speed up the normally slowest part of an application i.e. the single threaded part. There by speeding up the total run time of a process by a vary large amount.

----------------------------

It seems as though the Linux Kernel is programed to use every other core when it is not using all 8. Like, it looks like it "tries" to use 0,2,4,6. This is the worst thing possible for a Nehalem CPU. However, it is probably good behavior for a CPU that doesn't use Turbo-Mode.

----------------------------
----------------------------

Also, this is not about whether the Linux CPU scheduler is "Fine" or if it is "Fast enough" or "Still vary fast".

It is about the fact that the Linux Kernel "Should" but getting a whole lot more power out of the CPU.

Last edited by hunterthomson (2010-02-17 04:35:19)

R00KIE · 2010-02-17 13:02:35

So what you are saying is that the scheduler when confronted with 4 tasks should load only one or two cores so the cpu can overclock on it's own.

Then what you want is that 4 tasks will have to share the resources of one or two cores, things like L1 and L2 cache and force the cpus to do more context switches just so you can watch it overclock. Getting the whole power of the cpu means putting _all_ the cores and logic working.

Also spreading the tasks by the "real" cores is the logic choice, hyperthreading is just some fancy trickery to get some otherwise unused units in a core busy, thus gaining a bit more in performance (but not that much anyway).

Think of an analogy, 4 people doing one task, each at normal speed, do as much if not more work that 2 people doing 2 tasks each working at twice the normal speed.

You're not even getting automatic overclocks to twice the normal speed, so you still expect 2 overclocked cores to be faster than 4 at normal speed?. Anyway you could always try to boot with maxcpus=2 and get that nice turbo-boost.

hunterthomson · 2010-02-18 04:17:05

Well I found out the problem was worse then I originally thought it was. I was looking at the
"True Frequency *****Mhz (Intel specifies largest of below to be running Freq)"
However, that is not the running Freq. The Freq's reported for each of the cores really is the Freq that core is running at..... Bla bla,

Long explanation short. Most of the time my VM's were under load the core that was running their processes was really Under-clocked. Really the cores were running at owe 900Mhz! most of the time. After trying different things. I have now got a much better setup. Now I an average load of the core running the VM's at 2.6Ghz. Even with the load being split across more Physical cores then needed.

....Boy it is hard to put this stuff down in a short and understandable way....
Pcore = physical core
Lcore = Logical core

Before the only Pcore that would ever really get above the 1.6Ghz for any real task other then compiling on the Host system with 6 Threads was core0(I set the make.conf to use 6 threads not 8 or 9 I like to have a little head room to do other things) Most of the time all the Pcores would be clocked really low like <1Ghz ! even under load. It would take some really heavy lifting to get the Pcores up to there standard Freq of 1.6Ghz.

So, as it turns out running cpufreq and leaving it all at the defaults with the performance governor set fixes the problem. That is kind of strange given that the cpufreq-info says it can only clock up to 1.6Ghz..... Hum, maybe the cpufreq is needed to get it up to the 1.6Ghz then the Intel Turbo Boost adds on it's multiplier to bring the tottal Freq up to where it should be under load... >2Ghz.... Just guessing on that....

Now with that cpufreq setup. Anytime the core gets a load the Freq goes up quite a bit. Now if a VM with 2 Lcores is under heavy load... or any load really. The Pcores running the processes get Freq boost up to 2.6Ghz and says there the whole time until the processes is done. Or if I say start Firefox the cores doing that will get a Freq boost >2Ghz. If there is any load from anything the Freq gets boosted >2Ghz. Then when the load is gone the Freq drops back down to <1Ghz.

So, in conclusion running cpufreq with all default settings and with the performance governor makes the CPU behave as it should. The CPU secluding is still not as good as it should be but at least with this setup it dosen't seem to matter that much. Maybe with the added L1 and L2 caches having it be just 2.6Ghz insead of 2.8Ghz may be just as fast if not faster. However, without this cpufreq fix the CPU running at 900Mhz is WAY, way slower then if was running at 2.8Ghz and no amount of L1 or L2 cache in the world would change that.

Performance with EVERYTHING is 10x better. I can now watch flash video with no problem, in full screen too! everything is soooo much better.
----------------------------------------------------------------

ROOKIE: It sounds as though you think that the Logical Core Hyper-threading is just some trickery to make use of unused core resources.

Like if one Physical Core is running one tread and that thread is only using say 80% of the core's total resources. Then the CPU pretends there is another Logical core and gives it the rest, the 20%, of the Physical core's resources. For a total of 100%.

That is not how Hyper-threading is implemented in Nehalem.
Each Physical Core really can do 2 threads at the same time almost as fast as it can do 1 thread. You are right it dose share the L1 and L2 cache so it can run slightly slower but not by much at all. The L1,L2, and L3 caches were made extra big for the Nehalem CPUs because the CPU manufacturer is the one that designed the CPU to do this Hyper-threading. It is not some third party hack. Intel put more cache because each core can do 2 threads at the same time.

Last edited by hunterthomson (2010-02-18 05:28:24)

hunterthomson · 2010-02-18 11:46:20

Hum, after doing some more reading and getting no definitive answers yet.

It seems you are correct to an extent. While it dose mostly make use of otherwise unused CPU resources, it is also optimized for this activity. So, the total processor power is more with hyper-threading. On the Pentium 4, which is a much more infantile implementation, Intel claims a 30% gain. In the Nehalem I would imagine (I have not yet found any claims or definitive answers yet) a much higher gain.

When say 2 threads are run on just 1 Pcore, that Pcore would run at 2.8Ghz. Though, if they were run on 2 Pcores then each Pcore would run at 2.6Ghz. So ya, that is much better. 2 Physical-cores running at 2.6Ghz is better then 1 Pcore running at 2.8Ghz.

So, I guess Linux CPU scheduling is better then I thought.
It seems that all my problems are solved by running cpufreq with the Performance governor.

Before my system was running like crap because it was running at <1Ghz almost all the time. I was thinking then that Turbo Boost only worked if things were on as few cores as posable. I see now that is not the case. Now that the Turbo Boost is getting the CPU cores >2Ghz whenever it is working my system runs way faster.

Last edited by hunterthomson (2010-02-18 11:51:38)

Arch Linux

#1 2010-02-01 10:29:27

( FIXED ) Linux CPU Scheduler Optimized for Nehalem i7?

#2 2010-02-01 10:44:51

Re: ( FIXED ) Linux CPU Scheduler Optimized for Nehalem i7?

#3 2010-02-01 11:16:22

Re: ( FIXED ) Linux CPU Scheduler Optimized for Nehalem i7?

#4 2010-02-01 11:43:43

Re: ( FIXED ) Linux CPU Scheduler Optimized for Nehalem i7?

#5 2010-02-01 12:19:27

Re: ( FIXED ) Linux CPU Scheduler Optimized for Nehalem i7?

#6 2010-02-04 11:49:26

Re: ( FIXED ) Linux CPU Scheduler Optimized for Nehalem i7?

#7 2010-02-04 12:14:29

Re: ( FIXED ) Linux CPU Scheduler Optimized for Nehalem i7?

#8 2010-02-04 12:42:35

Re: ( FIXED ) Linux CPU Scheduler Optimized for Nehalem i7?

#9 2010-02-04 12:56:42

Re: ( FIXED ) Linux CPU Scheduler Optimized for Nehalem i7?

#10 2010-02-04 15:27:36

Re: ( FIXED ) Linux CPU Scheduler Optimized for Nehalem i7?

#11 2010-02-05 03:09:29

Re: ( FIXED ) Linux CPU Scheduler Optimized for Nehalem i7?

#12 2010-02-05 09:25:30

Re: ( FIXED ) Linux CPU Scheduler Optimized for Nehalem i7?

#13 2010-02-05 10:06:17

Re: ( FIXED ) Linux CPU Scheduler Optimized for Nehalem i7?

#14 2010-02-05 10:45:45

Re: ( FIXED ) Linux CPU Scheduler Optimized for Nehalem i7?

#15 2010-02-05 10:59:40

Re: ( FIXED ) Linux CPU Scheduler Optimized for Nehalem i7?

#16 2010-02-05 11:03:26

Re: ( FIXED ) Linux CPU Scheduler Optimized for Nehalem i7?

#17 2010-02-05 11:57:12

Re: ( FIXED ) Linux CPU Scheduler Optimized for Nehalem i7?

#18 2010-02-05 12:42:20

Re: ( FIXED ) Linux CPU Scheduler Optimized for Nehalem i7?

#19 2010-02-16 01:13:17

Re: ( FIXED ) Linux CPU Scheduler Optimized for Nehalem i7?

#20 2010-02-16 11:10:17

Re: ( FIXED ) Linux CPU Scheduler Optimized for Nehalem i7?

#21 2010-02-16 21:21:24

Re: ( FIXED ) Linux CPU Scheduler Optimized for Nehalem i7?

#22 2010-02-17 04:18:08

Re: ( FIXED ) Linux CPU Scheduler Optimized for Nehalem i7?

#23 2010-02-17 13:02:35

Re: ( FIXED ) Linux CPU Scheduler Optimized for Nehalem i7?

#24 2010-02-18 04:17:05

Re: ( FIXED ) Linux CPU Scheduler Optimized for Nehalem i7?

#25 2010-02-18 11:46:20

Re: ( FIXED ) Linux CPU Scheduler Optimized for Nehalem i7?

Board footer