You are not logged in.

#1 2012-12-29 04:42:28

[Steven_M]
Member
Registered: 2012-12-29
Posts: 6

MAKEFLAGS -J value

Hi all, I find the statement (here: https://wiki.archlinux.org/index.php/Ma … #MAKEFLAGS ) "Typically -j2, plus 1 for each additional core/processor is an adequate choice" confusing. For starters, what is meant by "additional cores", since everyone has to have at least one core, does "additional cores" refer to the total number of cores minus 1 core, or does it simply mean the total number of cores?

For example, my CPU has 4 cores; this could be taken to mean that I have 3 additional cores over and above my first core. In this scenario, if we follow the formula "-j2, plus 1 for each additional core/processor", then I should set the value of -J to -J5. Or if we take "additional cores" to in fact mean the total number of cores, then following the formula "-j2, plus 1 for each additional core/processor" I should set the -J value to -J6.

This confusion is further compounded by the fact that I remember a few years ago (back in my Gentoo days), the principle was to simply set the -J value to the number of cores plus 1. This would mean that since I have 4 cores, I would set the -J value to -J5.

Can anyone help me interpret these instructions?

Offline

#2 2012-12-29 05:05:46

anonymous_user
Member
Registered: 2009-08-28
Posts: 3,059

Re: MAKEFLAGS -J value

[Steven_M] wrote:

since everyone has to have at least one core, does "additional cores" refer to the total number of cores minus 1 core, or does it simply mean the total number of cores?

I believe it is the former. As you say:

[Steven_M] wrote:

I remember a few years ago (back in my Gentoo days), the principle was to simply set the -J value to the number of cores plus 1. This would mean that since I have 4 cores, I would set the -J value to -J5.

If you think it would help, you could just edit the wiki and restate the sentence to make it clearer. And if you think you are not "qualified" to make such a change, remember that the wiki is a community effort and other users can further edit it if necessary.

Offline

#3 2012-12-29 06:06:49

MrCode
Member
Registered: 2010-02-06
Posts: 373

Re: MAKEFLAGS -J value

I use -j4, and my CPU is an Intel Core i5 430M (dual-core with HT; 4 logical cores), but I also run linux-ck… tongue

To address the OP: I think it would be less ambiguously phrased as "make -j(n+1), where n is the number of (logical) cores your CPU has".

Last edited by MrCode (2012-12-29 06:08:18)

Offline

#4 2012-12-29 06:18:39

brebs
Member
Registered: 2007-04-03
Posts: 3,742

Re: MAKEFLAGS -J value

[Steven_M wrote:

plus 1

That's just because the default CPU scheduler is inferior to BFS, so it's shorthand for "Oh and add a bit to keep the CPUs busy, to make up for the CPU scheduler not properly doing its job of keeping the CPUs busy."

From FAQ:

This scheduler proves that the jobservers weren't at fault at all, because make -j4 on a quad core machine with BFS is faster than *any* choice of job numbers on CFS.

Your username contains square brackets - messes up quoting a bit neutral

Offline

#5 2012-12-29 07:45:36

[Steven_M]
Member
Registered: 2012-12-29
Posts: 6

Re: MAKEFLAGS -J value

Thanks to all of you for your responses, you've been a great help! smile

Offline

#6 2012-12-29 07:50:38

[Steven_M]
Member
Registered: 2012-12-29
Posts: 6

Re: MAKEFLAGS -J value

MrCode wrote:

To address the OP: I think it would be less ambiguously phrased as "make -j(n+1), where n is the number of (logical) cores your CPU has".

Thanks. Would you like to change the wiki page or shell I?

Offline

#7 2012-12-29 08:01:27

[Steven_M]
Member
Registered: 2012-12-29
Posts: 6

Re: MAKEFLAGS -J value

MrCode wrote:

but I also run linux-ck… tongue

What's linux-ck?

Offline

#8 2012-12-29 08:06:40

[Steven_M]
Member
Registered: 2012-12-29
Posts: 6

Re: MAKEFLAGS -J value

brebs wrote:

Your username contains square brackets - messes up quoting a bit neutral

Good point, I'll change it shortly. smile

Offline

#9 2012-12-29 10:09:15

graysky
Wiki Maintainer
From: :wq
Registered: 2008-12-01
Posts: 10,595
Website

Re: MAKEFLAGS -J value

[Steven_M] wrote:

What's linux-ck?

https://wiki.archlinux.org/index.php/Linux-ck


CPU-optimized Linux-ck packages @ Repo-ck  • AUR packagesZsh and other configs

Offline

#10 2012-12-29 12:26:25

Allan
Pacman
From: Brisbane, AU
Registered: 2007-06-09
Posts: 11,365
Website

Re: MAKEFLAGS -J value

MrCode wrote:

To address the OP: I think it would be less ambiguously phrased as "make -j(n+1), where n is the number of (logical) cores your CPU has".

I find that -j(1.5*n) works better for larger n (4 or more...)

Offline

#11 2012-12-29 16:42:28

anonymous_user
Member
Registered: 2009-08-28
Posts: 3,059

Re: MAKEFLAGS -J value

[Steven_M] wrote:

Good point, I'll change it shortly. smile

If you mean change your username, you cannot:

https://bbs.archlinux.org/viewtopic.php?id=47587

Offline

#12 2012-12-31 03:09:04

skiwi
Member
Registered: 2012-11-28
Posts: 32

Re: MAKEFLAGS -J value

Just use -jn with n the number of real cores you have. Note also the difference between "threads" and "cores". Threads are different items in /proc/cpuinfo, for the number of cores look at the number of different 'core id's in /proc/cpuinfo. Typically each cpu (=core) has a number of threads (typically 2 or 4), which are virtual cpus used to balance the workload. That means you can still do things, like use a browser, while compiling. At the same time a thread can use an entire cpu if no other thread is asking for it so you will not lose any performance. So, never use all threads, use all cpus. More is pointless imho. There might be a tiny 'start-up' time gain between processes by using n+1 if you compile something consisting of many small files, but I guess it would be negligible and you risk blocking the cpu/having a very slow computer during compilation.

-- edit : the additional cores in the original also refer additional to 1, so 4 cores implies -j4

Last edited by skiwi (2012-12-31 03:41:24)

Offline

#13 2012-12-31 06:07:39

Max-P
Member
Registered: 2011-06-11
Posts: 164

Re: MAKEFLAGS -J value

skiwi wrote:

Just use -jn with n the number of real cores you have. Note also the difference between "threads" and "cores". Threads are different items in /proc/cpuinfo, for the number of cores look at the number of different 'core id's in /proc/cpuinfo. Typically each cpu (=core) has a number of threads (typically 2 or 4), which are virtual cpus used to balance the workload. That means you can still do things, like use a browser, while compiling. At the same time a thread can use an entire cpu if no other thread is asking for it so you will not lose any performance. So, never use all threads, use all cpus. More is pointless imho. There might be a tiny 'start-up' time gain between processes by using n+1 if you compile something consisting of many small files, but I guess it would be negligible and you risk blocking the cpu/having a very slow computer during compilation.

-- edit : the additional cores in the original also refer additional to 1, so 4 cores implies -j4

Using -j8 on my hyperthreaded i7 CPU is somehow much faster than -j4. I have 4 cores, with 2 threads each. Using the total number of cores as seen by the OS can't be slower than using only real cores. Those CPUs can run multiple threads at the same time, as long as core ressources doesn't conflict. In the WORST case, it will be as slow as the real number of cores.

When building stuff, it looks like all threads can fully be used, and I have to put an extra fan near my laptop because the CPU gets extremely hot (97 degrees, 3 degrees under the shutdown point). When encoding x264 videos, it doesn't come near as hot, and ffmpeg never fill up all the CPUs, even with many jobs in the background. At some point, the total CPU load never seems to go higher than 50-60% of the 8 cores. That's when CPU ressources conflict, and in that case I'm stuck to my 4 physical cores. But when using make, it definitely uses all 8 threads of my CPU and greatly improve performance.

make -j 8 shows a nice 99% CPU usage on all cores in htop smile

Offline

#14 2012-12-31 13:03:52

graysky
Wiki Maintainer
From: :wq
Registered: 2008-12-01
Posts: 10,595
Website

Re: MAKEFLAGS -J value

This thread is full of mis-information... here is an experiment that answer the OP's question: run a series of compiles varying the make flags and inspect the results.  Presumably, faster compilation times will be a function of the make flags used when nothing else is changed.

Test machine: i7-3770K@3.5 GHz. This is a quad core chip with 8 logical CPUs (4 real + 4 hyperthreaded).  Note that I am running a patched kernel (linux-ck v3.7.1-3) which uses the Con Kolivas' BFS not the CFS in mainline.

Test compile: Samba v3.6.10 was compiled five times for each makeflag.  I looked at -j4, -j8, -j9, -j12, -j16, -j20, -j24, -j28, and -j30. 

Results:  Running make with either 8 or 9 threads gave the fastest compilation time; they are not statistically significant from each other.  All other values gave longer compile times.

Source code: Available on github.

Tables and figures:
table.jpg
barchart.jpg

EDIT: Read the 3rd paragraph in this document written by Kolivas explaining some of the confusion and consider running a BFS patched kernel.

Last edited by graysky (2012-12-31 14:41:46)


CPU-optimized Linux-ck packages @ Repo-ck  • AUR packagesZsh and other configs

Offline

#15 2012-12-31 14:05:21

mariusmeyer
Member
From: Norway
Registered: 2009-04-25
Posts: 244

Re: MAKEFLAGS -J value

Great, systematic review graysky smile

Offline

#16 2012-12-31 14:50:58

Ramses de Norre
Member
From: Leuven - Belgium
Registered: 2007-03-27
Posts: 1,289

Re: MAKEFLAGS -J value

skiwi wrote:

Just use -jn with n the number of real cores you have. Note also the difference between "threads" and "cores". Threads are different items in /proc/cpuinfo, for the number of cores look at the number of different 'core id's in /proc/cpuinfo. Typically each cpu (=core) has a number of threads (typically 2 or 4), which are virtual cpus used to balance the workload. That means you can still do things, like use a browser, while compiling. At the same time a thread can use an entire cpu if no other thread is asking for it so you will not lose any performance. So, never use all threads, use all cpus. More is pointless imho. There might be a tiny 'start-up' time gain between processes by using n+1 if you compile something consisting of many small files, but I guess it would be negligible and you risk blocking the cpu/having a very slow computer during compilation.

-- edit : the additional cores in the original also refer additional to 1, so 4 cores implies -j4

This is plainly wrong, you should read up on the purpose of hyperthreading. Roughly, the extra threads are used to keep the processor pipeline full when the currently running process stalls for IO, is preempted or terminates. Without hyperthreading there would be a delay during which the pipeline is filled but the processor has not enough data to operate, but hyperthreading keeps the processor busy during this delay by switching to another thread very fast.

Not using hyperthreading to keep your desktop responsive is like only using one core on a dual core cpu in case some network traffic comes in, it's unnecessary, detrimental to performance and should be taken care of by the scheduler.

Offline

Board footer

Powered by FluxBB