Am I wasting my time with from source compiles?

Chaniyth · 2011-11-21 16:44:39

Greetings all,

I have a question, and it may be silly. I'm a gamer but not a hardcore "zOMG I needZ moRe powAh tHan I actUAlly n33d!" type.

I've used Gentoo from 2002 - 2006.

I started using Arch Linux in fall of 2006 and have used it since.

Couple days ago I "upgraded" from my AMD 64 x2 - 2Ghz system, to a Intel Core 2 Duo E8400 3Ghz system with 4 GB Ram and Geforce 560 Ti. I installed Arch Linux and then I installed Gentoo just for nostalgic and curiosities sake, I had a working Gentoo install within a few hours. I'm not noticing that much of a difference in performance between the two, X seems to load a bit faster on Gentoo but other apps in general load the same, have same performance quality and games have about the same exact framerates. Reguardless if I use "default" USE flags or "all dependencies" USE flags (I tend to be an all dependencies type of guy -- but I did both types for my testings) im not seeing that much, if any at all, performance differences.

I should also note that: Phoronix Test Suite actually provided better performance numbers in Arch Linux than in Gentoo Linux.

Now my "silly" question and I want true answers and not just speculative answers.

Am I wasting my time with Gentoo and from source compiles in general? If I generally tend to do all dep compiles, and im not seeing any performance difference reguardless if I do or don't do the all deps USE flags approach, simular gaming framerates, etc... am I really wasting my time with all the compiles and compile time? Thanks in advance.

Last edited by Chaniyth (2011-11-21 17:08:26)

karol · 2011-11-21 16:51:51

From what I've read, using Gentoo gives you more control wrt the configuration and dependencies but you probably won't have a faster system.

Awebb · 2011-11-21 17:52:52

If you just use USE flags, then you're wasting your time. -march and -mtune are, where the magic happens. On a magic scale of 1 (Siegfried and Roy) to 10 (Hawaiian Cubensis), every single one I asked, reports a different number.

General function:

be t time;
f[waste of time] → ( t[compile] - t[relevant learning curve] ) / t[performance gain between compile sessions];
if f > 1, then you've wasted your time.

Chaniyth · 2011-11-21 18:07:22

Awebb wrote:

If you just use USE flags, then you're wasting your time. -march and -mtune are, where the magic happens. On a magic scale of 1 (Siegfried and Roy) to 10 (Hawaiian Cubensis), every single one I asked, reports a different number.

General function:
be t time;
f[waste of time] → ( t[compile] - t[relevant learning curve] ) / t[performance gain between compile sessions];
if f > 1, then you've wasted your time.

My CFLAG settings are -march=native -O2 -pipe

pedromj · 2011-11-21 18:28:19

From a pure theoretical point of view, current processors do internal code optimization, so you should not get any gain from playing with compiler flags. The only options to "defend" the all-custom-compiled approach are, on the one hand, making use of specific processor extensions like SSE, which is supported by some software, and, on the other hand, the possibility to configure the system to remove the unneeded/unwanted parts so the final result is lighter and, therefore, quicker.

nomilieu · 2011-11-21 19:56:18

Do you use the AUR?
If so, you're compiling from source, and that's not a waste in general. Even if the occasional program you need isn't there, it's still no big deal to compile and install it.
If you're doing that all of the time though, then yeah, Gentoo would probably be a better deal.
I wouldn't swap for performance though; it's not worth it.

Last edited by nomilieu (2011-11-21 20:00:39)

brebs · 2011-11-21 20:14:13

pedromj wrote:

so the final result is lighter and, therefore, quicker.

And easier to maintain (because there's less of it), and less susceptible to security flaws (again, because there's less of it).

Regarding performance, it's a crapshoot - any of the zillions of patches thrown around can have nasty side-effects. Comparing one distro to another is again a crapshoot because they're not even using the same app/library versions, never mind the same additional patchsets.

Compiling from source is just *one* of the ways to optimize your setup.

I do it, to have control over app versions. Devs break previously-working software functionality often - the latest version isn't always the best. It's nice to pick 'n' choose patches from Arch, Gentoo, Ubuntu, Fedora, openSuSE, etc.

Ya learn a lot, but it's a *lot* more effort.

sitquietly · 2011-11-21 21:56:17

Chaniyth wrote:

.....I installed Arch Linux and then I installed Gentoo just for nostalgic and curiosities sake, I had a working Gentoo install within a few hours. I'm not noticing that much of a difference in performance between the two.....I should also note that: Phoronix Test Suite actually provided better performance numbers in Arch Linux than in Gentoo Linux. ..... Am I wasting my time with .. source compiles..?

IME yes, if your goal is overall performance you ARE wasting your time. I have the same results in benchmarking stock Archlinux vs. the same system recompiled with CFLAGS="-march=native etc." on Sandy Bridge. The recompiled system ran slightly slower on some tests. Quite a bit faster on bzip2 compression. Better x264 performance. Much lower latencies, measured using the interbench package from AUR. But the lower latency depends primarily on the kernel. I use linux-lqx with arch=native; it is better than linux-ck or linux-pf in my tests.

I compile everything on my Archlinux system, using xmonad/lxpanel/rox as desktop or, when I'm in an alternate mood, KDE/rox. I don't compile from source for speed (except for math packages like sage-mathematics which can sometimes take advantage of the extra registers and SIMD instructions). I compile from source so that I have a complete system on hand. I once lived off-grid (in a cabin in the Ozark National Forest) and got in the habit of keeping my system ready for self-reliance. I enjoy having source code available for perusal and I modify a few of my packages. I use my own build script that uses meld to help me quickly bring forward my modifications into updated PKGBUILDs from the Arch repository (hey, there's no USE flags but I find that just modifying the PKGBUILD is actually cleaner, less baroque than the circular mess that can happen in Gentoo.)

One word of caution on your Phoronix benchmarks. Phoronix Test Suite was invented for comparing hardware. It uses its own binaries for many of the tests. When looking for the effect of software changes on the same hardware you should use the changed software (obviously). To do this with PTS you need to go into the ~/.phoronix-test-suite/installed-tests/pts/<test>/ folder and move aside the folder containing the phoronix version of the binary being tested and link to your /usr directory. For example, in the sqlite test folder I move aside "sqlite_" and "ln -s /usr sqlite_" You can figure how to get each test to run local software by looking at the script in the test folder. If you don't do this, IMO you're getting misleading results from the PTS.

JGC · 2011-11-21 22:25:07

For i686, you can get a big performance gain on modern CPUs when compiling native. For x86_64 the difference is not so big, as generic x86_64 has much more registers and instruction set extensions available than plain i686. You could gain some extra speed with multimedia programs like mplayer and x264 by recompiling native though. For programs like OpenOffice you won't gain much speed, as those programs are waiting for your input almost all the time.

fukawi2 · 2011-11-21 22:26:16

Chaniyth wrote:

... am I really wasting my time with all the compiles and compile time?

IMHO, if you're not compiling with additional options (that aren't in the binary builds) and/or you have other things to do with your time, then yes you are wasting your time.

If you save 3ms on starting application X, but you spent 30 minutes tweaking and compiling, how many times will you have to start application X to break-even on your time investment? (600,000 )

Of course, there's always just a certain satisfaction of not having bloat (eg, MySQL *and* PostgreSQL *and* SQLite support)

Last edited by fukawi2 (2011-11-21 22:27:04)

Gusar · 2011-11-21 23:00:55

JGC wrote:

You could gain some extra speed with multimedia programs like mplayer and x264 by recompiling native though.

People keep repeating that, but it's completely false. These multimedia apps gain all their speed from hand-written assembly, not from compiler flags. All this assembly is compiled in *always*, and then runtime cpu detection determines which codepaths to run.

pedromj · 2011-11-21 23:31:40

Gusar wrote:

People keep repeating that, but it's completely false. These multimedia apps gain all their speed from hand-written assembly, not from compiler flags. All this assembly is compiled in *always*, and then runtime cpu detection determines which codepaths to run.

That's true but to some extent. Some software needs specific build parameters to determine what optimization (assembly code) to use because they do not perform runtime detection or because they want to let the user decide what CPU extension to use. This situation was more frequent in the past but today it is still true for some software, even though the more common and widely used software use to perform the runtime detection.

sitquietly · 2011-11-21 23:39:22

Gusar wrote:

JGC wrote:
You could gain some extra speed with multimedia programs like mplayer and x264 by recompiling native though.
People keep repeating that, but it's completely false.

Benchmarks!? I was wondering if you have ever measured the difference.

skottish · 2011-11-22 03:35:57

Gusar wrote:

JGC wrote:
You could gain some extra speed with multimedia programs like mplayer and x264 by recompiling native though.
People keep repeating that, but it's completely false. These multimedia apps gain all their speed from hand-written assembly, not from compiler flags. All this assembly is compiled in *always*, and then runtime cpu detection determines which codepaths to run.

I remember reading something, somewhere when someone asked in some forum if it was worth using ICC for building x264. Dark Shikari answered the question in that there was some small gain in the C code, but the numbers that I remember ( less than 2% in some areas ) are along the lines of what fukawi2 said. So, the hand-written ASM code, where all of the action is, didn't change. But, the help file performed magnificently in rendering its options...

x264 is very specialized and has had a lot of time with ASM optimizations. Mplayer, being based on FFmpeg code, will vary in degrees on how much attention each codec, or shared code for them, has had. C optimizations will still help in many places there.

KingX · 2011-11-22 05:31:56

I agree with fukawi2 on the time-investment vs. performance gain argument. I have done a couple of gentoo installs and it takes considerably longer to get a fully working system compared with a binary install like Arch and the performance "gain" is un-noticeable.

If you want noticeable gains, look into compiling your own kernel and using a different scheduling algorithm like BFS (which I plan to soon!) the gains made here will be instantly noticeable. Tweaking stuff like loading firefox profile in RAM will make Firefox run a lot snappier. For faster boot times if you have a lot of daemons starting during boot, run them in the background using '@' in rc.conf. You will easily take a few seconds off of your boot time.

I think things like that make more of a difference then compiling programs from source. Sure some programs *may* run faster... but the majority will run about the same. Just my 2 cents.

ngoonee · 2011-11-22 06:36:41

Moving to Gnu/Linux Discussion since this discussion isn't actually about Arch (even though Arch is mentioned).

Gusar · 2011-11-22 10:27:49

sitquietly wrote:

Gusar wrote:
JGC wrote:
You could gain some extra speed with multimedia programs like mplayer and x264 by recompiling native though.
People keep repeating that, but it's completely false.
Benchmarks!? I was wondering if you have ever measured the difference.

I personally not. Others have. There is a difference, so it's not completely negligible, but the difference is very small. Whereas the people who say what I was responding to make it sound as if there's huge speedups to be gained.

Misfit138 · 2011-11-22 14:18:19

Am I wasting my time with from source compiles?

Yes.
It is a waste of time.

But, if you have the time and enjoy it, go for it. Just always make sure of the more important things.

Leonid.I · 2011-11-22 17:59:51

It's not only about performance, but security too. Stock apps are compiled with some generic set of options which are known not to break, but individual apps can be compiled with a much more restrictive flags.

Mr_ED-horsey · 2011-11-22 19:10:16

nomilieu wrote:

Do you use the AUR?
If so, you're compiling from source, and that's not a waste in general. Even if the occasional program you need isn't there, it's still no big deal to compile and install it.
If you're doing that all of the time though, then yeah, Gentoo would probably be a better deal.
I wouldn't swap for performance though; it's not worth it.

I usually try installing from AUR first, but there's quite a bit of packages on there that fail to install no matter how many times I edit the pkgbuild. Although the overwhelming majority of packages on the AUR work perfectly, but for the ones that don't, thankfully I have three years of Slackware experience under my belt, so compiling from source isn't much of a problem for me

Compiling from source is never a waste of time IMO. It's good thing to know how to do, and in my personal experience the programs built from source are a little faster and have less crashes. Again, personal experience. Other people may have different experiences with this.

I've only played around with Gentoo on Virtualbox. I've never installed Gentoo on an actual HDD, so I can't give a good comparison on speed, but from what I've heard there isn't a whole lot of differences.

nomilieu · 2011-11-22 20:41:55

Mr_ED-horsey wrote:

Compiling from source is never a waste of time IMO.

Never?
The only reason I even use Arch is because compiling software is a waste of time most of the time.
(Although I do have a Gentoo box.)

Mr_ED-horsey · 2011-11-22 21:40:13

nomilieu wrote:

Mr_ED-horsey wrote:
Compiling from source is never a waste of time IMO.
Never?
The only reason I even use Arch is because compiling software is a waste of time most of the time.
(Although I do have a Gentoo box.)

Again, it's all a matter of personal opinion, but I've never found running:

tar -xzvf
cd
/.configure
make
make install

to be very time consuming. EXCEPT when the source tarball doesn't have an up to date list of required dependencies in it's README file... Then, yes, it is annoying as hell

But I do agree, Arch and the AUR is much easier and most of the time I don't compile from source anymore, but learning/knowing how to do it can be very valuable. That's what I meant by saying it's never a waste of time.

nomilieu · 2011-11-22 22:09:42

We're talking about two different things it seems. By waste of time, I meant the time waiting on the compilation to actually complete.

Leonid.I · 2011-11-22 22:47:30

Mr_ED-horsey wrote:

EXCEPT when the source tarball doesn't have an up to date list of required dependencies in it's README file... Then, yes, it is annoying as hell
But I do agree, Arch and the AUR is much easier and most of the time I don't compile from source anymore, but learning/knowing how to do it can be very valuable. That's what I meant by saying it's never a waste of time.

Agreed. Especially if you have to hunt bugs (just finished compiling evince 2.32 -- it kept segfaulting on every pdf created in windows until I merged some patches from ubuntu).

However, AUR has to be used with caution IMHO because it is mostly broken/PKGBUILDS are badly written or contain untrustable sources, like mainteiner's personal ftp site.

ngoonee · 2011-11-23 01:50:30

Mr_ED-horsey wrote:

nomilieu wrote:
Do you use the AUR?
If so, you're compiling from source, and that's not a waste in general. Even if the occasional program you need isn't there, it's still no big deal to compile and install it.
If you're doing that all of the time though, then yeah, Gentoo would probably be a better deal.
I wouldn't swap for performance though; it's not worth it.
I usually try installing from AUR first, but there's quite a bit of packages on there that fail to install no matter how many times I edit the pkgbuild. Although the overwhelming majority of packages on the AUR work perfectly, but for the ones that don't, thankfully I have three years of Slackware experience under my belt, so compiling from source isn't much of a problem for me
Compiling from source is never a waste of time IMO. It's good thing to know how to do, and in my personal experience the programs built from source are a little faster and have less crashes. Again, personal experience. Other people may have different experiences with this.
I've only played around with Gentoo on Virtualbox. I've never installed Gentoo on an actual HDD, so I can't give a good comparison on speed, but from what I've heard there isn't a whole lot of differences.

I'd be interested to know of specific cases where compiling from source works and putting the same commands into the PKGBUILD doesn't. In most such cases upstream is doing something wrong (hardcoded directories etc., not using the standard tools).

Arch Linux

#1 2011-11-21 16:44:39

Am I wasting my time with from source compiles?

#2 2011-11-21 16:51:51

Re: Am I wasting my time with from source compiles?

#3 2011-11-21 17:52:52

Re: Am I wasting my time with from source compiles?

#4 2011-11-21 18:07:22

Re: Am I wasting my time with from source compiles?

#5 2011-11-21 18:28:19

Re: Am I wasting my time with from source compiles?

#6 2011-11-21 19:56:18

Re: Am I wasting my time with from source compiles?

#7 2011-11-21 20:14:13

Re: Am I wasting my time with from source compiles?

#8 2011-11-21 21:56:17

Re: Am I wasting my time with from source compiles?

#9 2011-11-21 22:25:07

Re: Am I wasting my time with from source compiles?

#10 2011-11-21 22:26:16

Re: Am I wasting my time with from source compiles?

#11 2011-11-21 23:00:55

Re: Am I wasting my time with from source compiles?

#12 2011-11-21 23:31:40

Re: Am I wasting my time with from source compiles?

#13 2011-11-21 23:39:22

Re: Am I wasting my time with from source compiles?

#14 2011-11-22 03:35:57

Re: Am I wasting my time with from source compiles?

#15 2011-11-22 05:31:56

Re: Am I wasting my time with from source compiles?

#16 2011-11-22 06:36:41

Re: Am I wasting my time with from source compiles?

#17 2011-11-22 10:27:49

Re: Am I wasting my time with from source compiles?

#18 2011-11-22 14:18:19

Re: Am I wasting my time with from source compiles?

#19 2011-11-22 17:59:51

Re: Am I wasting my time with from source compiles?

#20 2011-11-22 19:10:16

Re: Am I wasting my time with from source compiles?

#21 2011-11-22 20:41:55

Re: Am I wasting my time with from source compiles?

#22 2011-11-22 21:40:13

Re: Am I wasting my time with from source compiles?

#23 2011-11-22 22:09:42

Re: Am I wasting my time with from source compiles?

#24 2011-11-22 22:47:30

Re: Am I wasting my time with from source compiles?

#25 2011-11-23 01:50:30

Re: Am I wasting my time with from source compiles?

Board footer