You are not logged in.

#1 2014-06-18 05:26:57

HalosGhost
Member
From: Twin Cities, MN
Registered: 2012-06-22
Posts: 1,485
Website

Autotools vs the alternatives

Autotools. Oh my lord, autotools. When a script to make sure that everything under the sun exists for your build (including a bunch of tools that are not needed at all) takes less time to run than the compilation itself, I might be able to understand. But, even then, writing and reading Makefile.amS is an incredibly sad and gruesome experience. Grr…

All the best,

-HG


-- mod edit: I've split this out from the Grr thread as it is an interesting discussion and is better suited to a dedicated thread...

Last edited by jasonwryan (2014-06-18 21:29:13)


"All errors are ᴘᴇʙᴋᴀᴄ errors—It's just a matter of narrowing down which keyboard and chair." -Trilby
\ldots

Offline

#2 2014-06-18 09:15:46

jakobcreutzfeldt
Member
Registered: 2011-05-12
Posts: 1,033

Re: Autotools vs the alternatives

HalosGhost wrote:

Autotools. Oh my lord, autotools. When a script to make sure that everything under the sun exists for your build (including a bunch of tools that are not needed at all) takes less time to run than the compilation itself, I might be able to understand. But, even then, writing and reading Makefile.amS is an incredibly sad and gruesome experience. Grr…

All the best,

-HG

Because as soon as someone tries to build that software on some non-GNU/Linux system and it doesn't build/run, they go and post a bug. Those configure scripts weren't made just to annoy you. They were made to solve a very real problem: inconsistent features between POSIX or POSIX-like systems. The fact that they take so long because they're doing so many tests is a testament to the sorry state of compatability between POSIX systems. Likewise with Automake and the Makefiles that it generates: they aren't made to be read by humans, they're made to do the right thing correctly every time, for every possible corner case on every possible system. As someone who manages a couple software build systems (Ports-like systems), I absolutely cringe and curse every time I have to deal with someone's shoddy, incomplete, inflexible, handwritten Makefiles or install scripts. Packages that use Automake, on the other hand, are an absolute breeze to handle.

And to clarify, a lot of those checks aren't just for, say, GNU extensions to libc. They're for verifying that some standard function behaves  the way you expect it to behave.  You should look at gnulib (probably the source of most of those compatibility checks in configure scripts). A ton of work has gone into making sure that C programmers can guarantee that their software runs exactly the same on as many systems as possible without any intervention by the user. The trade-off is that the user has  to spend 30 seconds of their day waiting for the configure script to finish running. I think that's a fair trade-off.

Edit: and for years, people have wanted configure scripts to be able to run checks in parallel, but it's a hard problem and to date, no one has done more than just complain about it and request the feature.

Last edited by jakobcreutzfeldt (2014-06-18 09:16:53)

Offline

#3 2014-06-18 11:29:02

Blµb
Member
Registered: 2008-02-10
Posts: 223

Re: Autotools vs the alternatives

Okay fine, but why do these packages have to be shipped with build scripts covering heaps of possible systems which are then outdated or simply don't know about some system yet. (eg freebsd switch to version 10, with configure scripts checking for version '9*' and '1*', so that '10' hits the version 1 case).
Why not just say: You need compilers, libraries and headers installed already, so why not also install a damn build system so that it doesn't also need to wrongly guess how to compile shared libraries.
Eg don't ship a damn libtool script, but simply require libtool to be installed and error when it's not, and use the local one. After all, the local one is supposed to know better!

Last edited by Blµb (2014-06-18 11:29:11)


You know you're paranoid when you start thinking random letters while typing a password.
A good post about vim
Python has no multithreading.

Offline

#4 2014-06-18 12:11:07

drcouzelis
Member
From: Connecticut, USA
Registered: 2009-11-09
Posts: 3,423
Website

Re: Autotools vs the alternatives

Autotools is a paradox, simultaneously being the absolute best thing in the open source software world and the very worst.

Vim and Perl tie for second.

Offline

#5 2014-06-18 12:33:56

GloW_on_dub
Member
Registered: 2013-03-13
Posts: 326

Re: Autotools vs the alternatives

@HalosGhost : switch to CMake then, it is so much easier to write a CMakeLists... still take some time to configure and generate though, but it is worth it !

Offline

#6 2014-06-18 12:45:48

jakobcreutzfeldt
Member
Registered: 2011-05-12
Posts: 1,033

Re: Autotools vs the alternatives

Blµb wrote:

Okay fine, but why do these packages have to be shipped with build scripts covering heaps of possible systems which are then outdated or simply don't know about some system yet. (eg freebsd switch to version 10, with configure scripts checking for version '9*' and '1*', so that '10' hits the version 1 case).

That's a bug in their configure.ac that needs to be reported. No software is perfect. You would complain just as loudly if the software didn't try to account for any differences between BSD and GNU/Linux and ended up failing to build or not running correctly.

Why not just say: You need compilers, libraries and headers installed already, so why not also install a damn build system so that it doesn't also need to wrongly guess how to compile shared libraries.
Eg don't ship a damn libtool script, but simply require libtool to be installed and error when it's not, and use the local one. After all, the local one is supposed to know better!

Because there are many different kernels and libraries out there and even though they ostensibly cover the same functionality, there may be subtle differences in how they do it or even bugs that need to be accounted for. And anyway, are you serious? If I'm a developer and I use a different libc than you, you think it's ok for me to say, "OK, if you want to use my software, you must install this libc, not that piece of crap you're using."?   Give me a break.

As for libtool, how do you propose that a local one will know better than one included in a software package? And if your system is so weird that it's difficult to deduce how to correctly build shared libraries, then you are in fact an excellent example of why we need things like Autoconf (albeit an example that it's a neverending struggle to account for all machines).

Offline

#7 2014-06-18 12:51:36

jakobcreutzfeldt
Member
Registered: 2011-05-12
Posts: 1,033

Re: Autotools vs the alternatives

GloW_on_dub wrote:

@HalosGhost : switch to CMake then, it is so much easier to write a CMakeLists... still take some time to configure and generate though, but it is worth it !

Except CMake is much more opaque to the user than it is to the developer. With Autoconf, a simple "./configure --help" gives you a complete list of options available and they're set in a familiar fashion. With CMake, I'm still not entirely sure how best to configure the build.  Looking at its help output, I guess it involves some combination of "--help-variable", "--help-variable-list", and "-D" but I don't know.  Also, it's then up to the developer to make sure that the software is configurable to run on multiple, diverse platforms.

In anycase, whether you go with CMake or Autoconf/Automake, the fact is that they serve a purpose more than just to annoy people.  So  what if something takes a bit longer to build? If it makes the whole development/packaging/building/installation process smoother between all developers, packagers and users, then that's a very good thing, not something to grr about.

Offline

#8 2014-06-18 13:29:47

GloW_on_dub
Member
Registered: 2013-03-13
Posts: 326

Re: Autotools vs the alternatives

I'm using ccmake ( or cmake-ncurses ), and well it is very NOT opaque at all this way. Options are directly visible.

Offline

#9 2014-06-18 13:49:39

ANOKNUSA
Member
Registered: 2010-10-22
Posts: 2,141

Re: Autotools vs the alternatives

GloW_on_dub wrote:

I'm using ccmake ( or cmake-ncurses ), and well it is very NOT opaque at all this way. Options are directly visible.

I'll humbly admit that I probably don't belong in this conversation---the pros and cons of build systems only matter to me as an end-user, not a developer---but regardless of what the topic of discussion is, I shudder every time somone argues that x is the best solution to an issue, provided you also account for y, z and aa. An analogy: A hammer is the best tool for driving in a screw, provided you use a grinding wheel to remove the threading from all the screws you use. But now you've introduced a new tool and you no longer have screws, but nails. The hammer is the best tool, but only after you change the initial situation to make it the best tool, and the "solution" is therefore more convoluted than it needed to be. Could've just gone with a screwdriver or power drill to begin with.

Offline

#10 2014-06-18 14:55:30

Blµb
Member
Registered: 2008-02-10
Posts: 223

Re: Autotools vs the alternatives

jakobcreutzfeldt wrote:

Because there are many different kernels and libraries out there and even though they ostensibly cover the same functionality, there may be subtle differences in how they do it or even bugs that need to be accounted for. And anyway, are you serious? If I'm a developer and I use a different libc than you, you think it's ok for me to say, "OK, if you want to use my software, you must install this libc, not that piece of crap you're using."?   Give me a break.

Talking about compiling software, not running it. If I have libtool installed on my system, it's setup to be capable of compiling software for my system. If you ship your own libtool, it'll definitely not know better. The interface of how the libtool program is to be invoked is the same in both scenarios. After all I said you're supposed to have the buildsystem installed. I'm not talking about implementations of runtime libraries. Ie: You provide instructions on how to build your software in a way the buildsystem you use wants you to. Makefile.am, CMakeLists, etc, and the buildsystem translates them to compile-commands suitable for the system I'm compiling on.
The problem is NOT the configure.ac script as many of the checks ./configure do come from external macros. Occasionally you can just run autoreconf, sometimes even that bugs out.
Turning your argument around: Are you saying I need to install a compiler that uses the exact same commandline switches yours does otherwise mine's a piece of crap? There are many more differences than just the runtime or compile time even. For instance on FreeBSD libraries only get 1 version suffix to their filename, and many packages get that one wrong.

jakobcreutzfeldt wrote:

As for libtool, how do you propose that a local one will know better than one included in a software package? And if your system is so weird that it's difficult to deduce how to correctly build shared libraries, then you are in fact an excellent example of why we need things like Autoconf (albeit an example that it's a neverending struggle to account for all machines).

Above example: many packages don't get library versioning right because of libtool.
The ports system is full of similar hacks.
See → https://github.com/freebsd/freebsd-port … ls.mk#L389
https://github.com/freebsd/freebsd-port … libtool.mk

But I can still compile shared libraries the exact same way as I do on linux: gcc -shared, clang -shared, and yes, also by using libtool with the very same commandline switches.

Now what.

EDIT:
On a related note: ArchBSD ships a patched libtool that actually gets library versions in their filenames right. As in: it knows better than your provided one!

Last edited by Blµb (2014-06-18 14:58:25)


You know you're paranoid when you start thinking random letters while typing a password.
A good post about vim
Python has no multithreading.

Offline

#11 2014-06-18 15:55:00

jakobcreutzfeldt
Member
Registered: 2011-05-12
Posts: 1,033

Re: Autotools vs the alternatives

Blµb wrote:

Above example: many packages don't get library versioning right because of libtool.
The ports system is full of similar hacks.
See → https://github.com/freebsd/freebsd-port … ls.mk#L389
https://github.com/freebsd/freebsd-port … libtool.mk

But I can still compile shared libraries the exact same way as I do on linux: gcc -shared, clang -shared, and yes, also by using libtool with the very same commandline switches.

Then I guess I didn't really know what you were talking about. After all, you seem to just be grring about bugs in libtool, not about the nature of Autotools and other build systems, which is what the discussion was about.  I fail to understand how any of your comments have been salient to the discussion now. If libtool needs patching to work properly, then file a bug report or submit a patch.  As for libtool itself, I have no experience with it but as I understand it, developing and maintaining a shared library is a pain without it. BTW, are there actual examples of packages shipping their own libtool?

Now what.

Is this the new "come at me, bro"? Relax. It's not a fight.

Offline

#12 2014-06-18 21:43:08

Trilby
Forum Moderator
From: Massachusetts, USA
Registered: 2011-11-29
Posts: 13,485
Website

Re: Autotools vs the alternatives

HalosGhost wrote:

Autotools. Oh my lord, autotools.

Odd that this makes you Grr ... I just tried out Shaman: you used autotools for it!


InterrobangSlider
• How's my coding? See this page.
• How's my moderating? Feel free to email any concerns, complaints, or objections.

Offline

#13 2014-06-19 00:54:14

HalosGhost
Member
From: Twin Cities, MN
Registered: 2012-06-22
Posts: 1,485
Website

Re: Autotools vs the alternatives

Trilby wrote:

Odd that this makes you Grr ... I just tried out Shaman: you used autotools for it!

I do, for the moment. Though, if you go through the commit history; you'll notice that it's a recent thing. And I only merged it at the recommendation of another Archer. I'll be replacing it with tup or reverting to hand-written makefiles in a short while.

All the best,

-HG


"All errors are ᴘᴇʙᴋᴀᴄ errors—It's just a matter of narrowing down which keyboard and chair." -Trilby
\ldots

Offline

#14 2014-06-20 08:42:33

Blµb
Member
Registered: 2008-02-10
Posts: 223

Re: Autotools vs the alternatives

jakobcreutzfeldt wrote:

BTW, are there actual examples of packages shipping their own libtool?

Every autotools based package that includes the ./configure script instead of just the configure.ac.

If you run aclocal/automake/autoconf etc. you generate the scripts from your local system, but the ready-to-execute ./configure file and its dependencies come from whoever does `make dist` to create the package archive. When they use libtool, these then also include their libtool and ltmain.sh scripts.
Instead of the hacks used by ports for example you can also run ./configure, then replace all libtool and ltmain.sh files with local ones before running make, and the build will behave correctly as well. I'm just mostly targeting libtool because it's much easier to find differences with libraries than with anything else (another one immediately coming to mind: freebsd will ignore libfoo.so.X.Y.Z in your library path because there are too many version suffixes. */me points to mesas Makefile.am refering to library files with 3 version suffixes that don't actually end up being used in the filename... [1]*). For many other things on a whole lot of systems including BSDs, haiku, and even windows (if you have eg cygwin installed) you can usually `gcc foo.c -lbar -lbaz` and it'll compile your program provided the required libraries/headers etc are installed.

And while I was mostly targeting libtool here, the ./configure script and all the m4 macros you get have (or at least can have) similar issues. Eg the version-matching I mentioned wasn't in libtool but in ./configure scripts, and not even coming from the developers configure.ac, but from them running `make dist` on a system that didn't know about freebsd's version 10 yet. (So even once that's fixed, it takes a while for all the software's developers to update their local build files and provide a new package.)

The problem with not shipping ./configure scripts is that then whoever wants to compile the package needs autotools installed - and IMO this should not actually be seen as a problem at all.
On the other hand you might end up having to install multiple autotools versions (see core/autoconf vs extra/autoconf2.13 and some more versions found in the AUR)

Personally, I pretty much hate every build system. And as you may have noticed I can get really passionate about showing it wink

[1]https://github.com/ArchBSD/abs/blob/mas … GBUILD#L43


You know you're paranoid when you start thinking random letters while typing a password.
A good post about vim
Python has no multithreading.

Offline

#15 2014-06-20 12:54:15

jakobcreutzfeldt
Member
Registered: 2011-05-12
Posts: 1,033

Re: Autotools vs the alternatives

Blµb wrote:

Every autotools based package that includes the ./configure script instead of just the configure.ac.

If you run aclocal/automake/autoconf etc. you generate the scripts from your local system, but the ready-to-execute ./configure file and its dependencies come from whoever does `make dist` to create the package archive. When they use libtool, these then also include their libtool and ltmain.sh scripts.

OK, I see what you're saying now.  I've never developed a library so I haven't had to deal with libtool personally, and I've never noticed that happening from a user-perspective.

As I understand it, so-versioning and distributing shared libraries in the face of ABI changes, etc., is a difficult problem and libtool is one of the only (or is it *the* only) attempts at a solution to it. When trying to solve that problem across many systems, I can imagine that it only gets more difficult, so bugs are to be expected.

And while I was mostly targeting libtool here, the ./configure script and all the m4 macros you get have (or at least can have) similar issues. Eg the version-matching I mentioned wasn't in libtool but in ./configure scripts, and not even coming from the developers configure.ac, but from them running `make dist` on a system that didn't know about freebsd's version 10 yet. (So even once that's fixed, it takes a while for all the software's developers to update their local build files and provide a new package.)

Yeah, that's a problem but I don't see any way around that: if you're going to try to automatically customize the software based on detected features, it will always be difficult to future-proof that process.  That's beyond Autotools...that's for any case.  All that can be done is to report bugs.

The problem with not shipping ./configure scripts is that then whoever wants to compile the package needs autotools installed - and IMO this should not actually be seen as a problem at all.
On the other hand you might end up having to install multiple autotools versions (see core/autoconf vs extra/autoconf2.13 and some more versions found in the AUR)

Yes, the reason configure scripts are shipped is the assumption that the user should not be expected to have the Autotools installed; they're viewed as developer tools. As for having to use old autotools versions (presumably due to use of deprecated macros), I think that's a  bad idea.  It's better to patch the configure.ac/Makefile.am files to not require the old version, to bug the developers to update, or to find other software.  The same can be said for requiring ancient libraries...

Personally, I pretty much hate every build system. And as you may have noticed I can get really passionate about showing it wink

I'm ambivalent towards them in general but now that I've become involved in writing a lot of build scripts for a couple of  Ports-like systems that I maintain, I've grown to really appreciate their use.  The biggest advantage is that they end up presenting a standard interface to the user. Sure, there might be a few varieties: Autotools, CMake, or language-specific scripts (e.g. Python's distutils setup.py scripts). But once you recognize the variety, it's simple to automate the build. As soon as you run into a hand-written solution, you suddenly have to jump through a bunch of hoops to install the software.

Case in point: f***ing bioinformatics software. Probably the majority of it has no build system to speak of. In some cases, you're lucky if they even give you a Makefile. They almost always want you to edit some files to configure the software (esp. paths, which are often hard-coded to their development systems). It's a pain. So when I encounter one that uses Autotools, I feel like sending a bottle of champagne to the devs in thanks. When they just give a handwritten Makefile, I generally have to edit all of the paths, I have to make it support staged installations (DESTDIR), I might have to modify environment variables, etc. The user/packager shouldn't have to deal with that. I just want to run a couple commands (configure/make, cmake/make, whatever) and be done with it.

Offline

#16 2014-06-20 13:22:27

Blµb
Member
Registered: 2008-02-10
Posts: 223

Re: Autotools vs the alternatives

jakobcreutzfeldt wrote:

I'm ambivalent towards them in general but now that I've become involved in writing a lot of build scripts for a couple of  Ports-like systems that I maintain, I've grown to really appreciate their use.  The biggest advantage is that they end up presenting a standard interface to the user.

Yes, the standardized interface is *the* one good thing about them.

jakobcreutzfeldt wrote:

Case in point: f***ing bioinformatics software. Probably the majority of it has no build system to speak of. In some cases, you're lucky if they even give you a Makefile.

Lately I'm getting the feeling anything somewhat related to medical or bio IT is of unreasonably poor quality... (I shouldn't have chosen medical IT but it's too late to switch curriculum now tongue)


You know you're paranoid when you start thinking random letters while typing a password.
A good post about vim
Python has no multithreading.

Offline

#17 2014-06-20 15:06:36

Trilby
Forum Moderator
From: Massachusetts, USA
Registered: 2011-11-29
Posts: 13,485
Website

Re: Autotools vs the alternatives

jakobcreutzfeldt wrote:

It's better to patch the configure.ac/Makefile.am files to not require the old version, to bug the developers to update, or to find other software.

If one needs to occasionally patch configure.ac/Makefile.am files, and/or bug the upstream dev for fixes in them, I fail to see how this is then any better than occasionally needing to patch a clear and concise hand-written Makefile.  But more importantly:

jakobcreutzfeldt wrote:

When they just give a handwritten Makefile, I generally have to edit all of the paths, I have to make it support staged installations (DESTDIR), I might have to modify environment variables, etc.

While I really don't know enough about autotools to make informed comparisons myself, it seems like a lot of the comparisons here are between a good use of autotools and a piss-poor use of hand-written Makefiles.

Poorly written Makefiles are a problem.  But from my current state of knowledge (or ignorance) the sensible solution is well written makefiles, not a whole new system of complexity.  Or in short, I suspect there may be benefits of autotools, but in a discussion of the pros/cons I think it'd only be fair to compare autotools to good Makefiles.  Autotools are a GNU product - but the GNU already has good standards for Makefiles, that - if followed - may alleviate (most of) the need for autotools.


InterrobangSlider
• How's my coding? See this page.
• How's my moderating? Feel free to email any concerns, complaints, or objections.

Offline

#18 2014-06-24 17:09:51

Franek
Member
Registered: 2010-05-16
Posts: 97

Re: Autotools vs the alternatives

For reference, there is an older thread about build systems where several interesting options are mentioned, perhaps most notably premake.

Thanks for bringing up tup I did not know about that one.

Offline

#19 2014-06-24 19:57:14

sitquietly
Member
From: Moscow, Tennessee
Registered: 2010-07-12
Posts: 216

Re: Autotools vs the alternatives

jakobcreutzfeldt wrote:
GloW_on_dub wrote:

@HalosGhost : switch to CMake then, it is so much easier to write a CMakeLists... still take some time to configure and generate though, but it is worth it !

Except CMake is much more opaque to the user than it is to the developer. With Autoconf, a simple "./configure --help" gives you a complete list of options available and they're set in a familiar fashion. .....

I didn't see an answer when scanning this thread.

cmake -LH gives me all the info I need for configuring a package using cmake; it is at least as helpful (pun?) as ./configure --help is for packages using autotools.  As example, my build script automatically creates a README file for every package I build with READMEs accumulated from the source code and the cmake or configure help messages.  In part that script has

# search for configure or CMakeLists.txt and generate configuration help:
  for dir in src/*/ src/*/*/; do
    if find $dir -maxdepth 1 -type f -name "README*" 2>/dev/null | grep -q . ; then
    echo "===== $dir/README ====="
    cat $dir/README*
    echo "----------"
    echo
    fi >>"${Buildfolder}/$Pkgbase/README"
    if [ -r $dir/configure ]; then
      echo "===== $dir/configure --help ====="
      pushd $dir
      ./configure --help
      popd
      echo "----------"
      echo
    fi >>"${Buildfolder}/$Pkgbase/README"
    if [ -r $dir/CMakeLists.txt ]; then
      echo "===== $dir/cmake -LH ====="
      mkdir src/build_tmp
      cd src/build_tmp
      cmake -LH "${Buildfolder}/${Pkgbase}_build/$dir/" 2>/dev/null
      cd ../..
      rm -rf src/build_tmp
      echo "----------"
      echo
      break ## only use topmost level cmake
    fi >>"${Buildfolder}/$Pkgbase/README"

Offline

Board footer

Powered by FluxBB