Non-Recursive Automake is the Best Alternative to Automake

People often complain that autotools is complicated. But the alternatives generally involve a learning curve, require large changes to existing projects, and don’t provide the features or the command-line interface that we’ve become used to with autotools, making life difficult for people building tarballs, and for distros’ packaging tools.

One of the biggest annoyances with traditional autotools has been the need for a Makefile.am file in each sub-directory, and the need to create (and link) non-installed convenience libraries in each one. That leads to lots of repetitive Makefile.am code. More code means more errors, less clarity, and difficult refactoring. I had forgotten how much I hated that about autotools when I first learned it.

One of the advantages often mentioned for alternatives such as cmake is the ability to define the build in just one text file. However, automake has supported the non-recursive way for a while. Now you can have just one top-level Makefile.am. The configure.ac is still separate, but that’s fine with me.

Daniel Elstner changed Glom to use a single Makefile.am, removing 47 annoying little Makefile.am files while preserving our special stuff for client-only, Maemo, and Windows builds, with no disruption for developers using the source from git or for people building from the tarball. It’s a great improvement that shows how attractive non-recursive automake can be. OK, so Daniel is an autotools expert, but I’d still rather move from autotools to non-recursive autotools than take the leap of faith needed to move from autotools to something completely different.

Apparently this is also more efficient, leading to faster build times, particularly when building in parallel with the -j option, with more correct dependencies. And there’s no need to mention those convenience libraries repeatedly to work around linker errors.

Together with autoreconf (replacing hand-built autogen.sh files), autotools can be much nicer these days.

21 thoughts on “Non-Recursive Automake is the Best Alternative to Automake

  1. I completely agree with your post. I (and some coworkers) have wrote a build-system template using autotools with “state of the art” techniques and one of them is, for sure, the single Makefile.am approach.

    The template is well documented and should be useful to whoever is starting a project or wants to migrate to a modern autotools build-system, take a look:

    http://svn.ademar.org/code/trunk/autotools-build_system-template/

    It’s very complete but yet modular. It supports pkgconfig, libtool, check (unit-tests in C), lcov (coverage reports for C/C++), doxygen, gcc warnings, debug mode, etc.

    Patches are always welcome as well.

    Cheers,
    – Ademar

  2. Getting rid of libtool or at least making it a complete no-op on all *sane* configurations (w/ shared library linking & recursive lookups) and do funny stuff on the architechtures that don’t would be a huge improvement. This dumbing down of linking for 99% of all users is terrible.
    Automake can be painful, but libtool is just completely bogus nowadays.

  3. I am at least glad that there is movement in the Autoconf section too.

    For the longest time I thought autoconf devs dont care at all about complaints due to “legacy” reasons.

    I just hope that the net result of all those moves – including binutils gold linker – will eventually lead to an
    improved build time experience for BOTH developers, distribution packagers and “powerusers” who often
    feel left out in decisions imposed onto them by upstream (just look at the craziness of autoconf cmake scons
    libtool and so forth).

    Computers get fast and faster but software concepts get worse and worse – or dont improve as fast.

  4. “.”: That’s what dolt does (unfortunately there’s no project page, bug google for “dolt libtool”). And yes, my template already includes it. :-)

  5. @Stefan: Ooops, thanks for the note. I didn’t realize the page had moved. I have corrected the link in the comment now.

    @pvanhoof: No, you can’t do that anymore. That’s about the only downside. However, there is nothing preventing you from creating a named phony target for a logical subset of your targets, if you need that ability.

    Also note that this isn’t an all-or-nothing decision. You can combine recursive and non-recursive make as you like. Even the new build system of glom still recurses into the po/ and docs/user-guide/ directories, because the associated tools for these parts ship with their own makefiles. (Note that it is possible to list docs/user-guide in the toplevel Makefile.am — it does not have to be an immediate child.)

  6. @mark: To put things into perspective:

    This is by no means a recent development in the Autotools world. Automake has included explicit support for non-recursive make for a couple of years now! Actually, you got your history exactly the wrong way around. :-) Years ago, the Autotools developers pissed off a lot of people mightily when they made the incompatible switch from Autoconf 2.13 to Autoconf 2.50 and from Automake 1.4 to Automake 1.5. What we now call the “modern” Autotools interface actually had its roots back then. Anything since then has mainly been an incremental improvement upon those major new releases years ago.

    The main problem is that a lot of configure.ac (or even still configure.in) files and Makefile.am files I see in the wild do their work the old way from about ten years ago, except for the necessary compatibility adjustments to make things barely work with the newer releases. And this is not surprising given that new projects usually copy the build infrastructure of some other existing project. And that’s fine, not everyone is a crazy nutcase like me who actually enjoys all this stuff. The trick is to know where to copy from. ;-)

  7. ah thanks, that’s a useful trick; I didn’t know, even if I’ve been using autotools for years.

    Autotools may have its problems and peculiarities (like combining the sh, m4 and make languages), but as of yet, I haven’t see anything that could take it’s place — maybe this quagmire thing? I can sympathize with people from a non-Unix background who suddenly have to get up to speed with autools… it will take a lot of googling, copy-pasting, and things that just don’t work…

    Once you’re past that, it’s not too bad.

  8. Quite a few projects changed from autotools to cmake and they are quite happy with it. and it is really easyer than autotools after a day or 2 of studying. and one *really* large project changed quite a time ago, making it a guinea pig for testing and succeded. ( and it’s nice to see the percentage of the build gently scrolling up )

  9. World needs no alternative to Autoconf/Automake. Both “tools” should simply be gone. Most of the time, they aren’t helping, just adding to the mess.

  10. For me, the ultimate bliss would be to have the Makefile.ams in subdirectories, and automake creates a single, big top-level, non-recursive Makefile.in out of them. (And puts stub Makefiles in the subdirectories so that I can run make there.)

  11. OK, as you are now the second person asking for the ability to run make in a subdirectory, I’d interested in your use case(s) for this. Specifically, would you need to ignore targets not in the chosen directory, or is it merely a convenience for “make -C ..”? It would be trivial to write a script to automatically do the latter.

  12. @Daniel: perhaps a bit unconventional, but I keep my (unit) test cases within the same folder of the library code. When writing new or adjusting existing code I am only interested in that part of the source tree, nothing else. I also use the `subdir-objects’ over the use of convenience libraries as it has come the my notice the latter can introduce a fair amount of additional bloat.

    I do think however, that in case your project has somewhat reached a maintenance only stage, a single rooted Makefile.am might be more beneficial.

  13. When looking for “non-recursive makefile” on internet, I found examples of still structured but non-recursive layout. Ie, you still define piece of Makefile in each directory containing a “buildable”. This piece of Makefile is then read by a global Makefile.

    This is really interesting because a single big BIG Makefile.am is not an easy way to maintain the building process (IMHO). Spliting the building process in many files, containing rules related to files hosted in the same directory is really much more simpler to maintain.

    Does someone know a way to do such thing with automake?

  14. I’m curious how people have dealt with being able to run make in each subdirectory. I’m assuming a layout where there’s one top level Makefile.am that includes Makefile.am’s in each subdirectory. The build directory ends up with a Makefile but subdirectories in the build directory don’t. That’s where I’d like Makefile’s to end up.

    Do people add Makefile.in’s to each subdirectory that do something like this (untested):

    all:
    make -C @top_builddir@

    (with the appropriate AC_CONFIG_FILES call in configure.ac)

    What else have people tried?

    Thanks.

    -DB

  15. I have started using a non-recursive make system in which each directory contains two makefiles:
    A file “makefile.inc” contains variables with lists of header and source files in that directory and
    its subdirectories. A file “makefile” can be used to compile only the code in a directory and it subdirectories. Each makefile.inc file includes the makefile.inc files defined in its subdirectories,
    so that the makefile.inc file in the root source directory has access to a complete tree, as in any
    distributed non-recursive make system.

    In my version, each makefile.inc file defines several makefile variables that contain lists of the
    header, source, and object files in that directory and its subdirectories. To allow the variable
    names to be generated automatically, the variable names for directory are constructed from
    the path from the source directory to the current directory, by replacing directory separators
    (/) by underscores (_).

    For example, consider project with a directory tree

    src/
    sim/
    util/
    timer/
    random/
    random.h
    random.cpp

    in which I have shown only directories and the header and source files in subdirectory
    src/util/random. The file makefile.inc in src/util/random, which has no subdirectories,
    would contain the variable definitions:

    util_random_HDRS=$(SRC_DIR)/util/random.h
    util_random_SRCS=$(SRC_DIR)/util/random.cpp
    util_random_OBJS=$(util_SRCS:.cpp.o)

    The makefile.inc in the directory src/util, which has two subdirectories but no
    source files, would contain

    util_HDRS=util_random_HDRS util_timer_HDRS
    util_SRCS=util_random_SRCS util_timer_SRCS
    util_OBJS=$(util_SRCS:.cpp.o)

    None of the makefile.inc files define the variable $(SRC_DIR). Instead,
    the makefile for each directory contains a definition of $(SRC_DIR) as a relative
    path to the root source directory, which generally consists of one or more
    concatentated “../” strings. Each makefile also includes the makefile.inc in
    the root src/ directory, so that it has access to variables that contain all the
    header, source, and object files in the project, defined as paths relative to
    the current directory.

    The makefile in each directory has an “all” target that makes all the object
    files in the *_OBJS variable defined in the same directory, which contains
    all the object files in that directory and its subdirectories.

    The build rule for *.o files also uses the $(SRC_DIR) variable to define an
    include “-I $(SRC_DIR)”, and uses includes that are all defined using the
    #include notation in which the paths within the brackets are paths
    defined from the root source directory. This guarantees that the same
    form is used for each path in the preprocessor as that used in the makefile.

    The goal of this is a makefile system in which entering “make all” from any
    directory causes compilation all of the source files in the current directory
    and its subdirectories, and nothing else, while having access to a directed
    acyclic graph for the entire project. I wanted to have local makefiles during
    development in order to make it easy to recompile a file that I am working
    on without leaving the directory containing the file.

    The proliferation of makefile variable names based on directory paths is
    manageable because all of my makefiles are generated by a python script
    that descends the directory tree recursively and includes all the header and
    source files that it finds. I can always rebuild the whole system of makefiles
    by rerunning this script. The current version of the script reads a third file
    that is present in each directory that contains a list of subdirectories with
    source code, and that specifies the path $(SRC_DIR) from the directory to
    the root source directory and to this directory from the root source directory,
    though I could probably have generated this automatically as well. The script
    allows me to add executable files and/or libraries as additional targets in
    some directories.

    Now that I have the required scripts written, this system is working well for
    me, and appears to be easily portable to other projects. It does have a few
    limitations that spring from the fact that all the path names are defined
    relative to $(SRC_DIR). These are:

    i) I am using the gcc compiler for dependency generation, but have to do some
    processing of the output in order to get the desired form of the path for the
    target. This requires another script that is called for automatic dependency
    generation, which generates a *.d dependency file for each *.cpp source file.
    The scheme requires that you write a script for dependency generation that
    can generate the path to the target in terms of the variable $(SRC_DIR).

    ii) It is easy to enter “make all” from any directory, but harder to specify a single
    file as a target. To enter a single file as a target, you would have to use the exact
    form of the target expected by the makefile in the current directory, which is
    generally something like ../../util/random/random.o with one or more ../ strings
    in order to refer to the root source directory. In practice, this has not been a
    problem, since I am usually working on one file at a time, and entering “make all”
    from a directory normally just compiles the source file I am working on, if none
    of the other source files in my current working directory or its sub-directories
    have changed. If I get in a situation in which I have temporarily broken one source
    file, so it will not compile, but am working on another, I have to temporarily remove
    the broken source file from the directory_name_OBJS variable in the makefile.inc
    file in order to allow myself to continue to use “make all”.

    iii) The mangling of the directory paths to create variable names requires that
    I avoid directory names with underscores. I don’t think that file names with
    underscores would cause any trouble, though I haven’t tried this.

    This is the first time I have created a build system for a multi-directory project.
    I developed this after several unworkable attempts, and after reading articles on
    non-recursive make and automatic dependency generation. I think the main
    difference from other systems is probably my insistence on being able to type
    “make all” from any directory during development and have it do what I expect.
    I’d appreciate comments about the extent to which it is similar to what others
    have come up with, how it might be improved, or if there is some other
    system that uses a non-recursive make but also makes it easy to compile only
    part of a source tree.

Leave a Reply to Javier Jardón Cancel reply

Your email address will not be published. Required fields are marked *