Out of source build fails in autotools

Question

This might be a noobish question but I just recently started using autotools to simplify my requirements on things. Namely, I like the idea that the end user really doesn't need to go get anything to build my project on their local system.

In order to keep my source directory kind of clean I have extra configuration files saved in a config directory and I want to mkdir a build directory to keep my configuration files for the build snugly kept away. Here's my current project tree:

.
├── aclocal.m4
├── AUTHORS
├── autom4te.cache
│   ├── output.0
│   ├── output.1
│   ├── requests
│   ├── traces.0
│   └── traces.1
├── bin
├── build
├── Changelog
├── config
│   ├── compile
│   ├── install-sh
│   └── missing
├── config.h.in
├── configure
├── configure.ac
├── doc
├── lib
├── Makefile.am
├── Makefile.in
├── NEWS
├── README
├── README.html
├── README.md
├── src
│   ├── Makefile.am
│   ├── main.c
└── test

Makefile.am

SUBDIRS = src
dist_doc_data = README

src/Makefile.am

bin_PROGRAMS = testprog
testprog_SOURCES = main.c

configure.ac

AC_PREREQ(2.59)

AC_INIT([testprog], [1.0], [[email protected]])

AC_CONFIG_AUX_DIR([config])
AM_INIT_AUTOMAKE([-Wall -Werror foreign])

AC_CONFIG_HEADERS([config.h])

AC_PROG_CC

AC_CONFIG_FILES([Makefile])
AC_OUTPUT

And finally the result of building:

mkdir build
cd build
../configure
make

Configure output

checking for a BSD-compatible install... /usr/bin/install -c
checking whether build environment is sane... yes
checking for a thread-safe mkdir -p... /usr/bin/mkdir -p
checking for gawk... gawk
checking whether make sets $(MAKE)... yes
checking whether make supports nested variables... yes
checking for gcc... /usr/bin/clang
checking whether the C compiler works... yes
checking for C compiler default output file name... a.out
checking for suffix of executables... 
checking whether we are cross compiling... no
checking for suffix of object files... o
checking whether we are using the GNU C compiler... yes
checking whether /usr/bin/clang accepts -g... yes
checking for /usr/bin/clang option to accept ISO C89... none needed
checking whether /usr/bin/clang understands -c and -o together... yes
checking for style of include used by make... GNU
checking dependency style of /usr/bin/clang... none
checking that generated files are newer than configure... done
configure: creating ./config.status
config.status: creating Makefile
config.status: creating config.h
config.status: executing depfiles commands

Make output

CDPATH="${ZSH_VERSION+.}:" && cd .. && /bin/sh /home/me/projects/testprog/config/missing aclocal-1.15 
 cd .. && /bin/sh /home/me/projects/testprog/config/missing automake-1.15 --foreign
CDPATH="${ZSH_VERSION+.}:" && cd .. && /bin/sh /home/me/projects/testprog/config/missing autoconf
/bin/sh ./config.status --recheck
running CONFIG_SHELL=/bin/sh /bin/sh ../configure CC=/usr/bin/clang --no-create --no-recursion
checking for a BSD-compatible install... /usr/bin/install -c
checking whether build environment is sane... yes
checking for a thread-safe mkdir -p... /usr/bin/mkdir -p
checking for gawk... gawk
checking whether make sets $(MAKE)... yes
checking whether make supports nested variables... yes
checking for gcc... /usr/bin/clang
checking whether the C compiler works... yes
checking for C compiler default output file name... a.out
checking for suffix of executables... 
checking whether we are cross compiling... no
checking for suffix of object files... o
checking whether we are using the GNU C compiler... yes
checking whether /usr/bin/clang accepts -g... yes
checking for /usr/bin/clang option to accept ISO C89... none needed
checking whether /usr/bin/clang understands -c and -o together... yes
checking for style of include used by make... GNU
checking dependency style of /usr/bin/clang... none
checking that generated files are newer than configure... done
configure: creating ./config.status
 /bin/sh ./config.status
config.status: creating Makefile
config.status: creating config.h
config.status: config.h is unchanged
config.status: executing depfiles commands
(CDPATH="${ZSH_VERSION+.}:" && cd .. && /bin/sh /home/me/projects/testprog/config/missing autoheader)
rm -f stamp-h1
touch ../config.h.in
cd . && /bin/sh ./config.status config.h
config.status: creating config.h
config.status: config.h is unchanged
make  all-recursive
make[1]: Entering directory '/home/me/projects/testprog/build'
Making all in src
/bin/sh: line 20: cd: src: No such file or directory
make[1]: *** [Makefile:352: all-recursive] Error 1
make[1]: Leaving directory '/home/me/projects/testprog/build'
make: *** [Makefile:293: all] Error 2

I have purchased a copy of Autotools as well as followed the Automake canon on gnu.org. I'm unsure where to go and neither of these sources have helped me solve it.

From what it appears it is looking for src inside of the build directory and naturally it does not exist. But every resource I look at for out of source builds doesn't seem to indicate this problem can even happen. What am I doing wrong?

Answer 1

Since you invite an explanation from someone more experienced, I'll start by recapitulating my comment on your answer: you need to tell Autoconf which files the configure script is supposed to create, and this is the function of the AC_CONFIG_FILES macro. You having failed to tell it that you wanted a src/Makefile was an error for both in-source and out-of-source building.

But you seem to want more explanation of what's going on under the covers, so here goes.

Your project's source tree is the directory tree rooted at the directory containing the top-level configure script. A build tree is rooted at the working directory from which you run configure. When a build tree differs from the source tree, you are performing an out-of-source build. configure emits all its output into the build tree, from temporary files created by its tests to its log files to the output files designated via AC_CONFIG_FILES macros. When appropriate, it also creates any subdirectories specified for output files, such as src for your src/Makefile. It does not copy anything from the source directory per se, nor does it create more directories than are needed to place its output files.

The magic of out-of-source builds happens in make. It relies on the "VPATH" feature of some make implementations, including GNU Make, by which make can be given a search path for prerequisites and targets of build rules. Make always looks first for these relative to its (then) working directory, before consulting the VPATH, but in a fresh out-of-source build, the only things it can initially find there are files created by configure. (As the build progresses, it may find files it previously built, too.)

But VPATH is orthogonal to recursive make, such as you are leveraging via Automake's SUBDIRS feature. In fact, recursive make is not so much an explicit feature of make at all but rather a special case of ordinary make behavior (though GNU make, at least, recognizes it and provides a little bit of special support). Recursive make operates simply by putting a make command into the recipe of a regular make rule, but VPATH has nothing (directly) to do with the commands in recipes.

In particular, VPATH is of no help if, say, make tries to recurse into a directory that does not exist in the build tree. This is not a problem for a correct Autotools build system in which all Makefiles are generated by configure, because make will recurse only into a directories that (it expects will) have Makefiles, and configure will have created those directories along with the Makefiles in them. make, however, knows only about the Makefile itself, not about its genesis. If it tries to recurse into a directory that does not exist, all it knows is that a command it tried to execute failed for that reason.

That's why you got a less-than-helpful error message in your case. But also because out-of-source building is a capability to engage after your build already works in-source. You would have gotten a somewhat more helpful error message from an in-source build, and Automake-based Makefiles are very good about cleanup. make clean and / or make distclean and / or make maintainer-clean work as advertised.

Bonus

Now to figure out how to make it put the binary in bin instead of src...

Why? Seriously, what's the point, especially when you're already performing an out-of-source build? make knows where to find the binaries when you're ready to install them (or to clean them up). And, of course, the place make initially puts them is not intended to be their installed location.

But yes, Automake can do this. You just need to tell it to do. With the layout you present, the main part of that would look like this:

src/Makefile.am

bin_PROGRAMS = ../bin/testprog
___bin_testprog_SOURCES = main.c

You may also need to induce the top-level Makefile to ensure the bin/ directory present before it recurses into src/. If that's indeed needed, then I think you can achieve it with this variation:

Makefile.am

SUBDIRS = . src
dist_doc_data = README

all-local:
    mkdir -p bin

clean-local:
    -rm -rf bin

See also:

• Recursive make considered harmful
• Autotools Mythbuster: non-recursive Automake