Makefiles, "configure" files, and other compilation tools -- How do they work? Why do they work the way they do?

Question

I'm still new to the UNIX/Linux world, and, in particular, to related tools, such as the GCC compiler. Namely, I'm still new to makefiles and things like that (I use MinGW on Windows), since so far, most of my development has been with IDEs like Visual Studio and Eclipse.

When I open a typical project folder, I see files like these:

configure
make
INSTALL
install-sh  (or a variant)

There's quite a few things I don't understand here, and my main questions are these:

1. What's the difference between these, and why do we need each one? (In the IDE world, all you have is a project file; that's it. So I'm confused why we have more than just a makefile here.)

2. How are these files generated? For small projects, you could probably write them by hand, but for big projects like GCC, it would be ridiculous to even try. I've found editing these files to be a pain, and I've come to the conclusion that I'm not supposed to be modifying them by hand. But if so, then what tools do people typically use for adding and modifying these files?

3. Why don't they use wildcards inside a makefile? Why is there a single line for every single object file? Is this because of a restriction, or is there an advantage to it?

4. What's the difference between having a shell script that calls the compiler with every file, and having a makefile that does the same thing? In Windows, I'd be very tempted to just create a batch file inside the folder, and compile everything with that -- no need for more than just one file. What's the difference? Why isn't there just a single .sh file, to compile everything?

Bonus question:

• Is there a "standard" makefile format? I've seen different make tools that don't accept each other's formats... how do I know what to use? Is GCC just the usual tool here, or is there some standard everyone should follow?

I'll probably have more questions as I learn more about the structure of projects like this, but for now, these are my biggest questions. :)

Answer 1

1. In your average package, those files each have an individual purpose. This is concordant with the unix philosophy of "each program should do one thing and do that thing well". In most projects, you'll see files like:

   • configure
   • configure.ac
   • Makefile
   • Makefile.in
   • Makefile.am
   • install-sh
   • INSTALL

   configure is (usually) a shell script that checks your system for all required features before building anything. Makefile.in is a template for a Makefile. The results of the configure tests are substituted into Makefile.in to generate Makefile. This is to deal with people having things (compilers, headers, libraries) in obscure paths, cross-compiling (e.g., building for ARM on x86), optional library support (some programs have additional features that can be switched on or off), compiling with different options, and so on. Writing a one-size-fits-all Makefile is actually really hard.

   As you've noticed, the configure script itself is a mess. It's not meant to be seen by mortal eyes, nor edited by mortal hands. It's actually the result of compiling configure.ac, using a program called autoconf. autoconf is a package of macros for and a wrapper around the m4 macro processor, which was about the only good tool for this sort of thing at the time (autoconf is really quite old software, but has aged remarkably well). autoconf lets the developer easily write tests to check for headers, libraries or programs that are required to build the software (and this changes from program to program).

   If you dig a little deeper, you'll notice that Makefile.in also tends to be a little ugly. This is because writing good Makefiles is often a lot of boilerplate, and that's inspired another tool, automake. automake compiles Makefile.am (which is often short and declarative) into Makefile.in (which is enormous), which is then compiled into Makefile by configure (essentially).

   install-sh is a script that is distributed with automake but copied into other packages. It exists as a replacement if the version of install on the system is crap (install copies files into the installation directory. Some really old systems had broken versions of install, and automake is pretty conservative about dropping warnings for old systems). Some other scripts that fulfil similar roles are compile, depcomp and ylwrap.

   INSTALL is just a document that describes how to install the package. It's usually boilerplate content copied into the package by automake.

2. I've answered this inline above, but here is the summary:

   • configure.ac ==[autoconf]=> configure
   • Makefile.am ==[automake]=> Makefile.in ==[configure]=> Makefile

   Where the program responsible is inside the arrow. To understand this in detail, I recommend this autotools tutorial. Don't be put off by the page count, most of it is diagrams appearing piece by piece.

3. Wildcards in Makefiles are sometimes used. GNU Make, for example, supports a $(wildcard) function, where you can write something like:

   SOURCES := $(wildcard src/*.c)

   The main reason features like $(wildcard) aren't used is that they are extensions, and automake tries very hard to generate Makefiles that will work with any POSIX-compliant make. After a project becomes mature, the list of files to compile doesn't change all that much, anyway.

   A second reason files are listed explicitly is when programs get optional features. Wildcards are no longer appropriate, and you instead have to list the conditions under which additional features are to be compiled in.

4. A Makefile tracks dependencies between files, where a shell script cannot (not without significant effort, anyway).

If you have a Makefile rule like:

foo.out: foo.in
        generate-foo foo.in

It tells make that if foo.in is newer than foo.out, a new foo.out can be created by executing generate-foo foo.in. This saves lots of redundant work on big projects, where you might only change one or two files between recompilations.

Your bonus question appears a little ill-posed. The most common make is probably GNU Make, although I'd guess that the BSD make would be a close second, followed by the various proprietary make versions supplied with Solaris, AIX and so on.

These all accept the same basic structure in Makefile (because POSIX says so), but might have vendor-specific extensions to the syntax.

GCC is not a build tool like make. GCC is a command-line compiler, akin to cl.exe on windows.

Answer 2

There are a lot of reasons for different things, but in the end it boils down to portability. During the unix wars, every platform was different, and people tried to make software that would work on as many as possible. So common denominators had to be found. sh was one such commonality, so configure scripts were written to use maximally portable sh. Maximal portability implies using a minimal set of features. Different implementations of make vary greatly in the features they support, so Makefiles were written that used as few features as possible. Things are slightly better today, but portability concerns still drive a lot of the things you are seeing.

Answer 3

1. configure is a script that sets up the build environment and possibly generates the makefile. With make you build the project, with make install or install.sh or similar, you install the compiled files. If you want to test something locally, you might want to leave the final part out and so it's seperate.
2. You can use GNU autotools.
3. Can you give specific examples? Makefiles do support wildcards and they are used from time to time.
4. Makefile handles dependencies and automatic recompilation of only the needed files if you change something in a source file and then decide to rebuild the project. With a batch file you'd be recompiling everything, with Makefiles you're only compiling what was changed.