Why doesn't gnu make's "override" pass through to sub-makes?

Question

Note: This question was originally posted as a rant by a now-deleted user, but there was a valid question behind the rant; this is my attempt to provide an answer.

Given the Makefile:

ifeq "$(MAKELEVEL)" "0"

# Override the command-line specification of "foo".
override foo=replaced
export foo

all::
    @echo outer: foo is "$(foo)"
    @$(MAKE)

else

# Variable 'foo' was "exported" from the top-level Makefile.
all::
    @echo inner: foo is "$(foo)"

endif

The expectation is that export foo will cause make to export the value defined in the override declaration. But it doesn't:

$ make -s foo=original
outer: foo is replaced
inner: foo is original

Answer 1

The expectation is probably reasonable, but it turns out that this is not the way Gnu make works. It could well be that the make documentation could be improved to clarify the process, but the hints all seem to be there.

How variables get their values

A variable can be set by the programmer in three ways:

• On the command line with a var=value command-line argument
• Explicitly in the make file
• From the environment

The above list is the normal priority order; the first definition found in the list "wins". However, you can use the override directive to swap the priorities of the first two methods. (You can also use the -e flag to make to swap the priorities of the last two methods. The -e flag is required by Posix, but its use is discouraged and it does not interact will with override.)

How variables are passed to sub-makes

make is very similar to a shell in that the environment is used to pass variable values. If a variable is marked as exported, then its value is placed into the environment for any processes initiated by make, including sub-makes. As with the shell, a variable is marked as exported if its definition came from the environment or if it is explicitly marked as exported with the export directive. Variables are also exported if they were set on the command line.

However, there is another mechanism by which variables on the command-line are passed to subprocesses: the MAKEFLAGS exported variable.. MAKEFLAGS contains (most) command-line options as well as all of the command-line variable overrides. If make finds MAKEFLAGS in the environment, it merges the settings in that variable with the ones actually specified on its command line. That means that command-line variable settings in a make will also take priority in a sub-make.

Since command-line variable settings are passed through the MAKEFLAGS variable, they are not subject to any changes in the makefile. A makefile can unexport or override a variable set on the command-line, but that will only affect the value (or presence) of the variable in the environment; it does not remove or change the value from MAKEFLAGS.

Resolution

So if the intent is to override (or modify) a command-line variable both in the make itself and in the environment of sub-makes, it is necessary to use both an override and an explicit modification of MAKEFLAGS. (As explained in the make manual, MAKEFLAGS is actually recursively composed using the MAKEOVERRIDES variable, so we actually modify that variable.)

ifeq "$(MAKELEVEL)" "0"

# Override the command-line specification of "foo".
override foo=replaced
MAKEOVERRIDES += foo=replaced

all::
    @echo outer: foo is "$(foo)"
    @$(MAKE) -s

else

# Variable 'foo' was "exported" from the top-level Makefile.
all::
    @echo inner: foo is "$(foo)"

endif

And now we get the expected result:

$ make -s foo=original
outer: foo is replaced
inner: foo is replaced

Real-life application: dealing with whitespace

The primary intention of overrides is to allow the makefile to append words to a variable possibly provided on the command line. The example provided in the gnu make manual is insisting that CFLAGS always includes the -g flag, even if it were specified on the make command line:

override CFLAGS += -g

Passing the append through to a sub-make needs a little caution; in particular, the obvious:

MAKEOVERRIDES += CFLAGS=$(CFLAGS)  # Don't do this

won't work because the whitespace inside the CFLAGS variable will not be escaped when it is added to MAKEFLAGS; the result will be that MAKEFLAGS will look something like this:

-- CFLAGS=-O3 CFLAGS=-O3 -g

instead of the desired

-- CFLAGS=-O3 CFLAGS=-O3\ -g

If the value assigned to CFLAGS on the command line included whitespace, the whitespace is escaped in MAKEFLAGS. The particular escaping mechanism used is not documented, and Posix only requires that there be some mechanism; apparently, Gnu make uses backslash. It would be possible to manually backslash escape the whitespace, resulting in something like this:

# Don't do this either
MAKEOVERRIDES += CFLAGS=$(subst $(space),\ ,$(CFLAGS))

(The definition and use of space is based on an example in the gnu make manual.)

But it is actually easier to just use an append assignment in MAKEOVERRIDES, which is undocumented but appears to work. It works on the command line, too.

override         CFLAGS+=-g
MAKEOVERRIDES += CFLAGS+=-g

Important Note as of make v4.1: A bit of testing revealed that the above stanza will only work if CFLAGS (or some other variable) is actually set on the command-line. I reported this bug as Savannah issue 46013, with a very simple fix in the bug report. In the meantime, if you really want to do this, use the following workaround:

override         CFLAGS+=-g
MAKEOVERRIDES += CFLAGS+=-g

# This line is necessary in case there were no command-line overrides.
# In effect, it produces a command-line override, although that value
# will not be passed on to sub-makes.
MAKEFLAGS     += dummy=dummy

Update May 19, 2019: Today I was informed that a fix for the bug referenced above has been committed, so it should be fixed in the next gmake release.