Why export variables with shell command in Makefile takes a very long time?

Question

Here is my test Makefile:

VERSION ?= $(shell cat VERSION | grep "VERSION" | cut -d'=' -f2)
VERSION_MAJOR ?= $(shell echo $(VERSION) | cut -d'.' -f1)
VERSION_MINOR ?= $(shell echo $(VERSION) | cut -d'.' -f2)
VERSION_PATCH ?= $(shell echo $(VERSION) | cut -d'.' -f3)

DATE ?= "$(shell date +"%Y-%m-%dT%H:%M")"

SOME_NESTED ?= "-X=aaa=$(VERSION_MAJOR) -X=bbb=$(VERSION_MINOR) -X=ccc=$(VERSION_PATCH) -X=ddd=$(DATE)"

#.EXPORT_ALL_VARIABLES:


t:
        echo "Test $(SOME_NESTED)"

And a test file VERSION:

echo "VERSION=1.2.3" > VERSION

This makefile works well until I export the variables with shell call.

# use the original makefile
$ time make t
echo "Test "-X=aaa=1 -X=bbb=2 -X=ccc=3 -X=ddd="2023-10-26T10:00"""
Test -X=aaa=1 -X=bbb=2 -X=ccc=3 -X=ddd=2023-10-26T10:00
make t  0.02s user 0.00s system 116% cpu 0.021 total

# uncomment the .EXPORT_ALL_VARIABLES
$ time make t
echo "Test "-X=aaa=1 -X=bbb=2 -X=ccc=3 -X=ddd="2023-10-26T10:09"""
### (hanged for seconds here)
Test -X=aaa=1 -X=bbb=2 -X=ccc=3 -X=ddd=2023-10-26T10:09
make t  5.15s user 0.41s system 111% cpu 4.979 total

Uncomment the .EXPORT_ALL_VARIABLES and the command takes 5s to execute.

With make -d t I can see massive log like this:

Makefile:1: not recursively expanding VERSION to export to shell function
Makefile:8: not recursively expanding SOME_NESTED to export to shell function
Makefile:2: not recursively expanding VERSION_MAJOR to export to shell function
Makefile:3: not recursively expanding VERSION_MINOR to export to shell function
Makefile:4: not recursively expanding VERSION_PATCH to export to shell function
Makefile:6: not recursively expanding DATE to export to shell function
Makefile:2: not recursively expanding VERSION_MAJOR to export to shell function
Makefile:3: not recursively expanding VERSION_MINOR to export to shell function
Makefile:1: not recursively expanding VERSION to export to shell function
Makefile:6: not recursively expanding DATE to export to shell function
Makefile:1: not recursively expanding VERSION to export to shell function
Makefile:2: not recursively expanding VERSION_MAJOR to export to shell function
Makefile:3: not recursively expanding VERSION_MINOR to export to shell function
Makefile:4: not recursively expanding VERSION_PATCH to export to shell function
Makefile:6: not recursively expanding DATE to export to shell function
Makefile:2: not recursively expanding VERSION_MAJOR to export to shell function
Makefile:3: not recursively expanding VERSION_MINOR to export to shell function
Makefile:4: not recursively expanding VERSION_PATCH to export to shell function

It seems that make is doing something recursively here.

I noticed that the command to be executed echo "Test "-X=aaa=1 -X=bbb=2 -X=ccc=3 -X=ddd="2023-10-26T10:09""" is displayed with little delay but the actual output delayed for seconds.

Am I using some bad-practice? Why this happened?

My make version:

$ make --version
GNU Make 4.4.1
Built for x86_64-pc-linux-gnu
Copyright (C) 1988-2023 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <https://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.

Answer 1

If a make variable does not already have a value then setting it with ?= makes it a recursively expanded variable. That means that variable and function references in its value are not expanded until the variable itself is expanded. Instead, that is done, recursively, every time the variable itself is expanded. This is the traditional handling of make variables.

For example, when your VERSION variable has recursively expanded value $(shell cat VERSION | grep "VERSION" | cut -d'=' -f2), every appearance of $(VERSION) causes make to launch a shell, and in it to run cat, grep, and cut.

These activities are individually quick on the timescale of an interactive user, but they are nevertheless slow enough that it's not too hard to consume non-trivial time by performing enough iterations. And even without .EXPORT_ALL_VARIABLES, you are performing many iterations: to expand $(SOME_NESTED) once, I count 8 shells, 8 cuts, 4 cats, 4 greps, and 1 date (and 4 echos, but these are likely much cheaper on account of being a shell builtin). That's already pretty heavy.

But if you .EXPORT_ALL_VARIABLES then make expands each one of your variables every time it launches a child process. That includes for every line of every recipe, as well as for more or less each run of the shell function. Even though make is smart enough to recognize and break the recursion inherent in that, you've still massively multiplied the number of processes being run.

Am I using some bad-practice? Why this happened?

At a high level, you're trying to write a makefile as if it were a script. This is a fundamental error that leads to all kinds of trouble, though you can often push through to get such a thing to work.

From a strategy perspective, it's probably unwise to export all variables, but that's moot if you really do want to export the particular variables involved.

At the detail level, GNU make "functions" such as $(shell) are an extension to POSIX make. If you want portability then it is a mistake to use any of them at all. Even if portability is not a concern for you, most GNU extensions are best used sparingly and carefully, and $(shell) especially so.

With that said, it's a bit of a rabbit hole. Supposing that you don't particularly want the deferred recursive expansion of your variables, you can rescue your performance by engaging another extension to make your variables immediately expanded by assigning to them with :=, like so:

VERSION := $(shell cat VERSION | grep "VERSION" | cut -d'=' -f2)
VERSION_MAJOR := $(shell echo $(VERSION) | cut -d'.' -f1)
VERSION_MINOR := $(shell echo $(VERSION) | cut -d'.' -f2)
VERSION_PATCH := $(shell echo $(VERSION) | cut -d'.' -f3)

DATE := "$(shell date +"%Y-%m-%dT%H:%M")"

SOME_NESTED := "-X=aaa=$(VERSION_MAJOR) -X=bbb=$(VERSION_MINOR) -X=ccc=$(VERSION_PATCH) -X=ddd=$(DATE)"

That breaks the conditional assignment semantic, however. If you want to preserve the behavior of accepting values for all of these variables from the environment (or from earlier assignments in the same makefile) while also giving them immediately expanded values, then you need to engage yet another GNU extension: conditionals. For example,

ifndef VERSION
  VERSION := $(shell cat VERSION | grep "VERSION" | cut -d'=' -f2)
endif

ifndef VERSION_MAJOR
  VERSION_MAJOR := $(shell echo $(VERSION) | cut -d'.' -f1)
endif

# ...

But even that's not all the way to your goal, because a variable that obtains its value from the environment is recursively expanded and can contain variable and function references (nasty!). If you want to absolutely ensure that these variables are recursively expanded then you need to follow this model for each one:

ifdef VERSION
  VERSION := $(VERSION)
else
  VERSION := $(shell cat VERSION | grep "VERSION" | cut -d'=' -f2)
endif

Alternative

The traditional way to do this sort of thing would be either

• to use some kind of preparation script in front of make that creates data files or even the makefile itself with appropriate content. For example, the configure script of an Autotools-based project.

  and / or

• to have make store build-time generated content in files instead of in variables. Example:

  t: unnested
          echo "Test $$(<unnested)"
  
  .PHONY: unnested
  unnested:
          @awk -F= '$$1 == "VERSION" { split($$2, v, "."); printf "-X=aaa=%s -X=bbb=%s -X=ccc=%s", v[1], v[2], v[3] }' VERSION > $@
          @echo " -X=ddd=$$(date +'%Y-%m-%dT%H:%M')" >> $@
  

  Along with that, make good use (in recipes) of the standard tools you can rely upon to be present in your build environment. awk, for instance.