Download list of files from list of URLs as prerequisites in makefile

Question

I'd like to have a makefile for compiling a project, and I need some files (binaries) to be downloaded if they are missing, as it's not a good idea to track them with git.

I have a list of URLs and a list of given names for them (and I'd like to be able to choose the name to save them, as I cannot control the URL name and mostly names are quite awful and non-descriptive). Let's say that the files will always be the same or that we don't care if they change in the server (we do not need to download them again).

So my idea was to write something like this in my makefile:

files = one two three four

main : $(files)
  command to do when files are present

but I want each of them to be downloaded if they are not already present, and each one from its own URL, so I'd need something like an iteration over the elements of $(files).

I'll write the idea I have in my mind mixing python code and what I know from makefiles. I know it's rubbish, but I don't know any other way to do it (the question is how to write it well, basically) and I think it's quite easy to understand.

urls = url1 url2 url3 url4

for i in len(files):
    $(files)[i] :
        curl -Lo $(files)[i] $(urls)[i]

Therefore the question is: how should I do this?

I've been looking the make documentation for a while and I cannot find the proper way to do it without writing the filename several times (which I think should be avoided and variables should be used instead). Maybe canned recipes are the way, but I cannot see how.

Answer 1

There are several ways to do what you want. Warning: they are a bit tricky because they use advanced features of GNU make:

Solution #1:

files := one two three four
urls := url1 url2 url3 url4

main: $(files)
    @echo 'command to do when files are present'

# $(1): file name
# $(2): url
define DOWNLOAD_rule
$(1):
    @echo 'curl -Lo $(1) $(2)'
endef
$(foreach f,$(files),\
  $(eval $(call DOWNLOAD_rule,$(f),$(firstword $(urls))))\
  $(eval urls := $(wordlist 2,$(words $(urls)),$(urls)))\
)

I replaced the recipes by echos for easier testing:

$ make -j
curl -Lo one url1
curl -Lo four url4
curl -Lo three url3
curl -Lo two url2
command to do when files are present

Explanations:

The DOWNLOAD_rule multi-line variable is a template rule for a download operation where $(1) represents the file name and $(2) represents the corresponding URL.

You can perform the substitution of $(1) and $(2) in DOWNLOAD_rule, and instantiate the result as make syntax with:

$(eval $(call DOWNLOAD_rule,one,url1))

This is the same as if you wrote:

one:
    @echo 'curl -Lo one url1'

The foreach function allows to loop over a list of words. So:

$(foreach f,$(files),$(eval $(call DOWNLOAD_rule,$(f),url1)))

would instantiate the rule for all files... except that the URL would be the same (url1) for all. Not what you want.

In order to get the URL corresponding to each file, we can put two different calls to the eval function in our foreach loop:

• a first one that instantiates the rule with the current file name and the first URL in $(urls),
• a second one that removes the first word from $(urls) and re-assigns the result to urls.

Note the := assignments, they are essential. The default = (recursive expansion) would not work here.

$(wordlist 2,$(words $(urls)),$(urls)) may look complicated but it is not:

• $(wordlist s,e,l) expands as words number s to e in list l (words are numbered from 1 to length of l)
• $(words l) expands as the number of words in list l

So, if $(urls) is, e.g., url2 url3 url4:

• $(words $(urls)) expands as 3
• $(wordlist 2,$(words $(urls)),$(urls)) expands as url3 url4 because it is the same as $(wordlist 2,3,url2 url3 url4)

Solution #2:

It is also possible to pack the second eval in the DOWNLOAD_rule variable, but there is another aspect to consider: the recipe is expanded by make just before being passed to the shell. A target-specific variable (url), which is expanded during the first pass of analysis, solves this:

files := one two three four
urls := url1 url2 url3 url4

main: $(files)
    @echo 'command to do when files are present'

# $(1): file name
define DOWNLOAD_rule
$(1): url := $$(firstword $$(urls))
$(1):
    @echo 'curl -Lo $(1) $$(url)'
urls := $$(wordlist 2,$$(words $$(urls)),$$(urls))
endef
$(foreach f,$(files),$(eval $(call DOWNLOAD_rule,$(f))))

Note the $$ in the definition of DOWNLOAD_rule, they are needed because eval expands its parameter and make will expand it a second time when analysing the result as regular make syntax. The $$ is a way to protect our variable references from the first expansion, such that what is instantiated by eval during the four iterations of foreach is:

one: url := $(firstword $(urls))
one:
    @echo 'curl -Lo one $(url)'
urls := $(wordlist 2,$(words $(urls)),$(urls))

two: url := $(firstword $(urls))
two:
    @echo 'curl -Lo two $(url)'
urls := $(wordlist 2,$(words $(urls)),$(urls))

three: url := $(firstword $(urls))
three:
    @echo 'curl -Lo three $(url)'
urls := $(wordlist 2,$(words $(urls)),$(urls))

four: url := $(firstword $(urls))
four:
    @echo 'curl -Lo four $(url)'
urls := $(wordlist 2,$(words $(urls)),$(urls))

which will do exactly what we want. While without the $$ it would be:

one: url := url1
one:
    @echo 'curl -Lo one '
urls := url2 url3 url4

two: url := url1
two:
    @echo 'curl -Lo two '
urls := url2 url3 url4

three: url := url1
three:
    @echo 'curl -Lo three '
urls := url2 url3 url4

four: url := url1
four:
    @echo 'curl -Lo four '
urls := url2 url3 url4

Remember this: when using eval there are two expansions and $$ is frequently needed to escape the first one.

Solution #3:

We can also declare one variable per file (<file>-url) to store the URL, and a macro to set these variables and the list of files (files):

# Set file's URL and update files' list
# Syntax: $(call set_url,<file>,<url>)
set_url = $(eval $(1)-url := $(2))$(eval files := $$(files) $(1))

$(call set_url,one,url1)
$(call set_url,two,url2)
$(call set_url,three,url3)
$(call set_url,four,url4)

main: $(files)
    @echo 'command to do when files are present'

# $(1): file name
define DOWNLOAD_rule
$(1):
    @echo 'curl -Lo $(1) $$($(1)-url)'
endef
$(foreach f,$(files),$(eval $(call DOWNLOAD_rule,$(f))))

Solution #4:

Finally, we can use the excellent GNU Make Standard Library (gmsl) by John Graham-Cumming and its associative arrays to do about the same as in solution #3. We can, for instance, define an associative array named urls with the file names as keys and the URLs as values:

include gmsl

$(call set,urls,one,url1)
$(call set,urls,two,url2)
$(call set,urls,three,url3)
$(call set,urls,four,url4)

files := $(call keys,urls)

main: $(files)
    @echo 'command to do when files are present'

# $(1): file name
define DOWNLOAD_rule
$(1):
    @echo 'curl -Lo $(1) $$(call get,urls,$(1))'
endef
$(foreach f,$(files),$(eval $(call DOWNLOAD_rule,$(f))))

Answer 2

FILES := file1 file2 file3

main : $(FILES)
    command to do when files are present

file1: firstuglyurl
file2: seconduglyurl
file3: thirduglyurl

$(FILES):
    curl -Lo $@ $<

[EDIT] THAT WAS A VERY STUPID SOLUTION, I DON'T KNOW WHAT I WAS THINKING. Try this:

FILES := file1 file2 file3

main : $(FILES)
    command to do when files are present

file1: URL:=firstuglyurl
file2: URL:=seconduglyurl
file3: URL:=thirduglyurl

$(FILES):
    curl -Lo $@ $(URL)

If your map (URL=>filename) is in a file and you don't want to maintain it in the makefile by hand, you can have Make import it without too much trouble, just tell us the format.

Note that this does not check for files that are present but outdated, i.e. a more recent version of the file exists at the URL (which can perhaps be done, but it's tricky...).