make: Run several tasks in parallel and wait for completion

Question

One target in my makefile is a very CPU and time consuming task. But I can split the workload and run the task several times in parallel to speed up the entire process.

My problem is that make doesn't wait for all processes to complete.

Consider this simple script, named myTask.sh:

#!/bin/bash

echo "Sleeping $1 seconds"
sleep $1
echo "$1 are over!"

Now, let's call this from a bash script, and use wait to wait for all tasks to complete:

#!/bin/bash

echo "START"
./myTask.sh 5 &
./myTask.sh 15 &
./myTask.sh 10 &

wait  # Wait for all tasks to complete

echo "DONE"

The output is as expected:

START
Sleeping 15 seconds
Sleeping 5 seconds
Sleeping 10 seconds
5 are over!
10 are over!
15 are over!
DONE

But when trying the same in a Makefile:

test:
    echo "START"
    ./myTask.sh 5 &
    ./myTask.sh 15 &
    ./myTask.sh 10 &
    wait
    echo "DONE"

it doesn't work:

START
Sleeping 5 seconds
Sleeping 15 seconds
Sleeping 10 seconds
DONE
sweber@pc:~/testwait $5 are over!
10 are over!
15 are over!

Of course, I could create multiple targets which can be "built" in parallel by make, or let make just run the bash script which runs the tasks. But is there a way to do it more like what I already tried?

Answer 1

Each individual logical line in a recipe is invoked in its own shell. So your makefile is basically running this:

test:
        /bin/sh -c 'echo "START"'
        /bin/sh -c './myTask.sh 5 &'
        /bin/sh -c './myTask.sh 15 &'
        /bin/sh -c './myTask.sh 10 &'
        /bin/sh -c 'wait'
        /bin/sh -c 'echo "DONE"'

You should make it all run in a single shell using semicolons and backslash/newlines to combine the physical lines into one logical line:

test:
        echo "START"; \
        ./myTask.sh 5 & \
        ./myTask.sh 15 & \
        ./myTask.sh 10 & \
        wait; \
        echo "DONE"

Answer 2

Why not use the mechanisms already built in make? Something like this:

task%:
        sh task.sh $*

test: task5 task15 task10
        echo "DONE"

You will be able to adjust the level of parallelism on the make command line, instead of having it hard-coded into your makefile (e.g use 'make -j 2 test' if you have 2 cores available and 'make -j 32 test' if you have 32 cores)