Using make to execute independent tasks in parallel

Question

I have a bunch of commands I would like to execute in parallel. The commands are nearly identical. They can be expected to take about the same time, and can run completely independently. They may look like:

command -n 1 > log.1
command -n 2 > log.2
command -n 3 > log.3
...
command -n 4096 > log.4096

I could launch all of them in parallel in a shell script, but the system would try to load more than strictly necessary to keep the CPU(s) busy (each task takes 100% of one core until it has finished). This would cause the disk to thrash and make the whole thing slower than a less greedy approach to execution.

The best approach is probably to keep about n tasks executing, where n is the number of available cores.

I am keen not to reinvent the wheel. This problem has already been solved in the Unix make program (when used with the -j n option). I was wondering if perhaps it was possible to write generic Makefile rules for the above, so as to avoid the linear-size Makefile that would look like:

all: log.1 log.2 ...
log.1:
        command -n 1 > log.1
log.2:
        command -n 2 > log.2
...

If the best solution is not to use make but another program/utility, I am open to that as long as the dependencies are reasonable (make was very good in this regard).

Answer 1

Here is more portable shell code that does not depend on brace expansion:

LOGS := $(shell seq 1 1024)

Note the use of := to define a more efficient variable: the simply expanded "flavor".