Logo Questions Linux Laravel Mysql Ubuntu Git Menu
 

How can I write a makefile to auto-detect and parallelize the build with GNU Make?

Not sure if this is possible in one Makefile alone, but I was hoping to write a Makefile in a way such that trying to build any target in the file auto-magically detects the number of processors on the current system and builds the target in parallel for the number of processors.

Something like the below "pseudo-code" examples, but much cleaner?

all:     @make -j$(NUM_PROCESSORS) all 

Or:

all: .inparallel     ... build all here ...  .inparallel:     @make -j$(NUM_PROCESSORS) $(ORIGINAL_TARGET) 

In both cases, all you would have to type is:

% make all 

Hopefully that makes sense.

UPDATE: Still hoping for an example Makefile for the above. Not really interested in finding the number of processes, but interested in how to write a makefile to build in parallel without the -j command line option.

like image 558
dlamotte Avatar asked Mar 26 '10 23:03

dlamotte


People also ask

How do you parallelize a makefile?

To start GNU Make in parallel mode it's enough to specify either the -j or --jobs option on the command-line. The argument to the option is the maximum number of processes that GNU Make will run in parallel. For example, typing make --jobs=4 will allow GNU Make to run up to four subprocesses in parallel.

What is $@ and in makefile?

The $@ and $< are called automatic variables. The variable $@ represents the name of the target and $< represents the first prerequisite required to create the output file. For example: hello.o: hello.c hello.h gcc -c $< -o $@ Here, hello.o is the output file.

What is $( make in makefile?

And in your scenario, $MAKE is used in commands part (recipe) of makefile. It means whenever there is a change in dependency, make executes the command make --no-print-directory post-build in whichever directory you are on.

What is $( q in makefile?

Q is defined somewhere after those line. Since makefile have peculiar concept of variable (which is expandable), it can be implement in anywhere. Q is used to whether show message or not (Q maybe for Quiet).


1 Answers

The detection part is going to be OS dependent. Here's a fragment that will work on Linux and Mac OS X:

NPROCS:=1 OS:=$(shell uname -s)  ifeq($(OS),Linux)   NPROCS:=$(shell grep -c ^processor /proc/cpuinfo) endif ifeq($(OS),Darwin) # Assume Mac OS X   NPROCS:=$(shell system_profiler | awk '/Number Of CPUs/{print $4}{next;}') endif 

To get it working you are probably going to have to re-invoke make. Then your problem is preventing infinite recursion. You could manage that by having two makefiles (the first only resetting the -j value), but it is probably possible to finesse it.

like image 186
dmckee --- ex-moderator kitten Avatar answered Sep 27 '22 21:09

dmckee --- ex-moderator kitten