Gnu make on multiple host machine?

Question

In clearmake there is an option to pass host names so that to run multiple jobs on these hosts but in gmake there is no option to pass multiple hosts although multiple jobs can be passed. I want to know how i can mimic this clearmake functionality?

Answer 1

The usual answer for people looking to distribute builds run with standard make tools like GNU make across multiple systems is to use something like distcc.

Answer 2

As sfjac writes, GNU Make does not provide any support for distributing processes across machines. So you'd have to set up such support yourself.

One way to do this is by changing the $(SHELL) to something that can take a shell command and execute it on a different host, and also picks the host to use, such that the total work is distributed in a reasonable way. You could write a script that invokes ssh and does some bookkeeping to pick hosts.

Another issue is how to communicate the prerequisites and results across the machines. Let's assume all your Makefile ever does is look at files and create other files depending on their contents.

One option is to put everything on a shared filesystem, but that may produce synchronization issues (depending on how you share the files and how your recipes deal with such issues). Another option is to let each host use a local filesystem, and somehow mirror it (with rsync?) before and after execution of each command (but only the files read or written by the command). How does clearmake solve this problem?

So how easy it is to make this work reliably strongly depends on how complicated your Makefiles and their recipes are.

I notice there exist some so-called distributed shells that may or may not make life easier - I've never used them.