How do I avoid concurrency problems when using GNU make for parallel builds of archive files?

Question

I'm using GNU make to build a group of static libraries, using the implicit make rules for doing so. These rules run the ar(1) command to update the library / archive. Profiling has shown that the build time would be reduced if I used the -j option to make to run parallel jobs during the build.

Unfortunately, the GNU make manual has a section http://www.gnu.org/software/make/manual/html_node/Archive-Pitfalls.html that pretty much says that make provides no concurrency guards for running ar(1), and thus it can (and does) corrupt the archive. The manual further teases that this may be fixed in the future.

One solution to this is to use http://code.google.com/p/ipcmd, which basically does semaphore locking before running a command, thus serializing the ar(1) commands building the archive. This particular solution isn't good for me because I'm building with mingw based cross-compilation tools on Windows.

Is there a simpler or better solution to this problem?

Answer 1

Try the following -

AR := flock make.lock $(AR)

clean::
    rm -f make.lock

Now ar(1) will execute with an exclusive lock to the file make.lock, thereby serializing access to the library.

You can add a command to delete the file make.lock after the ranlib command.

Add export AR to propagate the definition to sub-makes, if necessary.

Answer 2

Do the archiving as a single step, rather than trying to update the archive incrementally:

libfoo.a: $(OBJS)
        -rm -f $@
        $(AR) rc $@ $^
        $(RANLIB) $@