How to test a Makefile for missing dependencies?

Question

Is there a way to test for missing dependencies that shows up when compiling a project with multiple jobs (-jN where N > 1)?

I often encounter packages, mostly open source, where the build process works fine as long as I use -j1, or -jN where N is a relatively low value such as 4 or 8 but if I used higher values likes 48, a little uncommon, it starts to fail due to missing dependencies.

I attempted to build myself a bash script that would, given a target, figure out all the dependencies and try to build explicitly each of those dependency with -j1 in order to validate that none are missing dependencies on their own. It appears to work with small / medium package but fails on more important one like uClibc for example.

I am sharing my script in here, as some people may understand better what I mean by reading code. I also hope that a more robust solution exists and could be shared back.

#!/bin/bash
TARGETS=$*
echo "TARGETS=$TARGETS"

for target in $TARGETS
do
    MAKE="make"
    RULE=`make -j1 -n -p | grep "^$target:"`
    if [ -z "$RULE" ]; then
        continue
    fi

    NEWTARGETS=${RULE#* }
    if [ -z "$NEWTARGETS" ]; then
        continue
    fi

    if [ "${NEWTARGETS}" = "${RULE}" ]; then
        # leaf target, we do not want to test.
        continue
    fi

    echo "RULE=$RULE"
#   echo "NEWTARGETS=$NEWTARGETS"
    $0 $NEWTARGETS
    if [ $? -ne 0 ]; then
        exit 1
    fi

    echo "Testing target $target"
    make clean && make -j1 $target 
    if [ $? -ne 0 ]; then
        echo "Make parallel will fail with target $target"
        exit 1
    fi
done

Answer 1

I'm not aware of any open source solution, but this is exactly the problem that ElectricAccelerator, a high-performance implementation of GNU make, was created to solve. It will execute the build in parallel and dynamically detect and correct for missing dependencies, so that the build output is the same as if it had been run serially. It can produce an annotated build log which includes details about the missing dependencies. For example, this simple makefile has an undeclared dependency between abc and def:

all: abc def

abc:
        echo PASS > abc

def:
        cat abc

Run this with emake instead of gmake and enable --emake-annodetail=history, and the resulting annotation file includes this:

<job id="Jf42015c0" thread="f4bfeb40" start="5" end="6" type="rule" name="def" file="Makefile" line="6" neededby="Jf42015f8">
<command line="7">
<argv>cat abc</argv>
<output>cat abc
</output>
<output src="prog">PASS
</output>
</command>
<depList>
<dep writejob="Jf4201588" file="/tmp/foo/abc"/>
</depList>
<timing invoked="0.356803" completed="0.362634" node="chester-1"/>
</job>

In particular the <depList> section shows that this job, Jf42015c0 (in other words, def), depends on job Jf4201588, because the latter modified the file /tmp/foo/abc.

You can give it a try for free with ElectricAccelerator Huddle.

(Disclaimer: I'm the architect of ElectricAccelerator)