How to get ccache to not pass the full path to the compiler to distcc

Question

(This is different to the question ccache and absolute path as I want only the command path to not be expanded on the ccache host machine)

When using ccache and distcc together ccache is expanding the compiler to an absolute path, and then distcc cannot use the PATH on the remote machine to choose which compiler to use.

e.g. I call CCACHE_PREFIX=distcc ccache g++ foo.cc and ccache expands this into a local preprocessing step and cache check and then a call to distcc as distcc /usr/bin/g++, which is the wrong version (g++ lives in the path on the remote before /usr/bin, but this doesn't give it the chance to search the path at all).

I have various different machines being used as distcc hosts, and they have the same version of gcc/g++ installed in different locations (yes, this problem goes away if I put them all in somewhere like /usr/local, but I can't do that at the moment).

Is there a setting to get ccache to pass just g++ to distcc rather than expanding the path to the absolute path of the local compiler? I'm not completely against patching ccache if there is no setting yet, but that's a last resort :)

Answer 1

Turns out there's a simple way to do this : just use a wrapper for CCACHE_PREFIX instead of distcc directly, with something like this:

File : distcc-wrap.sh

#!/bin/sh
compiler=$(basename $1)
shift
exec distcc "$compiler" "$@"

export CCACHE_PREFIX=distcc-wrap.sh and then this allows the remote compiler to live at a different place and distcc will search the PATH for it.

(Thanks to Joel on the ccache mailing list for this answer; see http://www.mail-archive.com/[email protected]/msg00670.html for original message)

Answer 2

I try David's solution but encounter problem of "distcc seems to have invoked itself recursively!" in distcc plain mode. This is because when host distcc do preprocessing (cpp), it will invoke host ccache, but distcc-wrap intercept and spawn a nested distcc, and forms a recursive call:

g++ -> ccache -> distcc -> distcc-wrap -> preprocess using g++ -> ccache -> distcc -> .... and so on.

My solution is to use DISTCC_CMDLIST, from man distccd:

DISTCC_CMDLIST

If the environment variable DISTCC_CMDLIST is set, load a list of supported commands from the file named by DISTCC_CMDLIST, and refuse to serve any command whose last DISTCC_CMDLIST_MATCHWORDS last words do not match those of a command in that list. See the comments in src/serve.c.

Assuming on remote machine you want to use /usr/local/ccache/g++ (which is a simulink to /usr/bin/ccache) to do the compilation, instead of using the absolute path expanded by host machine, you can do like this:

1. create a file /path/to/.distcc/DISTCC_CMDLIST with this line:

   /usr/local/ccache/g++

2. export DISTCC_CMDLIST=/path/to/.distcc/DISTCC_CMDLIST

3. restart distccd daemon distccd --no-detach -a <host IPs> --daemon

What will happen is whenever distcc remote server receive expanded command from host lke /usr/bin/g++ main.cc -c, it will maps the real compiler from /usr/bin/g++ to /usr/local/ccache/g++. The mapping is done by:

1. retrieve the basename from the compiler path in receiving command ( g++ in this case)

2. Lookup DIST_CMDLIST file to see if any line has basename equals to g++. In this case, it will be /usr/local/ccache/g++

3. overwrite the command to /usr/local/ccache/g++ main.cc -c. which will invoke ccache on remote server.

The above is just an example, and you can extend the compiler mapping by changing the value of DISTCC_CMDLIST_NUMWORDS from 1 to other values to do more tricks.