caffe cmake error by ccache

Question

When I successfully 'cmake ..' caffe build, I preceed with 'make all' command but came up with an error below. I don't know if there is something wrong with NVCC or gcc.

[  1%] Built target proto
[  1%] Building NVCC (Device) object src/caffe/CMakeFiles/cuda_compile.dir/layers/cuda_compile_generated_split_layer.cu.o
/usr/bin/ccache: invalid option -- 'E'
Usage:
ccache [options]
ccache compiler [compiler options]
compiler [compiler options]          (via symbolic link)

Options:
-c, --cleanup         delete old files and recalculate size counters
                      (normally not needed as this is done automatically)
-C, --clear           clear the cache completely
-F, --max-files=N     set maximum number of files in cache to N (use 0 for
                      no limit)
-M, --max-size=SIZE   set maximum size of cache to SIZE (use 0 for no
                      limit; available suffixes: G, M and K; default
                      suffix: G)
-s, --show-stats      show statistics summary
-z, --zero-stats      zero statistics counters

-h, --help            print this help text
-V, --version         print version and copyright information

See also <http://ccache.samba.org>.
CMake Error at cuda_compile_generated_split_layer.cu.o.cmake:206 (message):
Error generating
/home/gpuusr/lpq/caffe-332/build/src/caffe/CMakeFiles/cuda_compile.dir/layers/./cuda_compile_generated_split_layer.cu.o


make[2]: *** [src/caffe/CMakeFiles/cuda_compile.dir/layers/cuda_compile_generated_split_layer.cu.o] Error 1
make[1]: *** [src/caffe/CMakeFiles/caffe.dir/all] Error 2
make: *** [all] Error 2

Answer 1

I had a similar issue trying to build OpenCV with CUDA support. There's some odd interaction between CMake, CUDA, and ccache that I don't understand, it's trying to call ccache with invalid arguments. You can bypass this by specifying your system compiler for CUDA instead:

cmake -DCUDA_HOST_COMPILER=/usr/bin/g++ ..

(Or whatever compiler you're using.)

Answer 2

This used to be a bug in CMake. It was fixed with CMake release 3.4.0.

Proof

The code from here can be compiled with the following CMakeLists.txt.

cmake_minimum_required (VERSION 2.8.9 FATAL_ERROR)
project(CudaTest)

find_package(CUDA REQUIRED)
cuda_add_executable(hello_cuda hello.cu)

• With /usr/lib/ccache on the PATH.
• Building once with CMake 3.2 and once with CMake 3.7.1.

• CMake 3.2:

  $ make
  ...
  /path/to/build-3.2
  [100%] Building NVCC (Device) object CMakeFiles/hello_cuda.dir/hello_cuda_generated_hello.cu.o
  /usr/bin/ccache: invalid option -- 'E'
  ...
  

• CMake 3.7.1:

  $ make
  [ 50%] Building NVCC (Device) object CMakeFiles/hello_cuda.dir/hello_cuda_generated_hello.cu.o
  [100%] Linking CXX executable hello_cuda
  [100%] Built target hello_cuda
  

For the record

A small git bisect run over CMake 3.2.2 -- 3.7.1 shows that this is the commit that fixed it:

commit b405f01daaeaeda98d448e2f0d71ea685757a5bd
Author: Bill Hoffman <[email protected]>
Date:   Fri Jun 12 14:16:09 2015 -0400

    FindCUDA: Resolve a host compiler symlink only if it is Apple cc - clang

    Otherwise using a "cc -> ccache" or similar symlink as the compiler
    causes FindCUDA to select ccache as the host compiler.  Update the logic
    added by commit v3.1.0-rc1~354^2 (FindCUDA: Fix OSX Clang & no C
    language enabled, 2014-06-12) to apply only in the specific case it is
    needed.

This commit went into CMake 3.4.0. Compare git rev-list v3.4.0 | grep b405f01 with git rev-list v3.3.2 | grep b405f01.