clang is using the wrong system include directory

Question

My workstation is RHEL 6.5, which came with gcc 4.4.7 preinstalled. We are doing C++11 work, so we purchased the RedHat Developer's Toolset set of repositories, which includdes G++ 4.8.2. Both versions of G++ are installed on this machine, but 4.8.2 is the version that is invoked when launched from the command line and via a build:

john@hurricane /home/john/repo/phoenix [PHX-29|✚5]  $ which g++
/opt/rh/devtoolset-2/root/usr/bin/g++
john@hurricane /home/john/repo/phoenix [PHX-29|✚5]  $ g++ --version
g++ (GCC) 4.8.2 20140120 (Red Hat 4.8.2-15)
Copyright (C) 2013 Free Software Foundation, Inc.
This is free software; see the source for copying conditions.  There is NO
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.

Note the unusual install location.

I've also built clang from source and installed it. However when I build C++11 projects using clang, clang is picking up the wrong system includes. It's picking up the system headers from gcc 4.4.7, rather than from 4.8.2.

G++ configuration:

john@hurricane /home/john/repo/phoenix [PHX-29|✚5]  $ g++ -E -x c++ - -v < /dev/null 
Using built-in specs.
COLLECT_GCC=g++
Target: x86_64-redhat-linux
Configured with: ../configure --prefix=/opt/rh/devtoolset-2/root/usr --mandir=/opt/rh/devtoolset-2/root/usr/share/man --infodir=/opt/rh/devtoolset-2/root/usr/share/info --with-bugurl=http://bugzilla.redhat.com/bugzilla --enable-bootstrap --enable-shared --enable-threads=posix --enable-checking=release --with-system-zlib --enable-__cxa_atexit --disable-libunwind-exceptions --enable-gnu-unique-object --enable-linker-build-id --enable-languages=c,c++,fortran,lto --enable-plugin --with-linker-hash-style=gnu --enable-initfini-array --disable-libgcj --with-isl=/builddir/build/BUILD/gcc-4.8.2-20140120/obj-x86_64-redhat-linux/isl-install --with-cloog=/builddir/build/BUILD/gcc-4.8.2-20140120/obj-x86_64-redhat-linux/cloog-install --with-mpc=/builddir/build/BUILD/gcc-4.8.2-20140120/obj-x86_64-redhat-linux/mpc-install --with-tune=generic --with-arch_32=i686 --build=x86_64-redhat-linux
Thread model: posix
gcc version 4.8.2 20140120 (Red Hat 4.8.2-15) (GCC) 
COLLECT_GCC_OPTIONS='-E' '-v' '-shared-libgcc' '-mtune=generic' '-march=x86-64'
 /opt/rh/devtoolset-2/root/usr/libexec/gcc/x86_64-redhat-linux/4.8.2/cc1plus -E -quiet -v -D_GNU_SOURCE - -mtune=generic -march=x86-64
ignoring nonexistent directory "/opt/rh/devtoolset-2/root/usr/lib/gcc/x86_64-redhat-linux/4.8.2/include-fixed"
ignoring nonexistent directory "/opt/rh/devtoolset-2/root/usr/lib/gcc/x86_64-redhat-linux/4.8.2/../../../../x86_64-redhat-linux/include"
#include "..." search starts here:
#include <...> search starts here:
 /opt/rh/devtoolset-2/root/usr/lib/gcc/x86_64-redhat-linux/4.8.2/../../../../include/c++/4.8.2
 /opt/rh/devtoolset-2/root/usr/lib/gcc/x86_64-redhat-linux/4.8.2/../../../../include/c++/4.8.2/x86_64-redhat-linux
 /opt/rh/devtoolset-2/root/usr/lib/gcc/x86_64-redhat-linux/4.8.2/../../../../include/c++/4.8.2/backward
 /opt/rh/devtoolset-2/root/usr/lib/gcc/x86_64-redhat-linux/4.8.2/include
 /usr/local/include
 /opt/rh/devtoolset-2/root/usr/include
 /usr/include
End of search list.
# 1 "<stdin>"
# 1 "<command-line>"
# 1 "<stdin>"
COMPILER_PATH=/opt/rh/devtoolset-2/root/usr/libexec/gcc/x86_64-redhat-linux/4.8.2/:/opt/rh/devtoolset-2/root/usr/libexec/gcc/x86_64-redhat-linux/4.8.2/:/opt/rh/devtoolset-2/root/usr/libexec/gcc/x86_64-redhat-linux/:/opt/rh/devtoolset-2/root/usr/lib/gcc/x86_64-redhat-linux/4.8.2/:/opt/rh/devtoolset-2/root/usr/lib/gcc/x86_64-redhat-linux/
LIBRARY_PATH=/opt/rh/devtoolset-2/root/usr/lib/gcc/x86_64-redhat-linux/4.8.2/:/opt/rh/devtoolset-2/root/usr/lib/gcc/x86_64-redhat-linux/4.8.2/../../../../lib64/:/lib/../lib64/:/usr/lib/../lib64/:/opt/rh/devtoolset-2/root/usr/lib/gcc/x86_64-redhat-linux/4.8.2/../../../:/lib/:/usr/lib/
COLLECT_GCC_OPTIONS='-E' '-v' '-shared-libgcc' '-mtune=generic' '-march=x86-64'

clang++ configuration:

john@hurricane /home/john/repo/phoenix [PHX-29|✚5]  $ clang++ -E -x c++ - -v < /dev/null 
clang version 3.6.0 (trunk 222833) (llvm/trunk 222832)
Target: x86_64-unknown-linux-gnu
Thread model: posix
Found candidate GCC installation: /usr/lib/gcc/x86_64-redhat-linux/4.4.4
Found candidate GCC installation: /usr/lib/gcc/x86_64-redhat-linux/4.4.7
Selected GCC installation: /usr/lib/gcc/x86_64-redhat-linux/4.4.7
Candidate multilib: .;@m64
Candidate multilib: 32;@m32
Selected multilib: .;@m64
 "/usr/local/bin/clang" -cc1 -triple x86_64-unknown-linux-gnu -E -disable-free -disable-llvm-verifier -main-file-name - -mrelocation-model static -mthread-model posix -mdisable-fp-elim -fmath-errno -masm-verbose -mconstructor-aliases -munwind-tables -target-cpu x86-64 -target-linker-version 2.23.52.0.1 -v -dwarf-column-info -resource-dir /usr/local/bin/../lib/clang/3.6.0 -internal-isystem /usr/lib/gcc/x86_64-redhat-linux/4.4.7/../../../../include/c++/4.4.7 -internal-isystem /usr/lib/gcc/x86_64-redhat-linux/4.4.7/../../../../include/c++/4.4.7/x86_64-redhat-linux -internal-isystem /usr/lib/gcc/x86_64-redhat-linux/4.4.7/../../../../include/c++/4.4.7/backward -internal-isystem /usr/local/include -internal-isystem /usr/local/bin/../lib/clang/3.6.0/include -internal-externc-isystem /include -internal-externc-isystem /usr/include -fdeprecated-macro -fdebug-compilation-dir /home/john/repo/phoenix -ferror-limit 19 -fmessage-length 271 -mstackrealign -fobjc-runtime=gcc -fcxx-exceptions -fexceptions -fdiagnostics-show-option -fcolor-diagnostics -o - -x c++ -
clang -cc1 version 3.6.0 based upon LLVM 3.6.0svn default target x86_64-unknown-linux-gnu
ignoring nonexistent directory "/include"
#include "..." search starts here:
#include <...> search starts here:
 /usr/lib/gcc/x86_64-redhat-linux/4.4.7/../../../../include/c++/4.4.7
 /usr/lib/gcc/x86_64-redhat-linux/4.4.7/../../../../include/c++/4.4.7/x86_64-redhat-linux
 /usr/lib/gcc/x86_64-redhat-linux/4.4.7/../../../../include/c++/4.4.7/backward
 /usr/local/include
 /usr/local/bin/../lib/clang/3.6.0/include
 /usr/include
End of search list.
# 1 "<stdin>"
# 1 "<built-in>" 1
# 1 "<built-in>" 3
# 320 "<built-in>" 3
# 1 "<command line>" 1
# 1 "<built-in>" 2
# 1 "<stdin>" 2

john@hurricane /home/john/repo/phoenix [PHX-29|✚5]  $ 

I'm sure this can be fixed by passing the right parameters to clang's configure.sh, but I'm not sure what to pass?

Answer 1

I believe the solution to this problem is the --gcc-toolchain option.

Without the option, clang++ will use the gcc includes in /usr/lib:

[elemental@jsaxton-dev sockeye (FixClangStaticAnalysis)]$ clang++ -E -x c++ - -v < /dev/null
clang version 3.4.2 (tags/RELEASE_34/dot2-final)
Target: x86_64-redhat-linux-gnu
Thread model: posix
Found candidate GCC installation: /usr/bin/../lib/gcc/x86_64-redhat-linux/4.4.4
Found candidate GCC installation: /usr/bin/../lib/gcc/x86_64-redhat-linux/4.4.7
Found candidate GCC installation: /usr/lib/gcc/x86_64-redhat-linux/4.4.4
Found candidate GCC installation: /usr/lib/gcc/x86_64-redhat-linux/4.4.7
Selected GCC installation: /usr/bin/../lib/gcc/x86_64-redhat-linux/4.4.7
 "/usr/bin/clang" -cc1 -triple x86_64-redhat-linux-gnu -E -disable-free -disable-llvm-verifier -main-file-name - -mrelocation-model static -mdisable-fp-elim -fmath-errno -masm-verbose -mconstructor-aliases -munwind-tables -target-cpu x86-64 -target-linker-version 2.20.51.0.2 -v -resource-dir /usr/bin/../lib/clang/3.4.2 -internal-isystem /usr/bin/../lib/gcc/x86_64-redhat-linux/4.4.7/../../../../include/c++/4.4.7 -internal-isystem /usr/bin/../lib/gcc/x86_64-redhat-linux/4.4.7/../../../../include/c++/4.4.7/x86_64-redhat-linux -internal-isystem /usr/bin/../lib/gcc/x86_64-redhat-linux/4.4.7/../../../../include/c++/4.4.7/backward -internal-isystem /usr/bin/../lib/gcc/x86_64-redhat-linux/4.4.7/../../../../include/x86_64-redhat-linux/c++/4.4.7 -internal-isystem /usr/local/include -internal-isystem /usr/bin/../lib/clang/3.4.2/include -internal-externc-isystem /include -internal-externc-isystem /usr/include -fdeprecated-macro -fdebug-compilation-dir /data/home/elemental/sandbox/master2/sockeye -ferror-limit 19 -fmessage-length 237 -mstackrealign -fobjc-runtime=gcc -fcxx-exceptions -fexceptions -fdiagnostics-show-option -fcolor-diagnostics -vectorize-slp -o - -x c++ -
clang -cc1 version 3.4.2 based upon LLVM 3.4.2 default target x86_64-redhat-linux-gnu
ignoring nonexistent directory "/usr/bin/../lib/gcc/x86_64-redhat-linux/4.4.7/../../../../include/x86_64-redhat-linux/c++/4.4.7"
ignoring nonexistent directory "/include"
#include "..." search starts here:
#include <...> search starts here:
 /usr/bin/../lib/gcc/x86_64-redhat-linux/4.4.7/../../../../include/c++/4.4.7
 /usr/bin/../lib/gcc/x86_64-redhat-linux/4.4.7/../../../../include/c++/4.4.7/x86_64-redhat-linux
 /usr/bin/../lib/gcc/x86_64-redhat-linux/4.4.7/../../../../include/c++/4.4.7/backward
 /usr/local/include
 /usr/bin/../lib/clang/3.4.2/include
 /usr/include
End of search list.
# 1 "<stdin>"
# 1 "<built-in>" 1
# 1 "<built-in>" 3
# 167 "<built-in>" 3
# 1 "<command line>" 1
# 1 "<built-in>" 2
# 1 "<stdin>" 2

With the --gcc-toolchain option, clang++ will use the gcc includes in /opt/rh/devtoolset-2:

[elemental@jsaxton-dev sockeye (FixClangStaticAnalysis)]$ clang++ -E -x c++ --gcc-toolchain=/opt/rh/devtoolset-2/root/usr/ - -v < /dev/null
clang version 3.4.2 (tags/RELEASE_34/dot2-final)
Target: x86_64-redhat-linux-gnu
Thread model: posix
Found candidate GCC installation: /opt/rh/devtoolset-2/root/usr/lib/gcc/x86_64-redhat-linux/4.8.2
Selected GCC installation: /opt/rh/devtoolset-2/root/usr/lib/gcc/x86_64-redhat-linux/4.8.2
 "/usr/bin/clang" -cc1 -triple x86_64-redhat-linux-gnu -E -disable-free -disable-llvm-verifier -main-file-name - -mrelocation-model static -mdisable-fp-elim -fmath-errno -masm-verbose -mconstructor-aliases -munwind-tables -fuse-init-arra
y -target-cpu x86-64 -target-linker-version 2.20.51.0.2 -v -resource-dir /usr/bin/../lib/clang/3.4.2 -internal-isystem /opt/rh/devtoolset-2/root/usr/lib/gcc/x86_64-redhat-linux/4.8.2/../../../../include/c++/4.8.2 -internal-isystem /opt/r
h/devtoolset-2/root/usr/lib/gcc/x86_64-redhat-linux/4.8.2/../../../../include/c++/4.8.2/x86_64-redhat-linux -internal-isystem /opt/rh/devtoolset-2/root/usr/lib/gcc/x86_64-redhat-linux/4.8.2/../../../../include/c++/4.8.2/backward -interna
l-isystem /opt/rh/devtoolset-2/root/usr/lib/gcc/x86_64-redhat-linux/4.8.2/../../../../include/x86_64-redhat-linux/c++/4.8.2 -internal-isystem /usr/local/include -internal-isystem /usr/bin/../lib/clang/3.4.2/include -internal-externc-isys
tem /include -internal-externc-isystem /usr/include -fdeprecated-macro -fdebug-compilation-dir /data/home/elemental/sandbox/master2/sockeye -ferror-limit 19 -fmessage-length 237 -mstackrealign -fobjc-runtime=gcc -fcxx-exceptions -fexcept
ions -fdiagnostics-show-option -fcolor-diagnostics -vectorize-slp -o - -x c++ -
clang -cc1 version 3.4.2 based upon LLVM 3.4.2 default target x86_64-redhat-linux-gnu
ignoring nonexistent directory "/opt/rh/devtoolset-2/root/usr/lib/gcc/x86_64-redhat-linux/4.8.2/../../../../include/x86_64-redhat-linux/c++/4.8.2"
ignoring nonexistent directory "/include"
#include "..." search starts here:
#include <...> search starts here:
 /opt/rh/devtoolset-2/root/usr/lib/gcc/x86_64-redhat-linux/4.8.2/../../../../include/c++/4.8.2
 /opt/rh/devtoolset-2/root/usr/lib/gcc/x86_64-redhat-linux/4.8.2/../../../../include/c++/4.8.2/x86_64-redhat-linux
 /opt/rh/devtoolset-2/root/usr/lib/gcc/x86_64-redhat-linux/4.8.2/../../../../include/c++/4.8.2/backward
 /usr/local/include
 /usr/bin/../lib/clang/3.4.2/include
 /usr/include
End of search list.
# 1 "<stdin>"
# 1 "<built-in>" 1
# 1 "<built-in>" 3
# 167 "<built-in>" 3
# 1 "<command line>" 1
# 1 "<built-in>" 2
# 1 "<stdin>" 2