Include search path on Mac OS X Yosemite 10.10.1

Question

I just to change the include search path order (I believe).

---

I'd like to change the include search path. Especially, I need /usr/local/include first.

But it doesn't change because of duplicate. How can I change it?

I suppose there's default setting, because paths I don't specify appears. Like /usr/include/c++/4.2.1, /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/../lib/clang/6.0/include

 "/Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/clang" -cc1 -triple x86_64-apple-macosx10.10.0 -emit-obj -mrelax-all -disable-free -disable-llvm-verifier -main-file-name test_common.cpp -mrelocation-model pic -pic-level 2 -mdisable-fp-elim -masm-verbose -munwind-tables -target-cpu core2 -target-linker-version 241.9 -v -gdwarf-2 -coverage-file /Users/machidahiroaki/RubymineProjects/caffe/.build_debug/src/caffe/test/test_common.o -resource-dir /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/../lib/clang/6.0 -D DEBUG -D CPU_ONLY -I /usr/local/include -I /usr/local/include -I /Users/machidahiroaki/anaconda/include -I /Users/machidahiroaki/anaconda/include/python2.7 -I /Users/machidahiroaki/anaconda/lib/python2.7/site-packages/numpy/core/include -I .build_debug/src -I ./src -I ./include -I /usr/local/atlas/include -stdlib=libstdc++ -O0 -Wall -Wno-sign-compare -Wno-unneeded-internal-declaration -fdeprecated-macro -fdebug-compilation-dir /Users/machidahiroaki/RubymineProjects/caffe -ferror-limit 19 -fmessage-length 0 -pthread -stack-protector 1 -mstackrealign -fblocks -fobjc-runtime=macosx-10.10.0 -fencode-extended-block-signature -fcxx-exceptions -fexceptions -fdiagnostics-show-option -vectorize-slp -o .build_debug/src/caffe/test/test_common.o -x c++ src/caffe/test/test_common.cpp
clang -cc1 version 6.0 based upon LLVM 3.5svn default target x86_64-apple-darwin14.0.0
ignoring nonexistent directory "/usr/include/c++/4.2.1/i686-apple-darwin10/x86_64"
ignoring nonexistent directory "/usr/include/c++/4.0.0"
ignoring nonexistent directory "/usr/include/c++/4.0.0/i686-apple-darwin8/"
ignoring nonexistent directory "/usr/include/c++/4.0.0/backward"
ignoring duplicate directory "/usr/local/include"
ignoring duplicate directory "/usr/local/include"
  as it is a non-system directory that duplicates a system directory
#include "..." search starts here:
#include <...> search starts here:
 /Users/machidahiroaki/anaconda/include
 /Users/machidahiroaki/anaconda/include/python2.7
 /Users/machidahiroaki/anaconda/lib/python2.7/site-packages/numpy/core/include
 .build_debug/src
 ./src
 ./include
 /usr/local/atlas/include
 /usr/include/c++/4.2.1
 /usr/include/c++/4.2.1/backward
 /usr/local/include
 /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/../lib/clang/6.0/include
 /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/include
 /usr/include
 /System/Library/Frameworks (framework directory)
 /Library/Frameworks (framework directory)
End of search list.

---

Library search paths

Machida-no-MacBook-Air:caffe machidahiroaki$ /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/ld -v ...
@(#)PROGRAM:ld  PROJECT:ld64-241.9
configured to support archs: armv6 armv7 armv7s arm64 i386 x86_64 x86_64h armv6m armv7m armv7em
Library search paths:
    /usr/lib
    /usr/local/lib
Framework search paths:
    /Library/Frameworks/
    /System/Library/Frameworks/

For sure this is the ld done in make.

 "/Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/ld" -demangle -dynamic -dylib -arch x86_64 -macosx_version_min 10.10.0 -t -o .build_debug/lib/libcaffe.so -L/usr/local/lib -L/usr/lib -L/usr/local/atlas/lib .build_debug/src/caffe/proto/caffe.pb.o .build_debug/src/caffe/proto/caffe_pretty_print.pb.o .build_debug/src/caffe/blob.o .build_debug/src/caffe/common.o .build_debug/src/caffe/data_transformer.o .build_debug/src/caffe/internal_thread.o .build_debug/src/caffe/layer_factory.o .build_debug/src/caffe/layers/absval_layer.o .build_debug/src/caffe/layers/accuracy_layer.o .build_debug/src/caffe/layers/argmax_layer.o .build_debug/src/caffe/layers/base_data_layer.o .build_debug/src/caffe/layers/bnll_layer.o .build_debug/src/caffe/layers/concat_layer.o .build_debug/src/caffe/layers/contrastive_loss_layer.o .build_debug/src/caffe/layers/conv_layer.o .build_debug/src/caffe/layers/cudnn_conv_layer.o .build_debug/src/caffe/layers/cudnn_pooling_layer.o .build_debug/src/caffe/layers/cudnn_relu_layer.o .build_debug/src/caffe/layers/cudnn_sigmoid_layer.o .build_debug/src/caffe/layers/cudnn_softmax_layer.o .build_debug/src/caffe/layers/cudnn_tanh_layer.o .build_debug/src/caffe/layers/data_layer.o .build_debug/src/caffe/layers/dropout_layer.o .build_debug/src/caffe/layers/dummy_data_layer.o .build_debug/src/caffe/layers/eltwise_layer.o .build_debug/src/caffe/layers/euclidean_loss_layer.o .build_debug/src/caffe/layers/flatten_layer.o .build_debug/src/caffe/layers/hdf5_data_layer.o .build_debug/src/caffe/layers/hdf5_output_layer.o .build_debug/src/caffe/layers/hinge_loss_layer.o .build_debug/src/caffe/layers/im2col_layer.o .build_debug/src/caffe/layers/image_data_layer.o .build_debug/src/caffe/layers/infogain_loss_layer.o .build_debug/src/caffe/layers/inner_product_layer.o .build_debug/src/caffe/layers/loss_layer.o .build_debug/src/caffe/layers/lrn_layer.o .build_debug/src/caffe/layers/memory_data_layer.o .build_debug/src/caffe/layers/multinomial_logistic_loss_layer.o .build_debug/src/caffe/layers/mvn_layer.o .build_debug/src/caffe/layers/neuron_layer.o .build_debug/src/caffe/layers/pooling_layer.o .build_debug/src/caffe/layers/power_layer.o .build_debug/src/caffe/layers/relu_layer.o .build_debug/src/caffe/layers/sigmoid_cross_entropy_loss_layer.o .build_debug/src/caffe/layers/sigmoid_layer.o .build_debug/src/caffe/layers/silence_layer.o .build_debug/src/caffe/layers/slice_layer.o .build_debug/src/caffe/layers/softmax_layer.o .build_debug/src/caffe/layers/softmax_loss_layer.o .build_debug/src/caffe/layers/split_layer.o .build_debug/src/caffe/layers/tanh_layer.o .build_debug/src/caffe/layers/threshold_layer.o .build_debug/src/caffe/layers/window_data_layer.o .build_debug/src/caffe/net.o .build_debug/src/caffe/solver.o .build_debug/src/caffe/syncedmem.o .build_debug/src/caffe/util/benchmark.o .build_debug/src/caffe/util/im2col.o .build_debug/src/caffe/util/insert_splits.o .build_debug/src/caffe/util/io.o .build_debug/src/caffe/util/math_functions.o .build_debug/src/caffe/util/upgrade_proto.o -framework Accelerate -lglog -lgflags -lprotobuf -lleveldb -lsnappy -llmdb -lboost_system -lhdf5_hl -lhdf5 -lopencv_core -lopencv_highgui -lopencv_imgproc -lpthread -lboost_thread-mt -lcblas -lstdc++ -lSystem /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/../lib/clang/6.0/lib/darwin/libclang_rt.osx.a

So... where does this path /usr/include/c++/4.2.1 come from?

---

Did I get some hints?

Machida-no-MacBook-Air:caffe machidahiroaki$ g++ -print-search-dirs
install: /usr/local/lib/gcc/x86_64-apple-darwin14.0.0/5.0.0/
programs: =/usr/local/libexec/gcc/x86_64-apple-darwin14.0.0/5.0.0/:/usr/local/libexec/gcc/x86_64-apple-darwin14.0.0/5.0.0/:/usr/local/libexec/gcc/x86_64-apple-darwin14.0.0/:/usr/local/lib/gcc/x86_64-apple-darwin14.0.0/5.0.0/:/usr/local/lib/gcc/x86_64-apple-darwin14.0.0/:/usr/local/lib/gcc/x86_64-apple-darwin14.0.0/5.0.0/../../../../x86_64-apple-darwin14.0.0/bin/x86_64-apple-darwin14.0.0/5.0.0/:/usr/local/lib/gcc/x86_64-apple-darwin14.0.0/5.0.0/../../../../x86_64-apple-darwin14.0.0/bin/
libraries: =/usr/local/lib/gcc/x86_64-apple-darwin14.0.0/5.0.0/:/usr/local/lib/gcc/x86_64-apple-darwin14.0.0/5.0.0/../../../../x86_64-apple-darwin14.0.0/lib/x86_64-apple-darwin14.0.0/5.0.0/:/usr/local/lib/gcc/x86_64-apple-darwin14.0.0/5.0.0/../../../../x86_64-apple-darwin14.0.0/lib/:/usr/local/lib/gcc/x86_64-apple-darwin14.0.0/5.0.0/../../../x86_64-apple-darwin14.0.0/5.0.0/:/usr/local/lib/gcc/x86_64-apple-darwin14.0.0/5.0.0/../../../:/lib/x86_64-apple-darwin14.0.0/5.0.0/:/lib/:/usr/lib/x86_64-apple-darwin14.0.0/5.0.0/:/usr/lib/
Machida-no-MacBook-Air:caffe machidahiroaki$ clang++ -print-search-dirs
programs: =/Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin:/Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin
libraries: =/Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/../lib/clang/6.0
M

I don't know if it has to do with the problem...

Machida-no-MacBook-Air:caffe machidahiroaki$ gcc -print-search-dirs
install: /usr/local/lib/gcc/x86_64-apple-darwin14.0.0/5.0.0/
programs: =/usr/local/libexec/gcc/x86_64-apple-darwin14.0.0/5.0.0/:/usr/local/libexec/gcc/x86_64-apple-darwin14.0.0/5.0.0/:/usr/local/libexec/gcc/x86_64-apple-darwin14.0.0/:/usr/local/lib/gcc/x86_64-apple-darwin14.0.0/5.0.0/:/usr/local/lib/gcc/x86_64-apple-darwin14.0.0/:/usr/local/lib/gcc/x86_64-apple-darwin14.0.0/5.0.0/../../../../x86_64-apple-darwin14.0.0/bin/x86_64-apple-darwin14.0.0/5.0.0/:/usr/local/lib/gcc/x86_64-apple-darwin14.0.0/5.0.0/../../../../x86_64-apple-darwin14.0.0/bin/
libraries: =/usr/local/lib/gcc/x86_64-apple-darwin14.0.0/5.0.0/:/usr/local/lib/gcc/x86_64-apple-darwin14.0.0/5.0.0/../../../../x86_64-apple-darwin14.0.0/lib/x86_64-apple-darwin14.0.0/5.0.0/:/usr/local/lib/gcc/x86_64-apple-darwin14.0.0/5.0.0/../../../../x86_64-apple-darwin14.0.0/lib/:/usr/local/lib/gcc/x86_64-apple-darwin14.0.0/5.0.0/../../../x86_64-apple-darwin14.0.0/5.0.0/:/usr/local/lib/gcc/x86_64-apple-darwin14.0.0/5.0.0/../../../:/lib/x86_64-apple-darwin14.0.0/5.0.0/:/lib/:/usr/lib/x86_64-apple-darwin14.0.0/5.0.0/:/usr/lib/
Machida-no-MacBook-Air:caffe machidahiroaki$ gcc --version
gcc (GCC) 5.0.0 20141005 (experimental)
Copyright (C) 2014 Free Software Foundation, Inc.
This is free software; see the source for copying conditions.  There is NO
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.

---

Wait, export CFLAGS="-I/path/to/preferred/HDF5/hdf5.h" would not work...

Specify nothing

Machida-no-MacBook-Air:caffe machidahiroaki$ echo | "/Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/clang" -Wp,-v -stdlib=libstdc++ -x c++ - -fsyntax-only
clang -cc1 version 6.0 based upon LLVM 3.5svn default target x86_64-apple-darwin14.0.0
#include "..." search starts here:
#include <...> search starts here:
 /usr/include/c++/4.2.1
 /usr/include/c++/4.2.1/backward
 /usr/local/include
 /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/../lib/clang/6.0/include
 /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/include
 /usr/include
 /System/Library/Frameworks (framework directory)
 /Library/Frameworks (framework directory)
End of search list.

Can't specify a header file!!

Machida-no-MacBook-Air:caffe machidahiroaki$ echo | "/Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/clang" -Wp,-v -stdlib=libstdc++ -x c++ - -fsyntax-only -I/usr/local/include/hdf5.h
clang -cc1 version 6.0 based upon LLVM 3.5svn default target x86_64-apple-darwin14.0.0
ignoring nonexistent directory "/usr/local/include/hdf5.h"
#include "..." search starts here:
#include <...> search starts here:
 /usr/include/c++/4.2.1
 /usr/include/c++/4.2.1/backward
 /usr/local/include
 /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/../lib/clang/6.0/include
 /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/include
 /usr/include
 /System/Library/Frameworks (framework directory)
 /Library/Frameworks (framework directory)
End of search list.

As expected, the duplicate directory is ignored and the order does not change.

Machida-no-MacBook-Air:caffe machidahiroaki$ echo | "/Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/clang" -Wp,-v -stdlib=libstdc++ -x c++ - -fsyntax-only -I/usr/local/include/
clang -cc1 version 6.0 based upon LLVM 3.5svn default target x86_64-apple-darwin14.0.0
ignoring duplicate directory "/usr/local/include"
  as it is a non-system directory that duplicates a system directory
#include "..." search starts here:
#include <...> search starts here:
 /usr/include/c++/4.2.1
 /usr/include/c++/4.2.1/backward
 /usr/local/include
 /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/../lib/clang/6.0/include
 /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/include
 /usr/include
 /System/Library/Frameworks (framework directory)
 /Library/Frameworks (framework directory)
End of search list.

---

But, I found clang's -include option, which is able to specify a file!

Answer 1

I'd like to change library search path...

At compile time, you augment the library search path with -L. You cannot delete paths; you can only add paths that have a "higher" preference than existing paths.

At runtime, you use DYLD_LIBRARY_PATH, DYLD_FALLBACK_LIBRARY_PATH and friends to change the library search path. See dyld(1) OS X Man Pages.

Usually you use DYLD_LIBRARY_PATH to ensure a particular library is loaded rather than the system one. Its a way to override default behavior. DYLD_FALLBACK_LIBRARY_PATH is used to provide a library that's not a system one. Its less intrusive because you don't displace system paths.

----------

I need /usr/local/include first...

Sounds a bit odd to me, but...

$ export DYLD_LIBRARY_PATH=/usr/local/include
$ clang ...

----------

IF you are cross-compiling from the command line, you should do something like this. The trick is to use --sysroot to automatically bring in the headers and libraries for the platform, rather than hacking them with -I, -L and -l.

# Put cross compile tools on the PATH first
export PATH="/Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/:$PATH"

# C compiler
export CC=/Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/clang

# C++ compiler
export CXX=/Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/clang++

# SYSROOT brings in platform headers and libraries, No need for -I, -L and -l.
SYSROOT=/Applications/Xcode.app/Contents/Developer/Platforms/iPhoneOS.platform/Developer/SDKs/iPhoneOS7.1.sdk

# Compiler flags
export CFLAGS="-march armv7 -march armv7s --sysroot=$SYSROOT"

# Compiler flags
export CXXFLAGS="-march armv7 -march armv7s --sysroot=$SYSROOT"

# Using defualt C++ runtime
$CXX $CXXFLAGS foo.c -o foo.exe

You can also specify GNU's runtime with -stdlib=libstdc++, and LLVM's runtime with -stdlib=libc++.

----------

Based on your updates:

<ld command and output omitted>
where does this path /usr/include/c++/4.2.1 come from??

/usr/include/c++/4.2.1 is not in your Library Search Path. Its just not there.

That's a header include path, and you specify it with -I.

If you want to see your header include paths for libstdc++ and libc++, do this:

# GNU C++ runtime
$ echo | /usr/local/bin/clang++ -Wp,-v -stdlib=ibstdc++ -x c++ - -fsyntax-only

And:

# LLVM C++ runtime
$ echo | /usr/local/bin/clang++ -Wp,-v -stdlib=libc++ -x c++ - -fsyntax-only

Here's what I get on OS X 10.8.5:

libstdc++ (GNU):

 /usr/include/c++/4.2.1
 /usr/include/c++/4.2.1/backward
 /usr/local/include
 /usr/local/bin/../lib/clang/3.5.0/include
 /usr/include
 /System/Library/Frameworks (framework directory)
 /Library/Frameworks (framework directory

libc++ (LLVM):

 /usr/local/include
 /usr/local/bin/../lib/clang/3.5.0/include
 /usr/include
 /System/Library/Frameworks (framework directory)
 /Library/Frameworks (framework directory)

So /usr/include/c++/4.2.1 is a built-in compiler path when using GNU's libstdc++ (but not LLVM's libc++). I'll go even further and tell you its hard coded in LLVM/Clang sources (I build LLVM/Clang from source often):

$ cd clang-sources/llvm
$ grep -R "/usr/include/c++/4.2.1" *
tools/clang/lib/Frontend/InitHeaderSearch.cpp:      AddGnuCPlusPlusIncludePaths("/usr/include/c++/4.2.1",
tools/clang/lib/Frontend/InitHeaderSearch.cpp:      AddGnuCPlusPlusIncludePaths("/usr/include/c++/4.2.1",
tools/clang/lib/Frontend/InitHeaderSearch.cpp:      AddGnuCPlusPlusIncludePaths("/usr/include/c++/4.2.1",
tools/clang/lib/Frontend/InitHeaderSearch.cpp:      AddGnuCPlusPlusIncludePaths("/usr/include/c++/4.2.1",
tools/clang/lib/Frontend/InitHeaderSearch.cpp:      AddGnuCPlusPlusIncludePaths("/usr/include/c++/4.2.1",

You can add additional include paths when building LLVM/Clang with --with-c-include-dirs.

----------

Based on your comment: "... If I delete ~/anaconda/include/hdf5.h...". If you need a new path for the HDF5 package, then use something like:

export CPPFLAGS="-I/path/to/preferred/HDF5/hdf5.h"
export CFLAGS="-I/path/to/preferred/HDF5/hdf5.h"
export CXXFLAGS="-I/path/to/preferred/HDF5/hdf5.h"
export LDFLAGS="-L/path/to/preferred/HDF5/lib"
./configure ...

You need it in CPPFLAGS too because the autotools might run some tests with it.

Then, at runtime:

export DYLD_LIBRARY_PATH=/path/to/preferred/HDF5/lib
./run_the_executable

And you can side step DYLD_LIBRARY_PATH IF you perform static linking against HDF5. But be aware the OS X linker always uses a shared object or dynalib if its available. Effectively, it ignores your request for static linking. You will have to mv shared object or dynalib off-path while leaving an archive on-path.

----------

To muddy the waters a little more... If you are saying "library" when you really want to say "framework", then see this Stack Overflow question about using the -framework option: Clang(LLVM) compile with frameworks.