When running clang built from source, how to specify location of libc++, or, someone explain to me what -stdlib=libc++ does

Question

I am developing plugins and tools using clang's provisions for doing so via plugins and clang's LibTooling. I am able to do the following things:

• Compile llvm with clang inside, from svn (Linux and OSX), by following the getting started page by running the configure script (without using cmake)

• Compile libcxx/libc++ on Linux (also from svn), and I have no reason to expect any trouble with this on OSX. The thing is that libc++ headers, already exist on my OSX system at

  /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/lib/c++/v1/
  

  and the libc++ dylib lives at /usr/lib/.

  Edit 4: I was able to compile libcxx just fine on OS X by following the directions. I've got a brand new libc++.1.0.dylib sitting here now.

• On OSX, use the Release+Asserts (and Debug+Asserts) builds of clang++ to compile C++ source code by appending

  -I/Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/lib/c++/v1/
  

  and without using -stdlib=libc++. Using this flag to explicitly specify libc++ as an include directory allows the clang that I built to "see" the standard c++ headers. Surprisingly, it seems to be happy with it while compiling a moderately basic source file (which still exercises a good amount of c++11 madness in it)

Based on this, you can see that I am hacking my freshly built clang version 3.6.0 (trunk 217905) to look up Apple's Xcode-packaged libc++. This is ostensibly working for now because Apple's libc++ that came with Xcode remains ABI-compatible with the compiler that I just built from source. What is still very curious to me is how my freshly compiled clang is able to figure out where to find the corresponding libc++ dylib! This raises the question later on of when I actually get libc++ compiled, how am I supposed to tell my new svn-compiled clang to look up and use the new svn-compiled libc++ dylib??

So, basically, I am still utterly confused about what I'm really supposed to do to set up libc++ properly. What, specifically, does clang actually do when you tell it -stdlib=libc++?

Is it a hardcoded include path? I probably want to be building libc++abi and libc++ from svn to use along with the clang that has been built from svn. That makes the most sense... Then how should I go about installing it? Having to put -I~/libcxx/include/c++/v1/ (or whatever it would be) into a build config is inelegant.

Presumably I can just set up my llvm build to build clang along with libc++abi and libc++ by also checking out libcxxabi and libcxx from svn, and my expectation is that installing it ought to make -stdlib=libc++ magically work. Note also that the clang that Apple gives you with Xcode doesn't really require you to use -stdlib=libc++. It just magically knows where to grab the library files.

However! The fly in the ointment, at least the only one so far that I know to look for: my machine already has a /usr/bin/clang++:

$ ls -la $(which clang++)
-rwxr-xr-x  1 root  wheel  14240 Mar 17  2014 /usr/bin/clang++

This is not a symlink to inside Xcode.app as I expected! I now have real concern about running make install from my llvm build directory! It could probably break my Xcode environment or prevent my Mac from booting (as the libc++ llvm doc clearly states will happen if something bad happens to /usr/lib/libc++.1.dylib)!

Hopefully there is not already documentation out there that I have missed that answers these questions, because I really should have found it by now.

Edit: I see in the sparse instructions found on http://libcxx.llvm.org that

clang++ -std=c++11 -stdlib=libc++ -nostdinc++ -I<path-to-libcxx>/include -L<path-to-libcxx>/lib test.cpp

might actually be the "proper way" right now. But what this fails at is explaining what -stdlib=libc++ causes clang to do, or what -nostdinc++ causes clang to do.

Edit 2: Thanks to @n.m. now we know that -nostdinc++ means no standard c++ include path searched. This means that standard include path is a thing. Is this path hardcoded when clang is built? I googled and found something referencing a variable CLANG_INCLUDEPATH, this appears like possibly a makefile variable. Unfortunately I am running full filesystem text search over my llvm source dir and finding no matches to such a string (so it can't be a config used by clang or during the process of building clang).

Edit 3: With the aid of --verbose, I can now see some helpful output:

Xcode's clang:

clang -cc1 version 5.1 based upon LLVM 3.4svn default target x86_64-apple-darwin13.4.0
ignoring nonexistent directory "/usr/include/c++/v1"
#include "..." search starts here:
#include <...> search starts here:
 /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/../lib/c++/v1
 /usr/local/include
 /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/../lib/clang/5.1/include
 /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/include
 /usr/include
 /System/Library/Frameworks (framework directory)
 /Library/Frameworks (framework directory)
End of search list.

svn fresh clang:

clang -cc1 version 3.6.0 based upon LLVM 3.6.0svn default target x86_64-apple-darwin13.4.0
ignoring nonexistent directory "/Users/username/Documents/llvmbuild/Release+Asserts/bin/../include/c++/v1"
ignoring nonexistent directory "/usr/include/c++/v1"
#include "..." search starts here:
#include <...> search starts here:
 /usr/local/include
 /Users/username/Documents/llvmbuild/Release+Asserts/bin/../lib/clang/3.6.0/include
 /usr/include
 /System/Library/Frameworks (framework directory)
 /Library/Frameworks (framework directory)
End of search list.

Okay, so this is painting a picture of a few places to deposit some libc++ files.

The presence of two "ignoring nonexistent directory" entries indicates to me that Xcode's clang is able to find a libc++ at its hardcoded /Apps/Xcode.app/.../c++/v1/ while my svn clang is not finding one at the place it wants to look (inside the Release+Asserts build dir, which seems silly but it may actually be where libcxx copies its headers into by the llvm build system).

No hints on where it's fetching the libc++ dylibs from though. Not even when looking through the strings that are in the clang executables, using string (which was a long shot anyway).

Not exactly crystal clear what exactly to do, but I do appear to sort of have the tools now to get going on my project. The main issue now is what's going on with the libc++.dylib. I can't tell if the freshly built clang is hardcoded to find it at /usr/lib/libc++.1.dylib, or what. I need to do this:

1. Not touch /usr/lib/libc++.1.dylib so as not to break my entire operating system
2. Point freshly compiled clang++ to use the libc++.1.dylib that's now been built, and is sitting in a different location than /usr/lib/libc++.1.dylib. I don't care where it goes at this point, but I am NOT about to run make install which will likely overwrite /usr/lib/libc++.1.dylib.

It's just not clear what makes sense to do. Does specifying -stdlib=libc++ cause clang++ to link the hardcoded /usr/lib/libc++.1.dylib? If so, can I just drop it, and -l<compiled libc++.dylib> explicitly? Otherwise, how do I compile clang in a way to modify this path to get it to use the right one?

For the time being I will use -stdlib=libc++ -nostdinc++ (see first edit) and pray that it actually means for clang to listen to the -I<new libc++ header path> -L<new libc++ dylib path>. I guess if it still doesn't work and the system files get used anyway despite these efforts, I'll still be happy as long as my programs continue to compile...

Answer 1

I have gotten far enough to satisfactorily answer my own questions, so I will summarize my findings here.

This may not be the whole picture, but it paints enough to get me the information that I was having trouble finding.

Around about line 606 of tools/clang/lib/Driver/ToolChains.cpp is code that just pushes -lc++ to the command arguments. This is in the toolchain implementation of clang (which is more or less part of the component(s) that let clang emulate the command line behavior of gcc, in order that it is compatible for building purposes).

What this means is that libc++.dylib/so gets loaded in using the exact same mechanism as all other libraries are. There is no special treatment for the C++ standard library. What this also means is that it is being done in a semi-hardcoded fashion. Perhaps this is not quite accurate, because I do expect that explicit library path specifications (-L) can take precedence over default root paths.

At any rate, rather than try to tinker with -L, there is a bigger hammer that can be used, --sysroot. By installing libc++ to a separate location from the system's /usr/lib, I can use --sysroot with clang invocations to force clang to not even think about looking in /usr/lib for libc++, or anything else, for that matter. This is something of a useful flag for e.g. cross-compiling jobs.

Okay, those are some ways to manipulate clang to do our bidding. Surely they are sufficiently powerful. Now I needed to be absolutely sure that the libc++ that I thought was being included and linked was actually being included and linked.

I went for something foolproof rather than something simple and clever. I made a few strategic edits to libc++ itself, and then installed it into an alternate directory (not /usr/lib -- see my question for why i do not want to mess around with my system libc++): this can be pretty arbitrary, I added

std::libcxx_custom_version_type::libcxx_custom_version_type() {
  libcxx_custom_version = "slu_libc++_svn_218071";
}

to src/typeinfo.cpp (in libc++) and correspondingly

struct libcxx_custom_version_type {
    char* libcxx_custom_version;
    libcxx_custom_version_type();
};

to include/typeinfo. The purpose this serves (yeah it's awful, oh well) is to make it unambiguously easy to verify that the libc++ I'm linking with is in fact the one I compiled from source. Indeed I can find the string slu_libc++_svn_218071 by using strings on the libc++.dylib. A small amount of test code in a test c++ file can now easily show whether the libc++ being linked is the one I just compiled.

To do the same verification for the actual libc++ headers is much more trivial, we just need to define a new macro and check for it during a compile.

Now I have all the tools to use to answer my questions.

Well, Actually, right now I am getting this error so I can't quite build anything just yet.

Undefined symbols for architecture x86_64:
  "std::terminate()", referenced from:
      ___clang_call_terminate in reflect-16fcd7.o
ld: symbol(s) not found for architecture x86_64

One step at a time, I guess...