Why does g++ look in LIBRARY_PATH/../lib64 and where is this documented?

Question

A similar question was asked here: g++ searches /lib/../lib/, then /lib/

These scary-looking search paths are determined at least in part when the compiler itself it built, for example during the configure phase. It's clear that it goes beyond environment variables because it's possible to have multiple copies of GCC installed and have each of them give different results for gcc --print-search-dirs. Also noting that g++ --print-search-dirs and gcc --print-search-dirs give different results points out that the g++ wrapper is also affecting the search path. Besides configure/build time differences, GCC is definitely aware of the path where its own executable is, and will search subdirectories of that path. A lot of this alchemy can be found in the GCC documentation:
http://gcc.gnu.org/onlinedocs/gcc-4.7.1/gcc/Directory-Options.html#Directory-Options
http://gcc.gnu.org/onlinedocs/gcc-4.7.1/gcc/Environment-Variables.html#Environment-Variables

As far as I know, the most forceful thing that you can without compiling your own copy of GCC is to specify your custom libraries using the -L option. The reason I say this is because -L is searched before e.g. LIBRARY_PATH (see the above link on environment variables). In order to make it more tolerable you could add an alias for g++ including the -L option in your .bashrc file.

If you want a definitive answer then downloading a copy of the GCC source code is one way. For example, in gcc.c the following highly suggestive comment appears:

/* Build a list of search directories from PATHS.
   PREFIX is a string to prepend to the list.
   If CHECK_DIR_P is true we ensure the directory exists.
   If DO_MULTI is true, multilib paths are output first, then
   non-multilib paths.
   This is used mostly by putenv_from_prefixes so we use `collect_obstack'.
   It is also used by the --print-search-dirs flag.  */

However the function that follows the comment is not very obvious.

This is multilib at work - a mechanism which allows for having libraries (but also the whole compilation and build toolchain) for multiple architectures on a single machine. This Wiki states that "The multilib suffix is appended to all directories searched for libraries by GCC and passed via -L options to the linker. The linker itself does not have any particular knowledge of multilibs, and will continue to consult its default search directories if a library is not found in the -L paths. If multiple orthogonal ABI-changing options are used in a single compilation, multiple multilib suffixes can be used in series.".

So, according to the above description, the architecture marker string or different variants thereof are appended to each library search path the compiler receives since it doesn't differentiate between default and custom paths. Your custom path is first in the row, but it undergoes the same "expansion" process as other paths.

Due to the need to handle i386 compatibility, multilib mechanisms seem to now be used by default on most x64 distros, which in practice means most of the installations out there.

I have the exact same problem on:

Fedora 17, gcc 4.7 and gcc 4.3
CentOS 6.3, gcc 4.4
Unubuntu 12, gcc 4.6

So it looks like this is a problem with most gcc versions. Probably this strange behavior first appeared in gcc 4.2 at least according to this.

I tried duping the specs and playing with them. It looks like the *multilib spec is used to append specific strings depending on the platform. For example my original spacs looked like:

*multilib:
. !m64 !m32;64:../lib64 m64 !m32;32:../lib !m64 m32;

When I changed 64:../lib64 to 64:../lib then instead of ../lib64 gcc appended ../lib. But I couldn't fully decipher the meaning of *multilib or any of the other specs.

This answer attempts to summarize the search path behaviors for both GCC and Clang.

GCC

Include path: for a command-line of the following format:

CPLUS_INCLUDE_PATH=EDIR g++ -IIDIR -isystemSDIR

the following list of directories are used as search paths for #include <...>:

IDIR                                           # '-I' directories.
SDIR                                           # '-isystem' directories.
EDIR                                           # *_INCLUDE_PATH directories.
GCCDIR/include/c++/GCCVER                      # libstdc++ directory (C++).
GCCDIR/include/c++/GCCVER/GCCARCH              # libstdc++ directory (C++).
GCCDIR/include/c++/GCCVER/backward             # libstdc++ directory (C++).
GCCDIR/lib/gcc/GCCARCH/GCCVER/include          # GCC arch-specific directory.
/usr/local/include/GCCARCH                     # Local arch-specific include directory.
/usr/local/include                             # Local include directory.
GCCDIR/include                                 # GCC include directory.
GCCDIR/lib/gcc/GCCARCH/GCCVER/include-fixed    # GCC include-fixed directory.
/usr/include/GCCARCH                           # System include arch-specific directory.
/usr/include                                   # System include directory.

Library path: for a command-line of the following format:

LIBRARY_PATH=EDIR gcc -BBDIR -LLDIR

the following arguments are passed to the linker:

-LLDIR                                         # '-L' directories.
-LBDIR                                         # '-B' directories.
-LEDIR/../libXX                                # Multilib directories from LIBRARY_PATH.
-LGCCDIR/lib/gcc/GCCARCH/GCCVER                # GCC arch-specific library directory.
-LGCCDIR/libXX                                 # GCC multilib library directory.
-L/libXX                                       # System multilib library directory.
-L/usr/libXX                                   # System multilib library directory. 
-LEDIR                                         # LIBRARY_PATH directories.
-LGCCDIR/lib                                   # Other GCC libraries.

Clang

Include path: for a command-line of the following format:

CPLUS_INCLUDE_PATH=EDIR clang++ --gcc-toolchain=GCCDIR -BBDIR -IIDIR -isystemSDIR

the following list of directories are used as search paths for #include <...>:

IDIR                                           # '-I' directories.
SDIR                                           # '-isystem' directories.
EDIR                                           # *_INCLUDE_PATH directories.
 # If -stdlib=libstdc++ is used:
   GCCDIR/include/c++/GCCVER                   # libstdc++ directory from the selected GCC toolchain (C++).
   GCCDIR/include/c++/GCCVER/GCCARCH           # libstdc++ directory from the selected GCC toolchain (C++).
   GCCDIR/include/c++/GCCVER/backward          # libstdc++ directory from the selected GCC toolchain (C++).
 # If -stdlib=libc++ is used:
   CLANGDIR/include/c++/v1                     # libc++ directory (C++).
/usr/local/include                             # Local include directory.
CLANGDIR/lib/clang/CLANGVER/include            # Clang include directory.
/include                                       # System include directory.
/usr/include                                   # System include directory.

Library path: for a command-line of the following format:

LIBRARY_PATH=EDIR clang --gcc-toolchain=GCCDIR -BBDIR -LLDIR    

the following arguments are passed to the linker:

-LLDIR                                         # '-L' directories.
-LGCCDIR/lib/gcc/GCCARCH/GCCVER                # GCC arch-specific library directory.
-LGCCDIR/libXX                                 # GCC multilib library directory.
-L/libXX                                       # System multilib library directory.
-L/usr/libXX                                   # System multilib library directory.
-LGCCDIR/lib                                   # Other GCC libraries.
-LCLANGDIR/lib                                 # Clang libraries.
-L/lib                                         # System library directory.
-L/usr/lib                                     # System library directory.
-LEDIR                                         # LIBRARY_PATH directories.

Summary

The search path for includes are pretty much the same in both GCC and Clang. C++-specific paths are omitted if the C frontend is used in both cases. Library search paths differ substantially between GCC and Clang, notably the presence of -B directories and the odd manipulation of LIBRARY_PATH in the GCC frontend.

The library search paths are the same for both C and C++ frontends. Other library search paths are introduced by the linker itself. The following excerpt comes from the vanilla linker script for GNU Binutils:

# Multilib library directories.
SEARCH_DIR("BINUTILSDIR/BINUTILSARCH/libXX");
SEARCH_DIR("BINUTILSDIR/libXX");
SEARCH_DIR("/usr/local/libXX");
SEARCH_DIR("/libXX");
SEARCH_DIR("/usr/libXX");
# Traditional library directories.
SEARCH_DIR("BINUTILSDIR/BINUTILSARCH/lib");
SEARCH_DIR("BINUTILSDIR/lib");
SEARCH_DIR("/usr/local/lib");
SEARCH_DIR("/lib");
SEARCH_DIR("/usr/lib");

It is also imperative to note that library dependencies are not searched for inside the directories listed above. These exclusively rely on -rpath and -rpath-link options passed to the linker, or else they get resolved from the default system library paths. Thus it might be useful to produce both -L and -rpath-link arguments in order to guarantee that the correct libraries are linked.

Finally, special files (such as the CRT objects) are searched for only in -B directories. In Clang, special files are also searched inside the selected GCC toolchain. Other factors (spec files, distribution-specific configuration) might change some or all of the above.