Convincing gcc to ignore system libraries in favour of locally installed libraries

Question

I am trying to build a simple executable that uses boost_serialization and boost_iostreams.

#include <fstream>
#include <iostream>
#include <boost/archive/xml_iarchive.hpp>
#include <boost/archive/xml_oarchive.hpp>
#include <boost/iostreams/filtering_stream.hpp>
#include <boost/iostreams/filter/gzip.hpp>
#include <boost/iostreams/device/file.hpp>

int main()
{
    using namespace boost::iostreams;
    filtering_ostream os;
    os.push(boost::iostreams::gzip_compressor());
    os.push(boost::iostreams::file_sink("emptyGzipBug.txt.gz"));
}

Unfortunately the system I am working with has a very outdated version of boost_serialization in /usr/lib/, and I have no way to change that.

I am fairly certain when I build the example using

g++ -o main main.cpp -lboost_serialization -lboost_iostreams

that the linker errors result because gcc uses the system version of boost_serialization rather than my locally installed version. Setting LIBRARY_PATH and LD_LIBRARY_PATH to /home/andrew/install/lib doesnt work. When i build using

g++ -o main main.cpp -L/home/andrew/install/lib -lboost_serialization -lboost_iostreams

then everything works.

My questions are:

1. How can I get gcc to tell me the filenames of the libraries its using?

2. Is it possible to setup the environment so that I dont have to specify the absolute path to my local boost on the command line of gcc.

Answer 1

PS After typing the below info, I thought I'd be kind and add what you need for your specific case:

g++ -Wl,-rpath,/home/andrew/install/lib -o main main.cpp -I/home/andrew/install/include -L/home/andrew/install/lib -lboost_serialization -lboost_iostreams

---

gcc itself doesn't care about the libraries. The linker does ;). Even though the linker needs to find the shared libraries so it can resolve symbols, it doesn't store the path of those libraries in the executable normally.

So, for a start, lets find out what is actually in the binary after you linked it:

$ readelf -d main | grep 'libboost'
 0x0000000000000001 (NEEDED)             Shared library: [libboost_serialization.so.1.54.0]
 0x0000000000000001 (NEEDED)             Shared library: [libboost_iostreams.so.1.54.0]

Just the names thus.

The libraries that are actually used are detemined by /lib/ld-linux.so.* at run time:

$ ldd main | grep libboost
        libboost_serialization.so.1.54.0 => /usr/lib/x86_64-linux-gnu/libboost_serialization.so.1.54.0 (0x00007fd8fa920000)
        libboost_iostreams.so.1.54.0 => /usr/lib/x86_64-linux-gnu/libboost_iostreams.so.1.54.0 (0x00007fd8fa700000)

The path is found by looking in /etc/ld.so.cache (which is normally compiled by running ldconfig). You can print its contents with:

ldconfig -p | grep libboost_iostreams
    libboost_iostreams.so.1.54.0 (libc6,x86-64) => /usr/lib/x86_64-linux-gnu/libboost_iostreams.so.1.54.0
    libboost_iostreams.so.1.49.0 (libc6,x86-64) => /usr/lib/libboost_iostreams.so.1.49.0
    libboost_iostreams.so (libc6,x86-64) => /usr/lib/x86_64-linux-gnu/libboost_iostreams.so

but since that is only the cached result of a previous look up, you are more interested in the output of:

$ ldconfig -v 2>/dev/null | egrep '^[^[:space:]]|libboost_iostreams'
/lib/i386-linux-gnu:
/usr/lib/i386-linux-gnu:
/usr/local/lib:
/lib/x86_64-linux-gnu:
/usr/lib/x86_64-linux-gnu:
    libboost_iostreams.so.1.54.0 -> libboost_iostreams.so.1.54.0
/lib32:
/usr/lib32:
/lib:
/usr/lib:
    libboost_iostreams.so.1.49.0 -> libboost_iostreams.so.1.49.0

which shows the paths that it looked in before finding a result. Note if you are linking a 64bit program and it would find a 32bit library first (or visa versa) then that would be skipped as being incompatible. Otherwise, the first one found is used.

The paths used to search are specified in /etc/ld.so.conf which is read (usually at boot time, or after installing something new) when running ldconfig as root.

However, precedence take paths specified as a colon separated list of paths in the environment variable LD_LIBRARY_PATH. For example, if I'd do:

$ export LD_LIBRARY_PATH=/tmp
$ cp /usr/lib/libboost_iostreams.so.1.49.0 /tmp/libboost_iostreams.so.1.54.0
$ ldd main | grep libboost_iostreams
    libboost_iostreams.so.1.54.0 => /tmp/libboost_iostreams.so.1.54.0 (0x00007f621add8000)

then it finds 'libboost_iostreams.so.1.54.0' in /tmp (even though it was a libboost_iostreams.so.1.49.0).

Note that you CAN hardcode a path in your executable by passing -rpath to the linker:

$ unset LD_LIBRARY_PATH
$ g++ -Wl,-rpath,/tmp -o main main.cpp -lboost_serialization -lboost_iostreams
$ ldd main | grep libboost_iostreams
    libboost_iostreams.so.1.54.0 => /tmp/libboost_iostreams.so.1.54.0 (0x00007fbd8bcd8000)

which can be made visible with

$ readelf -d main | grep RPATH
 0x000000000000000f (RPATH)              Library rpath: [/tmp]

Note that LD_LIBRARY_PATH even takes precedence over -rpath, unless you also passed -Wl,--disable-new-dtags, along with the -rpath and provided that you are linking an executable and your linker supports this flag.

You can show the search paths that gcc uses during compile(link) time with the -print-search-dirs command line option:

$ g++ -print-search-dirs  | grep libraries
libraries: =/usr/lib/gcc/x86_64-linux-gnu/4.7/:/usr/lib/gcc/x86_64-linux-gnu/4.7/../../../../x86_64-linux-gnu/lib/x86_64-linux-gnu/4.7/:/usr/lib/gcc/x86_64-linux-gnu/4.7/../../../../x86_64-linux-gnu/lib/x86_64-linux-gnu/:/usr/lib/gcc/x86_64-linux-gnu/4.7/../../../../x86_64-linux-gnu/lib/../lib/:/usr/lib/gcc/x86_64-linux-gnu/4.7/../../../x86_64-linux-gnu/4.7/:/usr/lib/gcc/x86_64-linux-gnu/4.7/../../../x86_64-linux-gnu/:/usr/lib/gcc/x86_64-linux-gnu/4.7/../../../../lib/:/lib/x86_64-linux-gnu/4.7/:/lib/x86_64-linux-gnu/:/lib/../lib/:/usr/lib/x86_64-linux-gnu/4.7/:/usr/lib/x86_64-linux-gnu/:/usr/lib/../lib/:/usr/lib/gcc/x86_64-linux-gnu/4.7/../../../../x86_64-linux-gnu/lib/:/usr/lib/gcc/x86_64-linux-gnu/4.7/../../../:/lib/:/usr/lib/

This can be influenced by adding -L command line options. If a library can't be found in a path specified with the -L option then it looks in paths found through the environment variable GCC_EXEC_PREFIX (see the man page for that) and if that fails it uses the environment variable LIBRARY_PATH.

When you run g++ with the -v option, it will print the LIBRARY_PATH used.

LIBRARY_PATH=/tmp/lib g++ -v -o main main.cpp -lboost_serialization -lboost_iostreams 2>&1 | grep LIBRARY_PATH
LIBRARY_PATH=/tmp/lib/../lib/:/usr/lib/gcc/x86_64-linux-gnu/4.7/:/usr/lib/gcc/x86_64-linux-gnu/4.7/../../../x86_64-linux-gnu/:/usr/lib/gcc/x86_64-linux-gnu/4.7/../../../../lib/:/lib/x86_64-linux-gnu/:/lib/../lib/:/usr/lib/x86_64-linux-gnu/:/usr/lib/../lib/:/tmp/lib/:/usr/lib/gcc/x86_64-linux-gnu/4.7/../../../:/lib/:/usr/lib/

Finally, note that especially for boost (but in general) you should use header files that match the correct version! So, if the library that you link with at run time is version xyz you should have used an -I command line option to get g++ to find the corresponding header files, or things might not link or worse, result in unexplainable crashes.