libraries which exist in a binary's elf RUNPATH are not being used?

Question

I have custom built gcc-4.7.2 in my environment. The system gcc is gcc-4.3.4.

I have patched the RUNPATH for all my custom gcc's binaries and shared libraries using patchelf --set-rpath

However, when I run ldd on my 4.7.2 cc1 it picks up the system libstdc++ instead of the one pointed to by the RUNPATH:

$ ldd /sdk/x86_64/2.11.1/gcc-4.7.2/libexec/gcc/x86_64-suse-linux/4.7.2/cc1
        libcloog-isl.so.1 => /sdk/x86_64/2.11.1/gcc-4.7.2/lib/libcloog-isl.so.1 (0x00007f072dce8000)
        ...
        libc.so.6 => /lib64/libc.so.6 (0x00007f072bfe0000)
   -->  libstdc++.so.6 => /usr/lib64/libstdc++.so.6 (0x00007f072bcd5000)
        libgcc_s.so.1 => /lib64/libgcc_s.so.1 (0x00007f072babe000)
        /lib64/ld-linux-x86-64.so.2 (0x00007f072df0d000)

As can be seen the RUNPATH specifies the gcc-4.7.2 library locations:

$ readelf -a /sdk/x86_64/2.11.1/gcc-4.7.2/libexec/gcc/x86_64-suse-linux/4.7.2/cc1 | grep PATH
 0x000000000000001d (RUNPATH)            Library runpath: \
    [/sdk/x86_64/2.11.1/gcc-4.7.2/lib64: \
     /sdk/x86_64/2.11.1/gcc-4.7.2/lib: \
     /sdk/x86_64/2.11.1/gcc-4.7.2/libexec/gcc/x86_64-suse-linux/lib64: \
     /sdk/x86_64/2.11.1/gcc-4.7.2/lib/gcc/x86_64-suse-linux/4.7.2: \
     /hostname/sig/lib64: \
     /hostname/sig/lib]

I know that libstdc++.so.6 exists in the first entry in the RUNPATH:

$ ls -l /sdk/x86_64/2.11.1/gcc-4.7.2/lib64/libstdc++.so*
lrwxrwxrwx .../sdk/x86_64/2.11.1/gcc-4.7.2/lib64/libstdc++.so -> libstdc++.so.6.0.17
lrwxrwxrwx .../sdk/x86_64/2.11.1/gcc-4.7.2/lib64/libstdc++.so.6 -> libstdc++.so.6.0.17
-rwxr-x--- .../sdk/x86_64/2.11.1/gcc-4.7.2/lib64/libstdc++.so.6.0.17
-rwxr-x--- .../sdk/x86_64/2.11.1/gcc-4.7.2/lib64/libstdc++.so.6.0.17-gdb.py

I don't have an LD_LIBRARY_PATH set in my environment:

$ echo $LD_LIBRARY_PATH

$

• How come it doesn't pick up the library found in RUNPATH?
• How can I force it to use the gcc-4.7.2 libraries?

Answer 1

The problem is that one of the prerequisites (libppl.so) also imports libstdc++. That prerequisite was built using the system gcc, and therefore finds /usr/lib64/libstdc++.so.6

$ ldd /sdk/x86_64/2.11.1/gcc-4.7.2/lib/libppl.so
        linux-vdso.so.1 =>  (0x00007fffd10db000)
        libgmpxx.so.4 => /sdk/x86_64/2.11.1/gcc-4.7.2/lib/libgmpxx.so.4 (0x00007f4716f92000)
        libgmp.so.10 => /sdk/x86_64/2.11.1/gcc-4.7.2/lib/libgmp.so.10 (0x00007f4716d26000)
    --> libstdc++.so.6 => /usr/lib64/libstdc++.so.6 (0x00007f4716a25000)
        libm.so.6 => /lib64/libm.so.6 (0x00007f47167a0000)
        libc.so.6 => /lib64/libc.so.6 (0x00007f4716441000)
        libgcc_s.so.1 => /usr/lib64/libgcc_s.so.1 (0x00007f471622c000)
        /lib64/ld-linux-x86-64.so.2 (0x00007f47174b4000)

Once a library has been located by the dynamic linker once, it will no longer be searched for; that location will be used for any subsequent requirements.

I resolved this by rebuilding the prerequisites with the new gcc.

$ ldd /sdk/x86_64/2.11.1/gcc-4.7.2/lib/libppl.so
        linux-vdso.so.1 =>  (0x00007fffd10db000)
        libgmpxx.so.4 => /sdk/x86_64/2.11.1/gcc-4.7.2/lib/libgmpxx.so.4 (0x00007f4716f92000)
        libgmp.so.10 => /sdk/x86_64/2.11.1/gcc-4.7.2/lib/libgmp.so.10 (0x00007f4716d26000)
    --> libstdc++.so.6 => /sdk/x86_64/2.11.1/gcc-4.7.2/lib/../lib64/libstdc++.so.6 (0x00007f4716a25000)
        libm.so.6 => /lib64/libm.so.6 (0x00007f47167a0000)
        libc.so.6 => /lib64/libc.so.6 (0x00007f4716441000)
        libgcc_s.so.1 => /sdk/x86_64/2.11.1/gcc-4.7.2/lib/../lib64/libgcc_s.so.1 (0x00007f471622c000)
        /lib64/ld-linux-x86-64.so.2 (0x00007f47174b4000)

I'm thinking the final step is to now rebuild gcc with the newly build prerequisites.

• build prerequisites with system gcc
• build new gcc
• rebuild prerequisites with new gcc
• rebuild gcc with rebuilt prerequisites

Whether the final step is necessary I'm not sure.

Answer 2

You need to set the LD_LIBRARY_PATH to point to the desired libstdc++. RUNPATH is evaluated after LD_LIBRARY_PATH.

Quoting from RPATH issue:

The dynamic linker will look for a matching library in the following locations, in this order, which can be changed (see the footnotes below): 
1. the DT_RPATH dynamic section attribute of the library causing the lookup 
2. the DT_RPATH dynamic section attribute of the executable 
3. the LD_LIBRARY_PATH environment variable, unless the executable is setuid/setgid. 
4. the DT_RUNPATH dynamic section attribute of the executable 
5. /etc/ld.so.cache 
6. base library directories (/lib and /usr/lib)