CMake project fails to find shared library

Question

I am using CMake to build a cross platform project. For the moment I am trying to run it on Linux. I have recently added a project for running tests, but it will not run because it cannot find one of the shared libraries, specifically libtbbmalloc.so.2:

/tests: error while loading shared libraries: libtbbmalloc.so.2: cannot open shared object file: No such file or directory`

When I run ldd on the executable I get the following:

linux-vdso.so.1 (0x00007fffeb572000)
libtbbmalloc_proxy.so.2 => /home/username/dev/tbb/libtbbmalloc_proxy.so.2 (0x00007f50afe00000)
libstdc++.so.6 => /usr/lib/x86_64-linux-gnu/libstdc++.so.6 (0x00007f50afa70000)
libm.so.6 => /lib/x86_64-linux-gnu/libm.so.6 (0x00007f50af6d0000)
libgcc_s.so.1 => /lib/x86_64-linux-gnu/libgcc_s.so.1 (0x00007f50af4b0000)
libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007f50af0a0000)
libdl.so.2 => /lib/x86_64-linux-gnu/libdl.so.2 (0x00007f50aee90000)
libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0 (0x00007f50aec70000)
librt.so.1 => /lib/x86_64-linux-gnu/librt.so.1 (0x00007f50aea60000)
/lib64/ld-linux-x86-64.so.2 (0x00007f50b0400000)
libtbbmalloc.so.2 => not found

The CMakeLists.txt for my test project looks like this:

set(test_sourcefiles main_tests.cpp)

add_executable(tests ${test_sourcefiles})

target_link_libraries(tests Catch2::Catch2 MyLib)

MyLib uses tbb, and I guess that is why my executable (tests) searches for it. When running ldd on MyLib it finds the library (libtbbmalloc.so.2):

(removed some output for readability)
libtbbmalloc_proxy.so.2 => /home/username/dev/tbb/libtbbmalloc_proxy.so.2 (0x00007f9af8110000)
libtbbmalloc.so.2 => /home/username/dev/tbb/libtbbmalloc.so.2 (0x00007f9ac4eb0000)

I have tried specifically adding libttbbmalloc.so.2 in my tests/CMakeLists.txt target_link_libraries(${project} /home/username/dev/tbb/libtbbmalloc.so.2), but it makes no difference.

If I add /home/username/dev/tbb/ to LD_LIBRARY_PATH, the program runs, and ldd reports that libtbbmalloc.so.2 is found.

Any ideas on what I might be doing wrong, and how can I get my program to run without setting LD_LIBRARY_PATH?

Update: I found out that it's possible to print runpath/rpath by using chrpath -l name-of-executable. When using this tool on my executable, it looks like the folder with libtbbmalloc.so.2 is added to runpath, but the program still won't run:

larjr@DESKTOP:~/dev/project/build/tests$ chrpath -l tests
tests: RUNPATH=/home/larsjr/dev/project/build/MyLib:/home/username/dev/tbb

Answer 1

The issue you are encountering does sound to be related to the search path for direct and indirect linkage of shared libraries. CMake may build with the new ld RUNPATH variable which will only work on direct dependents, where RPATH will work recursively on all indirect dependents. As shown in the snippet at bottom of this post, this may depend on your version of GCC. To check if your executable is using RPATH or RUNPATH, run:

$ readelf -d ./executable

and find the library rpath / runpath line after your dependencies. ie/eg: It may show either of the below:

a) 0x000000000000001d (RUNPATH)            Library runpath: [/some/dir/lib]
b) 0x000000000000001d (RPATH)              Library rpath:   [/some/dir/lib]

You showed you used the command chrpath -l ./executable, which actually spat out:

larjr@DESKTOP:~/dev/project/build/tests$ chrpath -l tests

tests: RUNPATH=/home/larsjr/dev/project/build/MyLib:/home/username/dev/tbb

It says here "RUNPATH" which is non-recursive - that's your error.

Refer to this excellent github issue where members are discussing indirect linking issues with CMake, highly suggest reading it all. It has a a great example regarding dependencies and their search paths. https://github.com/conan-io/conan/issues/2660

A snippet regarding new vs old linker behavior which can cause this 'issue':

The problem you are experiencing in conan-deptest (excellent isolated case, by the way!) is the behaviour that you are documenting, between DT_RPATH and DT_RUNPATH. It is CMake (and not conan), that passes the -rpath flag with the correct paths to the dependent libraries, when building the executable. The problem is that newer versions of the linker in some platforms, emit DT_RUNPATH where they used to emit DT_RPATH. Indeed this is the case as readelf -d is listing the RUNPATH tag.

There is also another stackoverflow post where a user asks about this change in behaviour in LD/GCC: How to set RPATH and RUNPATH with GCC/LD?

A conan member further elaborates on how to change this behavior:

The way to replicate this behaviour on versions of the linker that emit RUNPATH instead, is to pass the --disable-new-dtags flag to the linker.

And the resulting cmake:

target_link_libraries(target "-Wl,--disable-new-dtags")

Now if you rebuild, and check readelf -d on your executable, you should see the RPATH tag. You can run:

env LD_DEBUG=files,libs ./executable

to see RPATH being passed into the search path for each dependency:

31658:  file=libbar.so.1 [0];  needed by /home/user/projectA/lib/libfoo.so.2.5 [0]
31658:  find library=libbar.so.1 [0]; searching
31658:   search path=/home/user/projectA/lib        (RPATH from file ./executable)
31658:    trying file=/home/user/projectA/lib/libbar.so.1

Hopefully this fixes your issue!

Bonus: Link to tutorial on shared libraries. At the bottom of the aritcle shows runpath vs rpath + origin! https://amir.rachum.com/blog/2016/09/17/shared-libraries/

Answer 2

I believe you are looking for setting a runtime path on the executable:

set_target_properties(tests PROPERTIES
    INSTALL_RPATH "<your path to libs>"
    BUILD_WITH_INSTALL_RPATH 1
)

INSTALL_RPATH

Normally CMake uses the build tree for the RPATH when building executables etc on systems that use RPATH. When the software is installed the executables etc are relinked by CMake to have the install RPATH. If this variable is set to true then the software is always built with the install path for the RPATH and does not need to be relinked when installed.

BUILD_WITH_INSTALL_RPATH

Normally CMake uses the build tree for the RPATH when building executables etc on systems that use RPATH. When the software is installed the executables etc are relinked by CMake to have the install RPATH. If this variable is set to true then the software is always built with the install path for the RPATH and does not need to be relinked when installed.