Cross-compiling and linking to a library in a sysroot - what's wrong?

Question

I am not inexperienced but it's become clear to me that I lack some fundamental understanding in the area of cross-compilation and sysroots. I'm hoping someone can provide me that gem of information I need to continue.

I have downloaded a pre-built armhf gcc cross-compiler/toolchain and it came with a directory called arm-linux-gnueabihf that contains the following directories:

bin/
include/
lib/  - contains libstdc++.a, libgcc_s.so, but does NOT contain `libcrypt.so` or `libcrypt.so.1`, etc.
libc/

Curiously libc contains yet another set of directories, that looks a bit like a separate sysroot, but I'm not sure why it's here. I've looked in other toolchains, including one I built myself with crosstool-ng, and I've not seen anything similar:

libc/
     etc/
     lib/  - contains files like libcrypt.so.1 / libcrypt-2.24.so
     sbin/
     usr/
         bin/
         include/
         lib/  - contains files like libc.a, libc.so, libcrypt.a, libcrypt.so, 
         libexec/
         sbin/
         share/
     var/

Anyway, I'm not sure if that's a problem, or if I have to merge those two sysroots into one

I have installed it within a Docker container at the path /cross-pi-gcc-9.1.0-1. I am using cmake to cross-compile, and my toolchain.cmake file refers to this toolchain:

SET(CMAKE_SYSTEM_NAME Linux)
SET(CMAKE_SYSTEM_VERSION 1)
SET(CMAKE_C_COMPILER /cross-pi-gcc-9.1.0-1/bin/arm-linux-gnueabihf-gcc)
SET(CMAKE_CXX_COMPILER /cross-pi-gcc-9.1.0-1/bin/arm-linux-gnueabihf-g++)
SET(CMAKE_FIND_ROOT_PATH /cross-pi-gcc-9.1.0-1/arm-linux-gnueabihf/)
SET(CMAKE_FIND_ROOT_PATH_MODE_PROGRAM NEVER)
SET(CMAKE_FIND_ROOT_PATH_MODE_LIBRARY ONLY)
SET(CMAKE_FIND_ROOT_PATH_MODE_INCLUDE ONLY)

This seems to be sufficient to build most code with the cross-compiler (including Boost).

As part of this container, I wish to have WiringPi installed in the sysroot, so I can build and link against it.

To do this, I have created a custom CMakeLists.txt file that successfully builds and installs WiringPi:

cmake_minimum_required(VERSION 3.0)
project(WiringPi)
set(CMAKE_THREAD_PREFER_PTHREAD TRUE)
find_package(Threads REQUIRED)
add_library(wiringPi SHARED ads1115.c <snip a bunch of .c files>)
target_include_directories(wiringPi PUBLIC ${CMAKE_CURRENT_SOURCE_DIR})
target_link_libraries(wiringPi PUBLIC ${CMAKE_THREAD_LIBS_INIT} crypt m rt)
install(TARGETS wiringPi DESTINATION lib)
install(FILES ads1115.h <snip a bunch of .h files>
        DESTINATION include)

This indicates that the resultant libwiringpi.so should be linked with libcrypt, libpthread, libm and librt. And if I run the target's ldd tool on libwiringpi.so it does indeed show these libraries as dependencies:

$ ldd libwiringPi.so 
    linux-vdso.so.1 (0x7eee6000)
    /usr/lib/arm-linux-gnueabihf/libarmmem-${PLATFORM}.so => /usr/lib/arm-linux-gnueabihf/libarmmem-v7l.so (0x76f0a000)
    libpthread.so.0 => /lib/arm-linux-gnueabihf/libpthread.so.0 (0x76ee0000)
    libcrypt.so.1 => /lib/arm-linux-gnueabihf/libcrypt.so.1 (0x76ea0000)
    libm.so.6 => /lib/arm-linux-gnueabihf/libm.so.6 (0x76e1e000)
    librt.so.1 => /lib/arm-linux-gnueabihf/librt.so.1 (0x76e07000)
    libc.so.6 => /lib/arm-linux-gnueabihf/libc.so.6 (0x76cb9000)
    /lib/ld-linux-armhf.so.3 (0x76f43000)

The problem I'm having is that I have an application that links against wiringpi using a cmake line like this:

target_link_libraries(myapp wiringPi)

When I build this on a raspberry pi using the native toolchain, I don't need to explicitly link against libcrypt. But in my Docker/cross-compiler environment, I get this error at link time:

/cross-pi-gcc-9.1.0-1/bin/arm-linux-gnueabihf-g++     CMakeFiles/app.dir/main.cpp.o   -o myapp -lpthread -lwiringPi -lrt 
/cross-pi-gcc-9.1.0-1/bin/../lib/gcc/arm-linux-gnueabihf/9.1.0/../../../../arm-linux-gnueabihf/bin/ld: warning: libcrypt.so.1, needed by /cross-pi-gcc-9.1.0-1/bin/../lib/gcc/arm-linux-gnueabihf/9.1.0/../../../../arm-linux-gnueabihf/lib/libwiringPi.so, not found (try using -rpath or -rpath-link)
/cross-pi-gcc-9.1.0-1/bin/../lib/gcc/arm-linux-gnueabihf/9.1.0/../../../../arm-linux-gnueabihf/lib/libwiringPi.so: undefined reference to `crypt@GLIBC_2.4'
collect2: error: ld returned 1 exit status

Note that -lrt and -lpthread seem to have been automatically included in the link library list. But -lcrypt is not present.

If I copy/paste the make VERBOSE=1 output that corresponds to this error, and manually add -lcrypt to the end of the command, it links successfully and the application compilation is complete.

I realise this is a long description but what I'm ultimately trying to do is find the hole in my knowledge that is preventing me from understanding why I would need to explicitly link libcrypt into this application in this environment.

I would have thought, perhaps wrongly, that since libwiringpi.so is already linked against libcrypt, that it wouldn't need to be linked at the top level. If that's not how it works, is there anyone that can help me repair my mental model, please?

Note: I could just add target_link_libraries(myapp wiringPi crypt) however I don't think it's necessary (not needed when building natively) and I'd like to learn a bit more about the process rather than just finding a workaround.

Answer 1

Answering this myself...

It looks like the presence of the libc directory in the sysroot is unusual, and I checked other toolchains and it was always merged with lib. So I ended up ditching that particular toolchain and building my own with crosstool-ng instead. This avoided the problem entirely. It would be nice to understand how I might have got this working, but for now I must move on.

Note that I had success with just using CMAKE_SYSROOT only - no need in my case to specify CMAKE_FIND_ROOT_PATH and friends.