Undefined symbol when loading a shared library

Question

In my program I need to load a shared library dynamically with dlopen(). Both the program and the shared library are successfully cross-compiled for an ARM architecture with the cross-compiler installed on my x86. However, whenever the program tries to load the library at run time on ARM, it fails giving this error:

undefined symbol: _dl_hwcap

I cannot find the culprit of this error.

Let me give details on how the shared library (libmyplugin.so) is built on x86 first. I use the g++ cross-compiler as below:

/home/me/arm/gcc-arm-linux-gnueabihf/bin/arm-linux-gnueabihf-g++ -march=armv7-a -mfloat-abi=hard -c -s -fPIC -o build/module1.o module1.cpp
/home/me/arm/gcc-arm-linux-gnueabihf/bin/arm-linux-gnueabihf-g++ -march=armv7-a -mfloat-abi=hard -c -s -fPIC -o build/module2.o module2.cpp
/home/me/arm/gcc-arm-linux-gnueabihf/bin/arm-linux-gnueabihf-g++ -o dist/libmyplugin.so build/module1.o build/module2.o --sysroot /home/me/arm/sysroot/ -Wl,--no-as-needed -ldl -lX11 -lXext /home/me/arm/libstatic.a -shared -s -fPIC

Please pay attention to the following notes:

1. module1.cpp and module2.cpp are my source code files.
2. libstatic.a is a big archive of object .o files implementing the stuff directly invoked/referenced by module1.cpp and module2.cpp. These object files have been compiled by others for the same ARM architecture as mine, with the same compiler flags, but using a slightly more updated g++ compiler (v4.9 instead of my v4.8.3). Unfortunately, I have no control on the building of these objects.
3. --sysroot /home/me/arm/sysroot/ represents the remote filesystem of my ARM OS from which the local g++ cross-compiler can take the native libraries while linking.
4. -Wl,--no-as-needed -ldl -lX11 -lXext: these flags are required to force the dynamic loader to load the X11 libraries present on the system when my shared library is loaded by the program. In particular, --no-as-needed is required because the X11 libraries are NOT directly referenced by module1.o and module2.o; on the contrary the X11 libraries are referenced by the static library only.

Note that all the above setup works on x86. It's just that I don't understand what is the reason of the _dl_hwcap symbol not resolved when the program tried to load the library on ARM.

Do you have any idea how to investigate this issue?

Answer 1

There are a myriad of things that could be problematic, but here are four avenues of exploration. I am concentrating on the -shared in your link line, but the last item addresses that as well. (A nice HOWTO on shared libraries is here: http://tldp.org/HOWTO/Program-Library-HOWTO/shared-libraries.html

a) Check your environment variable LD_LIBRARY_PATH. Since you aren't using RPATH to the linker (RPATH embeds a full path to the .so so you can find it at runtime), then the only way the linker can find your code is to search the LD_LIBRARY_PATH. Make sure the .so or .0 you want is in the path.

b) Use the UNIX utility 'nm' to search .so (shared objects) and .a files for that symbol. For example, 'nm -D /usr/lib64/libpython2.6.so' will show all dynamic symbols in the libpython.so, and you can look for symbols of interest: For example, Is 'initgc' defined or used in libpython?

% nm -D /usr/lib64/libpython2.6.so | grep initgc
000003404300cf0 T initgc

The 'T' means TEXT or, yes, it is defined there. See if you can find the symbol in the module of interest using grep and nm. (A 'U' means undefined, which means it is defined in another module).

c) Another useful tool is 'ldd'. It shows all dynamic libraries that the library you are looking on depends on. For example:

% ldd /usr/lib64/libpython2.6.so
linux-vdso.so.1 =>  (0x00007fffa49ff000)
libpthread.so.0 => /lib64/libpthread.so.0 (0x00000033f0200000)
libdl.so.2 => /lib64/libdl.so.2 (0x00000033f0600000)
libutil.so.1 => /lib64/libutil.so.1 (0x00000033fea00000)
libm.so.6 => /lib64/libm.so.6 (0x00000033f0a00000)
libc.so.6 => /lib64/libc.so.6 (0x00000033efe00000)
/lib64/ld-linux-x86-64.so.2 (0x00000033efa00000)

If it can't find a library (because it's not on the LD_LIBRARY_PATH or wasn't specified in the RPATH), the library will turn up empty.

d) I am a little worried from your link line of seeing a '.a' file with a -shared option. Some compilers/linkers cannot use a '.a' (archive) file to create a '.so' file. '.so' files usually have to made from other '.so' files or '.o' files that have been compiled with -fPIC.

I would recommend (if you can), recompile /home/me/arm/libstatic.a so that it's a .so. If you can't do, you might have to make your final output a '.a' file as well. (In other words, get rid of the -shared command line option).

In summary: Check your LD_LIBRARY_PATH, use nm and ldd to look around at your .a and .so files, but I think the end result is that you may not be able to combine .so and .a files.

I hope this helps.