Debugging shared libraries with gdbserver

Question

I am using gdbserver on target and CodeSourcery IDE. My hardware is a gumstix with a omap3530.

I can step through code in my main application but if I attempt to step into a function in a shared library I get memory address and a debugger terminates.

This is my library that is compiled and copied to the /lib folder on the target system.(it does have debug symbols) I have attempted to use the .gbdinit file to set solib-absolute-prefix /lib

Here are the warnings from the gdb trace:

903,056 13-gdb-set sysroot-on-target /lib
903,065 13^done
903,065 (gdb) 
903,065 14-target-select remote 192.168.1.101:2345
903,114 =thread-group-started,id="i1",pid="42000"
903,114 =thread-created,id="1",group-id="i1"
903,115 15-list-thread-groups --available
903,120 16-list-thread-groups
903,128 &"warning: Unable to find dynamic linker breakpoint function.\nGDB will be unable to debug shared library initializers\nand track explicitly loaded dynamic code."
903,128 &"\n"

Which leads to

903,395 &"Error while mapping shared library sections:\n"
903,397 &"/lib/libCoreLib.so: Invalid argument.\n"
903,399 =library-loaded,id="/lib/libCoreLib.so",target-name="/lib/libCoreLib.so",hostname="/lib/libCoreLib.so",low-address="0x0",high-address="0x0",symbols-loaded="0",thread-group="i1"

Answer 1

You can debug with the library installed on your host, provided the debugging machine is also the development machine. In that case, you use set sysroot instead of set sysroot-on-target. For example :

set sysroot /home/username/.../rootfs/

where /home/username/.../rootfs/ contains a copy of your target filesystem

I think you should also specify / instead of /lib

Answer 2

Target with debug symbols

This is the simplest method to get working, and it is specially useful when you are developing one particular shared library.

First copy the test executable and shared library to the target with debug information:

• check with readelf ----debug-dump=decodedline libmyib.so: How can I tell if a library was compiled with -g?
• I recommend using an NFS server on host, so that the compiled output gets automatically uploaded after compilation

Then on target:

gdbserver --multi :1234 ./executable_name

Host:

arm-linux-gnueabihf-gdb -q -nh \
  -ex "target extended-remote target-hostname-or-ip:1234" \
  -ex "file ./executable_name" \
  -ex 'tb main' \
  -ex 'c' \
  -ex 'set solib-search-path .'

sharedlibrary libmylib.so also works.

The problem I had was that gdbserver stops at the dynamic loader, before main, and the dynamic libraries are not yet loaded at that point, and so GDB does not know where the symbols will go in memory yet.

GDB appears to have some mechanisms to automatically load shared library symbols, and if I compile for host, and run gdbserver locally, running to main is not needed. But on the ARM target, that is the most reliable thing to do.

Target gdbserver 7.12-6, host arm-linux-gnueabihf-gdb 7.6.1 from Linaro.

Target libraries without debug symbols

It is common to strip target libraries before deployment on embedded targets, since debug information makes them way larger.

For example, Buildroot does that by default, but you can disable it with BR2_STRIP_none=y.

You can identify this scenario by running:

info shared

Which shows something like:

From                To                  Syms Read   Shared Object Library
0x00007ffff7df7f90  0x00007ffff7dfcdd7  Yes (*)     target:/lib/ld64-uClibc.so.0
0x00007ffff7b3a9b0  0x00007ffff7bbe05d  Yes (*)     target:/lib/libc.so.0
(*): Shared library is missing debugging information.

so there are asterisks (*) for both of the libraries which says that debug information is missing.

If that is the case, then you have to tell GDB to use the shared libraries on host before they were stripped.

Buildroot for example makes that easy for us, as it maintains the staging directory which contains the shared libraries before they were stripped and in the same relative paths as in the target:

set sysroot buildroot/output/staging/

When this option is set, gdb immediately searches for libraries in the host instead of target, and finds /lib/libc.so.0 at the path buildroot/output/staging/ + /lib/libc.so.0:

Reading symbols from buildroot/output/staging/lib/ld64-uClibc.so.0...done.
Reading symbols from buildroot/output/staging/lib/libc.so.0...done.

TODO: I don't think you can set more than one sysroot, so all your shared libraries must be placed in their correct relative paths as in the target image.

If you check the bad default sysroot, you will see:

show sysroot

give:

target:

which means that gdb searches for shared libraries on the target root / by default.