What do I need to debug pthreads?

Question

I want to debug pthreads on my custom linux distribution but I am missing something. My host is Ubuntu 12.04, my target is an i486 custom embedded Linux built with a crosstool-NG cross compiler toolset, the rest of the OS is made with Buildroot.

I will list the facts:

• I can run multi-threaded applications on my target

• Google Breakpad fails to create a crash report when I run a multi-threaded application on the target. The exact same application with the exact same build of Breakpad libraries will succeed when I run it on my host.

• GDB fails to debug multithreaded applications on my target.

e.g.

$./gdb -n -ex "thread apply all backtrace" ./a.out --pid 716

dlopen failed on 'libthread_db.so.1' - /lib/libthread_db.so.1: undefined symbol: ps_lgetfpregs
GDB will not be able to debug pthreads.
GNU gdb 6.8

I don't think ps_lgetfpregs is a problem because of this.

• My crosstool build created the libthread_db.so file, and I put it on the target.

• My crosstool build created the gdb for my target, so it should have been linked against the same libraries that I run on the target.

• If I run gdb on my host, against my test app, I get a backtrace of each running thread.

I suspect the problem with Breakpad is related to the problem with GDB, but I cannot substantiate this. The only commonality is lack of multithreaded debug.

There is some crucial difference between my host and target that stops me from being able to debug pthreads on the target.

Does anyone know what it is?

EDIT:

Denys Dmytriyenko from TI says:

Normally, GDB is not very picky and you can mix and match different versions of gdb and gdbserver. But, unfortunately, if you need to debug multi-threaded apps, there are some dependencies for specific APIs...

For example, this is one of the messages you may see if you didn't build GDB properly for the thread support:

dlopen failed on 'libthread_db.so.1' - /lib/libthread_db.so.1: undefined symbol: ps_lgetfpregs GDB will not be able to debug pthreads.

Note that this error is the same as the one that I get but he doesn't go in to detail about how to build GDB "properly".

and the GDB FAQ says:

(Q) GDB does not see any threads besides the one in which crash occurred; or SIGTRAP kills my program when I set a breakpoint.

(A) This frequently happen on Linux, especially on embedded targets. There are two common causes:

• you are using glibc, and you have stripped libpthread.so.0

• mismatch between libpthread.so.0 and libthread_db.so.1

GDB itself does not know how to decode "thread control blocks" maintained by glibc and considered to be glibc private implementation detail. It uses libthread_db.so.1 (part of glibc) to help it do so. Therefore, libthread_db.so.1 and libpthread.so.0 must match in version and compilation flags. In addition, libthread_db.so.1 requires certain non-global symbols to be present in libpthread.so.0.

Solution: use strip --strip-debug libpthread.so.0 instead of strip libpthread.so.0.

I tried a non-stripped libpthread.so.0 but it didn't make a difference. I will investigate any mismatch between pthread and thread_db.

Answer 1

This:

dlopen failed on 'libthread_db.so.1' - /lib/libthread_db.so.1: undefined symbol: ps_lgetfpregs
GDB will not be able to debug pthreads.

means that the libthread_db.so.1 library was not able to find the symbol ps_lgetfpregs in gdb.

Why?

Because I built gdb using Crosstoolg-NG with the "Build a static native gdb" option and this adds the -static option to gcc.

The native gdb is built with the -rdynamic option and this populates the .dynsym symbol table in the ELF file with all symbols, even unused ones. libread_db uses this symbol table to find ps_lgetfpregs from gdb.

But -static strips the .dynsym table from the ELF file.

At this point there are two options:

1. Don't build a static native gdb if you want to debug threads.
2. Build a static gdb and a static libthread_db (not tested)

Edit:

By the way, this does not explain why Breakpad in unable to debug multithreaded applications on my target.