Core dump of multithreaded application shows only one thread

Question

I have a test application in c++ starting several threads in its main() and then sleeping in main() forever.

One of the threads is doing something that causes a segfault and a coredump is generated (ulimit -c unlimited was set previously).

I'm opening the core with gdb and see with thread apply all bt or info threads that I have only one thread (started in main()), which is impossible because at least the main() thread should be running as well.

The question is how is it possible the rest of the threads to be missing and what could cause it?

The backtrace of this lonely thread seems ok, no strange stuff in it.

The OS is Red Hat Enterprise 5.3, gdb-6.8.

Answer 1

It turned out to be kernel bug in default Red Hat Enterprise 5.3, fixed in later Red Hat version (5.4) - kernel-2.6.18-164.el5

http://docs.redhat.com/docs/en-US/Red_Hat_Enterprise_Linux/5/html-single/5.4_Technical_Notes/index.html

1.110.1. RHSA-2009:1193: Important security and bug fix update on 32-bit systems, core dumps for some multithreaded applications did not include all thread information. (BZ#505322)

https://bugzilla.redhat.com/show_bug.cgi?id=505322

Answer 2

The reason why you see only one thread is that GDB is not able to distinguish threads "by itself", it relies on an external library, the libthread_db, provided by the thread library.

This library must be enabled at beginning of the debugging session in order to monitor the thread activities (birth, death, ...) and communicates all the thread-related information to GDB during runtime.

You should be able to read

[Thread debugging using libthread_db enabled]

when you try to debug any file compiled with -lpthread, but GDB doesn't even try to enable libthread_db when you debug a core dump.