Determining the correct thread to debug in GDB

Question

I've run into some problems debugging a multi-threaded process using GDB. I have a multi-threaded process that splinters off into several (8 or 9) different threads, and I am trying to determine what the contents of variables are when the constructor for a class called XML_File_Data is called. However, I've run into a problem where, after I apply the correct function breakpoint to all threads and it's apparent one of the thread's break point is getting hit (the program temporarily halts execution), I'm not able to determine which thread hit the breakpoint. The command

(gdb) thread apply all where

is giving me shockingly useless information in the form:

#0  0x004ab410 in __kernel_vsyscall ()
#1  0x05268996 in nanosleep () from /lib/libc.so.6
#2  0x052a215c in usleep () from /lib/libc.so.6
#3  0x082ee313 in frame_clock_frame_end (clock=0xb4bfd2f8)
    at frame_clock.c:143
#4  0x003a349a in ?? ()
#5  0x00b5cfde in thread_proxy ()
   from /cets_development_libraries/install/lib/libboost_thread-gcc41-mt-1_38.so.1.38.0
#6  0x02c1f5ab in start_thread () from /lib/libpthread.so.0
#7  0x052a8cfe in clone () from /lib/libc.so.6

Of the 9 processes, 7 or so are giving me almost exactly that output, and the information about the last 2 isn't really much more helpful (functions far down the call stack have recognizable names, but any recent #0-#4 functions aren't recognizable).

This is what I have so far:

(gdb) gdb
(gdb) gdb attach <processid>
(gdb) thread apply all 'XML_File_Data::XML_File_Data()'

and (after the breakpoint is hit)

(gdb) thread apply all where

Could any experienced debuggers offer me some hints on what I am doing wrong or what is normally done in this situation?

Cheers, Charlie

EDIT: Fortunately, I was able to find out that the cause of the ??'s was optimized code being run through the debugger, in addition to not running the debugger in the directory of the executable file. Still not much success with the debugging though.

Answer 1

I find myself doing this all the time:

> t a a f

Short for:

> thread apply all frame

Of course, other variants are possible:

> t a a bt 3

Which prints the bottom 3 frames of each thread's stack. (You can also use negative numbers to get the top N frames of the stack)

Answer 2

You can use command thread or info threads to find out the current thread number after breakpoint hit

(gdb) thread
[Current thread is 1 (Thread 0xb790d6c0 (LWP 2519))]
(gdb)

(gdb) info threads
  17 Thread 0xb789cb90 (LWP 2536)  0xb7fc6402 in __kernel_vsyscall ()
  16 Thread 0xb769bb90 (LWP 2537)  0xb7fc6402 in __kernel_vsyscall ()
  15 Thread 0xb749ab90 (LWP 2543)  0xb7fc6402 in __kernel_vsyscall ()
  14 Thread 0xb7282b90 (LWP 2544)  0xb7fc6402 in __kernel_vsyscall ()
  13 Thread 0xb5827b90 (LWP 2707)  0xb7fc6402 in __kernel_vsyscall ()
  12 Thread 0xb5626b90 (LWP 2708)  0xb7fc6402 in __kernel_vsyscall ()
  11 Thread 0xb5425b90 (LWP 2709)  0xb7fc6402 in __kernel_vsyscall ()
  10 Thread 0xb5161b90 (LWP 2713)  0xb7fc6402 in __kernel_vsyscall ()
  9 Thread 0xb4ef9b90 (LWP 2715)  0xb7fc6402 in __kernel_vsyscall ()
  8 Thread 0xb4af7b90 (LWP 2717)  0xb7fc6402 in __kernel_vsyscall ()
  7 Thread 0xb46ffb90 (LWP 2718)  0xb7fc6402 in __kernel_vsyscall ()
  6 Thread 0xb44feb90 (LWP 2726)  0xb7fc6402 in __kernel_vsyscall ()
  5 Thread 0xb42fdb90 (LWP 2847)  0xb7fc6402 in __kernel_vsyscall ()
  4 Thread 0xb40fcb90 (LWP 2848)  0xb7fc6402 in __kernel_vsyscall ()
  3 Thread 0xb3efbb90 (LWP 2849)  0xb7fc6402 in __kernel_vsyscall ()
  2 Thread 0xb3cfab90 (LWP 2850)  0xb7fc6402 in __kernel_vsyscall ()
* 1 Thread 0xb790d6c0 (LWP 2519)  0xb7fc6402 in __kernel_vsyscall ()
(gdb)

An asterisk `*' to the left of the gdb thread number indicates the current thread. See here.