How can I find a global variable's destructor call using gdb?

Question

We Have very a big project.(about 200'000 files)

When the project exits, it crashes in a global variable's deconstruct.

But we can't find where the variable defined.

Does anybody known how to find where the global variable is?

Answer 1

But we can't find where the variable defined.

Presumably, you want to find out which global variable is causing trouble. If you knew which one it is, you could use grep, or GDB info variable foobar to find it.

Let's consider an example.

cat foo.cc
#include <stdio.h>
#include <string>

using std::string;

string foo("foo");
string bar("bar");
string baz("baz");

// It is not valid to call this function with a global string.
void invalid_with_global(string *str)
{
  printf("str: %s @%p)\n", str->c_str(), str);
  printf("before dtor\n");
  str->~string();
  printf("after dtor\n");
}

int main()
{
  invalid_with_global(&baz);
}

g++ -g foo.cc && gdb -q ./a.out

Reading symbols from /tmp/a.out...done.
(gdb) r
Starting program: /tmp/a.out
str: baz @0x601088)
before dtor
after dtor
*** glibc detected *** /tmp/a.out: double free or corruption (fasttop): 0x0000000000602070 ***
======= Backtrace: =========
/lib/x86_64-linux-gnu/libc.so.6(+0x7eb96)[0x7ffff7583b96]
/usr/lib/x86_64-linux-gnu/libstdc++.so.6(_ZNSsD1Ev+0x23)[0x7ffff7b78c13]
/lib/x86_64-linux-gnu/libc.so.6(+0x3b901)[0x7ffff7540901]
/lib/x86_64-linux-gnu/libc.so.6(+0x3b985)[0x7ffff7540985]
/lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xf4)[0x7ffff7526774]
/tmp/a.out[0x4007f9]
======= Memory map: ========
00400000-00401000 r-xp 00000000 ca:01 147024                             /tmp/a.out
00600000-00601000 r--p 00000000 ca:01 147024                             /tmp/a.out
00601000-00602000 rw-p 00001000 ca:01 147024                             /tmp/a.out
00602000-00623000 rw-p 00000000 00:00 0                                  [heap]
7ffff7209000-7ffff7304000 r-xp 00000000 ca:01 881564                     /lib/x86_64-linux-gnu/libm-2.15.so
7ffff7304000-7ffff7503000 ---p 000fb000 ca:01 881564                     /lib/x86_64-linux-gnu/libm-2.15.so
7ffff7503000-7ffff7504000 r--p 000fa000 ca:01 881564                     /lib/x86_64-linux-gnu/libm-2.15.so
7ffff7504000-7ffff7505000 rw-p 000fb000 ca:01 881564                     /lib/x86_64-linux-gnu/libm-2.15.so
7ffff7505000-7ffff76ba000 r-xp 00000000 ca:01 881572                     /lib/x86_64-linux-gnu/libc-2.15.so
7ffff76ba000-7ffff78b9000 ---p 001b5000 ca:01 881572                     /lib/x86_64-linux-gnu/libc-2.15.so
7ffff78b9000-7ffff78bd000 r--p 001b4000 ca:01 881572                     /lib/x86_64-linux-gnu/libc-2.15.so
7ffff78bd000-7ffff78bf000 rw-p 001b8000 ca:01 881572                     /lib/x86_64-linux-gnu/libc-2.15.so
7ffff78bf000-7ffff78c4000 rw-p 00000000 00:00 0
7ffff78c4000-7ffff78d9000 r-xp 00000000 ca:01 881501                     /lib/x86_64-linux-gnu/libgcc_s.so.1
7ffff78d9000-7ffff7ad8000 ---p 00015000 ca:01 881501                     /lib/x86_64-linux-gnu/libgcc_s.so.1
7ffff7ad8000-7ffff7ad9000 r--p 00014000 ca:01 881501                     /lib/x86_64-linux-gnu/libgcc_s.so.1
7ffff7ad9000-7ffff7ada000 rw-p 00015000 ca:01 881501                     /lib/x86_64-linux-gnu/libgcc_s.so.1
7ffff7ada000-7ffff7bbc000 r-xp 00000000 ca:01 1096828                    /usr/lib/x86_64-linux-gnu/libstdc++.so.6.0.16
7ffff7bbc000-7ffff7dbb000 ---p 000e2000 ca:01 1096828                    /usr/lib/x86_64-linux-gnu/libstdc++.so.6.0.16
7ffff7dbb000-7ffff7dc3000 r--p 000e1000 ca:01 1096828                    /usr/lib/x86_64-linux-gnu/libstdc++.so.6.0.16
7ffff7dc3000-7ffff7dc5000 rw-p 000e9000 ca:01 1096828                    /usr/lib/x86_64-linux-gnu/libstdc++.so.6.0.16
7ffff7dc5000-7ffff7dda000 rw-p 00000000 00:00 0
7ffff7dda000-7ffff7dfc000 r-xp 00000000 ca:01 881673                     /lib/x86_64-linux-gnu/ld-2.15.so
7ffff7fd8000-7ffff7fdd000 rw-p 00000000 00:00 0
7ffff7ff7000-7ffff7ffb000 rw-p 00000000 00:00 0
7ffff7ffb000-7ffff7ffc000 r-xp 00000000 00:00 0                          [vdso]
7ffff7ffc000-7ffff7ffd000 r--p 00022000 ca:01 881673                     /lib/x86_64-linux-gnu/ld-2.15.so
7ffff7ffd000-7ffff7fff000 rw-p 00023000 ca:01 881673                     /lib/x86_64-linux-gnu/ld-2.15.so
7ffffffdd000-7ffffffff000 rw-p 00000000 00:00 0                          [stack]
ffffffffff600000-ffffffffff601000 r-xp 00000000 00:00 0                  [vsyscall]

Program received signal SIGABRT, Aborted.
0x00007ffff753b425 in __GI_raise (sig=<optimized out>) at ../nptl/sysdeps/unix/sysv/linux/raise.c:64
64  ../nptl/sysdeps/unix/sysv/linux/raise.c: No such file or directory.
(gdb) bt
#0  0x00007ffff753b425 in __GI_raise (sig=<optimized out>) at ../nptl/sysdeps/unix/sysv/linux/raise.c:64
#1  0x00007ffff753eb8b in __GI_abort () at abort.c:91
#2  0x00007ffff757939e in __libc_message (do_abort=2, fmt=0x7ffff7683008 "*** glibc detected *** %s: %s: 0x%s ***\n") at ../sysdeps/unix/sysv/linux/libc_fatal.c:201
#3  0x00007ffff7583b96 in malloc_printerr (action=3, str=0x7ffff76831f8 "double free or corruption (fasttop)", ptr=<optimized out>) at malloc.c:5007
#4  0x00007ffff7b78c13 in std::basic_string<char, std::char_traits<char>, std::allocator<char> >::~basic_string() () from /usr/lib/x86_64-linux-gnu/libstdc++.so.6
#5  0x00007ffff7540901 in __run_exit_handlers (status=0, listp=0x7ffff78bd688 <__exit_funcs>, run_list_atexit=true) at exit.c:78
#6  0x00007ffff7540985 in __GI_exit (status=<optimized out>) at exit.c:100
#7  0x00007ffff7526774 in __libc_start_main (main=0x400904 <main()>, argc=1, ubp_av=0x7fffffffd828, init=<optimized out>, fini=<optimized out>, rtld_fini=<optimized out>, stack_end=0x7fffffffd818) at libc-start.c:258
#8  0x00000000004007f9 in _start ()

At this point, you know that one of your global 'std::basic_string<...>' objects is causing trouble, but you don't know which one.

It would be trivial to find out: just go to frame #4 and print this, right?

(gdb) fr 4
#4  0x00007ffff7b78c13 in std::basic_string<char, std::char_traits<char>, std::allocator<char> >::~basic_string() () from /usr/lib/x86_64-linux-gnu/libstdc++.so.6
(gdb) p this
No symbol "this" in current context.

Hmm, that didn't work so well, because I don't have debug symbols for libstdc++.so.6 installed (if the object that is causing you trouble is defined in your own sources, you shouldn't have that trouble).

Now, I can either install debug symbols for my libstdc++.so.6, or I can use my knowledge of the ABI. On Linux/x86_64, the first (this) parameter is passed in register %rdi. I can't use the current %rdi though, as it has already been used for something else. So instead I set a breakpoint on problem destructor: std::basic_string<char, std::char_traits<char>, std::allocator<char> >::~basic_string(), and restart.

In a real program, the destructor would be called a lot, but I'd only be interested in the call from exit. So I set the breakpoint on exit, and only when that breakpoint is hit, I set a new breakpoint on the destructor.

(gdb) b exit
(gdb) r
Starting program: /tmp/a.out
str: baz @0x601088)
before dtor
after dtor

Breakpoint 1, __GI_exit (status=0) at exit.c:100
100 exit.c: No such file or directory.
(gdb) b std::basic_string<char, std::char_traits<char>, std::allocator<char> >::~basic_string()
Breakpoint 2 at 0x7ffff7b78bf0
(gdb) c
Continuing.

Breakpoint 2, 0x00007ffff7b78bf0 in std::basic_string<char, std::char_traits<char>, std::allocator<char> >::~basic_string() () from /usr/lib/x86_64-linux-gnu/libstdc++.so.6
(gdb) bt
#0  0x00007ffff7b78bf0 in std::basic_string<char, std::char_traits<char>, std::allocator<char> >::~basic_string() () from /usr/lib/x86_64-linux-gnu/libstdc++.so.6
#1  0x00007ffff7540901 in __run_exit_handlers (status=0, listp=0x7ffff78bd688 <__exit_funcs>, run_list_atexit=true) at exit.c:78
#2  0x00007ffff7540985 in __GI_exit (status=<optimized out>) at exit.c:100
#3  0x00007ffff7526774 in __libc_start_main (main=0x400904 <main()>, argc=1, ubp_av=0x7fffffffd828, init=<optimized out>, fini=<optimized out>, rtld_fini=<optimized out>, stack_end=0x7fffffffd818) at libc-start.c:258
#4  0x00000000004007f9 in _start ()
(gdb) p/x $rdi
$1 = 0x601088
(gdb) c
Continuing.
*** glibc detected *** /tmp/a.out: double free or corruption (fasttop): 0x0000000000602070 ***
... as before, omitted ...


Program received signal SIGABRT, Aborted.
0x00007ffff753b425 in __GI_raise (sig=<optimized out>) at ../nptl/sysdeps/unix/sysv/linux/raise.c:64
64  ../nptl/sysdeps/unix/sysv/linux/raise.c: No such file or directory.

So now we have the address of "bad" global: 0x601088, which you'll note matches the value I printed.

Finally we are ready to answer the "which variable is it?" question:

(gdb) info sym 0x601088
baz in section .bss of /tmp/a.out

Voilà: the problem variable is baz. Finally:

(gdb) info var baz
All variables matching regular expression "baz":

File foo.cc:
std::string baz;

Answer 2

You could use nm -C on each object file (and grep its result to the destructor you are searching).

Perhaps the ptype command of gdb might help.

And you might even consider customizing your GCC e.g. with MELT (a domain specific language to extend GCC) for navigation or refactoring purposes (i.e. make a GCC extension in MELT to help you). For a big project like yours customizing the compiler (and learning MELT) is worth the effort (at least to help the development team, e.g. by customizing GCC with MELT for your coding rules, etc...)