Is there a way to dump stack trace with line number from a Linux release binary?

Question

I have a requirement of dumping stack traces when my c++ Linux application crashes. I was successfully able to do this using backtrace() and backtrace_symbols(). Now, additionally I would like to get the line numbers of the crash. How do it do it?

Answer 1

system() needs #include <stdlib.h>

Few other things seem missing.

$ g++-8 -g -o dump dump.cpp
dump.cpp: In function ‘void bt_sighandler(int, sigcontext)’:
dump.cpp:15:43: error: ‘struct sigcontext’ has no member named
‘eip’; did > you mean ‘rip’?
         "from %p\n", sig, ctx.cr2, ctx.eip);
                                        ^~~
                                        rip
dump.cpp:21:26: error: ‘struct sigcontext’ has no member
named ‘eip’;
did you mean ‘rip’?
  trace[1] = (void *)ctx.eip;
                         ^~~
                         rip
dump.cpp: In function ‘int main()’:
dump.cpp:64:19: error: invalid conversion from ‘void*’ to
   ‘__sighandler_t’ > {aka ‘void (*)(int)’} [-fpermissive]
   sa.sa_handler = (void *)bt_sighandler;
                   ^~~~~~~~~~~~~~~~~~~~~

Answer 2

I took help from

http://www.linuxjournal.com/files/linuxjournal.com/linuxjournal/articles/063/6391/6391l2.html and http://www.linuxjournal.com/article/6391?page=0,0 to come up with sample code which shows how you can achieve this.

Basically it is about putting a stack backtrace inside a signal handler and having the latter catch all the "bad" signals your program can receive (SIGSEGV, SIGBUS, SIGILL, SIGFPE and the like). This way, if your program unfortunately crashes and you were not running it with a debugger, you can get a stack trace and know where the fault happened. This technique also can be used to understand where your program is looping in case it stops responding...

Below code runs the external program addr2line for every address in the trace to convert it into a file name and a line number.

The source code below prints line numbers for all local functions. If a function from another library is called, you might see a couple of ??:0 instead of file names.

#include <stdio.h>
#include <signal.h>
#include <execinfo.h>

void bt_sighandler(int sig, struct sigcontext ctx) {

  void *trace[16];
  char **messages = (char **)NULL;
  int i, trace_size = 0;

  if (sig == SIGSEGV)
    printf("Got signal %d, faulty address is %p, "
           "from %p\n", sig, ctx.cr2, ctx.eip);
  else
    printf("Got signal %d\n", sig);

  trace_size = backtrace(trace, 16);
  /* overwrite sigaction with caller's address */
  trace[1] = (void *)ctx.eip;
  messages = backtrace_symbols(trace, trace_size);
  /* skip first stack frame (points here) */
  printf("[bt] Execution path:\n");
  for (i=1; i<trace_size; ++i)
  {
    printf("[bt] #%d %s\n", i, messages[i]);

    char syscom[256];
    sprintf(syscom,"addr2line %p -e sighandler", trace[i]); //last parameter is the name of this app
    system(syscom);
  }

  exit(0);
}


int func_a(int a, char b) {

  char *p = (char *)0xdeadbeef;

  a = a + b;
  *p = 10;  /* CRASH here!! */

  return 2*a;
}


int func_b() {

  int res, a = 5;

  res = 5 + func_a(a, 't');

  return res;
}


int main() {

  /* Install our signal handler */
  struct sigaction sa;

  sa.sa_handler = (void *)bt_sighandler;
  sigemptyset(&sa.sa_mask);
  sa.sa_flags = SA_RESTART;

  sigaction(SIGSEGV, &sa, NULL);
  sigaction(SIGUSR1, &sa, NULL);
  /* ... add any other signal here */

  /* Do something */
  printf("%d\n", func_b());
}

This code should be compiled as: gcc sighandler.c -o sighandler -rdynamic

The program outputs:

Got signal 11, faulty address is 0xdeadbeef, from 0x8048975
[bt] Execution path:
[bt] #1 ./sighandler(func_a+0x1d) [0x8048975]
/home/karl/workspace/stacktrace/sighandler.c:44
[bt] #2 ./sighandler(func_b+0x20) [0x804899f]
/home/karl/workspace/stacktrace/sighandler.c:54
[bt] #3 ./sighandler(main+0x6c) [0x8048a16]
/home/karl/workspace/stacktrace/sighandler.c:74
[bt] #4 /lib/tls/i686/cmov/libc.so.6(__libc_start_main+0xe6) [0x3fdbd6]
??:0
[bt] #5 ./sighandler() [0x8048781]
??:0

Answer 3

It is only possible if the program has been compiled with debugging information (i.e. with gcc -Wall -g or with g++ -Wall -g). Without -g the executable does not contain any source line information. And if using gcc you can compile with both optimization & debugging information (e.g. g++ -Wall -g -O2) but sometimes the line location would be "surprising".

^{The -Wall flag asks GCC to show all warnings. It is very useful (hence my recommendation to use it) but unrelated to -g or debugging information.}

As to how to extract the line number, the simplest way would be to fork a gdb process. Alternatively, you could get the debugging information (in DWARF format) and parse it, perhaps using libdwarf from the ELF tool chain. I am not sure it is worth the trouble...

To just get the backtrace, you might simply run your program thru gdb perhaps as gdb --args yourprogram itsarguments...

---

addenda

you could also use the libbacktrace from inside a recent GCC (actually it is Ian Taylor's libbacktrace), which is designed to solve your problem (it is "interpreting" the DWARF format of the current executable, which you would compile with g++ -O -g).