Scripting LLDB to obtain a stack trace after a crash

Question

I'm trying to add the capability of generating a stack trace from a core dump on mac automatically when one of our tests crashes.

I was able to do it pretty easily on linux with

gdb --batch --quiet -ex "thread apply all bt" -ex "quit" <binary> <core file> 2> /dev/null

But I'm having some trouble doing the same on mac (OSX 10.8) with lldb. To start off, the version of lldb I'm using is lldb-310.2.37.

My initial approach was to use the -s option and pass in a script file like so:

target create -c <core file> <binary>
thread backtrace all
quit

Initially I had some trouble which I think was caused by missing a newline at the end of the script file which caused lldb to not exit, but after that was fixed, I'm getting the following: Executing commands in 'lldbSource'.

(lldb)  target create -c <core file> <binary>
Core file '<core file>' (x86_64) was loaded.
(lldb)  thread backtrace all
error: Aborting reading of commands after command #1: 'thread backtrace all' failed with error: invalid thread
Aborting after_file command execution, command file: 'lldbSource' failed.

The funny thing is, after that happens, we're still running lldb, and issuing 'thread backtrace all' manually works just fine.

So, approach #2 was to create a python script and use their python API (I tried this before figuring out the initial blocker I described was due to a missing newline).

My script:

import lldb
debugger = lldb.SBDebugger.Create()
target = debugger.CreateTarget('<binary>')
if target:
 process = target.LoadCore('<core file>')
 if process:
  print process.exit_description
  for thread in process:
   print 'Thread %s:' % str(thread.GetThreadID())
   print '\n'.join(str(frame) for frame in thread)

The problem I'm having with this approach is that process.exit_description is returning None (and so is every other thing I've tried; LLDB's python API documentation is almost completely useless).

The output I'm looking for from that call is something similar to the following:

Process 0 stopped
* thread #1: tid = 0x0000, 0x00007fff8aca4670 libsystem_c.dylib`strlen + 16, stop reason = signal SIGSTOP
    frame #0: 0x00007fff8aca4670 libsystem_c.dylib`strlen + 16
libsystem_c.dylib`strlen + 16:
-> 0x7fff8aca4670:  pcmpeqb (%rdi), %xmm0
   0x7fff8aca4674:  andl   $0xf, %ecx
   0x7fff8aca4677:  shll   %cl, %eax
   0x7fff8aca4679:  pmovmskb %xmm0, %ecx

This is output automatically by LLDB proper when loading a core file. I don't necessarily need the assembly dump, but I want at least the thread, frame and reason.

I think the first approach I used, if it could be made to work, would be ideal, but either way is OK for me. I don't have control over the LLDB version that's going to be used unfortunately, so I can't just update to latest and see if it's a bug that was fixed.

Other approaches to get the desired output are also welcome. For context, this is going to be called from a perl script.

Answer 1

The --batch command line argument is supported in the version of lldb that ships with Xcode 7.2 (lldb-340.4.119), and possibly earlier.

It's not documented in the man page, but it is documented in lldb --help:

   -b 
   --batch 
        Tells the debugger to running the commands from -s, -S, -o & -O,
        and then quit.  However if any run command stopped due to a signal
        or crash, the debugger will return to the interactive prompt at the
        place of the crash.

These other command line options are useful for lldb automation:

   -o 
   --one-line 
        Tells the debugger to execute this one-line lldb command after any
        file provided on the command line has been loaded.

   -k 
   --one-line-on-crash 
        When in batch mode, tells the debugger to execute this one-line
        lldb command if the target crashes.

Using these facilities, I pieced together the following command:

ulimit -c unlimited && (<binary> || (lldb -c `ls -t /cores/* | head -n1` \
  --batch -o 'thread backtrace all' -o 'quit' && exit 1))

This command:

1. Enables core dumps
2. Runs the executable <binary>
3. If it fails, runs lldb on the most recently created core dump in /cores (hopefully the right one)
4. Prints the backtraces of all threads
5. Exits with a non-zero status, so that this command can be embedded in other workflows (CI scripts, Makefiles, etc.)

I would have preferred to run lldb on <binary> directly, so that the command does not rely on guessing at the correct core file. But lldb still appears to lack a way to cause it to exit with a non-zero exit status -- an equivalent of GDB's -return-child-result option or quit 1 command -- so I would have no way of knowing if the debugged program was successful or not. I filed an issue requesting such a feature.

Answer 2

TOT lldb from lldb.llvm.org has a new "--batch" mode to work pretty much like the gdb batch mode, and fixes some bugs that made command-line source commands behave better. If you can build your own lldb, you can get these fixes now, otherwise you'll have to wait till the next Xcode update.

The exit_description is None because your process didn't exit, it crashed. Note that at least on OS X several threads can have had simultaneous exceptions by the time the process gets around to crashing, it isn't really useful to say the process crashed. You have to ask the threads. The stop status that lldb prints out when a thread stops is available with the GetStatus method.

stream = lldb.SBStream()
process.threads[0].GetStatus(stream)
print stream.GetData()

That doesn't seem to have a very useful help string.