Showing the stack trace from a running Python application

Question

I have this Python application that gets stuck from time to time and I can't find out where.

Is there any way to signal Python interpreter to show you the exact code that's running?

Some kind of on-the-fly stacktrace?

Related questions:

• Print current call stack from a method in Python code
• Check what a running process is doing: print stack trace of an uninstrumented Python program

Answer 1

I have module I use for situations like this - where a process will be running for a long time but gets stuck sometimes for unknown and irreproducible reasons. Its a bit hacky, and only works on unix (requires signals):

import code, traceback, signal  def debug(sig, frame):     """Interrupt running process, and provide a python prompt for     interactive debugging."""     d={'_frame':frame}         # Allow access to frame object.     d.update(frame.f_globals)  # Unless shadowed by global     d.update(frame.f_locals)      i = code.InteractiveConsole(d)     message  = "Signal received : entering python shell.\nTraceback:\n"     message += ''.join(traceback.format_stack(frame))     i.interact(message)  def listen():     signal.signal(signal.SIGUSR1, debug)  # Register handler 

To use, just call the listen() function at some point when your program starts up (You could even stick it in site.py to have all python programs use it), and let it run. At any point, send the process a SIGUSR1 signal, using kill, or in python:

    os.kill(pid, signal.SIGUSR1) 

This will cause the program to break to a python console at the point it is currently at, showing you the stack trace, and letting you manipulate the variables. Use control-d (EOF) to continue running (though note that you will probably interrupt any I/O etc at the point you signal, so it isn't fully non-intrusive.

I've another script that does the same thing, except it communicates with the running process through a pipe (to allow for debugging backgrounded processes etc). Its a bit large to post here, but I've added it as a python cookbook recipe.

Answer 2

The suggestion to install a signal handler is a good one, and I use it a lot. For example, bzr by default installs a SIGQUIT handler that invokes pdb.set_trace() to immediately drop you into a pdb prompt. (See the bzrlib.breakin module's source for the exact details.) With pdb you can not only get the current stack trace (with the (w)here command) but also inspect variables, etc.

However, sometimes I need to debug a process that I didn't have the foresight to install the signal handler in. On linux, you can attach gdb to the process and get a python stack trace with some gdb macros. Put http://svn.python.org/projects/python/trunk/Misc/gdbinit in ~/.gdbinit, then:

• Attach gdb: gdb -p PID
• Get the python stack trace: pystack

It's not totally reliable unfortunately, but it works most of the time.

Finally, attaching strace can often give you a good idea what a process is doing.