Post mortem debugging of the multi-thread scripts

Question

I am trying to debugging multi-thread script. Once the exception is raised I want to:

1. report it to monitoring system (just print in following example)
2. stop whole script (including all other threads)
3. call post mortem debugger prompt in a perspective raised exception

I prepare pretty complicated example to show how I tried to solve it:

#!/usr/bin/env python

import threading
import inspect
import traceback
import sys
import os
import time


def POST_PORTEM_DEBUGGER(type, value, tb):
    traceback.print_exception(type, value, tb)
    print
    if hasattr(sys, 'ps1') or not sys.stderr.isatty():
        import rpdb
        rpdb.pdb.pm()
    else:
        import pdb
        pdb.pm()

sys.excepthook = POST_PORTEM_DEBUGGER



class MyThread(threading.Thread):

    def __init__(self):

        threading.Thread.__init__(self)
        self.exception = None
        self.info = None
        self.the_calling_script_name = os.path.abspath(inspect.currentframe().f_back.f_code.co_filename)

    def main(self):
        "Virtual method to be implemented by inherited worker"
        return self

    def run(self):
        try:
            self.main()
        except Exception as exception:
            self.exception = exception
            self.info = traceback.extract_tb(sys.exc_info()[2])[-1]
            # because of bug http://bugs.python.org/issue1230540
            # I cannot use just "raise" under threading.Thread
            sys.excepthook(*sys.exc_info())

    def __del__(self):
        print 'MyThread via {} catch "{}: {}" in {}() from {}:{}: {}'.format(self.the_calling_script_name, type(self.exception).__name__, str(self.exception), self.info[2], os.path.basename(self.info[0]), self.info[1], self.info[3])




class Worker(MyThread):

    def __init__(self):
        super(Worker, self).__init__()

    def main(self):
        """ worker job """
        counter = 0
        while True:
            counter += 1
            print self
            time.sleep(1.0)
            if counter == 3:
                pass # print 1/0


def main():

    Worker().start()

    counter = 1
    while True:
        counter += 1
        time.sleep(1.0)
        if counter == 3:
            pass # print 1/0

if __name__ == '__main__':
    main()

The trick with

sys.excepthook = POST_PORTEM_DEBUGGER

works perfectly if no threads are involved. I found that in case of multi-thread script I can use rpdb for debuggig by calling:

import rpdb; rpdb.set_trace()

It works perfectly for defined breakpoint but I want to debug multi-thread script post mortem (after the uncatched exception is raised). When I try to use rpdb in the POST_PORTEM_DEBUGGER function with multi-thread application I get following:

Exception in thread Thread-1:
Traceback (most recent call last):
    File "/usr/lib/python2.7/threading.py", line 552, in __bootstrap_inner
        self.run()
    File "./demo.py", line 49, in run
        sys.excepthook(*sys.exc_info())
    File "./demo.py", line 22, in POST_PORTEM_DEBUGGER
        pdb.pm()
    File "/usr/lib/python2.7/pdb.py", line 1270, in pm
        post_mortem(sys.last_traceback)
AttributeError: 'module' object has no attribute 'last_traceback'

I looks like the

sys.excepthook(*sys.exc_info())

did not set up all what the raise command does. I want the same behavior if the exception is raised in main() even under started thread.

Answer 1

(I haven't tested my answer, but it seems to me that...)

The call to pdb.pm (pm="post mortem") fails simply because there had been no "mortem" prior to it. I.e. the program is still running.

Looking at the pdb source code, you find the implementation of pdb.pm:

def pm():
    post_mortem(sys.last_traceback)

which makes me guess that what you actually want to do is call pdb.post_mortem() with no args. Looks like the default behavior does exactly what you need.

Some more source code (notice the t = sys.exc_info()[2] line):

def post_mortem(t=None):
    # handling the default
    if t is None:
        # sys.exc_info() returns (type, value, traceback) if an exception is
        # being handled, otherwise it returns None
        t = sys.exc_info()[2]
        if t is None:
            raise ValueError("A valid traceback must be passed if no "
                                               "exception is being handled")

    p = Pdb()
    p.reset()
    p.interaction(None, t)