Python: Getting a traceback from a multiprocessing.Process

Question

I am trying to get hold of a traceback object from a multiprocessing.Process. Unfortunately passing the exception info through a pipe does not work because traceback objects can not be pickled:

def foo(pipe_to_parent):     try:         raise Exception('xxx')     except:         pipe_to_parent.send(sys.exc_info())  to_child, to_self = multiprocessing.Pipe() process = multiprocessing.Process(target = foo, args = (to_self,)) process.start() exc_info = to_child.recv() process.join() print traceback.format_exception(*exc_info) to_child.close() to_self.close() 

Traceback:

Traceback (most recent call last):   File "/usr/lib/python2.6/multiprocessing/process.py", line 231, in _bootstrap     self.run()   File "/usr/lib/python2.6/multiprocessing/process.py", line 88, in run     self._target(*self._args, **self._kwargs)   File "foo", line 7, in foo     to_parent.send(sys.exc_info()) PicklingError: Can't pickle <type 'traceback'>: attribute lookup __builtin__.traceback failed 

Is there another way to access the exception info? I'd like to avoid passing the formatted string.

Answer 1

Using tblib you can pass wrapped exceptions and reraise them later:

import tblib.pickling_support tblib.pickling_support.install()  from multiprocessing import Pool import sys   class ExceptionWrapper(object):      def __init__(self, ee):         self.ee = ee         __, __, self.tb = sys.exc_info()      def re_raise(self):         raise self.ee.with_traceback(self.tb)         # for Python 2 replace the previous line by:         # raise self.ee, None, self.tb   # example of how to use ExceptionWrapper  def inverse(i):     """ will fail for i == 0 """     try:         return 1.0 / i     except Exception as e:         return ExceptionWrapper(e)   def main():     p = Pool(1)     results = p.map(inverse, [0, 1, 2, 3])     for result in results:         if isinstance(result, ExceptionWrapper):             result.re_raise()   if __name__ == "__main__":     main() 

So, if you catch an exception in your remote process, wrap it with ExceptionWrapper and then pass it back. Calling re_raise() in the main process will do the work.