What is the best way to get a stacktrace when using multiprocessing?

Question

I'm wondering about the best way to get a stacktrace when there is an exception inside a function executed via the multiprocessing module. Here's an example:

import multiprocessing

def square(x):
    raise Exception("Crash.")
    return x**2

if __name__ == '__main__':
    pool = multiprocessing.Pool(processes=4)
    results = pool.map_async(square, range(5))
    for result in results.get():
        print result

This prints:

Traceback (most recent call last):
  File "/extra/workspace/Playground/src/multiproc/multiproc_debug.py", line 11, in <module>
    for result in results.get():
  File "/extra/Python 2.6/lib/python2.6/multiprocessing/pool.py", line 422, in get
    raise self._value
Exception: Crash.

So there is no useful stacktrace, which is quite annoying. My current solution for this:

import multiprocessing
import traceback

def square(x):
    try:
        # some more code...
        raise Exception("Crash.")
    except Exception, exception:
        print exception
        traceback.print_exc()
        raise
    return x**2

Is there a way to get this behaviour without all the boilerplate code? If not, what is the reason for not including this feature?

Edit: One could use a decorator for the boilerplate code, but I don't know if such a decorator is included in the standard library?

Answer 1

It looks like you should avoid raising the exception from your main function. Instead, you can catch it, treat it as a value returned to the main program, then raise it there. Re-throwing exceptions in Python has more details.

Answer 2

In Python 3.4, full traceback is provided.

http://bugs.python.org/issue13831

Answer 3

Python 2

I've made a decorator implementation as below. Note the usage of functools.wraps, otherwise the multiprocessing would fail.

def full_traceback(func):
    import traceback, functools
    @functools.wraps(func)
    def wrapper(*args, **kwargs):
        try:
            return func(*args, **kwargs)
        except Exception as e:
            msg = "{}\n\nOriginal {}".format(e, traceback.format_exc())
            raise type(e)(msg)
    return wrapper

An example can be found in https://stackoverflow.com/a/43223455.

Python 3

As metioned by Paige Lo,now the get method of multiprocessing.pool.Async returns full traceback in Python 3, see http://bugs.python.org/issue13831.