Python 2.7.3 Multiprocessing Pool hanging

Question

I have the following code in ipython where the child process tries to do sys.exit(...) but causes the parent process to hang. Is this a bug? Any idea how to workaround this?

In [1]: from multiprocessing import Pool

In [2]: def f():
   ...:     import sys
   ...:     sys.exit('exiting system...')
   ...:    

In [3]: p = Pool(processes=2)

In [4]: r = p.apply_async(f, [])

In [5]: r.get()   <---- it is hanging here forever.

I have also tried to put raise SystemExit(...) instead of sys.exit(...) but it was the same thing. The only workaround I know is to put raise Exception(...) which turned out to work just fine.

I understand that sys.exit is essentially the same as raise SystemExit, but this exception should be delegated up to its parent process and thus r.get() should be able to receive this exception correct? But it seems to be getting stuck on recv call. Is this a bug in multiprocessing module?

Answer 1

You're causing the Pool worker process to actually exit when you call sys.exit(). The SystemExit exception is special-cased; when you raise it, the process you raised it in exits. The exception doesn't get propagated to the caller. So, that means that in your example, the worker process simply never returns anything back to the parent. And then the parent process will just wait around forever for the child to return something that's never going to get returned. See this question for a more in-depth discussion of that behavior.

I would argue that this is a bug, and that when the sub-process exits, the pool should be marked as broken, and all outstanding tasks should be aborted. This is how concurrent.futures.ProcessPoolExecutor behaves already. I've actually submitted a patch that adds this behavior to multiprocessing.Pool, but so far it hasn't been reviewed.

Now, back to your original question. It looks like you want the parent process to exit here, not just the child. To do that, you'll need to actually return some object back to the parent, and then have the parent exit when it receives that object. It looks like you've discovered you can do this by simply raising an Exception.