Kill Python Multiprocessing Pool

Question

I am running a Python program which uses the multiprocessing module to spawn some worker threads. Using Pool.map these digest a list of files.

At some point, I would like to stop everything and have the script die.

Normally Ctrl+C from the command line accomplishes this. But, in this instance, I think that just interrupts one of the workers and that a new worker is spawned.

So, I end up running ps aux | grep -i python and using kill -9 on the process ids in question.

Is there a better way to have the interrupt signal bring everything to a grinding halt?

Answer 1

SIGQUIT (Ctrl + \) will kill all processes even under Python 2.x.

You can also update to Python 3.x, where this behavior (only child gets the signal) seems to have been fixed.

Answer 2

Unfortunately in Python 2.x there really isn't a good solution for this problem. The best workaround that I know of is to use pool.map_async(...).get(timeout=<large number>) instead of pool.map. The problem is that pool.map makes a call to threading.Condition.wait(), which for some reason can't be interrupted by Ctrl+C in Python 2.x (it works in Python 3). When you use map_async(), it calls threading.Condition.wait(timeout=<large number>), which ends up doing a busy wait loop, which can be interrupted by Ctrl+C.

Try it for yourself:

c = threading.Condition() try:     c.acquire()     c.wait()  # You won't be able to interrupt this except KeyboardInterrupt:     print("Caught it")  c = threading.Condition() try:     c.acquire()     c.wait(timeout=100)  # You CAN interrupt this except KeyboardInterrupt:     print("Caught it") 

So, to make your map call interruptable, do this:

if __name__ == "__main__":     p = multiprocessing.Pool()     try:         p.map_async(func, iterable).get(timeout=10000000)     except KeyboardInterrupt:         print("Caught it")         # Optionally try to gracefully shut down the worker processes here.         p.close()         # DON'T join the pool. You'll end up hanging. 

Also note, as pointed out by phihag, this issue is fixed in Python 3.4 (and possibly earlier in 3.x).