Python multiprocessing Pool hangs on ubuntu server

Question

I'm running Django on an Ubuntu server with nginx and gunicorn. I'm trying to do some multiprocessing which is working on my local machine but hangs until the gunicorn worker times out on my server.

cpu_count = int(multiprocessing.cpu_count())
pool = Pool(processes = cpu_count)
result = pool.map_async(apiSimulAvail, rate_ranges)
result.wait()

...do some more stuff once all processes return

It hangs at pool = Pool(processes = cpu_count). I don't get any errors, the gunicorn worker just times out and reboots.

Any indication as to why this is happening and/or how I can solve it is greatly appreciated. Thanks.

Answer 1

This seems to be a variation on Using python's Multiprocessing makes response hang on gunicorn so possibly this is a dupe.

That said do you have to use multiprocessing (MP)? You might honestly be better served farming this out to something like Celery. MP might be getting killed with the gunicorn worker when it dies since it owns the MP process. Depending on the server config, that could happen pretty frequently. If you've got some very long running kind of job you can still farm that out to Celery, it's just a bit more configuration.

Answer 2

Are you using some async Gunicorn worker? If so, try the default sync worker and see if you can reproduce the problem.

If the problem can only be reproduced when using async workers, you should make sure the multiprocessing module is patched correctly.