Are Python multiprocessing Pool thread safe?

Question

I have Django project. If I make package variable that contains Pool() object, and will try to use that Pool from Django views (which run in parallel way), will be this way thread safe? Are there any others ways to do it?

from multiprocessing import Pool
general_executor_pool = Pool()

Answer 1

I found this question via Google as I'm asking the same question. Anecdotally I can say NO it is not, because I recently debugged a piece of software that suffered from race conditions. Here's how it went:

1. A master process ran in a loop and spawned a multiprocessing pool in a new thread eery 3 minutes with a list of ~1000 accounts to be acted on
2. The thread called multiprocessing.Pool(max_proccesses=32), pool.map(func, accounts). This would open 32 processes, and one by one apply each account to an available process.
3. Unbeknownst to the original author, this process took far longer to complete than 3 minutes. So what happened the next time a thread was spawned to create a multiprocessing pool? Did it spawn 32 new processes for a total of 64? No, in practice it did not. Instead my results were scrambled and showed indication that multiple threads were acting on my data in a non-deterministic way.

I'd love to trace through the multiprocessing module to see if it is un-thread-safe by design, or get an answer from someone in the know. Anecdotally, at least, I have witnessed first hand that it is not.