In the link below there is an explanation of the map
method on the Pool
class.
It seems that it blocks until the result is ready. This implies that there is no need to do pool.close(); pool.join()
after running pool.map
, however it is demo'd in this way in this blog.
Am I missing something or is there a need to do pool.close after running pool.map
(as opposed to pool.map_async
? Note I am using [multiprocessing.dummy][2]
, which is provides a similar api to multiprocessing, but uses threading under the covers.
https://docs.python.org/2/library/multiprocessing.html#multiprocessing.pool.multiprocessing.Pool.map
pool. close() makes sure that process pool does not accept new processes, and pool. join() waits for the processes to properly finish their work and return. So it is a good idea to use pool.
So Pool. close() is typically called when the parallelizable part of your main program is finished. Then the worker processes will terminate when all work already assigned has completed.
Pool allows multiple jobs per process, which may make it easier to parallel your program. If you have a numbers jobs to run in parallel, you can make a Pool with number of processes the same number of as CPU cores and after that pass the list of the numbers jobs to pool. map.
Results for issued tasks can then be retrieved synchronously, or we can retrieve the result of tasks later by using asynchronous versions of the functions such as apply_async() and map_async(). The process pool must be closed once we are finished with it in order to release the child worker processes.
pool.close
tells the pool not to accept any new job.
pool.join
tells the pool to wait until all jobs finished then exit, effectively cleaning up the pool.
So blocking the parent process is just a side effect of what pool.join
is doing.
It's true that when you call pool.map()
, the parent process is blocked until map returns result. But it's a good practice to close
and join
the pool
after using it, to better manage resource and control exceptions.
If you love us? You can donate to us via Paypal or buy me a coffee so we can maintain and grow! Thank you!
Donate Us With