Logo Questions Linux Laravel Mysql Ubuntu Git Menu
 

Purpose of pool.join, pool.close in multiprocessing?

In the link below there is an explanation of the map method on the Pool class.

It seems that it blocks until the result is ready. This implies that there is no need to do pool.close(); pool.join() after running pool.map, however it is demo'd in this way in this blog.

Am I missing something or is there a need to do pool.close after running pool.map (as opposed to pool.map_async? Note I am using [multiprocessing.dummy][2], which is provides a similar api to multiprocessing, but uses threading under the covers.

https://docs.python.org/2/library/multiprocessing.html#multiprocessing.pool.multiprocessing.Pool.map

like image 234
user3659451 Avatar asked Feb 29 '16 19:02

user3659451


People also ask

Why pool close pool join?

pool. close() makes sure that process pool does not accept new processes, and pool. join() waits for the processes to properly finish their work and return. So it is a good idea to use pool.

When should you close your pool?

So Pool. close() is typically called when the parallelizable part of your main program is finished. Then the worker processes will terminate when all work already assigned has completed.

How does a pool work in multiprocessing?

Pool allows multiple jobs per process, which may make it easier to parallel your program. If you have a numbers jobs to run in parallel, you can make a Pool with number of processes the same number of as CPU cores and after that pass the list of the numbers jobs to pool. map.

Do you need to close a pool Python?

Results for issued tasks can then be retrieved synchronously, or we can retrieve the result of tasks later by using asynchronous versions of the functions such as apply_async() and map_async(). The process pool must be closed once we are finished with it in order to release the child worker processes.


1 Answers

pool.close tells the pool not to accept any new job.

pool.join tells the pool to wait until all jobs finished then exit, effectively cleaning up the pool.

So blocking the parent process is just a side effect of what pool.join is doing.

It's true that when you call pool.map(), the parent process is blocked until map returns result. But it's a good practice to close and join the pool after using it, to better manage resource and control exceptions.

like image 191
THN Avatar answered Sep 18 '22 20:09

THN