python multiprocessing pool, wait for processes and restart custom processes

Question

I used python multiprocessing and do wait of all processes with this code:

...
            results = []
            for i in range(num_extract):
                url = queue.get(timeout=5)
                try:
                    print "START PROCESS!"
                    result = pool.apply_async(process, [host,url],callback=callback)
                    results.append(result)
                except Exception,e:

                    continue


            for r in results:
                r.get(timeout=7)
...

i try to use pool.join but get error:

Traceback (most recent call last):
  File "C:\workspace\sdl\lxchg\walker4.py", line 163, in <module>
    pool.join()
  File "C:\Python25\Lib\site-packages\multiprocessing\pool.py", line 338, in joi
n
    assert self._state in (CLOSE, TERMINATE)
AssertionError

Why join dont work? And what is the good way to wait all processes.

My second question is how can i restart certain process in pool? i need this in the reason of memory leak. Now In fact i do rebuild of all pool after all processes done their tasks (create new object pool to do process restarting).

What i need: for example i have 4 process in pool. Then process get his task, after task is done i need to kill process and start new (to refresh memory leak).

Answer 1

You are getting the error because you need to call pool.close() before calling pool.join()

I don't know of a good way to shut down a process started with apply_async but see if properly shutting down the pool doesn't make your memory leak go away.

The reason I think this is that the Pool class has a bunch of attributes that are threads running in daemon mode. All of these threads get cleaned up by the join method. The code you have now won't clean them up so if you create a new Pool, you'll still have all those threads running from the last one.