Multiprocessing in a loop, "Pool not running" error

Question

I'm trying to run some calculation in loop, each calculation creates, uses and closes a pool. But the calculation only runs once and then throws an error: "Pool not running". Of course the old one is not running, but shouldn't the new one be created?

Below is a simplified example, similar to my code. More freakishly, in my actual code calculation runs 7 times before crashing, so I'm really confused what's the problem. Any advice appreciated!

from pathos.multiprocessing import ProcessingPool as Pool

def add_two(number):  
    return (number + 2)

def parallel_function(numbers):
    pool = Pool(10)
    result = pool.imap(add_two, numbers)
    pool.close()
    pool.join()    
    return(result)

sets=[
    [1, 2, 3],
    [2, 3, 4],
    [3, 4, 5]
]

for one_set in sets:
    x = parallel_function(one_set)
    for i in x:
        print(i)

Answer 1

This is a pathos limitation which implements the Pool using the singleton pattern.

This is the related issue ticket.

I would recommend you to use another Pool of Workers implementation.

Answer 2

The following assumes that pathos acts the same as multiprocessing. The following would be the problem if you were using multiprocessing.

The problem is that your function closes the pool before the imap is finished:

def parallel_function(numbers):
    pool = Pool(10)
    result = pool.imap(add_two, numbers)
    pool.close()
    pool.join()    
    return(result)

This should be written as:

def parallel_function(numbers):
    with Pool(10) as pool:
       yield from pool.imap(add_two, numbers)