How to share a variable in 'joblib' Python library

Question

from joblib import Parallel, delayed

def func(v):
    temp.append(v)
    return

temp = []
Parallel(n_jobs=4)(delayed(func)(v) for v in range(10))
print temp

I want to make shared memory variable. But the value of temp is empty []. How can I do it?

For other method, I tried pickle.dump and load. But there is a lock problem. Please give me advice!

Answer 1

from joblib import Parallel, delayed

def func(v):
    return v

temp = Parallel(n_jobs=4)(delayed(func)(v) for v in range(10))
print temp

delayed collects the output returned by func in a list and returns it on completion.

Answer 2

You need to use multiprocessing.Manager.list, for example:

from joblib import Parallel, delayed
from multiprocessing import Manager

manager = Manager()
temp = manager.list()

def func(v, temp):
    temp.append(v)
    return

_ = Parallel(n_jobs=4)(delayed(func)(v, temp) for v in range(10))

temp[:]:

[0, 1, 2, 3, 4, 5, 6, 7, 8, 9]