I have a really similar problem to Python Multiprocessing Pool Map: AttributeError: Can't pickle local object
I think I understand where the problem is, I am just not sure how to fix it. "Pool.map" needs a top level function as input. But i dont know how i can rewrite this problem:
A simplified code version:
import os as os
from multiprocessing import Pool
import numpy as np
def opti_fun_data(prediction):
def opti_fun(x):
def error_fun(i):
return error_fun_opti(x,prediction,i)
try:
pool = Pool(np.max([os.cpu_count()-1,1]))
error = np.mean(pool.map(error_fun, range(M)))
finally: # To make sure processes are closed in the end, even if errors happen
pool.close()
pool.join()
return error
return opti_fun
If I run
opti_fun_data(prediction)(x0)
i get
Can't pickle local object 'opti_fun_data.<locals>.opti_fun.<locals>.error_fun'
I am new to the multiprocessing library and could use a helping hand. For those who are interested in a bit of background: I want to minimize the function "opti_fun" for a bunch of different scenarios/predictions. Calculating my error measure/benchmark ("error_fun_opti") is quiete computitional intensive, therefore i try to parallelize this step.
Lift the function to the top level and use functools.partial
to provide the scoped values rather than nested scope.
def error_fun(x, prediction, i):
return error_fun_opti(x, prediction, i)
error = np.mean(pool.map(functools.partial(error_fun, x, prediction), range(M)))
If you love us? You can donate to us via Paypal or buy me a coffee so we can maintain and grow! Thank you!
Donate Us With