multiprocessing Pool.map function issue

Question

I have a function like below, which takes a list and a path :

def my_function(items_list,directory):
     return(resulting number  of analyzing items_list for example [a,b] using specific file for example 'c:\\path')

to do parallel computation, im using multiprocessing module as follow:

from multiprocessing import Pool
def test_func(objs):
 pool= Pool(8) 
 result=pool.map(my_function,objs)
 return(result)


if __name__=='__main__':
    objects=[([a,b],'path1',),([c,d],'path2',),.....]
    result=test_funct(objects)

but it gives me the following error: TypeError: my_function() missing 1 required positional argument: 'directory'

I changed the objects list format several times but it keeps giving me the same error. does anybody know what the problem is? (im using python33 on windows 7)

Answer 1

multiprocessing.map will not unpack the variables in your tuple. So myfunction is receiving one tuple argument, instead of a list and a string.

If you're using Python 3.3+ (which it appears you are), you can use starmap, which will expand the tuple:

 result = pool.starmap(my_function,objs)

If you're using Python 3.2 or lower, the easiest thing would be to make my_function take a single argument, and just expand the tuple in the body of the function:

def my_function(tup)
    items_list, directory = tup

If you can't change my_function, add a helper function that will do the unpacking for you:

def my_function_helper(tup):
    return my_function(*tup)