multiprocessing numpy not defined error

Question

I am using the following test code:

from pathos.multiprocessing import ProcessingPool as Pool
import numpy

def foo(obj1, obj2):
   a = obj1**2
   b = numpy.asarray(range(1,5))
   return obj1, b

if __name__ == '__main__':
    p = Pool(5)
    res = p.map(foo, [1,2,3], [4,5,6])

It gives error:

File "C:\Python27\lib\site-packages\multiprocess\pool.py", line 567, in get
    raise self._value
NameError: global name 'numpy' is not defined

What am I doing wrong in the code?

Edit: Why was this question voted down twice?

I have numpy installed and my interpreter has been using it correctly until I try to do it for multiprocessing. I have been coding with same install for a while.

Answer 1

It seems like imports are not shared between processes. Therefore you need to import numpy in all your processes seperatly.

In your case this means adding the import numpy in your foo function. Processes are not light-weight so the import won't slow you down (at least not significantly).

The other alternative would be to pass the module to the functions (not recommended and I'm not sure if that will work):

if __name__ == '__main__':
    p = Pool(5)
    res = p.map(foo, numpy, [1,2,3], [4,5,6])

def foo(np, obj1, obj2):
   a = obj1**2
   b = np.asarray(range(1,5))
   return obj1, b