multiprocessing AttributeError module object has no attribute '__path__'

Question

i have a long script which at the end needs to run a function to all items of huge list which takes a long time,consider for example:

input_a= [1,2,3,4] # a lengthy computation on some data
print('test.1') # for testing how the script runs
input_b= [5,6,7,8]  # some other computation
print('test.2')

def input_analyzer(item_a): # analyzing item_a using input_a and input_b
     return(item_a * input_a[0]*input_b[2])

from multiprocessing import Pool
def analyzer_final(input_list):
    pool=Pool(7)
    result=pool.map(input_analyzer, input_list)
    return(result)

my_list= [10,20,30,40,1,2,2,3,4,5,6,7,8,9,90,1,2,3] # a huge list of inputs

if __name__=='__main__':
        result_final=analyzer_final(my_list)
        print(result_final)
    return(result)

the output of this codes, varies run to run but what all the runs has in common is several running of whole script, it seems that by assigning 7 as Pool, the whole script will run about 8 times!

im not sure if i got the concept of multiprocessing well, but i thought what it should do is just running the function 'input_analyzer' using several CPUs and not running the whole script several times. in case of my real code, its so long and it gives me a strange error :

without using multiprocessing i run this code just fine, i dont know what im doing wrong here especially with error "AttributeError module object has no attribute 'path'"i appreciate any help.

Answer 1

from multiprocessing import Pool as ThreadPool
import requests


API_URL = 'http://example.com/api'
pool = ThreadPool(4) # Hint...

def foo(x):
  params={'x': x}
  r = requests.get(API_URL, params=params)
  return r.json()

if __name__ == '__main__':
  num_iter = [1,2,3,4,5]
  out = pool.map(foo, num_iter)
  print(out)

Hint's Answer: This is why the exception is raised. Pool definition is outside if __name__ == '__main__'

Fixed...

from multiprocessing import Pool as ThreadPool
import requests


API_URL = 'http://example.com/api'

def foo(x):
  params={'x': x}
  r = requests.get(API_URL, params=params)
  return r.json()

if __name__ == '__main__':
  pool = ThreadPool(4) # Hint...
  num_iter = [1,2,3,4,5]
  out = pool.map(foo, num_iter)
  print(out)

The python docs touch on this scenario as well: https://docs.python.org/2/library/multiprocessing.html#using-a-pool-of-workers

I did not find this to be an issue when using multiprocessing.dummy at all.

Answer 2

Multiprocessing needs to be able to import your module, as stated at the top of the documentation.

You have a bunch of code sitting at module (global) scope, so this will be run every time the module is imported.

Put it within your if __name__ == '__main__' block, or better yet, in a function.