Evaluating a function parallely using itertools product

Question

I have to evaluate a function (let's call it my_func()) that takes 8 parameters as input and returns a scalar upon different matrix computations. Since I don't have any constrains on my_func(), I have no choice but to brute force all possibilities which amounts to 8^8 = 16777216. I started off using itertools's product functions and passed the generated sequences to the my_func() sequentially. Kindly take a look at the sample of my code below,

So far.....

from itertools import product
import numpy as np

def my_func(a,b,c,d,e,f,g,h): #Function that I have to evaluate
    #do some matrix computations and return some scalar q# 
    return q

def Evaluate():
    Range = [x for x in np.arange(0,3.60,0.5)] # A list that contains 8 elements
    it = itertools.product(Range, repeat=8) #Generate all possiblities
    Max = 0
    for i in it:
        Sum = my_func(*i)
        if Sum > Max:
            Max = present
    return Max

Result = Evaluate() #Output

Pitfalls...

Unfortunately, executing the above code sequentially takes ages to produce the output. This is because my_func is quite heavy. I have no choice but to somehow parallelize this code so that I can take advantage of multiple processors available to run my code.

Questions:

Since itertools.product is a generator I cannot parallelize this to evaluate my_func() simultaneously for different sets of parameters. Is there any way I can parallelize the code? if no is the asnswer, should I abondon the idea of using itertools and try someting else?

Kindly help me find a solution.

Thanks for suggesting any ideas.

Cheers!!

Answer 1

You can parallelize a generator ! You can use the multiprocessing library, which offers the Pool class which does exactly what you want. Here you can get more ample documentation, but essentially, what you want to do is :

with Pool(processes=4) as pool:
  pool.map(my_func, itertools.product(Range, repeat=8))

You should look into the alternatives to pool.map to find the one which suits you most.