Using DEAP (genetic algorithm library) with spark

Question

Is IT possible to use DEAP ( http://deap.readthedocs.io/en/master/) with a spark cluster to map the fitness evaluation function. I would like to run a GA but the fitness function is rather long and I was planning on distributing it on a spark cluster.

Answer 1

You should look at the Using Multiple Processors section in the DEAP documentation and at this example. They explain how to replace the map function in the DEAP toolbox by a map function of your choice.

To use pyspark to map the fitness evaluation function, you could do something like that:

from pyspark import SparkContext

sc = SparkContext(appName="DEAP")

def sparkMap(algorithm, population):
    return sc.parallelize(population).map(algorithm)

toolbox.register("map", sparkMap)