python DEAP genetic algorithm multi-core speed

Question

I am using Python's DEAP pacakge and I want to multi-core my code and I used the tutorial at http://deap.gel.ulaval.ca/doc/dev/tutorials/distribution.html to successfully do it using multiprocessing.

My question is the following: using 8 cores, how much speedup to I get in theory? The reason I ask is because I want to decide how many individuals and generations I can run in the same amount of time as the single-cored version. My code used to take ~200s to run and with 8 cores, it now takes ~0.5 seconds (this is a 400X speedup). Can I assume that anything will be sped up by 400X? I know it's complex, but your help will be very appreciated.

In general, if anybody can help, I wanted to understand how multicoring changes the flow of computation. Does it just map the evaluation of each individual over different cores for each generation? Or does it run generations in parallel? If you know of any documentation I could read about this, please let me know.

I did not provide a code example as it seems not necessary because this is a very high-level question.

Answer 1

Does it just map the evaluation of each individual over different cores for each generation, or does it run generations in parallel?

The example maps the evaluate operation thus...

fitnesses = toolbox.map(toolbox.evaluate, invalid_ind)

The lone running process hits the map: all the invalid_ind's are marshaled into a single queue and, as a core comes available, the next individual in the queue is assigned to that core to run the evaluate routine. When the queue is empty all the results are assembled into a list and assigned back to fitnesses. Thence the process continues on it's lonesome.

So:

• "Yes" it does map the evaluation of each individual over different cores and,
• "No" it does not run generations in parallel

At least that's what I surmise from when I asked this question. Depending on your application of course, in my experience with DEAP and cProfile, the top two consumers of CPU time was evaluating individuals and copying.