Python permutations threads

Question

I have generated permutations with the itertools.permutations function in python. The problem is that the result is very big and I would like to go through it with multiple threads but don't really know how to accomplish that here is what I have so far:

perms = itertools.permutations('1234', r=4)

#I would like to iterate through 'perms' with multiple threads
for perm in perms:
    print perm

Answer 1

If the work you want to do with the items from the permutation generator is CPU intensive, you probably want to use processes rather than threads. CPython's Global Interpreter Lock (GIL) makes multithreading of limited value when doing CPU bound work.

Instead, use the multiprocessing module's Pool class, like so:

import multiprocessing
import itertools

def do_stuff(perm):
    # whatever
    return list(reversed(perm))

if __name__ == "__main__":
    with multiprocessing.Pool() as pool: # default is optimal number of processes
        results = pool.map(do_stuff, itertools.permutations('1234', r=4))

        # do stuff with results

Note that if you will be iterating over results (rather than doing something with it as a list), you can use imap instead of map to get an iterator that you can use to work on the results as they are produced from the worker processes. If it doesn't matter what order the items are returned, you can use imap_unordered to (I think) save a bit of memory.

The if __name__ is "__main__" boilerplate is required on Windows, where the multiprocessing module has to work around the OS's limitations (no fork).