How do I access all computer cores for computation in python script?

Question

I have a python script that has to take many permutations of a large dataset, score each permutation, and retain only the highest scoring permutations. The dataset is so large that this script takes almost 3 days to run.

When I check my system resources in windows, only 12% of my CPU is being used and only 4 out of 8 cores are working at all. Even if I put the python.exe process at highest priority, this doesn't change.

My assumption is that dedicating more CPU usage to running the script could make it run faster, but my ultimate goal is to reduce the runtime by at least half. Is there a python module or some code that could help me do this? As an aside, does this sound like a problem that could benefit from a smarter algorithm?

Thank you in advance!

Answer 1

There are a few ways to go about this, but check out the multiprocessing module. This is a standard library module for creating multiple processes, similar to threads but without the limitations of the GIL.

You can also look into the excellent Celery library. This is a distrubuted task queue, and has a lot of great features. Its a pretty easy install, and easy to get started with.

Answer 2

I can answer a HOW-TO with a simple code sample. While this is running, run /bin/top and see your processes. Simple to do. Note, I've even included how to clean up afterwards from a keyboard interrupt - without that, your subprocesses will keep running and you'll have to kill them manually.

from multiprocessing import Process
import traceback
import logging 
import time

class AllDoneException(Exception):
    pass

class Dum(object):
    def __init__(self):
        self.numProcesses = 10    
        self.logger = logging.getLogger()
        self.logger.setLevel(logging.INFO)
        self.logger.addHandler(logging.StreamHandler())

    def myRoutineHere(self, processNumber):
        print "I'm in process number %d" % (processNumber)
        time.sleep(10)
        # optional: raise AllDoneException  

    def myRoutine(self):
        plist = []
        try:
            for pnum in range(0, self.numProcesses):
                p = Process(target=self.myRoutineHere, args=(pnum, ))
                p.start()
                plist.append(p)
            while 1:
                isAliveList = [p.is_alive() for p in plist]
                if not True in isAliveList:
                    break
                time.sleep(1)
        except KeyboardInterrupt:
            self.logger.warning("Caught keyboard interrupt, exiting.")
        except AllDoneException:
            self.logger.warning("Caught AllDoneException, Exiting normally.")
        except:
            self.logger.warning("Caught Exception, exiting: %s" % (traceback.format_exc()))
        for p in plist:
            p.terminate()

d = Dum()
d.myRoutine()