Multithreading performance overhead

Question

So basically I created this program that adds values to redis. So far I get this timing:

real 0m27.759s
user 0m18.129s
sys  0m5.580s

However when I tried to run multiple threads:

if __name__ == '__main__':
    try:
        for x in range(0, NUM_THREADS):
            Thread(None, startProgram, None,
                   (NUM_HOSTS/NUM_THREADS*x+1,
                    NUM_HOSTS/NUM_THREADS*(x+1))).start()
    except Exception as errtxt:
        print errtxt

I get this with NUM_THREADS set ot 10:

real 0m32.642s
user 0m22.953s
sys  0m11.473s

Why is my program running slower with more threads?

I'm running Linux Ubuntu 11.04 and Python 2.7.1.

Answer 1

The result depends on the Python implementation, cpython's GIL prevents parallel computations from being faster than sequential ones.

Consider using the multiprocessing module, which executes each thread in its own Python process, or alternative GIL-free Python implementations like IronPython and Jython.

Answer 2

You can use Parallel Python for this.

Here is an example of parallel sum:

http://www.parallelpython.com/content/view/17/31/#SUM_PRIMES