Multithreaded Python script taking longer than non-threaded script

Question

Disclaimer: I'm pretty terrible with multithreading, so it's entirely possible I'm doing something wrong.

I've written a very basic raytracer in Python, and I was looking for ways to possibly speed it up. Multithreading seemed like an option, so I decided to try it out. However, while the original script took ~85 seconds to process my sample scene, the multithreaded script ends up taking ~125 seconds, which seems pretty unintuitive.

Here's what the original looks like (I'm not going to copy the drawing logic and stuff in here. If someone thinks that would be needed to figure out the problem, I'll go ahead and put it back in):

def getPixelColor(x, y, scene):
    <some raytracing code>

def draw(outputFile, scene):
    <some file handling code>
    for y in range(scene.getHeight()):
        for x in range(scene.getWidth()):
            pixelColor = getPixelColor(x, y, scene)
            <write pixelColor to image file>

if __name__ == "__main__":
    scene = readScene()
    draw(scene)

And here's the multithreaded version:

import threading
import Queue

q = Queue.Queue()
pixelDict = dict()

class DrawThread(threading.Thread):
    def __init__(self, scene):
        self.scene = scene
        threading.Thread.__init__(self)

    def run(self):
        while True:
        try:
            n, x, y = q.get_nowait()
        except Queue.Empty:
            break
        pixelDict[n] = getPixelColor(x, y, self.scene)
        q.task_done()

    def getPixelColor(x, y, scene):
        <some raytracing code>

def draw(outputFile, scene):
    <some file handling code>
    n = 0
    work_threads = 4
    for y in range(scene.getHeight()):
        for x in range(scene.getWidth()):
            q.put_nowait((n, x, y))
            n += 1
    for i in range(work_threads):
        t = DrawThread(scene)
        t.start()
    q.join()
    for i in range(n)
        pixelColor = pixelDict[i]
        <write pixelColor to image file>

if __name__ == "__main__":
    scene = readScene()
    draw(scene)

Is there something obvious that I'm doing wrong? Or am I incorrect in assuming that multithreading would give a speed boost to a process like this?

Answer 1

I suspect the Python Global Interpreter Lock is preventing your code from running in two threads at once.

What is a global interpreter lock (GIL)?

Clearly you want to take advantage of multiple CPUs. Can you split the ray tracing across processes instead of threads?

The multithreaded version obviously does more "work" so I would expect it to be slower on a single CPU.

I also dislike subclassing Thread, and just construct a new thread with t = Thread(target=myfunc); t.run()

Answer 2

To directly answer your question, Python threading will not improve the performance and the GIL may actually make it worse.

In the larger scheme of things, I love python and ray tracing, but you should never combine them. A Python ray tracer would be at least 2 orders of magnitude slower than a C or even C++ version of the same.

So while your question is interesting from a Python programmers point of view, it is rather funny from a ray tracing point of view.