Threading In Python

Question

I'm new to threading and was wondering if it's bad to spawn a lot of threads for various tasks (in a server environment). Do threads take up a lot more memory/cpu compared to more linear programming?

Answer 1

You have to consider multiple things if you want to use multiple threads:

1. You can only run #processors threads simultaneously. (Obvious)
2. In Python each thread is a 'kernel thread' which normally takes a non-trivial amount of resources (8 mb stack by default on linux)
3. Python has a global interpreter lock, which means only one python instructions can be processed at once independently of the number of processors. This lock is however released if one of your threads waits on IO.

The conclusion I take from that:

1. If you are doing IO (Turbogears, Twisted) or are using properly coded extension modules (numpy) use threads.
2. If you want to execute python code concurrently use processes (easiest with multiprocess module)