How to increase python CPU usage

Question

I am a newbie programmer, so a little lost here.

I've written a small python function that is rather time consuming to run depending on my search area, which is expected.

After a certain threshold, about a million records my program always seems to get lost. Python shell is not responding, but CPU usage remains contstant at about 13%. Also I can't stop the program execution by KeyboardInterrupt

So my question: Is it possible to increase the CPU usage for my python program to make it run faster?

A side question: Any help on why python shell becomes unresponsive? I am not using excessive memory, I just need to be able to cycle through a large set as quickly as possible.

Answer 1

Is your cpu a multicore cpu? If yes there are several ways to use multiple cores with python.

Built-in is the multiprocessing module. The multiprocessing.Pool class provides vectorization across multiple CPUs with the map() and related methods. There is a trade-off in here though. If you have to communicate large amounts of data between the processes, that overhead might negate the advantage of multiple cores. Use a suitable build of numpy. If numpy is built with a multithreading ATLAS library, it will be faster on large problems. Use extension modules like numexpr, parallel python, corepy or Copenhagen Vector Byte Code.

Note that the threading module isn't all that useful in this regard. To keep memory management simple, the global interpreter lock ("GIL") enforces that only one thread at a time can be executing python bytecode. External modules like numpy can use multiple threads internally, though.