Menu
I use Python 2.5.4. My computer: CPU AMD Phenom X3 720BE, Mainboard 780G, 4GB RAM, Windows 7 32 bit.
I use Python threading but can not make every python.exe process consume 100% CPU. Why are they using only about 33-34% on average?.
I wish to direct all available computer resources toward these large calculations so as to complete them as quickly as possible.
EDIT: Thanks everybody. Now I'm using Parallel Python and everything works well. My CPU now always at 100%. Thanks all!
213
Key Takeaways. Python is NOT a single-threaded language. Python processes typically use a single thread because of the GIL. Despite the GIL, libraries that perform computationally heavy tasks like numpy, scipy and pytorch utilise C-based implementations under the hood, allowing the use of multiple cores.
CPU and RAMThere is not much to say about the CPU as most CPUs will run Python. A stronger CPU will always be faster than a weak one so purchase the CPU you can afford. Python is already a slow language. No need to make it even slower with a poor CPU.
In Python, single-CPU use is caused by the global interpreter lock (GIL), which allows only one thread to carry the Python interpreter at any given time. The GIL was implemented to handle a memory management issue, but as a result, Python is limited to using a single processor.
Only if your Python program uses multiprocessing, which in fact starts up multiple instances of the Python interpreter and lets them perform your tasks truly parallel, you can take advantage of multiple virtual cores/CPU threads.
Using only 25% of the CPU The bottom line is that when you see a single program consistently take up 25% of the CPU on a quad-core processor, you can almost bet that it’s software that only knows how to use a single processor at a time.
The Python interpreter is an application which only runs as one single process by default and is therefore not able to take advantage of more than one virtual core. Even if the code you run with it uses multithreading, it will still only use one CPU thread/virtual core, because of the GIL (global interpreter lock).
I've tried limiting the number of processes to be the number of CPUs minus 1, as described in How to limit the number of processors that Python uses: pool = Pool (processes=max (multiprocessing.cpu_count ()-1, 1) for p in pool.imap (func, iterable): ... This does reduce the total number of running processes.
It appears that you have a 3-core CPU. If you want to use more than one CPU core in native Python code, you have to spawn multiple processes. (Two or more Python threads cannot run concurrently on different CPUs)
As R. Pate said, Python's multiprocessing module is one way. However, I would suggest looking at Parallel Python instead. It takes care of distributing tasks and message-passing. You can even run tasks on many separate computers with little change to your code.
Using it is quite simple:
import pp
def parallel_function(arg):
return arg
job_server = pp.Server()
# Define your jobs
job1 = job_server.submit(parallel_function, ("foo",))
job2 = job_server.submit(parallel_function, ("bar",))
# Compute and retrieve answers for the jobs.
print job1()
print job2()
If you love us? You can donate to us via Paypal or buy me a coffee so we can maintain and grow! Thank you!
Donate Us With
