Both os
and multiprocessing
modules define a cpu_count
function.
os.cpu_count
is documented as follows:
Return the number of CPUs in the system. Returns None if undetermined.
and multiprocessing.cpu_count
's documentation says:
Return the number of CPUs in the system. May raise NotImplementedError. See also os.cpu_count()
On my machine, they both return the same result:
>>> import os
>>> import multiprocessing as mp
>>> os.cpu_count()
8
>>> mp.cpu_count()
8
I would have thought that multiprocessing.cpu_count
would be a mere reference to os.cpu_count
, but it is not:
>>> os.cpu_count is mp.cpu_count
False
So what is the difference between them? Am I guaranteed that they'll always return the same result?
Moreover, if I want to specify a number of processes to create for multiprocessing.Pool
, should I use os
or multiprocessing
's function?
help(multiprocessing) One of the useful functions in multiprocessing is cpu_count() . This returns the number of CPUs (computer cores) available on your computer to be used for a parallel program. Type into ipython. print(multiprocessing.cpu_count())
multiprocessing is a package that supports spawning processes using an API similar to the threading module. The multiprocessing package offers both local and remote concurrency, effectively side-stepping the Global Interpreter Lock by using subprocesses instead of threads.
cpu_count() method in Python is used to get the number of CPUs in the system. This method returns None if number of CPUs in the system is undetermined. Parameter: No parameter is required. Return Type: This method returns an integer value which denotes the number of CPUs in the system.
In Python, single-CPU use is caused by the global interpreter lock (GIL), which allows only one thread to carry the Python interpreter at any given time. The GIL was implemented to handle a memory management issue, but as a result, Python is limited to using a single processor.
The answer lies in multiprocessing.context
, which defines BaseContext.cpu_count
as follows:
# cpython/Lib/multiprocessing/context.py
class BaseContext(object):
def cpu_count(self):
'''Returns the number of CPUs in the system'''
num = os.cpu_count()
if num is None:
raise NotImplementedError('cannot determine number of cpus')
else:
return num
Then, this cpu_count
method is exposed by multiprocessing
:
# cpython/Lib/multiprocessing/__init__.py
__all__ = [x for x in dir(context._default_context) if not x.startswith('_')]
globals().update((name, getattr(context._default_context, name)) for name in __all__)
So in the end, multiprocessing.cpu_count
is nothing but a wrapper around os.cpu_count
.
If you love us? You can donate to us via Paypal or buy me a coffee so we can maintain and grow! Thank you!
Donate Us With