Logo Questions Linux Laravel Mysql Ubuntu Git Menu
 

How do you stop numpy from multithreading? [duplicate]

I have to run jobs on a regular basis on compute servers that I share with others in the department and when I start 10 jobs, I really would like it to just take 10 cores and not more; I don't care if it takes a bit longer with a single core per run: I just don't want it to encroach on the others' territory, which would require me to renice the jobs and so on. I just want to have 10 solid cores and that's all.

I am using Enthought 7.3-1 on Redhat, which is based on Python 2.7.3 and numpy 1.6.1, but the question is more general.

like image 917
MasDaddy Avatar asked Jun 11 '13 20:06

MasDaddy


People also ask

Is numpy single threaded?

Key Takeaways. Python is NOT a single-threaded language. Python processes typically use a single thread because of the GIL. Despite the GIL, libraries that perform computationally heavy tasks like numpy, scipy and pytorch utilise C-based implementations under the hood, allowing the use of multiple cores.

Are numpy operations multithreaded?

First, numpy supports multithreading, and this can give you a speed boost in multicore environments!

Does numpy automatically use multiple cores?

NumPy does not run in parallel. On the other hand Numba fully utilizes the parallel execution capabilities of your computer. NumPy functions are not going to use multiple CPU cores, never mind the GPU.

Does Python really support multithreading?

Python doesn't support multi-threading because Python on the Cpython interpreter does not support true multi-core execution via multithreading. However, Python does have a threading library.


1 Answers

Only hopefully this fixes all scenarios and system you may be on.

  1. Use numpy.__config__.show() to see if you are using OpenBLAS or MKL

From this point on there are a few ways you can do this.

2.1. The terminal route export OPENBLAS_NUM_THREADS=1 or export MKL_NUM_THREADS=1

2.2 (This is my preferred way) In your python script import os and add the line os.environ['OPENBLAS_NUM_THREADS'] = '1' or os.environ['MKL_NUM_THREADS'] = '1'.

NOTE when setting os.environ[VAR] the number of threads must be a string! Also, you may need to set this environment variable before importing numpy/scipy.

There are probably other options besides openBLAS or MKL but step 1 will help you figure that out.

like image 163
SARose Avatar answered Oct 18 '22 09:10

SARose