When using Dask with Dask array I suddenly get the following error and my kernel that dies / restarts.
The console says:
BLAS : Program is Terminated. Because you tried to allocate too many memory regions
I'm using Anaconda on mac with OpenBLAS. Any idea?
Note: this question was given to me through other means, I'm repeating it here for future reference
Dask and OpenBLAS are both trying to parallelize at the same time. This triggers a check in OpenBLAS which causes a hard failure. A simple way to avoid this is to set the following environment variable:
export OMP_NUM_THREADS=1
Dask is calling OpenBLAS functions (through NumPy) many times in parallel. OpenBLAS in turn starts up many threads to operate in parallel. So if you had four cores then you might end up with sixteen threads all trying to do work. This can cause poor performance. OpenBLAS intelligently identifies this situation and warns you about it. Unfortunately its mechanism to warn you is to fail hard.
Setting OMP_NUM_THREADS=1
tells OpenBLAS to only use one thread at a time. This will help avoid contention, but may result in slower performance when not parallelizing with some other library like Dask or Joblib.
If you love us? You can donate to us via Paypal or buy me a coffee so we can maintain and grow! Thank you!
Donate Us With