OpenBLAS blas_thread_init: pthread_create: Resource temporarily unavailable

Question

I am facing a problem now, unable to run any program in the cluster. It gives error.

OpenBLAS blas_thread_init: pthread_create: Resource temporarily unavailable
OpenBLAS blas_thread_init: RLIMIT_NPROC 64 current, 64 max
OpenBLAS blas_thread_init: pthread_create: Resource temporarily unavailable
OpenBLAS blas_thread_init: RLIMIT_NPROC 64 current, 64 max
OpenBLAS blas_thread_init: pthread_create: Resource temporarily unavailable
OpenBLAS blas_thread_init: RLIMIT_NPROC 64 current, 64 max
OpenBLAS blas_thread_init: pthread_create: Resource temporarily unavailable
OpenBLAS blas_thread_init: RLIMIT_NPROC 64 current, 64 max
Traceback (most recent call last):
  File "hello-world.py", line 1, in <module>
    from keras.models import Sequential
  File "/home/amalli2s/anaconda3/lib/python3.6/site-packages/keras/__init__.py", line 3, in <module>
    from . import utils
  File "/home/amalli2s/anaconda3/lib/python3.6/site-packages/keras/utils/__init__.py", line 2, in <module>
    from . import np_utils
  File "/home/amalli2s/anaconda3/lib/python3.6/site-packages/keras/utils/np_utils.py", line 6, in <module>
    import numpy as np
  File "/home/amalli2s/.local/lib/python3.6/site-packages/numpy/__init__.py", line 142, in <module>
    from . import add_newdocs
  File "/home/amalli2s/.local/lib/python3.6/site-packages/numpy/add_newdocs.py", line 13, in <module>
    from numpy.lib import add_newdoc
  File "/home/amalli2s/.local/lib/python3.6/site-packages/numpy/lib/__init__.py", line 8, in <module>
    from .type_check import *
  File "/home/amalli2s/.local/lib/python3.6/site-packages/numpy/lib/type_check.py", line 11, in <module>
    import numpy.core.numeric as _nx
  File "/home/amalli2s/.local/lib/python3.6/site-packages/numpy/core/__init__.py", line 16, in <module>
    from . import multiarray
SystemError: initialization of multiarray raised unreported exception

This problem, i assume to be same as this one Multiple instances of Python running simultaneously limited to 35

So according to the solution when I set export OPENBLAS_NUM_THREADS=1

then I end up getting following error:

terminate called after throwing an instance of 'std::system_error'
  what():  Resource temporarily unavailable
Aborted

Is there anybody else facing same issue or has idea on how to solve this ? Thank you.

EDIT: Ok, Seems like this happened because of some config restrictions the admins were trying to implement. Now it works again.

Answer 1

I had this problem running numpy on an ubuntu server. I got all of the following errors, depending on whether I tried to import numpy in a shell or running my django app:

• PyCapsule_Import could not import module "datetime" from
• numpy.core._multiarray_umath import ( OpenBLAS blas_thread_init:
• pthread_create failed for thread 25 of 32: Resource temporarily unavailable

I'm posting this answer since it drove me crazy. What helped for me was to add:

import os
os.environ['OPENBLAS_NUM_THREADS'] = '1'

before

import numpy as np

I guess the server had some limit for the amount of threads it allows(?). Hope it helps someone!

Answer 2

This is for others in the future who encounter this error. The cluster setup most likely limits the number processes that can be run by a user on an interactive node. The clue is in the second line of the error:

OpenBLAS blas_thread_init: pthread_create: Resource temporarily unavailable
OpenBLAS blas_thread_init: RLIMIT_NPROC 64 current, 64 max

Here the limit is set to 64. While this is quite sufficient for normal CLI use, it's probably not sufficient for interactively running Keras jobs (like the OP); or in my case, trying to run an interactive Dask cluster.

It may be possible to increase the limit from your shell using, say ulimit -u 10000, but that's not guaranteed to work. It's best to notify the admins like the OP.