RuntimeError at / cannot cache function '__shear_dense': no locator available for file '/home/...site-packages/librosa/util/utils.py'

Question

I am trying to host django application with apache2. But getting the following error.

RuntimeError at / cannot cache function '__shear_dense': no locator available for file '/home/username/project/env/lib/python3.6/site-packages/librosa/util/utils.py'

When running the Django server, no such error is encountered but in case of apache2 server, this error is thrown.

Similar question can be found here : RuntimeError: cannot cache function '__jaccard': no locator available for file '/usr/local/lib/python3.7/site-packages/librosa/util/matching.py'

The problem is a wsgi error and appears to be due to import of librosa and numba. I have been stuck on these days. Any pointers on how to approach this problem will be highly appreciated.

Answer 1

After spending a couple of days banging my head against this, and reading all I could google on this, I figured it out. Here goes.

TL;DR: Make very sure you set the NUMBA_CACHE_DIR environment variable to something your app can write to, and make sure that the variable is actually propagated to your app, and your app sees it. In some environments, this may appear so in local testing but may be silently lost when you deploy. Really, test it! I read this advice probably a dozen times, and I thought I checked everything, and my problem was elsewhere but in the end I was wrong.

Details. The culprit is the location of caching directories, and the corresponding lack of write permissions for these directories in the numba package, which is a dependency for librosa. Librosa tries to cache some functions using numba decorators. Numba has four locator classes, which inform where the cache is to be written.

I think Numba tries to be clever and uses fallback strategies depending on what user specified (e.g. a dedicated cache directory), and what might be available in the system to write cache to. As a result, it usually works but when it doesn't, it might seem that you specified a perfectly good caching location, it got lost or overridden by the fallback strategy and then fail.

I have noticed that some of these fallback caching location strategies include trying to cache inside the library's root directory (in this case, librosa's), and to cache to /root/something... But I am now pretty sure that if you set NUMBA_CACHE_DIR correctly, it will be fine.

Below is my specific case: using librosa in AWS Lambda. What helped me was to add debugging printouts in various places in the locator classes in numba/core/caching.py

My use case: AWS Lambda

If you get this, chances are you are using some restrictive environment with somewhat unusual defaults.

In my case, it was AWS Lambda, the root of the docker container with the app is mounted read-only. So, one of strategies to cache to the library root dir was not an option.

The caching directory did not default to /tmp by itself. Eventually, I set it explicitly via NUMBA_CACHE_DIR: /tmp in the CloudFormation template, and it tested successfully when invoked locally but when I deployed it via ZIP file manually to AWS for testing, I forgot to set it again in the console, and it came to the app as None, and failed.

Once I specified the caching dir env var in the lambda console, it worked.

Various Sources that Helped

https://github.com/numba/numba/issues/5566

https://github.com/numba/numba/issues/4032