How to install scikit-learn on heroku cedar?

Question

I'v successfully installed numpy and scipy using the method described in this answer. Then I wanted to add scikit-learn so at first I tried adding scikit-learn==0.11 to the requirements.txt and when pushed to heroku I got an error message:

ImportError: liblapack.so.3gf: cannot open shared object file: No such file or directory

So I've added to LD_LIBRARY_PATH the path where I have liblapack.so.3gf but then I got this:

ImportError: libgfortran.so.3: cannot open shared object file: No such file or directory

I believe that heroku doesn't have fortran compiler, but maybe I wrong. How can I resolve this?

Answer 1

Based on these pointers, I just finished an installation of scikit-learn on heroku. I was quite happy to see that there is no need to get custom binaries, but that setting some environments did the trick :)

You can find the additional custom step in my fork of the buildpack of wyn: https://github.com/ToonTimbermont/heroku-buildpack-python

The key was to set the proper values for LD_LIBRARY_PATH: export LD_LIBRARY_PATH=$(pwd)/vendor/lib:$(pwd)/vendor/lib/atlas- base:$(pwd)/vendor/lib/atlas-base/atlas

I also added these paths to the heroku config: heroku config:add LD_LIBRARY_PATH=/app/.heroku/vendor/lib/atlas-base/atlas:/app/.heroku/vendor/lib/atlas-base:/app/.heroku/vendor/lib/

Answer 2

Another good option is the conda buildpack, which allows you to add any of the free Linux64 packages available through Anaconda/Miniconda to a Heroku app. Some of the most popular packages include numpy, scipy, scikit-learn, statsmodels, and pandas. While the buildpack makes it fairly simple to add packages to an app, the downsides are that the buildback takes up a lot of space and that you have to wait on Anaconda to update the libraries in the repository.

If you are starting a new Python app on Heroku, you can add the conda buildpack using the command:

$ heroku create YOUR_APP_NAME --buildpack https://github.com/kennethreitz/conda-buildpack.git

If you have already setup a Python app on Heroku, you can add the conda buildpack to the existing app using the command:

$ heroku config:add BUILDPACK_URL=https://github.com/kennethreitz/conda-buildpack.git

Or, if you need to specify the app by name:

$ heroku config:add BUILDPACK_URL=https://github.com/kennethreitz/conda-buildpack.git --app YOUR_APP_NAME

To use the buildpack, you will need to include two text files in the app directory, requirements.txt and conda-requirements.txt. Just as with the standard Python buildpack, the requirements.txt file lists packages that should be installed using pip. Packages that should be installed using conda are listed in the conda-requirements.txt file. Some of the most useful scientific packages include numpy, scipy, scikit-learn, statsmodels, pandas, and cvxopt. The full list of available conda packages can be found at repo.continuum.io.

For example:

$ cat requirements.txt
gunicorn==0.14.2
requests==0.11.1

$ cat conda-requirements.txt
scipy
numpy
cvxopt

That’s it! You can now add Anaconda packages to a Python app on Heroku.

Answer 3

You might be able to package all of the libraries you require in the application itself, but the more elegant solution is to clone the Heroku Python Buildpack on git, and modify it to include the libraries. Then you can instruct your app to use your modified buildpack with the --buildpack flag on the command-line client.

Edit: I didn't click through to the other answer originally, but it sounds like you're already using a custom buildpack. The buildpack you're using has a variety of custom steps which download custom binaries. The binaries are compiled under 64-bit Debian.

You should be able to dissect one of the other custom binaries the buildpack is using to find out the --prefix with which you can ./configure and build the extra libraries you want. It's not exactly easy or convenient, but it should work the same as numpy and scipy worked.

Answer 4

If you came here like me having issues while installing scikit learn and getting error,

ImportError: numpy is not installed.
scikit-learn requires numpy >= 1.11.0.

What you have to do is check the version of your local/dev environment and use that specific versions while deploying, even the python version.

Either use pip freeze as mentioned by zenpoy or following.

import scipy
import sklearn
import numpy

print(scipy.__version__)
print(sklearn.__version__)
print(numpy.__version__)

Add this versions specifically to requirements.txt

scipy==1.4.1
scikit-learn==0.22.2.post1
numpy==1.19.5

In heroku to set the Python runtime, add a runtime.txt file to your app’s root directory that declares the exact version number to use:

python-3.7.10