scipy ImportError on travis-ci

Question

I'm setting up Travis-CI for the first time. I install scipy in what I believe is the standard way:

language: python
python:
  - "2.7"
# command to install dependencies
before_install:
  - sudo apt-get -qq update
  - sudo apt-get -qq install python-numpy python-scipy python-opencv
  - sudo apt-get -qq install libhdf5-serial-dev hdf5-tools
install:
   - "pip install numexpr"
   - "pip install cython"
   - "pip install -r requirements.txt --use-mirrors"
# command to run tests
script: nosetests

Everything builds. But when the nosetests begin, I get

ImportError: No module named scipy.ndimage

Update: Here is a more direct demonstration of the problem.

$ sudo apt-get install python-numpy python-scipy python-opencv
$ python -c 'import scipy'
Traceback (most recent call last):
  File "<string>", line 1, in <module>

ImportError: No module named scipy

The command "python -c 'import scipy'" failed and exited with 1 during install.

I tried installing scipy using pip also. I tried installing gfortran first. Here is one example of a failed build. Any suggestions?

Another Update: Travis has since added official documentation on using conda with Travis. See ostrokach's answer.

Answer 1

I found two ways around this difficulty:

1. As @unutbu suggested, build your own virtual environment and install everything using pip inside that environment. I got the build to pass, but installing scipy from source this way is very slow.

2. Following the approach used by the pandas project in this .travis.yml file and the shell scripts that it calls, force travis to use system-wide site-packages, and install numpy and scipy using apt-get. This is much faster. The key lines are

   virtualenv:
       system_site_packages: true
   

   in travis.yml before the before_install group, followed by these shell commands

   SITE_PKG_DIR=$VIRTUAL_ENV/lib/python$TRAVIS_PYTHON_VERSION/site-packages
   rm -f $VIRTUAL_ENV/lib/python$TRAVIS_PYTHON_VERSION/no-global-site-packages.txt  
   

   and then finally

   apt-get install python-numpy
   apt-get install python-scipy
   

   which will be found when nosetests tries to import them.

Update

I now prefer a conda-based build, which is faster than either of the strategies above. Here is one example on a project I maintain.

Answer 2

This is covered in the official conda documentation: Using conda with Travis CI.

---

The .travis.yml file

The following shows how to modify the .travis.yml file to use Miniconda for a project that supports Python 2.6, 2.7, 3.3, and 3.4.

NOTE: Please see the Travis CI website for information about the basic configuration for Travis.

language: python
python:
  # We don't actually use the Travis Python, but this keeps it organized.
  - "2.6"
  - "2.7"
  - "3.3"
  - "3.4"
install:
  - sudo apt-get update
  # We do this conditionally because it saves us some downloading if the
  # version is the same.
  - if [[ "$TRAVIS_PYTHON_VERSION" == "2.7" ]]; then
      wget https://repo.continuum.io/miniconda/Miniconda-latest-Linux-x86_64.sh -O miniconda.sh;
    else
      wget https://repo.continuum.io/miniconda/Miniconda3-latest-Linux-x86_64.sh -O miniconda.sh;
    fi
  - bash miniconda.sh -b -p $HOME/miniconda
  - export PATH="$HOME/miniconda/bin:$PATH"
  - hash -r
  - conda config --set always_yes yes --set changeps1 no
  - conda update -q conda
  # Useful for debugging any issues with conda
  - conda info -a

  # Replace dep1 dep2 ... with your dependencies
  - conda create -q -n test-environment python=$TRAVIS_PYTHON_VERSION dep1 dep2 ...
  - source activate test-environment
  - python setup.py install

script:
  # Your test script goes here

Answer 3

I found this approach to work:

http://danielnouri.org/notes/2012/11/23/use-apt-get-to-install-python-dependencies-for-travis-ci/

Add these lines to your Travis configuration to use a virtualenv with --system-site-packages:

virtualenv:
  system_site_packages: true

You can thus install Python packages via apt-get in the before_install section, and use them in your virtualenv:

before_install:
 - sudo apt-get install -qq python-numpy python-scipy

A real-world use of this approach can be found in nolearn.

Answer 4

As Dan Allan pointed out in his update, he now prefers a conda-based build. Here is a gist courtesy Dan Blanchard giving a full .travis.yml file example that will pre-install scipy on the test machine:

language: python
python:
  - 2.7
  - 3.3
notifications:
  email: false

# Setup anaconda
before_install:
  - wget http://repo.continuum.io/miniconda/Miniconda-latest-Linux-x86_64.sh -O miniconda.sh
  - chmod +x miniconda.sh
  - ./miniconda.sh -b
  - export PATH=/home/travis/miniconda/bin:$PATH
  - conda update --yes conda
  # The next couple lines fix a crash with multiprocessing on Travis and are not specific to using Miniconda
  - sudo rm -rf /dev/shm
  - sudo ln -s /run/shm /dev/shm
# Install packages
install:
  - conda install --yes python=$TRAVIS_PYTHON_VERSION atlas numpy scipy matplotlib nose dateutil pandas statsmodels
  # Coverage packages are on my binstar channel
  - conda install --yes -c dan_blanchard python-coveralls nose-cov
  - python setup.py install

# Run test
script:
  - nosetests --with-cov --cov YOUR_PACKAGE_NAME_HERE --cov-config .coveragerc --logging-level=INFO

# Calculate coverage
after_success:
  - coveralls --config_file .coveragerc