Error when using source in a python Tox ini file

Question

I'm trying to get Tox and Conda to play together well. Mainly because I have a lot of non-python dependencies that need to be installed and it was easy to create Conda distributions. Then I can install everything with a simple conda install.

However, I'm having trouble with activating the conda environment.

[tox]
envlist = py27

[testenv]
whitelist_externals =
    conda
    source
    py.test
setenv =
    PYTHONPATH = {toxinidir}:{toxinidir}/damlarces
install_command =
    python build_env.py --conda-env {toxworkdir}/conda {packages}
commands =
    source activate {toxworkdir}/conda
    py.test --basetemp={envtmpdir}

The python build_env.py --conda-env {toxworkdir}/conda {packages} takes care of creating the environment (if needed), installing the packages, etc. The problem comes at the source activate {toxworkdir}/conda line. I get an ERROR: InvocationError: could not find executable 'source' error. Typing the command directly into the command-line works fine.

For those who are interested. The build_env.py is in this Gist: https://gist.github.com/JudoWill/70450979353fa2d12823 ... Currently its just installing Python dependencies but in its intended environment it will be installing Conda repo's that are not necessarily python libraries. Any thoughts?

Answer 1

source is not a command. It's a shell builtin. source script.sh causes script.sh to be run inside the current shell. This is necessary for activate, because it modifies the PATH, and you want those modifications to affect the shell itself (normally, when you run a script, it runs in a subshell, which has its own environment that cannot affect the calling shell's environment).

I don't know if tox supports setting environment variables in the commands. If so, you can just use

export PATH={toxworkdir}/conda/bin:$PATH

Otherwise, just use the absolute path to all your commands, like

{toxworkdir}/conda/bin/py.test --basetemp={envtmpdir}

Answer 2

After plenty of tinkering I figured out a work-around. Its probably brittle to any changes in Tox but as long as they run tests in the order provided by envlist then it should work.

As suggested by @asmeurer in his answer, the trick is to somehow change the PATH as recognized by tox. But in order to generate the conda/bin I need to run conda create. First, I tried using the export PATH={toxworkdir}/conda/bin:$PATH suggested by asmeurer, but this ran into the same InvocationError issue just with export instead of source.

Next, I tried using the setenv tox section to change the path. This resulted in a chicken & egg problem. I couldn't add the bin directory until I ran the conda create command. Because of the default order of commands in Tox it doesn't seem like I can get the setenv to differ (or re-run) after the install_command.

The work-around I came up with was to create a new env to test against and then share this environment with the subsequent tests.

[tox]
envlist = setup,py27

[testenv]
whitelist_externals =
    conda
    source
    py.test

[testenv:setup]
setenv =
    PYTHONPATH = {toxinidir}:{toxinidir}/damlarces
commands =
    conda config --add channels judowill
    python build_env.py --conda-env {toxworkdir}/conda {packages}

[testenv:py27]
setenv =
    PYTHONPATH = {toxinidir}:{toxinidir}/damlarces
    PATH={toxworkdir}/conda/bin:$PATH
commands =
    {toxworkdir}/conda/bin/py.test --basetemp={envtmpdir}

This works. I'm not sure how difficult it would be to generalize this to a multiple python environment, but it works for me.