Tox and lib and lib64 and site-packages

Question

I'm using tox and coverage.py to run tests of my Python project in my continuous build server. I also have a package pkg_x from a vendor (not available on PyPI) that I've installed using python3.5 setup.py install, which puts it in /usr/lib/python3.5/site-packages. Now I need to make that package available to the testing code.

My current tox.ini looks like this:

[tox]
envlist = py35

[testenv]
deps = nose
       coverage
commands = coverage run -m nose []
sitepackages = True

and I run the tests like so:

python3.5 -m tox -- --verbose --with-doctest

That fails spectacularly - none of the dependency packages listed in my local setup.py (e.g. public stuff like more_itertools) can be found, even though it does create directories like .tox/py35/lib/python3.5/site-packages/more_itertools that seem to contain the relevant packages. If I fire up .tox/py35/bin/python3.5, sys.path looks like this:

>>> [re.compile('.*\\.tox').sub('.tox', x) for x in sys.path]
['',
 '.tox/py35/lib64/python35.zip',
 '.tox/py35/lib64/python3.5', 
 '.tox/py35/lib64/python3.5/plat-linux',
 '.tox/py35/lib64/python3.5/lib-dynload',
 '/usr/lib64/python3.5',
 '/usr/lib/python3.5',
 '.tox/py35/lib/python3.5/site-packages']

If I remove the sitepackages = True line from my tox.ini, then I do get farther, in that packages like more_itertools and the rest of the stuff in my setup.py dependencies can now be found, but the vendor package pkg_x I mentioned above still can't be found. And sys.path looks like this:

>>> [re.compile('.*\\.tox').sub('.tox', x) for x in sys.path]
['',
 '.tox/py35/lib64/python35.zip',
 '.tox/py35/lib64/python3.5', 
 '.tox/py35/lib64/python3.5/plat-linux',
 '.tox/py35/lib64/python3.5/lib-dynload',
 '/usr/lib64/python3.5',
 '/usr/lib/python3.5',
 '.tox/py35/lib/python3.5/site-packages',
 '/usr/lib64/python3.5/site-packages',
 '/usr/lib/python3.5/site-packages']

In neither case does .tox/py35/ seem to contain the vendor package pkg_x anywhere. And although the directory /usr/lib/python3.5/site-packages is listed when I fire up .tox/py35/bin/python3.5 manually, pkg_x isn't actually found when running the tests.

It also looks like sitepackages = True has the opposite effect from what it's documented to do at http://tox.readthedocs.io/en/latest/config.html#confval-sitepackages=True|False , right?

Advice very appreciated!

Answer 1

Tox creates a virtualenv, then runs it's tests from inside that environment.

The --sitepackages argument is a switch determining whether the virtualenv gets access to globally installed packages.

The 'normal' way to run tox would be to simply say tox; installing it via pip or via an OS package should put it into your path. ie:

$ tox

Which is the same as saying:

$ tox -c tox.ini

Where you invoke tox directly, python -m tox, it may do something, but this is a red flag to me. This command seems unlikely to activate the relevant virtual environment, which would explain your package availability woes. It fits because when you omit sitepackages, it actually adds the global packages because it thinks it is inside a virtualenv and so adds, what it thinks are the local sitepackages, although they're actually global. The reverse happens when you make it true because when it goes looking for global packages it can't find them. Whatever, because you're not invoking tox as expected it's as confused as we are.

So just use the tox command provided.

But wait there's more: You say you have a package which is required, but not available on pypi. So how is tox going to install it? The tox docs suggest a couple of ways (use a requirements.txt), but the most direct to describe here is to activate the env and install it manually.

Also good if you need to debug further: step into the .tox directory and activate the venv manually. eg:

$ source .tox/testenv/bin/activate

(where testenv is the name you used between brackets in the tox.ini).

Now install the package however you did it before eg: pip install pkg_x

Deactivate the venv when you're done with it this way:

$ deactivate

Try tox now?

If we're on the right track learn more about virtualenv here