flake8/pylint fails in Tox testing environment, raises InvocationError

Question

I've been learning about how to do testing in tox for my python project.

I have (what should be) a fairly standard tox initialization file that looks like the following:

[tox]
envlist=py27,flake8
...
[testenv:flake8]
deps=flake8
commands=flake8 library # 'library' is temp. name of project

Everything looks normal, all the test works, and even the flake8 output comes through (output below). However, tox raises an InvocationError (it does the same for testing using pylint)

flake8 recreate: /Users/shostakovich/projects/project_templates/library/.tox/flake8
flake8 installdeps: flake8
flake8 inst: /Users/shostakovich/projects/project_templates/library/.tox/dist/library-0.1.0.zip
flake8 installed: flake8==2.4.1,library==0.1.0,mccabe==0.3,pep8==1.5.7,pyflakes==0.8.1,wheel==0.24.0
library/__main__.py:12:1: F401 'os' imported but unused
library/__main__.py:13:1: F401 're' imported but unused
...
ERROR: InvocationError: '/Users/shostakovich/projects/project_templates/library/.tox/flake8/bin/flake8 library'

I am running tox 2.0.2 on MaxOSX 10.9.5. The problem goes away if I just call flake8 or pylint directly (the version of flake8 is shown above).

Answer 1

tox doesn't fail, it works!

Your flake8 source code check has findings and therefore tox exits with failures, that's your test result. Fix the findings and your done!

You may configure the flake8 run to ignore specific codes with a section in your tox.ini. From the flake8 docs:

[flake8]
ignore = E226,E302,E41

There are more options you may be interested in, e.g. select = ... for whitelisting enabled code checks.

Answer 2

You can also tell flake8 to exit without failure, even if the test results are not perfect. This will suppress the misleading InvocationError. Just add to your command --exit-zero, so for example:

commands=flake8 library --exit-zero