Getting tox to use the Python version set by pyenv

Question

I can't seem to wrap my head around managing Python versions. When I run tox, I can immediately see that it's using Python 3.7.9:

$ tox
py39: commands[0]> coverage run -m pytest
================================================================================== test session starts ==================================================================================
platform darwin -- Python 3.7.9, pytest-6.2.5, py-1.11.0, pluggy-1.0.0 -- /usr/local/bin/python3

But it's configured to use 3.9:

[tox]
envlist = py39,manifest,check-formatting,lint
skipsdist = True
usedevelop = True
indexserver =
    spotify = https://artifactory.spotify.net/artifactory/api/pypi/pypi/simple

[testenv]
basepython = python3.9
deps =
    :spotify:-r{toxinidir}/dev-requirements.txt
commands =
    coverage run -m pytest {posargs}
allowlist_externals = coverage

[testenv:manifest]
; a safety check for source distributions
basepython = python3.9
deps = check-manifest
skip_install = true
commands = check-manifest

Here's what I see with which:

$ pyenv local 3.9.10
$ which python
/Users/acheong/.pyenv/shims/python
$ which python3
/Library/Frameworks/Python.framework/Versions/3.7/bin/python3
$ pyenv which python
/Users/acheong/.pyenv/versions/3.9.10/bin/python

pytest also uses the wrong version:

$ pytest tests
================================================================================== test session starts ==================================================================================
platform darwin -- Python 3.7.9, pytest-6.2.5, py-1.11.0, pluggy-1.0.0 -- /Library/Frameworks/Python.framework/Versions/3.7/bin/python3
cachedir: .pytest_cache
rootdir: /Users/acheong/src/spotify/protean/ezmode-cli, configfile: tox.ini, testpaths: tests
plugins: mock-3.10.0, cov-2.10.0

But in this case I learned I can do this:

$ pyenv exec pytest tests
================================================================================== test session starts ==================================================================================
platform darwin -- Python 3.9.10, pytest-6.2.5, py-1.11.0, pluggy-1.0.0 -- /Users/acheong/.pyenv/versions/3.9.10/bin/python
cachedir: .pytest_cache
rootdir: /Users/acheong/src/spotify/protean/ezmode-cli, configfile: tox.ini, testpaths: tests
plugins: mock-3.10.0, cov-2.10.0

But when I try that with tox, I get an error:

$ pyenv exec tox
Traceback (most recent call last):
  File "/Users/acheong/.pyenv/versions/3.9.10/bin/tox", line 8, in <module>
    sys.exit(run())
  File "/Users/acheong/.pyenv/versions/3.9.10/lib/python3.9/site-packages/tox/run.py", line 19, in run
    result = main(sys.argv[1:] if args is None else args)
  File "/Users/acheong/.pyenv/versions/3.9.10/lib/python3.9/site-packages/tox/run.py", line 38, in main
    state = setup_state(args)
  File "/Users/acheong/.pyenv/versions/3.9.10/lib/python3.9/site-packages/tox/run.py", line 53, in setup_state
    options = get_options(*args)
  File "/Users/acheong/.pyenv/versions/3.9.10/lib/python3.9/site-packages/tox/config/cli/parse.py", line 38, in get_options
    guess_verbosity, log_handler, source = _get_base(args)
  File "/Users/acheong/.pyenv/versions/3.9.10/lib/python3.9/site-packages/tox/config/cli/parse.py", line 61, in _get_base
    MANAGER.load_plugins(source.path)
  File "/Users/acheong/.pyenv/versions/3.9.10/lib/python3.9/site-packages/tox/plugin/manager.py", line 90, in load_plugins
    self._register_plugins(inline)
  File "/Users/acheong/.pyenv/versions/3.9.10/lib/python3.9/site-packages/tox/plugin/manager.py", line 38, in _register_plugins
    self.manager.load_setuptools_entrypoints(NAME)
  File "/Users/acheong/.pyenv/versions/3.9.10/lib/python3.9/site-packages/pluggy/_manager.py", line 287, in load_setuptools_entrypoints
    plugin = ep.load()
  File "/Users/acheong/.pyenv/versions/3.9.10/lib/python3.9/importlib/metadata.py", line 77, in load
    module = import_module(match.group('module'))
  File "/Users/acheong/.pyenv/versions/3.9.10/lib/python3.9/importlib/__init__.py", line 127, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "<frozen importlib._bootstrap>", line 1030, in _gcd_import
  File "<frozen importlib._bootstrap>", line 1007, in _find_and_load
  File "<frozen importlib._bootstrap>", line 986, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 680, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 850, in exec_module
  File "<frozen importlib._bootstrap>", line 228, in _call_with_frames_removed
  File "/Users/acheong/.pyenv/versions/3.9.10/lib/python3.9/site-packages/tox_pyenv.py", line 48, in <module>
    from tox import hookimpl as tox_hookimpl
ImportError: cannot import name 'hookimpl' from 'tox' (/Users/acheong/.pyenv/versions/3.9.10/lib/python3.9/site-packages/tox/__init__.py)

I've tried a lot of things I found online but I'm afraid if anything I've only messed up my environment even more. What steps can I take to diagnose the problem and get tox to use my pyenv version?

Answer 1

Although this is an old question and it has already been marked as solved, I dare to give another answer since this post appears on the first page of Google search results of the "tox pyenv" query.

As already noted, tox-pyenv is not compatible with tox version 4. Moreover, tox no longer discovers Python executables by itself, this job is now delegated to virtualenv. That being said, a special tox plugin like tox-pyenv is no longer needed, the discovery machinery is extended via virtualenv plugins, not tox plugins. Here is an explanation in the tox-pyenv issues, and here is a more detailed migration guide.

For those who don't want to configure virtualenv and are looking for a simpler solution, there is another tox plugin — tox-pyenv-redux (not related to tox-pyenv). It is compatible with tox 4, uses virtualenv-pyenv under the hood, and does not require any configuration to work (though, it has a configuration setting).

Answer 2

Oh, turns out I was so close. Apparently Tox 4 broke tox-pyenv. Once I reverted to Tox 3, it worked.

$ pyenv exec python -m pip install "tox<4"
...
Installing collected packages: tox
  Attempting uninstall: tox
    Found existing installation: tox 4.4.6
    Uninstalling tox-4.4.6:
      Successfully uninstalled tox-4.4.6
Successfully installed tox-3.28.0
$ pyenv exec tox
py39 create: /Users/acheong/src/spotify/protean/ezmode-cli/.tox/py39
py39 installdeps: :spotify:-r/Users/acheong/src/spotify/protean/ezmode-cli/dev-requirements.txt
py39 installed: attrs==22.2.0,bump2version==1.0.1,bumpversion==0.6.0,CacheControl==0.12.11,cachetools==5.3.0,certifi==2022.12.7,chardet==4.0.0,click==7.1.2,colorama==0.4.6,coverage==5.2,event-definitions-protos-python==0.6.156,-e git+ssh://[email protected]/protean/ezmode-cli.git@32fc2a748278862efd15e431363d4aa5c95916e4#egg=ezmode_cli,ezmode-common==0.2.5,gabito-python==2.2.0,ghp-import==2.1.0,google-auth==2.16.2,grpcio==1.51.3,idna==2.10,importlib-metadata==5.0.0,iniconfig==2.0.0,interrogate==1.5.0,Jinja2==3.1.2,Markdown==3.3.7,MarkupSafe==2.1.2,mergedeep==1.3.4,mkdocs==1.4.2,msgpack==1.0.5,packaging==23.0,pluggy==1.0.0,protobuf==4.22.1,py==1.11.0,pyasn1==0.4.8,pyasn1-modules==0.2.8,pytest==6.2.5,pytest-cov==2.10.0,pytest-mock==3.10.0,python-dateutil==2.8.2,PyYAML==5.1,pyyaml_env_tag==0.1,requests==2.25.1,rsa==4.9,six==1.16.0,spotify-python-metrics==0.0.26,spotify-service-auth==0.3.9,tabulate==0.9.0,techdocs-cli==0.16.3,toml==0.10.2,tqdm==4.65.0,urllib3==1.26.15,watchdog==2.3.1,zipp==3.15.0
py39 run-test-pre: PYTHONHASHSEED='658999387'
py39 run-test: commands[0] | coverage run -m pytest
================================================================================== test session starts ==================================================================================
platform darwin -- Python 3.9.10, pytest-6.2.5, py-1.11.0, pluggy-1.0.0 -- /Users/acheong/src/spotify/protean/ezmode-cli/.tox/py39/bin/python
cachedir: .tox/py39/.pytest_cache
rootdir: /Users/acheong/src/spotify/protean/ezmode-cli, configfile: tox.ini, testpaths: tests
plugins: mock-3.10.0, cov-2.10.0