Python packaging: Generate a python file at installation time, have this work with tox

Question

I want to generate a python file at installation time.

I want this work with both python setup.py develop, python setup.py install and pip install. So far so good.

However I also want this to work with tox. This is where I am having problems.

The approach I using is to tweak the develop and install commands to generate source code in setup.py like so:

# make code as python 3 compatible as possible
from __future__ import absolute_import, division, print_function, unicode_literals

import subprocess
import setuptools
import os.path
import distutils.core

from setuptools.command.develop import develop
from setuptools.command.install import install


# Build anltr files on installation
#   this is such a mess... it looks like there are
#   no common steps to develop and install

class AntlrDevelopCommand(develop):
    def run(self):
        compile_grammar()
        develop.run(self)

class AntlrInstallCommand(install):
    def run(self):
        compile_grammar()
        install.run(self)

def compile_grammar():
    here = os.path.dirname(__file__) or '.'
    package_dir = os.path.join(here, 'latex2sympy')
    subprocess.check_output(['antlr4',  'PS.g4', '-o', 'gen'], cwd=package_dir)

setuptools.setup(
    name='latex2sympy',
    version=0.1,
    author='august.codes',
    author_email='[email protected]',
    description='Parse latex markup into sympy: suitable for programmatic modifcation',
    license='GPLv3',
    keywords='MIT',
    url='',
    packages=['latex2sympy'],
    classifiers=[
],
    install_requires=['antlr-ast',  'sympy'],
    cmdclass=dict(
        install=AntlrInstallCommand,
        develop=AntlrDevelopCommand),
    test_suite='nose.collector'
)

However tox's method of installation seems to somehow run setup.py away from my source code and the magic blackbox that tox represents makes it kind of infuriating to work out what's going on.

The problem seems to be down to this voodoo magic which pulls in setup.py runs it via an exec.... for some reason.

Command "/home/tom/active/latex2sympy/.tox/py35/bin/python3.5 -u -c "import setuptools, tokenize;__file__='/tmp/pip-e698cucb-build/setup.py';f=getattr(tokenize, 'open', open)(__file__);code=f.read().replace('\r\n', '\n');f.close();exec(compile(code, __file__, 'exec'))" install --record /tmp/pip-lu2idbzz-record/install-record.txt --single-version-externally-managed --compile --install-headers /home/tom/active/latex2sympy/.tox/py35/include/site/python3.5/latex2sympy" failed with error code 1 in /tmp/pip-e698cucb-build/

Things I've tried:

• Running with -v -v -v -v
• Reruning the pip command manually
• Adding a pdb.set_trace (command hangs and I don't see output)
• Adding an ipython shell (ipython no installed even when in install_required)
• Running a strace -F this shows that setup.py is indeed in the expected location relative to the source code

Things I've considered trying:

• Creating a network backdoor shell at run time (too lazy)

Answer 1

In the tox.ini-file of your project, you can add commands to be run in the test environments. A simple example looks something like this:

[tox]
envlist = py27,py34,py35,py36

[testenv]
deps=
    pytest
    ; ... other dependencies
commands= 
    pytest --basetemp={envtmpdir} {posargs}
    ; Add your command here?

Is it possible for you to add a command to make tox do what you want? (The command will be run for each of the environments).