Logo Questions Linux Laravel Mysql Ubuntu Git Menu
 

Should I include tests and .pyc files when building package with setuptools?

I'm trying to install a local package using pip install . in a conda environment. Everything looks okay, except that when I run this command, there are .pyc files being included by default (I'm guessing this is due to the build process). Explicitly excluding them in MANIFEST.in doesn't help. First of all, is this a problem? The setup.py is actually very simple. Mainly, the only relevant thing is include_package_data = True that I need in order to include some static files.

setup.py

from setuptools import setup, find_packages
setup(
...
    include_package_data = True,
...
)

and this is the MANIFEST.in

MANIFEST.in

recursive-include <my-project>/static *
global-exclude *.pyc
global-exclude .git

Another question: Should I include my test folder (recursive-include <my-project>/test/ *) in the installed package? I have some tests using nose and other client-side tests which are included as static files. What is the recommended way to deal with tests in this case?

like image 935
Robert Smith Avatar asked Sep 04 '15 05:09

Robert Smith


People also ask

Does PIP require setuptools?

A Python file that relies only on the standard library can be redistributed and reused without the need to use setuptools. But for projects that consist of multiple files, need additional libraries, or need a specific version of Python, setuptools will be required.

What does setuptools setup do?

setuptools allows you to install a package without copying any files to your interpreter directory (e.g. the site-packages directory). This allows you to modify your source code and have the changes take effect without you having to rebuild and reinstall.

Is setuptools installed by default with Python?

Usually, Yes. the setuptools is not part of the python vanilla codebase, hence not a vanilla modules. python.org installers or mac homebrew will install it for you, but if someone compile the python by himself or install it on some linux distribution he may not get it and will need to install it by himself.

What is setuptools package in Python?

Setuptools is a collection of enhancements to the Python distutils that allow developers to more easily build and distribute Python packages, especially ones that have dependencies on other packages. Packages built and distributed using setuptools look to the user like ordinary Python packages based on the distutils .


1 Answers

The discussion is getting long, so I'll just write up my thoughts here.

Everything looks okay, except that when I run this command, there are .pyc files being included by default (I'm guessing this is due to the build process).

Python sources are precompiled on installation, and .pyc files are put along with the .py files in the installation location by setuptools. It's a normal behavior.

Explicitly excluding them in MANIFEST.in doesn't help.

MANIFEST.in governs distribution, not installation. That is, it tells setuptools which files to include in the archive when python setup.py sdist is run. It is commonly used to include documentation, tests, license and other accompanying files that are part of the package, but are not actually installed. The files that are installed (in addition to .py files) are specified in the package_data keyword argument to setup().

Another question: Should I include my test folder (recursive-include /test/ *) in the installed package?

There are two approaches possible:

  1. Include tests in the distribution, but not install them. To do this, you put them in a separate folder and include all required files (including the .py ones) explicitly in the MANIFEST.in. As an example, see this excerpt from the manifest of one of my projects:

    recursive-include test *.py
    include test/pylintrc
    include test/.coveragerc
    
  2. Include tests in the package itself, so that you can run them as import package; package.test() (for instance, numpy does that). No need to tinker with the MANIFEST.in, just put them in the main package directory (as a subpackage or whatever) and treat them as a regular subpackage/module.

Edit: answering the additional questions:

What about the option test_suite (e.g. nose.readthedocs.org/en/latest/setuptools_integration.html)?

I haven't used this particular option (maybe I should), but I have a similar setup in another project, which also makes python setup.py test available — used by Travis.CI for automatic test runs. Since I'm a fan of py.test, it requires some interfacing; for nose it would look simpler.

In general, if you can do it, do it, it's a good thing to have (for continuous integration in particular).

Does including tests as a subpackage cause issues?

Well, numpy and many other packages are doing fine. The only downsides that I see are:

  • the size of the installation is increased (but in our age of terabyte HDDs it is probably not that important)
  • tests are somewhat harder to run from the command line
  • you don't have a clear separation between the package itself and the tests (very arguable)

In conclusion, I don't have anything against it and may very well use this kind of setup for my next project.

The Python Packaging User Guide includes an example that does this: packages=find_packages(exclude=['docs', 'tests*'])

This is done to include all Python packages for installation, but skip the folders with .py files that are not supposed to be installed as packages. You can either do that, or supply the list of packages manually. It's just an example of how to implement the scenario 1 from my answer above. In the scenario 2 your tests folder will be inside your main package folder, and will be picked up by find_packages() normally.

like image 157
fjarri Avatar answered Sep 30 '22 12:09

fjarri