packaging with numpy and test suite

Question

Introduction

Disclaimer: I'm very new to python packaging with distutils. So far I've just stashed everything into modules, and packages manually and developed on top of that. I never wrote a setup.py file before.

I have a Fortran module that I want to use in my python code with numpy. I figured the best way to do that would be f2py, since it is included in numpy. To automate the build process I want to use distutils and the corresponding numpy enhancement, which includes convenience functions for f2py wrappers.

I do not understand how I should organize my files, and how to include my test suite.

What I want is the possibility to use ./setup.py for building, installing, and testing, and developing.

My directory structure looks as follows:

volterra
├── setup.py
└── volterra
    ├── __init__.py
    ├── integral.f90
    ├── test
    │   ├── __init__.py
    │   └── test_volterra.py
    └── volterra.f90

And the setup.py file contains this:

def configuration(parent_package='', top_path=None):
    from numpy.distutils.misc_util import Configuration
    config = Configuration('volterra', parent_package, top_path)
    config.add_extension('_volterra',
                         sources=['volterra/integral.f90', 'volterra/volterra.f90'])
    return config


if __name__ == '__main__':
    from numpy.distutils.core import setup
    setup(**configuration(top_path='').todict())

After running ./setup.py build I get.

build/lib.linux-x86_64-2.7/
└── volterra
    └── _volterra.so

Which includes neither the __init__.py file, nor the tests.

Questions

• Is it really necessary to add the path to every single source file of the extension? (I.e. volterra/integral.f90) Can't I give a parameter which says, look for stuff in volterra/? The top_path, and package_dir parameters didn't do the trick.
• Currently, the __init__.py file is not included in the build. Why is that?
• How can I run my tests in this setup?
• What's the best workflow for doing development in such an environment? I don't want to install my package for every single change I do. How do you do development in the source directory when you need to compile some extension modules?

Answer 1

Here is a setup.py that works for me:

# pkg - A fancy software package
# Copyright (C) 2013  author (email)
#
# This program is free software: you can redistribute it and/or modify
# it under the terms of the GNU General Public License as published by
# the Free Software Foundation, either version 3 of the License, or
# (at your option) any later version.
#
# This program is distributed in the hope that it will be useful,
# but WITHOUT ANY WARRANTY; without even the implied warranty of
# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
# GNU General Public License for more details.
#
# You should have received a copy of the GNU General Public License
# along with this program.  If not, see http://www.gnu.org/licenses/gpl.html.
"""pkg: a software suite for 

Hey look at me I'm a long description
But how long am I?

"""

from __future__ import division, print_function

#ideas for setup/f2py came from:
#    -numpy setup.py: https://github.com/numpy/numpy/blob/master/setup.py 2013-11-07
#    -winpython setup.py: http://code.google.com/p/winpython/source/browse/setup.py 2013-11-07
#    -needing to use 
#        import setuptools; from numpy.distutils.core import setup, Extension: 
#        http://comments.gmane.org/gmane.comp.python.f2py.user/707 2013-11-07
#    -wrapping FORTRAN code with f2py: http://www2-pcmdi.llnl.gov/cdat/tutorials/f2py-wrapping-fortran-code 2013-11-07
#    -numpy disutils: http://docs.scipy.org/doc/numpy/reference/distutils.html 2013-11-07
#    -manifest files in disutils: 
#        'distutils doesn't properly update MANIFEST. when the contents of directories change.'
#        https://github.com/numpy/numpy/blob/master/setup.py         
#    -if things are not woring try deleting build, sdist, egg directories  and try again: 
#        https://stackoverflow.com/a/9982133/2530083 2013-11-07
#    -getting fortran extensions to be installed in their appropriate sub package
#        i.e. "my_ext = Extension(name = 'my_pack._fortran', sources = ['my_pack/code.f90'])" 
#        Note that sources is a list even if one file: 
#        http://numpy-discussion.10968.n7.nabble.com/f2py-and-setup-py-how-can-I-specify-where-the-so-file-goes-tp34490p34497.html 2013-11-07
#    -install fortran source files into their appropriate sub-package 
#        i.e. "package_data={'': ['*.f95','*.f90']}# Note it's a dict and list":
#        https://stackoverflow.com/a/19373744/2530083 2013-11-07
#    -Chapter 9 Fortran Programming with NumPy Arrays: 
#        Langtangen, Hans Petter. 2013. Python Scripting for Computational Science. 3rd edition. Springer.
#    -Hitchhikers guide to packaging :
#        http://guide.python-distribute.org/
#    -Python Packaging: Hate, hate, hate everywhere : 
#        http://lucumr.pocoo.org/2012/6/22/hate-hate-hate-everywhere/
#    -How To Package Your Python Code: 
#        http://www.scotttorborg.com/python-packaging/
#    -install testing requirements: 
#        https://stackoverflow.com/a/7747140/2530083 2013-11-07

import setuptools
from numpy.distutils.core import setup, Extension
import os
import os.path as osp

def readme(filename='README.rst'):
    with open('README.rst') as f:
        text=f.read()
    f.close()
    return text

def get_package_data(name, extlist):
    """Return data files for package *name* with extensions in *extlist*"""
    #modified slightly from taken from http://code.google.com/p/winpython/source/browse/setup.py 2013-11-7
    flist = []
    # Workaround to replace os.path.relpath (not available until Python 2.6):
    offset = len(name)+len(os.pathsep)
    for dirpath, _dirnames, filenames in os.walk(name):
        for fname in filenames:            
            if not fname.startswith('.') and osp.splitext(fname)[1] in extlist:
#                flist.append(osp.join(dirpath, fname[offset:]))
                flist.append(osp.join(dirpath, fname))
    return flist

DOCLINES = __doc__.split("\n")
CLASSIFIERS = """\
Development Status :: 1 - Planning
License :: OSI Approved :: GNU Lesser General Public License v3 or later (LGPLv3+)
Programming Language :: Python :: 2.7
Topic :: Scientific/Engineering
"""

NAME = 'pkg'
MAINTAINER = "me"
MAINTAINER_EMAIL = "[email protected]"
DESCRIPTION = DOCLINES[0]
LONG_DESCRIPTION = "\n".join(DOCLINES[2:])#readme('readme.rst')
URL = "http://meeeee.mmemem"
DOWNLOAD_URL = "https://github.com/rtrwalker/geotecha.git"
LICENSE = 'GNU General Public License v3 or later (GPLv3+)'
CLASSIFIERS = [_f for _f in CLASSIFIERS.split('\n') if _f]
KEYWORDS=''
AUTHOR = "me"
AUTHOR_EMAIL = "me.com"
PLATFORMS = ["Windows"]#, "Linux", "Solaris", "Mac OS-X", "Unix"]
MAJOR = 0
MINOR = 1
MICRO = 0
ISRELEASED = False
VERSION = '%d.%d.%d' % (MAJOR, MINOR, MICRO)

INSTALL_REQUIRES=[]
ZIP_SAFE=False
TEST_SUITE='nose.collector'
TESTS_REQUIRE=['nose']

DATA_FILES = [(NAME, ['LICENSE.txt','README.rst'])]
PACKAGES=setuptools.find_packages()
PACKAGES.remove('tools')

PACKAGE_DATA={'': ['*.f95','*f90']}               
ext_files = get_package_data(NAME,['.f90', '.f95','.F90', '.F95'])
ext_module_names = ['.'.join(osp.splitext(v)[0].split(osp.sep)) for v in ext_files]
EXT_MODULES = [Extension(name=x,sources=[y]) for x, y in zip(ext_module_names, ext_files)]      


setup(
    name=NAME,
    version=VERSION,
    maintainer=MAINTAINER,
    maintainer_email=MAINTAINER_EMAIL,
    description=DESCRIPTION,
    long_description=LONG_DESCRIPTION,
    url=URL,
    download_url=DOWNLOAD_URL,
    license=LICENSE,
    classifiers=CLASSIFIERS,
    author=AUTHOR,
    author_email=AUTHOR_EMAIL,
    platforms=PLATFORMS,
    packages=PACKAGES,
    data_files=DATA_FILES,
    install_requires=INSTALL_REQUIRES,
    zip_safe=ZIP_SAFE,
    test_suite=TEST_SUITE,
    tests_require=TESTS_REQUIRE,
    package_data=PACKAGE_DATA,    
    ext_modules=EXT_MODULES,
    )

To install, at the command line I use:

python setup.py install
python setup.py clean --all

The only issue I seem to have is a minor one. when I look in site-packages for my package it is installed inside the egg folder C:\Python27\Lib\site-packages\pkg-0.1.0-py2.7-win32.egg\pkg. Most other packages I see there have a C:\Python27\Lib\site-packages\pkg folder separate to the egg folder. Does anyone know how to get that separation?

As for testing, after installing, I type the following at the command line:

nosetests package_name -v

Try investigating python setup.py develop (Python setup.py develop vs install) for not having to install the package after every change.

As I commented in the code I found the following useful:

• numpy setup.py: https://github.com/numpy/numpy/blob/master/setup.py 2013-11-07
• winpython setup.py: http://code.google.com/p/winpython/source/browse/setup.py 2013-11-07
• needing to use import setuptools; from numpy.distutils.core import setup, Extension: http://comments.gmane.org/gmane.comp.python.f2py.user/707 2013-11-07
• wrapping FORTRAN code with f2py: http://www2-pcmdi.llnl.gov/cdat/tutorials/f2py-wrapping-fortran-code 2013-11-07
• numpy disutils: http://docs.scipy.org/doc/numpy/reference/distutils.html 2013-11-07
• manifest files in disutils: 'distutils doesn't properly update MANIFEST. when the contents of directories change.' https://github.com/numpy/numpy/blob/master/setup.py
• if things are not woring try deleting build, sdist, egg directories and try again: https://stackoverflow.com/a/9982133/2530083 2013-11-07
• getting fortran extensions to be installed in their appropriate sub package i.e. "my_ext = Extension(name = 'my_pack._fortran', sources = ['my_pack/code.f90'])" Note that sources is a list even if one file: http://numpy-discussion.10968.n7.nabble.com/f2py-and-setup-py-how-can-I-specify-where-the-so-file-goes-tp34490p34497.html 2013-11-07
• install fortran source files into their appropriate sub-package i.e. "package_data={'': ['.f95','.f90']}# Note it's a dict and list": https://stackoverflow.com/a/19373744/2530083 2013-11-07
• Chapter 9 Fortran Programming with NumPy Arrays: Langtangen, Hans Petter. 2013. Python Scripting for Computational Science. 3rd edition. Springer.
• Hitchhikers guide to packaging : http://guide.python-distribute.org/
• Python Packaging: Hate, hate, hate everywhere : http://lucumr.pocoo.org/2012/6/22/hate-hate-hate-everywhere/
• How To Package Your Python Code: http://www.scotttorborg.com/python-packaging/
• install testing requirements: https://stackoverflow.com/a/7747140/2530083 2013-11-07
• 'python setup.py develop' : https://stackoverflow.com/a/19048754/2530083

Answer 2

Here is setup.py from a project I made. I have found figuring out setup.py / packaging to be frustrating with no solid answers and definitely not pythonic in the sense of having one and only one obvious way to do something. Hopefully this will help a little.

The points you may find useful are:

• find_packages which removes the drudgery of including lots of files or messing around with generating manifest.
• package_data which allows you to easily specify non .py files to be included
• install_requires / tests_require

You'll need to find the source for distribute_setup.py if you don't have it already.

• Is it really necessary to add the path to every single source file of the extension? (I.e. volterra/integral.f90) Can't I give a parameter which says, look for stuff in volterra/? The top_path, and package_dir parameters didn't do the trick.
• Currently, the init.py file is not included in the build. Why is that?

Hopefully find_packages() will solve both of those. I don't have much experience packaging but I haven't had to go back to manual inclusion yet.

• How can I run my tests in this setup?

I think this is probably a different question with many answers depending on how you are doing tests. Maybe you can ask it separately?

As a side note, I am under the impression that the standard is to put your tests directory at the top level. I.e. volterra/volterra and volterra/tests.

• What's the best workflow for doing development in such an environment? I don't want to install my package for every single change I do. How do you do development in the source directory when you need to compile some extension modules?

This might be worth another question as well. I don't see why you would need to install your package for every single change. If you are uploading the package, just don't install it on your dev system (except to test installation) and work directly from your development copy. Maybe I'm missing something though since I don't work with compiled extensions.

---

Here is the example

try:
    from setuptools import setup, find_packages
except ImportError:
    from distribute_setup import use_setuptools
    use_setuptools()
    from setuptools import setup, find_packages


setup(
    # ... other stuff
    py_modules=['distribute_setup'],
    packages=find_packages(),
    package_data={'': ['*.png']},  # for me to include anything with png
    install_requires=['numpy', 'treenode', 'investigators'],
    tests_require=['mock', 'numpy', 'treenode', 'investigators'],
)