Why doesn't setuptools copy modules in a subfolder?

Question

I have a project called hwrt (see GitHub, PyPI) with the following structure:

.
├── bin
├── docs
├── hwrt
│   ├── datasets
│   │   ├── crohme_eval.py
│   │   ├── __init__.py
│   │   ├── inkml.py
│   │   ├── README.md
│   │   └── results.csv
│   ├── __init__.py
│   ├── misc: Not important for this question
│   ├── selfcheck.py
│   ├── serve.py
│   ├── ... (more Python modules)
│   ├── templates: Not important for this question
│   └── view.py
├── LICENSE
├── Makefile
├── MANIFEST.in
├── README.md
├── requirements.txt
├── setup.cfg
├── setup.py
└── tests: Not important for this question

My problem is that

$ python
>>> from hwrt.datasets import inkml
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
ImportError: No module named datasets

root/of/project$ python
>>> from hwrt.datasets import inkml
>>> 

Note that the datasets folder has __init__.py. So that is not the problem. One (the?) problem seems to be that setuptools does not copy the datasets folder. Do I have to do anything else than put a __init__.py in the folder?

setup.py

try:
    from setuptools import setup
except ImportError:
    from distutils.core import setup

config = {
    'name': 'hwrt',
    'version': '0.1.217',
    'author': 'Martin Thoma',
    'author_email': '[email protected]',
    'maintainer': 'Martin Thoma',
    'maintainer_email': '[email protected]',
    'packages': ['hwrt'],
    'scripts': ['bin/hwrt', 'bin/backup.py',
                'bin/test.py', 'bin/train.py',
                'bin/create_testset_online_once.py'],
    'package_data': {'hwrt': ['templates/*', 'misc/*']},
    'platforms': ['Linux', 'MacOS X', 'Windows'],
    'url': 'https://github.com/MartinThoma/hwrt',
    'license': 'MIT',
    'description': 'Handwriting Recognition Tools',
    'long_description': ("A tookit for handwriting recognition. It was "
                         "developed as part of the bachelors thesis of "
                         "Martin Thoma."),
    'install_requires': [
        "argparse",
        "theano",
        "nose",
        "natsort",
        "PyYAML",
        "matplotlib",
        "nntoolkit",
        "h5py",
        "flask",
        "flask-bootstrap",
        "requests",
        "six"
    ],
    'keywords': ['HWRT', 'recognition', 'handwriting', 'on-line'],
    'download_url': 'https://github.com/MartinThoma/hwrt',
    'classifiers': ['Development Status :: 3 - Alpha',
                    'Environment :: Console',
                    'Intended Audience :: Developers',
                    'Intended Audience :: Science/Research',
                    'License :: OSI Approved :: MIT License',
                    'Natural Language :: English',
                    'Programming Language :: Python :: 2.7',
                    'Programming Language :: Python :: 3',
                    'Programming Language :: Python :: 3.3',
                    'Programming Language :: Python :: 3.4',
                    'Topic :: Scientific/Engineering :: Artificial Intelligence',
                    'Topic :: Software Development',
                    'Topic :: Utilities'],
    'zip_safe': False,
    'test_suite': 'nose.collector'
}

setup(**config)

Answer 1

By default the "packages" keyword in setup() does not include all subpackages. It only looks for packages contained in the same folder as the setup.py. You can simply add "hwrt.datasets" to your packages list.

However you can run into trouble later on if you decide to add more packages to your project so the typical use case is to use the helper function find_packages.

e.g:

from setuptools import setup, find_packages

setup(
    # ...
    packages=find_packages(".")
)

See http://setuptools.readthedocs.io/en/latest/setuptools.html#using-find-packages