How can I use Sphinx with subpackages without duplicating everything?

Question

I have the following package structure as a minimal example (for convenience, all is uploaded here):

.
├── sphinx
│   ├── build
│   ├── Makefile
│   └── source
│       ├── conf.py
│       ├── index.rst
│       └── train.rst
└── train
    ├── __init__.py
    └── train.py

When writing Python packages, one must specifiy the __all__ constant in the __init__.py of any package in order for Sphinx to be able to map a reference such as train.DatasetMeta to train.train.DatasetMeta or similar. However, sphinx-apidoc generates the following sections for these packages:

train package
=============

Submodules
----------

train.train module
------------------

.. automodule:: train.train
    :members:
    :undoc-members:
    :show-inheritance:


Module contents
---------------

.. automodule:: train
    :members:
    :undoc-members:
    :show-inheritance:

Which duplicates the entire documentation as it contains .. automodule:: module.file as well as .. automodule:: module, which refer to the same thing. Removing either of these sections results in undefined reference warnings (turned into errors when using -n to SPHINXOPTS).

sphinx_test/train/train.py:docstring of train.DatasetMeta:1:py:class reference target not found: train.train.DatasetMeta

How can I solve this?

train/train.py

from collections import namedtuple


class DatasetMeta(namedtuple('DatasetMeta', ['dataset', 'num_classes', 'shape'])):
    @property
    def size(self):
        '''int: Number of examples in the dataset'''
        return self.shape[0]

train/__init__.py

from .train import *

__all__ = ['DatasetMeta']

sphinx/source/conf.py

import os
import sys
sys.path.insert(0, os.path.abspath('.'))
sys.path.insert(0, os.path.abspath('../../'))


project = 'test'
copyright = ''
author = ''

version = ''
release = '0'

extensions = [
    'sphinx.ext.autodoc',
]

source_suffix = '.rst'
master_doc = 'index'

I just cannot figure out what the logic is here.

Answer 1

One thing we can do to make the situation simpler is a minor rename:

class DatasetMeta(namedtuple('DatasetMetaBase', ['dataset', 'num_classes', 'shape'])):

which should make it obvious that the missing reference is train.train.DatasetMetaBase when you remove the train.train block from the rst file generated by sphinx-apidoc. The documentation for train.DatasetMeta and train.train.DatasetMeta is going to refer to train.train.DatasetMetaBase; I don't know way to hack around that without patching autodoc or adding your own directives.

From here, I see a few options:

(1) Move DatasetMetaBase to a different module that is not imported in __init__.py. For example

from .abstract import DatasetMetaBase
class DatasetMeta(DatasetMetaBase):

That way the autodoc for DatasetMeta refers to train.abstract.DatasetMetaBase, which should be a unique ref in your case.

(2) Create a separate rst file (say, hidden.rst) that renders the docs for train.train.DatasetMetaBase, but hidden from the main rst.

# hidden.rst
.. autodata:: train.train.DatasetMetaBase

That should be enough to add train.train.DatasetMetaBase to sphinx and resolve the class reference target not found warning.