Why is CPython not using `sphinx.autodoc` for the standard library? [closed]

Question

I am developing a python library and I am using sphinx.autodoc to generate the documentation as I think that this is a good way to don`t repeat yourself and to have documentation and code in agreement.

In a comment to Emit reStructuredText from sphinx autodoc? I learned that "The CPython docs build process doesn't have autodoc enabled (by deliberate choice)".

I am wondering why CPython is not using it and what are the disadvantages of using sphinx.autodoc?

Answer 1

It's mostly a matter of history, but also a question of personal (and project) preference. These days, you can get quite usable docs by relying primarily on docstrings and then maybe adding additional prose around them.

CPython's documentation however, long predates the existence of Sphinx (in fact, Georg Brandl wrote the initial version of Sphinx to replace CPython's old documentation system).

So, as a matter of policy, the docstrings and the prose documentation are still maintained separately without relying on the use of autodoc.

We also don't allow the use of reStucturedText docstrings in the standard library which further reduces the benefits of using autodoc. (See Q 10 in the PEP 287 Q & A: http://www.python.org/dev/peps/pep-0287/#questions-answers)

Finally, Georg Brandl pointed out that CPython is in a somewhat unique position that you need to be careful to ensure that the version of the standard library that is providing the docstrings when Sphinx runs is the exact same one you're generating the documentation for. It would be far too easy to accidentally bring in the wrong version, as well as creating a strong dependency between having a working Python build and being able to regenerate the documentation.

On the autodoc side, you do have the problem that when editing autodoc-based documentation, you can't easily see the docstring contents inline, so it can be difficult to make sure the docstring text and the additional prose read well together. That problem can be mitigated through an automatic browser refresh solution like http://pymolurus.blogspot.com.au/2012/01/documentation-viewer-for-sphinx.html

autodoc also has a problem with dependencies on rebuilds, since it doesn't automatically add the right dependencies on the Python source files. I've definitely had issues where the docstrings have changed but Sphinx didn't regenerate the relevant output files. (I don't believe this has been fixed, but if it's been resolved in more recent Sphinx releases let me know in the comments and I'll remove this observation).

While I think you get better documentation and better docstrings by maintaining them separately (because the two writing styles aren't exactly the same and raw docstrings are generally easier to read as plain text than they are when marked up with reStructuredText), it's an approach that comes with rather high costs in additional maintenance effort and an increased risk of inconsistency.

Accordingly, for most third party Python projects, my advice would actually be to avoid following the standard library's example and instead:

• use reRestructuredText docstrings (see PEP 287: http://www.python.org/dev/peps/pep-0287/)
• use apidoc/autodoc
• add additional prose documentation around the automatically embedded docstrings rather than as a substitute
• use an autoupdating approach such as that linked above to see how well the docstrings read as part of the documentation

While it's not a perfect solution, this approach does save quite a bit of duplicated effort in keeping both docstrings and prose documentation up to date.