Unit testing infrastructure for a python module

Question

I'm writing a python module and I would like to unit test it. I am new to python and somewhat bamboozled by the options available.

Currently, I would like to write my tests as doctests as I like the declarative rather than imperative style (however, feel free to disabuse me of this preference if it is misinformed). This raises a few questions, however:

1. Where should I put the tests? In the same file as the code they are testing (or in docstrings for doctests)? Or is it considered better to separate them out into their own directory?
2. How can I run all the tests in the whole module from the command-line in one go?
3. How can I report the code coverage of the test suite?
4. Any other best-practices I should be aware of for unit testing in python?

Answer 1

feel free to disabuse me of this preference if it is misinformed

I believe I used doctest more extensively (way stretching its intended use boundaries) than any other open source developer, at least within a single project -- all the tests in my gmpy project are doctests. It was brand new at the time gmpy was starting, it seemed a great little trick, and if something is worth doing it's worth doing in excess -- right?-)

Wrong. Except for gmpy, where redoing everything as proper unit tests would be too much rework, I've never made that mistake again: these days, I use unit tests as unit tests, and doctests just to check my docs, as they've always been meant to be used. What doctests do (compare an expected with an actual result for equality -- that's all) is just not a good or sound basis to build a solid test suite on. It was never intended otherwise.

I would recommend you look at nose. The unittest module in the new Python 2.7 is much richer and nicer, and if you're stuck on 2.4, 2.5 or 2.6 you can still use the new features with the unittest2 which you can download and install; nose complements unittest quite well.

If you can't stand unittest (but -- give it a try, it grows on you!-), maybe try py.test, an alternative package with a pretty different philosophy.

But, please, don't stretch doctest to test stuff other than examples in docs! The exact-equality comparison will stand in your way far too often, as I've had to learn at my (metaphorical;-) expense in gmpy...

Answer 2

I don't like doctests for these reasons:

• You can't run a subset of the tests. When a test fails, it's useful to run just one test. Doctest provides no way to do that.
• If a failure happens in the middle of the doctest, the whole thing stops. I'd rather see all the results to decide how to tackle a breakage.
• The coding style is stylized, and has to have printable results.
• Your code is executed in a special way, so it's harder to reason about how it will be executed, harder to add helpers, and harder to program around the tests.

This list was taken from my blog post Things I don't like about doctest, where there's more, and a long thread of comments debating the points.

About coverage: I don't believe there's a coverage tool for Python that will measure coverage within doctests. But since they are simply long lists of statements with no branches or loops, is that a problem?