In Django, when should I use doctests instead of unit testing?

Question

From Django docs:

...the database is not refreshed between doctests, so if your doctest requires a certain state you should consider flushing the database or loading a fixture.

Quite frankly, 90% of my testing is currently done in doctests. A colleague of mine thought that was peculiar. In all honesty, I do very little testing so I do not pretend to be a master of that field.

Does anyone have a rule of thumb they use when deciding how to test?

Non-SO answer

A colleague of mine suggested to test model functions and constraints as doctests and views via unit tests. How does that sound for a rule of thumb?

Answer 1

As your project grows, you'll find that unittests are far better for testing your code.

The Django project itself is in the process of converting all doctests to unittests (we'll be done by the 1.3 release). The reason we're doing this is that prior to this conversion, the order of execution in the test suite would sometimes cause hard to reproduce errors. Sometimes a bit of code would accidentally depend on previously run doctest code. Additionally, switching to unittests has speed the overall test time up, since we can be more judicious about how and when we clear the database.

The other advantage unittests have is that they are MUCH easier to maintain. Because the entire test case is self-contained, you either write another test case, or modify the small, targeted test function to suit.

Doctests tend to work by evolution - you get an instance of your widget, add green fur, make sure the fur is green, add 4 legs, make sure you have 4 legs and green fur, add a thumb, make sure you have a thumb, 4 legs, and green fur, etc... This means that if you want to add a test right after the green fur stage, you have to modify the expected results for every other test case following.

You don't want to do all this rewriting, so you add your new test at the end. Then you add another, and then after a while your tests are so hopelessly jumbled you can't figure out whether or not a specific feature is even tested! With unittests, since each test embodies a specific, concrete and limited idea, it's much easier to re-order the tests logically, and to add a new test that doesn't depend on all the previous ones. Additionally, if you change the way add_green_fur() works, you don't have to modify dozens of test case results.

The other advantage is that unittests (when written well) tell you precisely where your code failed. Failed: MyWidget.tests.test_green_fur() is a lot easier to debug than "widget test failed at line 384", which is often dozens to hundreds of lines away from the actual point of failure.

In general, unittests are the better way to test.

Edit:

In response to your colleague's idea, I respectfully suggest that he hasn't worked on a large project with many doctests. Doctests in models are just as bad as in views. They have precisely the same problems (though if anything, doctests are worse in models because flush is very expensive and absolutely necessary for thorough doctesting). Don't underestimate the cost of the time taken by running tests.

Also, don't mix your test types unless you have a VERY good reason to do so. If you do, you'll very quickly find yourself doubling up tests, or assuming that a function is tested in whichever test suite you don't happen to be looking at.

Doctests are often touted as "providing documentation" for how your code is supposed to work. That's nice, but it's not a substitute for writing readable code with good legible inline comments. If you want further documentation, write it out separately!

You can't write good tests that also function as good documentation.

Answer 2

Doctests are great for making sure your documentation is up-to-date, but I wouldn't really use them to test code. It's really easy for your documentation to become out of date as you make changes to your code.

In short, use unit tests to test code and doctests to test documentation.