How to do a meaningful code-coverage analysis of my unit-tests?

Question

I manage the testing for a very large financial pricing system. Recently our HQ have insisted that we verify that every single part of our project has a meaningful test in place. At the very least they want a system which guarantees that when we change something we can spot unintentional changes to other sub-systems. Preferably they want something which validates the correctness of every component in our system.

That's obviously going to be quite a lot of work! It could take years, but for this kind of project it's worth it.

I need to find out which parts of our code are not covered by any of our unit-tests. If I knew which parts of my system were untested then I could set about developing new tests which would eventually approach towards my goal of complete test-coverage.

So how can I go about running this kind of analysis. What tools are available to me?

I use Python 2.4 on Windows 32bit XP

UPDATE0:

Just to clarify: We have a very comprehensive unit-test suite (plus a seperate and very comprehensive regtest suite which is outside the scope of this exercise). We also have a very stable continuous integration platform (built with Hudson) which is designed to split-up and run standard python unit-tests across our test facility: Approx 20 PCs built to the company spec.

The object of this exercise is to plug any gaps in our python unittest suite (only) suite so that every component has some degree of unittest coverage. Other developers will be taking responsibility for non Python components of the project (which are also outside of scope).

"Component" is intentionally vague: Sometime it will be a class, other time an entire module or assembly of modules. It might even refer to a single financial concept (e.g. a single type of financial option or a financial model used by many types of option). This cake can be cut in many ways.

"Meaningful" tests (to me) are ones which validate that the function does what the developer originally intended. We do not want to simply reproduce the regtests in pure python. Often the developer's intent is not immediatly obvious, hence the need to research and clarify anything which looks vague and then enshrine this knowledge in a unit-test which makes the original intent quite explicit.

Answer 1

For the code coverage alone, you could use coverage.py.

As for coverage.py vs figleaf:

figleaf differs from the gold standard of Python coverage tools ('coverage.py') in several ways. First and foremost, figleaf uses the same criterion for "interesting" lines of code as the sys.settrace function, which obviates some of the complexity in coverage.py (but does mean that your "loc" count goes down). Second, figleaf does not record code executed in the Python standard library, which results in a significant speedup. And third, the format in which the coverage format is saved is very simple and easy to work with.

You might want to use figleaf if you're recording coverage from multiple types of tests and need to aggregate the coverage in interesting ways, and/or control when coverage is recorded. coverage.py is a better choice for command-line execution, and its reporting is a fair bit nicer.

I guess both have their pros and cons.

Answer 2

First step would be writing meaningfull tests. If you'll be writing tests only meant to reach full coverage, you'll be counter-productive; it will probably mean you'll focus on unit's implementation details instead of it's expectations.

BTW, I'd use nose as unittest framework (http://somethingaboutorange.com/mrl/projects/nose/0.11.1/); it's plugin system is very good and leaves coverage option to you (--with-coverage for Ned's coverage, --with-figleaf for Titus one; support for coverage3 should be coming), and you can write plugisn for your own build system, too.