Philisophical Questions about Test-Driven Development

Question

I have been perpetually intrigued by test-driven development, but I can never follow through with it when I try it on real projects. I have a couple of philosophical questions that continually arise when I try it:

1. How do you handle large changes? When it comes to testing single functions (some parameters, a result value, few side effects), TDD is a no-brainer. But what about when you need to thoroughly overhaul something large, e.g. switching from a SAX parsing library to a DOM parsing library? How do you keep to the test-code-refactor cycle when your code is in an intermediate state? Once you start making the change , you will get a bevy of failed tests until you've fully finished the overhaul (unless you maintain some kind of mongrel class that uses both DOM and SAX until you're done converting, but that's pretty weird). What happens to the small-step test-code-refactor cycle in this case? During this whole process you will no longer be moving in small, fully-tested steps. There must be some way people deal with this.
2. When testing GUI or database code with mocks, what are you really testing? Mocks are built to return exactly the answer you want, so how do you know that your code will work with the real-world database? What is the benefit of automated tests for this kind of thing? It improves confidence somewhat, but a) it doesn't give you the same level of confidence that a complete unit test ought to, and b) to a certain extent, aren't you simply verifying that your assumptions work with your code rather than that your code works with the DB or GUI?

Can anyone point me to good case studies on using test-driven development in large projects? It's frustrating that I can basically only find TDD examples for single classes.

Thanks!

Answer 1

How do you handle large changes?

As small as needed.

Sometimes refactorings have a big surface but are trivial in detail. These can be done in quite big steps. Putting too much effort in trying to break them down will be waste.

I would argue that a XML library change is in this category. You're putting XML in and get some representation out. As long as your representation does not change (from a graph representing the state to a event stream) the library switch is easy.

Most of the time refactorings are not trivial and have to be broken down. The problem is when to do bigger steps and when the smaller ones. My observation is that I'm quite bad at estimating the impact of a change. Most software is complicated enough that you may think they change is easily manageable but then there is the whole fine print that has to work again. So I do start with an amount of change. But I'm prepared to rollback everything if it starts to get unpredictable. I would say this happens in one out of ten refactorings. But this one will be hard. You have to track down the part of the system that does not behave as you expect. The problem has to be split now in multiple smaller problems. I do solve one problem at a time and check in when it's done. (Multiple iterations of revert and splitting are not uncommon.)

If you change the XML parser and representation in your code this should be definitely be at least two separate refactorings.

Mock testing

You're testing a communication protocol between objects/layers with mock objects.

The whole mock approach can be though of as a communication model like the OSI model. When layer X gets as call with parameter x it will call layer Z with parameters a and b. Your test specifies this communication protocol.

As useful as mock test can be, test as few functionality with them as possible. The best option are state-based tests: setup fixture, call system under test, check state of system under test and pure functional (as in functional programming) tests: call with x returns a.

Try to design your system in a way that most of its functionality is loosely coupled. Some of the functionality has to be tested with mock tests (a fully decoupled system is useless).

Integration tests are no option to test your system. They should only be used to test aspects of the system that can break with the integration of multiple units. If you try to test your system with integration tests you'll enter into the permutation casino.

So your strategy for GUI testing should be clear. The parts of the GUI code that cannot be tested in isolation should be tested with a mock tests (when this button is pressed service X is called with parameter y).

Databases muddy the water a bit. You cannot mock a database, unless you're going to reimplement the behavior of every database you would like to support. But this is not a unit test as you're integrating an external system. I've made peace with this conceptional problem and think of the DAO and database as one inseparable unit that can be tested with a state-based test approach. (Sadly, this unit behaves differently when it has its oracle day compared to its mysql day. And it may break in the middle and tell you that it cannot talk to itself.)

Answer 2

When testing GUI or database code with mocks, what are you really testing? Mocks are built to return exactly the answer you want, so how do you know that your code will work with the real-world database? What is the benefit of automated tests for this kind of thing? It improves confidence somewhat, but a) it doesn't give you the same level of confidence that a complete unit test ought to, and b) to a certain extent, aren't you simply verifying that your assumptions work with your code rather than that your code works with the DB or GUI?

This is my approach: For database access layer (DAL), I don't use mock for my unit test. Instead, I run the tests on a real database, albeit a different one than the production database. So in this sense you can say that I don't run unit test on database. For NHibernate applications, I maintain two databases with same schema, but different database type (ORM makes this easy). I use sqlite for my automated testing, and a real MySQL or SQL server database for ad-hoc testing.

Only once did I use mock for unit testing the DAL; and that's when I was using strongly typed dataset as the ORM ( a big mistake!). The way I did this was to have Typemock returned me a mocked copy of the complete table so that I can perform select * on it. Later as I looked back I wished I never do this, but that was long time ago, and I wished I used a proper ORM.

As for the GUI, it is possible to unit test the GUI interaction. The way I did this was to use the MVP pattern to separate out the Model, View and Presenter. Actually for this type of application I only test on the Presenter and the Model, in which I use Typemock ( or dependency injection) to isolate the different layers so that at one time I can concentrate on only one layer. I don't test the view, but I do test Presenter ( where the majority of interaction and bugs are happening) a lot .