While doing TDD and adding features incrementally, do I design database ahead or add tables and columns while coding?

Question

One of the most important opportunity TDD gives us, from my point of view, is to develop projects incrementally, adding features one by one, which means ideally we have working system at every point in time.
What I am asking is, when the project involves working with a database, can we use this incremental approach for creating database structure or should we work the structure out before start writing code? I know it's hard to predict what the structure of database will be like in 1 year from now, but generally, what's the best practice on it?

Answer 1

The benefit of TDD and YAGNI is that it explicitly adresses the issue that we, as developers, can't predict future requirements. That is just as true for relational database design as it is for object-oriented code.

The database is an implementation detail. Its only purpose is to support the application by providing persistence services. If you don't know what your code is going to do three months from now, it would be illusory to think that you know what your database is going to look like.

Answer 2

For me, this is a question with a "theoretical" answer and a "real world" answer.

In theory, you add a column as and when you need it, and you refactor your database as you go, because that's agile.

In the real world, your DBAs will kill you if they have to rebuild your test data every five minutes because you've changed the schema again. And in a smaller project, you'll get personally sick of having to spend half your time maintaining an unstable database.

As skaffman alluded to in a comment: database maintenance is generally more expensive than code maintenance. This is doubly true for rollout: you can roll an entire new application without a hitch, but try planning a live database upgrade without breaking your data.

It's a difficult discussion, because agile purists will insist that everything should be done "just in time." But, as in most things agile, the reality is that someone needs to be looking ahead of the next release. Priorities do change, but if there's not at least a vague idea of what the product will look like in 6 months then you've got bigger problems than development methodology...

The role of an architect (or tech lead, or chief DBA, or whatever flavour you have) is to be looking ahead those few months and planning for what you are 90% sure is coming, and part of that will be defining the data you're going to need and where it's likely to live.

So, perhaps instead of adding a column at a time, add a table at a time. Find the balance that suits your project and your development process, without doubling your workload.