Too much end-to-end testing? [closed]

Question

I am really frustrated by this whole situation, and here is why:

I inherited a completely untested legacy system for keeping many different client databases and one master database (with a different schema) in sync. The system was only partially finished when it was given to me, with many defects preventing it from working correctly ~90% of the time.

This system also has six different types of synchronizations allowed, each synchronizing different (sometimes overlapping) tables, since the databases can be fairly large, so the clients can prioritize the most important tables first depending on their state.

I started with a few end-to-end tests, setting up a master and several client databases locally with certain data then calling the different synchronization methods and verifying the right data showed up in the right databases in the right format.

I was pressed for time, and since this system has at least a hundred different ways that data can move from one database to another and is only a few thousand lines of code, I just kept making more and more end-to-end tests, basically 1-2 per defect that existed when I took over the project. I finished the system with 16 unit tests (TDD'd from the code I added) and 113 end-to-end tests, many of which were directly based off prior defects.

I finished the system, and it has been in production for several months without incident.

Recently, we decided to convert the client database to a new database, and when I ran my tests (which have been running nightly in a CI server this whole time) with the new database, about 100 of 113 failed. (The unit tests all pass, of course).

I have been fixing the failing end-to-end tests, and frankly, most failed for one or two simple reasons, (like the new database rounding dates differently) but I am frustrated by the fact that my tests were so brittle. While they correctly failed, I only needed one or two to tell me that, not 100. The problem is, there is not that much code to unit test, because most of it is just selecting data out of one table based on dates, then selecting the same data out of another database, merging the two, then inserting/updating appropriately.

There is no way I could have completed this system without these tests, but the pain of maintaining them is basically what leads me to this question: any suggestions how I should proceed/or what I could have done better? Did I waste too much time writing these end-to-end tests the first time? I read Working Effectively with Legacy Code, but I felt like there was not really a good answer in there for the sort of pain I was feeling, other than: "just refactor and write more unit tests" which I feel is not really much of an option do to the unique nature of this system being very little code and a lot of database conversion.

Answer 1

You can create database proxy classes, making sure they are the only classes that talk to the real database. Use dependency-injection to unit test all your logic code without talking to the actual database. Create as few end-to-end tests as possible to make sure the proxies can read/write to the database correctly.

End-to-end tests are usually fragile by nature. So create as few of them as possible, and if you need to create a lot you can create an abstraction layer for setting up fixtures and assertions. Duplication in test cases is just as difficult to maintain as duplication in code.

Working Effectively with Legacy Code is a good start, and I recommend xUnit Test Patterns, which is basically the bible of unit testing with lots of good advice including a section on testing with databases.

Edit: TDD is all about isolating logic. What I mean is control flow statements, regular expressions, algorithms, math, should be outside your proxies where you can unit test them easily. You had 113 tests which makes me suspect there is logic that can be extracted and unit tested.

If you are creating SQL commands, you can use the Builder pattern to verify the commands are created correctly, and if you need to change the SQL dialect, there is only one place to make the changes.

Making your code testable may mean you need to refactor somewhat aggressively. The hard part will be determining how much is worth it, based on the importance and longevity of the project.