When to use test scripts over unit testing?

Question

I am currently working on a project that has been in production for over two years. The project makes extensive use of unit testing and scripted UI tests. Initialy unit tests covered the system framework, business rules and state transitions (or workflow). Test scripts are used for black box testing. However, over time the cost of maintaining our full set of unit tests has become increasingly expensive especially those relating to state.

After a bit of investigation we have found that test scripts are more effective (that is, provide better coverage) and are cheaper to maintain than unit tests relating to workflow. This isn't to say the value of unit tests has been completely negated but it does raise this question whether some classes of unit tests can be dropped in favour of test scripts.

Our project is run on an iterative incremental model.

Answer 1

In more ways than one I have experienced the same kind of pain that you have vis-a-vis unit tests, especially in projects wherein the members are not at all enthusiastic about unit testing and many of them simply ignore or comment-out tests to be able to cheat source control, save time, etc. One former colleague even coined the term "Deadline Driven Development" for this.

In my opinion when facing this kind of challenge, the following are some guidelines vis-a-vis unit testing:

• Discard obsolete tests - Sometimes it is pointless in trying to update hundreds to thousands of lines of tests if they are, in essence, inaccurate or irrelevant. Discard tests immediately. Do not "Ignore" them, do not comment them out. Delete them completely.
• Write tests for new functionality - Any new functionality still needs to be unit tested and written in a unit-testable manner.
• Write tests for bug fixes - When regression testing the application, it might be relevant to ensure that bug fixes have unit tests that ensure that the bug has been fixed.
• To hell with code coverage - This might earn a few downvotes, I'm sure, but there is a fine line between ensuring functionality and using tests as an excuse to delay work. The focus should be on ensuring core functionality than following any arbitrary code coverage percentage.

That being said, I still think that unit testing should not be discarded completely. Test scripts and unit tests have their own purposes. But a balance should be struck between an over-zealous attempt to maintain TDD and facing the realities of enterprise application development.

Answer 2

One of the answers on SO to the question of 'the limitations of Unit Testing' was that a unit testing becomes convoluted when it's used to test anything to do with INTEGRATION rather than function. Connecting to and using external services (database, SSH'ing to a another server, etc.) and User Interfaces were two of the examples used.

It's not that you CANT use Unit Testing for these things, its just that the difficulty involved in covering all the bases makes using this method of testing not worth it except in cases where reliability is paramount.

I use "script tests" for all my custom JavaScript UI code (template engine, effects and animations, etc.) and I find it quick and reliable if done right.

Answer 3

You normally use Unit Tests to do exactly this: test units. More exactly, to test if a unit complies to its interface specification/contract. If a unit test fails, you know exactly where the problem is: it is within the tested unit. This makes it easier to debug, especially since the result of a unit test can be processed automatically. Automatic regression tests come to mind here.

You start using scripted UI tests if you either want to leave the scope of unit tests or want to test things that cannot be tested well with unit tests. E.g. when you test code that interfaces with lots of external APIs that you cannot mock. Now you can provoke certain errors but tracking down where exactly in the code the failure is buried is much harder.