Coverage analysis for Functional Tests

Question

In the project that I am working on, we have functional tests written over Selenium. The application undergoes functional changes with each feature release.

Is there a tool / mechanism that we can keep track of the gaps in automated functional tests, so that at least the manual testers can keep an eye for these areas?

Note: we are not doing FTDD, so the functional test coverage can be quite poor, even though we ensure high unit test coverage. We use NCover to check unit test coverage.

Answer 1

There are at least two (commercial but cheap) tools that I know of that allow you to attach to the IIS process to capture coverage data for IIS applications.

NCover:

NCover includes the //iis command line switch. This switch sets up the coverage environment within IIS and restarts the web server. You’ll run NCover like this to analyze coverage for your web applications:

NCover.Console.exe nunit-console.exe TestAssembly.dll //iis When you run NCover in this way, IIS will be restarted to allow NCover to monitor your coverage, and your tests will be run. Once finished, NCover will stop IIS and detach itself.

See: http://docs.ncover.com/how-to/code-coverage-of-asp-net-applications-on-iis/

DotCover by jetbrains:

Dotcover has visual studio integration which allows you to attach to an IIS application in the same way you would do if you'd want to trace your IIS application. This probably can also be started with the commandline dotCover tool although i've never actually tried this.

See http://www.jetbrains.com/dotcover/

I think Rational and Microsoft Teamsystem also have solutions but they're a bit more expensive.

Answer 2

We use a system for testing that involves a person creating a text narrative - a test script - for manual testing of functionality. This is enumerated in some fashion (e.g. [functionality]-001). Then our Selenium tests are noted as covering one or more of these enumerations.

When new functionality is constructed, new test scripts are written and enumerated. Then with Selenium testing we are able to compare its list of what's automated relative to the enumerated test scripts - the delta is what must be tested manually.