Optimizing regression testing in a C++ environnement

Question

In order to avoid too much testing, I would like to provide the Quality Assurance (QA) team with hints on which features have to be regression tested after a development iteration. Do you know tools that could do that on a C++ and Subversion (and visual studio) dev environment ?

Details about the use case:

1. Features would be defined by the development team in terms of entry points, typically classes or class methods. Say, feature "excel file import" is defined by method ImportExcelFile(...) of class FileImporter.
2. During the development iteration, the development team commits some changes on some methods of some classes. Say, one of these classes is indirectly used by method ImportExcelFile()
3. At the end of the iteration, all commits are analysed by the tool and a report is produced and delivered to the QA team. In our example, the QA team is informed that feature "excel file import" must be tested, and that other features X Y & Z are unchanged.

Very probably this tool would use static code analysis and consume subversion APIs. But does it exist ?

Answer 1

G'day,

What you are describing isn't really regression testing. You're just testing new features.

Regression testing is where you specifically run your complete test suite to see if the code supporting your new feature has broken previously working code.

I'd highly recommend reading Martin Fowler's excellent paper "Continuous Integration" which covers some of the aspects you are talking about.

It may also provide you with a better way of working, specifically the CI aspects Martin talks about in his paper.

Edit: Especially because CI has some hidden little traps that are obvious in hindsight. Such things as stopping testers trying to test a version that has not had all the files implementing a new feature committed yet. (You verify that there have been no commits in the last five minutes).

Another big point is the loss of time if you have a broken build and aren't aware that it is broken until someone checks out the code and then tries to build it so that they can test it.

If it's broken, you now have:

• a tester sitting around unable to do the scheduled tests,
• a developer interrupting their current work to go back to previous work to sort out what's causing the broken build. More probably it is developers because the problem is an interaction between two separate pieces, each of which worked on their own.
• time loss due to the developer(s) having to get back into the mindset for that previous piece of work, and
• time loss for the developer to get back into the mindset of the new piece of work that they were working on before the interruption to investigate.

The basic idea of CI is to do several builds of the complete product during the day so that you trap a broken build as early as possible. You may even select a few tests to check that the basic functionality of your product is still working. Once again to notify as soon as possible that there is a problem with the current state of your build.

Edit: As for your question, what about tagging the repository when you've done your testing, e.g. TESTS_COMPLETE_2009_12_16. Then when you're ready to work out what the next set of tests do an "svn diff -r" between that latest tests finished tag and HEAD?

HTH

BTW I'll update this answer with some further suggestions as I think of them.

cheers,