How to gauge the quality of a software product

Question

I have a product, X, which we deliver to a client, C every month, including bugfixes, enhancements, new development etc.) Each month, I am asked to err "guarantee" the quality of the product.

For this we use a number of statistics garnered from the tests that we do, such as:

• reopen rate (number of bugs reopened/number of corrected bugs tested)
• new bug rate (number of new, including regressions, bugs found during testing/number of corrected bugs tested)
• for each new enhancement, the new bug rate (the number of bugs found for this enhancement/number of mandays)

and various other figures.

It is impossible, for reasons we shan't go into, to test everything every time.

So, my question is:

How do I estimate the number and type of bugs that remain in my software? What testing strategies do I have to follow to make sure that the product is good?

I know this is a bit of an open question, but hey, I also know that there are no simple solutions.

Thanks.

Answer 1

I don't think you can ever really estimate the number of bugs in your app. Unless you use a language and process that allows formal proofs, you can never really be sure. Your time is probably better spent setting up processes to minimize bugs than trying to estimate how many you have.

One of the most important things you can do is have a good QA team and good work item tracking. You may not be able to do full regression testing every time, but if you have a list of the changes you've made to the app since the last release, then your QA people (or person) can focus their testing on the parts of the app that are expected to be affected.

Another thing that would be helpful is unit tests. The more of your codebase you have covered the more confident you can be that changes in one area didn't inadvertently affected another area. I've found this quite useful, as sometimes I'll change something and forget that it would affect another part of the app, and the unit tests showed the problem right away. Passed unit tests won't guarantee that you haven't broken anything, but they can help increase confidence that changes you make are working.

Also, this is a bit redundant and obvious, but make sure you have good bug tracking software. :)