Test Reporting

Question

We are migrating our test report data (unit, regression, integration, etc..) from an XML format to a database format for better analysis. Right now the majority of our test analysis is done using the CruiseControl.NET dashboard, but this is limited to primarily the most recent test data. Older test data can be accessed but not easily compared to new test data. We want to pin point problem components and better narrow down bugs. With the onset of tons of information brought on by our newly implemented regression and integration testing I would like to see some better metrics generated (possibly performance and the like). Have you worked with any business intelligence systems that will provide a framework for accurately and easily implementing some sort of analysis and reporting?

I have looked into JasperReports and Pentaho but I'm struggling with implemetation of Pentaho at the moment. Should I continue my fight with the system? Is this what I'm looking for?

Answer 1

You could always just use SQL Server Reporting Services and Report Builder (MS's web based designer) or Report Designer (component of Visual studio). It's pretty easy to get this set up too.

• Report Builder: http://msdn.microsoft.com/en-us/library/ms155933.aspx
• Report Designer: http://msdn.microsoft.com/en-us/library/ms157166.aspx
• Tutorial: http://www.simple-talk.com/sql/learn-sql-server/beginning-sql-server-2005-reporting-services-part-1/
• How to add Reporting Services to an existing SQL Server: http://www.mssqltips.com/tip.asp?tip=1444

There are a few end user reporting solutions around as well that make it easier to dynamically create reports, if you're willing to invest a bit of cash.

My company produce one: http://www.rsinteract.com has a very cheap standard edition with a limited number of reports (30 day free trial). It reports directly off SQL server with Reporting Services installed. It won best of TechEd 2006 - http://windowsitpro.com/article/articleid/53944/best-of-tech-ed-2006-winners.html

We actually use ours to analyse the support requests from clients i.e. which component is failing most, who reports the most bugs etc. Not tried it on test data.

There's also Proclarity, ApexSQL Report, and Tableau all of which are good.

Answer 2

You could try looking at rolling your own (if you know what you're looking for) using Processing written by Ben Fry. It's best accompanied by his book "Visualizing Data".

The tool is free and I guess you can get a free 45 day trial of O'Reilly Books Online to get a head start and see if its right for you. I do know there are chapters on reading and crunching data from all kinds of sources (including XML and databases) and then making meaningful and useful visualisations from them.

I'm currently using it to get my head round the dependency complexities of an inherited code base and its been massively useful.