What do you need from a test harness?

Question

I'm one of the people involved in the Test Anything Protocol (TAP) IETF group (if interested, feel free to join the mailing list). Many programming languages are starting to adopt TAP as their primary testing protocol and they want more from it than what we currently offer. As a result, we'd like to get feedback from people who have a background in xUnit, TestNG or any other testing framework/methodology.

Basically, aside from a simple pass/fail, what information do you need from a test harness? Just to give you some examples:

• Filename and line number (if applicable)
• Start and end time
• Diagnostic output such as the difference between what you got and what you expected.

And so on ...

Answer 1

Most definitely all things from your list for each individual item:

• Filename
• Line number
• Namespace/class/function name
• Test coverage
• Start time and end time
• And/or total time (this would be more useful for me than the top two items)
• Diagnostic output such as the difference between what you got and what you expected.

From the top of my head not much else but for the group of tests I would like to know

• group name
• total execution time

Answer 2

It must be very, very easy to write a test, and equally easy to run them. That, to me, is the single most important feature of a testing harness. If someone has to fire up a GUI or jump through a bunch of hoops to write a test, they won't use it.