Best way to measure execution time in automated regression tests

Question

I have some code that I want to measure the speed of while it runs continuously in automated regression tests. The purpose of this would be to alert me to changes made to the code which have had a negative impact on performance.

In pseudo-code, I want something like this:

cpuTimer.start
runTest
cpuTimer.stop

diff = cpuTimer.getDuration

if diff > prevDiff // Perhaps to within a tolerance
    failTest

I'm looking at ThreadMXBean#getCurrentThreadCpuTime() for this, but the key problem is that the automated tests will be run on a wide range of different developer's pcs, and will be automatically farmed out to test servers with a range of different hardware and capabilities.

Will this work, or will the numbers go awry?

How should this problem be solved? Is there a better way? Is there a standard tool for this sort of caper?

Answer 1

You might try to look at Perf4J. I haven't used it yet, but it's on my list of things to investigate.

From their homepage:

Perf4J is to System.currentTimeMillis() as log4j is to System.out.println()

The Developer Guide is a nice introduction.