exact way to measure performance on individual methods from inside Java?

Question

I'd like to measure the execution time of a set of unit tests to be able to automatically monitor and compare performance when changes are made. Are there any suitable performance counters that can be reached easily from Java itself? Ie, something like:

count1 = <call performance counter func>
executeUnitTest();
count2 = <call performance counter func>
testPerformance = count2 - count1;

with these additional requirements:

• gives the same answer independent of system load, even if the test runner is run in a virtualized guest OS (this seems to disqualify nanoTime() and friends, or even current thread's cpu time, due to varying load on the host machine)
• gives the same answer independent of what hardware it is run on (counting number of executed bytecode instructions would be excellent!)
• ideally, can separate GC execution count from actual method execution (in case GC is done in-thread)

Answer 1

The only way to do this is to instrument your code to add a platform independent counter. You can add into your code where ever you have unit(s) of work.

Use AtomicLong/AtomicInteger if you want it to be thread safe.

 public static int counter;


 // instrumented into the code.
 counter++;

 // big piece of work
 counter+=5;

You can then compare how much the counter increases by. This will be independent of the speed of the CPU, GCs, even hibernating the application in the middle might not matter.

---

The approach I would take is to compare many results with another similar reference implementation I wouldn't expect to change. You can expect a certain ratio and this ration will only vary by a small amount regardless of the platform you are running on.

Answer 2

There are a couple of approaches you could consider.

'Instrumenting' happens at compile- or load-time, and alters the class files so that they keep counters of what is going on. Typically, this is used for doing things like measuring unit tests' code coverage, but you could probably modify a code coverage tool to just keep execution counts. From memory, think that Emma, the code coverage tool, is a good open-source place to start. Since GC doesn't get instrumented, that won't get counted. There may well be something that does it out-of-the-box.

Profiling tools probably can give you at least some of the data you are after. There are heaps available, including the standard 'java' executable being able to generate a lot of information that you might be able to use.

There's also quite sophisticated profiling hooks available in the VM, the API is reachable via the standard JVMTI api, which is as of IIRC 1.5 a part of standard API.