How do to perform a good performance comparations test?

Question

To write a good comparations test test you have to run it several thousands (millions) times. It will level (in most cases) other programs' influence.

But if a JVM can influence on the results. For example:

First solution is:

    final StringBuilder stringBuilder = new StringBuilder();
    stringBuilder.append(getStrOne());
    stringBuilder.append(getStrTwo());
    final String result1 = stringBuilder.toString();

And second is:

    final String result2 = getStrOne() + getStrTwo();

I do not know which one is better because JVM can influence on the results. How to know which one is better?

UPDATE: I don't mean exactly that appending comporation test. I'm asking about such a hard to test situation.

Answer 1

I recently performed some benchmark tests that relied on the excellent IBM article found here: http://www.ibm.com/developerworks/java/library/j-benchmark1/index.html.

The article describes many of the pitfalls that can affect accuracy of results, for example:

• Runtime optimisation/re-compilation of your code.
• Dead code elimination (i.e. unused results can cause your test code to be removed)
• Garbage collection
• Caching
• ...

Finally, the article links to a site where a framework can be downloaded. The framework does a very good job of spooling up a test method, looking for evidence of recompilation and waiting for the execution time to settle.