How can I be sure that the compiler doesn't optimize away my performance test?

Question

I have a class that does some time-consuming calculations. I'm trying to performance test it:

int numValues = 1000000;
Random random = new Random();
startMeasuringTime();
double result;
for (int i = 0; i < numValues; i++) {
    result = calculatorInstance.doSomeTimeConsumingCalculationsOn(random.nextDouble());
}
stopMeasuringTime();

I'm using random values so the compiler won't optimize the calculations for being a million times the same. But what about the results? Does the compiler see it isn't used any more and leaves out the call (but then, can it see any side effects the method call could have?)

I don't want to put the results somewhere (into a file, array or to System.out), because I think this will slow down the test with work that I don't want to measure. Or produce an OutOfMemoryError.

Thanks in advance.

EDIT: changed the title a bit

Answer 1

But what about the results? Does the compiler see it isn't used any more and leaves out the call (but then, can it see any side effects the method call could have?)

It depends. If the JIT compiler can detect that the method call has no side-effects, then it is entitled to optimize it away. Especially since the result value is not used. In this case, you might just be measuring the calls to random.nextDouble() ... or possibly an empty loop.

To be sure you should it cannot be optimized away, you should probably write it like this:

int numValues = 1000000;
Random random = new Random();
startMeasuringTime();
double result;
for (int i = 0; i < numValues; i++) {
    result = result +
        calculatorInstance.doSomeCalculationsOn(random.nextDouble());
}
stopMeasuringTime();
System.err.println(result);  // Force result to be computed.

(I'm assuming that the time consuming calculation does depend on the argument ...)

---

You also need to take JVM warmup into account; i.e. run that benchmark code multiple times in the JVM until the measured time stabilizes.

---

Saying that the compiler is "over-optimizing" is kind of wrong. The compiler is actually doing its job correctly. If anything, the fault is in your code; i.e. it does "nothing useful".

Answer 2

Make sure that the result is used in some way, for example by summing it and printing it at the end. Summing is a good choice because addition is a very cheap operation.