How to be reasonably sure a code block has been JIT compiled?

Question

When conducting performance testing of java code, you want to test JIT compiled code, rather than raw bytecode. To cause the bytecode to be compiled, you must trigger the compilation to occur by executing the code a number of times, and also allow enough yield time for the background thread to complete the compilation.

• What is the minimum number of "warm up" executions of a code path required to be "very confident" that the code will be JIT compiled?
• What is the minimum sleep time of the main thread to be "very confident" that the compilation has completed (assuming a smallish code block)?

I'm looking for a threshold that would safely apply in any modern OS, say Mac OS or Windows for the development environment and Linux for CI/production.

Answer 1

Since OP's intent is not actually figuring out whether the block is JIT-compiled, but rather make sure to measure the optimized code, I think OP needs to watch some of these benchmarking talks.

TL;DR version: There is no reliable way to figure out whether you hit the "steady state":

• You can only measure for a long time to get the ball-park estimate for usual time your concrete system takes to reach some state you can claim "steady".

• Observing -XX:+PrintCompilation is not reliable, because you may be in the phase when counters are still in flux, and JIT is standing by to compile the next batch of now-hot methods. You can easily have a number of warmup plateaus because of that. The method can even recompile multiple times, depending on how many tiered compilers are standing by.

• While one could argue about the invocation thresholds, these things are not reliable either, since tiered compilation may be involved, method might get inlined sooner in the caller, the probabilistic counters can miss the updates, etc. That is, common wisdom about -XX:CompileThreshold=# is unreliable.

• JIT compilation is not the only warmup effect you are after. Automatic GC heuristics, scheduler heuristics, etc. also require warmup.

Get a microbenchmark harness which will make the task easier for you!

Answer 2

To begin with, the results will most likely differ for a JVM run in client or in server mode. Second of all, this number depends highly on the complexity of your code and I am afraid that you will have to exploratively estimate a number for each test case. In general, the more complex your byte code is, the more optimization can be applied to it and therefore your code must get relatively hotter in order to make the JVM reach deep into its tool box. A JVM might recompile a segment of code a dozen of times.

Furthermore, a "real world" compilation depends on the context in which your byte code is run. A compilation might for example occur when a monomorphic call site is promoted to a megamorphic one such that an observed compilation actually represents a de-optimization. Therefore, be careful when assuming that your micro benachmark reflects the code's actual performance.

Instead of the suggested flag, I suggest you to use CompilationMXBean which allows you to check the amount of time a JVM still spends with compilation. If that time is too high, rerun your test until the value gets stable long enough. (Be patient!) Frameworks can help you with creating good benchmarks. Personally, I like caliper. However, never trust your benchmark.

From my experience, custom byte code performs best when you immitate the idioms of javac. To mention the one anecdote I can tell on this matter, I once wrote custom byte code for the Java source code equivalent of:

int[] array = {1, 2, 3};

javac creates the array and uses dup for assigning each value but I stored the array reference in a local variable and loaded it back on the operand stack for assigning each value. The array had a bigger size than that and there was a noteable performance difference.

Finally, I recommend this article before writing a benchmark.