Is it possible to run a JMH benchmark under an external profiler?

Question

I'm currently performance hunting. In order to measure the throughput, and to enforce that we don't regress, I'm using the wonderful JMH.

When I come across something that is slow though, I want to start profiling to see what is going on, according to this link the author of JMH writes:

While JMH profilers can provide the aid in analyzing, I don't think they are the substitute for proper profiling. E.g. the "stack" profiler is good to glance over the profiles, but not for the serious work.

Run the workload for longer, and attach your profiler of choice to the running VM.

I was secretly hoping I could tweak the JMH test from the command line, then attach something like visualvm to that... but so far I've been able to get that to work. I guess that is a bad idea? It would be ideal if I could share the code I use to profile, as well as the code I use to enforce.

Answer 1

Yes it is possible, it is simple to do, and I recommend it as a best practice to do it all the time! It is always good to have data available to explain what is going on without the need to spend extra time to reproduce the issue.

I wrote my own JMH profiler to use JFR (Java Flight Recorder): JmhFlightRecorderProfiler which will start JMH profiling and save the data to a file when your benchmark ends. You can open this file with Java Mission control shipped with the JDK.

Here is how you can start your benchmarks and use it (complete example):

Options opt = new OptionsBuilder()
        .include(".*")
        .addProfiler(JmhFlightRecorderProfiler.class)
        .jvmArgs("-Xmx256m", "-Xms256m", "-XX:+UnlockCommercialFeatures",
                "-Djmh.stack.profiles=" + destinationFolder,
                "-Djmh.executor=FJP",
                "-Djmh.fr.options=defaultrecording=true,settings=" + profile)
        .result(destinationFolder + "/" + "benchmarkResults.csv")
        .resultFormat(ResultFormatType.CSV)
        .warmupIterations(10)
        .measurementIterations(10)
        .forks(1)
        .build();
 new Runner(opt).run();

I run benchmarks during the standard build, I use Jenkins JMH plugin to monitor/trend benchmark result/build and if any degradation is observed I have the Benchmark profiles available to investigate the potential cause.