How to run JMH from inside JUnit tests?

Question

How can I run JMH benchmarks inside my existing project using JUnit tests? The official documentation recommends making a separate project, using Maven shade plugin, and launching JMH inside the main method. Is this necessary and why is it recommended?

Answer 1

I've been running JMH inside my existing Maven project using JUnit with no apparent ill effects. I cannot answer why the authors recommend doing things differently. I have not observed a difference in results. JMH launches a separate JVM to run benchmarks to isolate them. Here is what I do:

• Add the JMH dependencies to your POM:

  <dependency>   <groupId>org.openjdk.jmh</groupId>   <artifactId>jmh-core</artifactId>   <version>1.21</version>   <scope>test</scope> </dependency> <dependency>   <groupId>org.openjdk.jmh</groupId>   <artifactId>jmh-generator-annprocess</artifactId>   <version>1.21</version>   <scope>test</scope> </dependency> 

  Note that I've placed them in scope test.

  In Eclipse, you may need to configure the annotation processor manually. NetBeans handles this automatically.

• Create your JUnit and JMH class. I've chosen to combine both into a single class, but that is up to you. Notice that OptionsBuilder.include is what actually determines which benchmarks will be run from your JUnit test!

  import java.util.ArrayList; import java.util.List; import java.util.Random; import java.util.concurrent.TimeUnit; import org.junit.Test; import org.openjdk.jmh.annotations.*; import org.openjdk.jmh.infra.Blackhole; import org.openjdk.jmh.runner.Runner; import org.openjdk.jmh.runner.options.*;   public class TestBenchmark  {        @Test public void      launchBenchmark() throws Exception {              Options opt = new OptionsBuilder()                 // Specify which benchmarks to run.                  // You can be more specific if you'd like to run only one benchmark per test.                 .include(this.getClass().getName() + ".*")                 // Set the following options as needed                 .mode (Mode.AverageTime)                 .timeUnit(TimeUnit.MICROSECONDS)                 .warmupTime(TimeValue.seconds(1))                 .warmupIterations(2)                 .measurementTime(TimeValue.seconds(1))                 .measurementIterations(2)                 .threads(2)                 .forks(1)                 .shouldFailOnError(true)                 .shouldDoGC(true)                 //.jvmArgs("-XX:+UnlockDiagnosticVMOptions", "-XX:+PrintInlining")                 //.addProfiler(WinPerfAsmProfiler.class)                 .build();              new Runner(opt).run();         }      // The JMH samples are the best documentation for how to use it     // http://hg.openjdk.java.net/code-tools/jmh/file/tip/jmh-samples/src/main/java/org/openjdk/jmh/samples/     @State (Scope.Thread)     public static class BenchmarkState     {         List<Integer> list;            @Setup (Level.Trial) public void         initialize() {                  Random rand = new Random();                  list = new ArrayList<>();                 for (int i = 0; i < 1000; i++)                     list.add (rand.nextInt());             }     }        @Benchmark public void      benchmark1 (BenchmarkState state, Blackhole bh) {              List<Integer> list = state.list;              for (int i = 0; i < 1000; i++)                 bh.consume (list.get (i));         } } 

• JMH's annotation processor seems to not work well with compile-on-save in NetBeans. You may need to do a full Clean and Build whenever you modify the benchmarks. (Any suggestions appreciated!)

• Run your launchBenchmark test and watch the results!

  -------------------------------------------------------  T E S T S ------------------------------------------------------- Running com.Foo # JMH version: 1.21 # VM version: JDK 1.8.0_172, Java HotSpot(TM) 64-Bit Server VM, 25.172-b11 # VM invoker: /usr/lib/jvm/java-8-jdk/jre/bin/java # VM options: <none> # Warmup: 2 iterations, 1 s each # Measurement: 2 iterations, 1 s each # Timeout: 10 min per iteration # Threads: 2 threads, will synchronize iterations # Benchmark mode: Average time, time/op # Benchmark: com.Foo.benchmark1  # Run progress: 0.00% complete, ETA 00:00:04 # Fork: 1 of 1 # Warmup Iteration   1: 4.258 us/op # Warmup Iteration   2: 4.359 us/op Iteration   1: 4.121 us/op Iteration   2: 4.029 us/op   Result "benchmark1":   4.075 us/op   # Run complete. Total time: 00:00:06  REMEMBER: The numbers below are just data. To gain reusable insights, you need to follow up on why the numbers are the way they are. Use profilers (see -prof, -lprof), design factorial experiments, perform baseline and negative tests that provide experimental control, make sure the benchmarking environment is safe on JVM/OS/HW level, ask for reviews from the domain experts. Do not assume the numbers tell you what you want them to tell.  Benchmark                                Mode  Cnt  Score   Error  Units Foo.benchmark1                           avgt    2  4.075          us/op Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 7.013 sec 

• Runner.run even returns RunResult objects on which you can do assertions, etc.