How can I use JMH for Scala benchmarks together with sbt?

Question

I have tried to use jmh together with sbt, but so far I have not managed to set it up properly so that .scala based benchmarks work.

As the combination sbt + .java based benchmarks works, I tried to start from that base. I am using sbt 0.13.1.

---

.java based benchmarks using sbt

build.sbt

import AssemblyKeys._

name := "scala-benchmark"

version := "1.0"

scalaVersion := "2.10.3"

scalacOptions += "-deprecation"

libraryDependencies += "org.openjdk.jmh" % "jmh-core" % "0.5.5"

libraryDependencies += "org.openjdk.jmh" % "jmh-java-benchmark-archetype" % "0.5.5"

libraryDependencies += "org.openjdk.jmh" % "jmh-generator-annprocess" % "0.5.5"

libraryDependencies += "org.openjdk.jmh" % "jmh-generator-bytecode" % "0.5.5"

assemblySettings

jarName in assembly := "microbenchmarks.jar"

test in assembly := {}

mainClass in assembly := Some("org.openjdk.jmh.Main")

To get a single "fat" jar at the end, the sbt-assembly plugin is required:

project/assembly.sbt

addSbtPlugin("com.eed3si9n" % "sbt-assembly" % "0.11.2")

A simple benchmark:

src/main/java/app/benchmark/java/benchmark2/Benchmark2.java

package app.benchmark.java.benchmark2;

import org.openjdk.jmh.annotations.GenerateMicroBenchmark;
import org.openjdk.jmh.runner.Runner;
import org.openjdk.jmh.runner.RunnerException;
import org.openjdk.jmh.runner.options.Options;
import org.openjdk.jmh.runner.options.OptionsBuilder;

public class Benchmark2 {
    @GenerateMicroBenchmark
    public int run() {
        int result = 0;
        for (int i = 0; i < 10; i++) {
            result += i * i;
        }
        return result;
    }
}

Running sbt assembly gives this output:

$ sbt assembly
[...]
[info] Compiling 2 Scala sources and 2 Java sources to ...
[warn] warning: Supported source version 'RELEASE_6' from annotation processor 'org.openjdk.jmh.generators.GenerateMicroBenchmarkProcessor' less than -source '1.8'
[warn] 1 warning
[info] Including: jmh-java-benchmark-archetype-0.5.5.jar
[info] Including: jmh-generator-bytecode-0.5.5.jar
[info] Including: jopt-simple-4.6.jar
[info] Including: jmh-generator-reflection-0.5.5.jar
[info] Including: jmh-generator-annprocess-0.5.5.jar
[info] Including: asm-4.2.jar
[info] Including: commons-math3-3.2.jar
[info] Including: jmh-core-0.5.5.jar
[info] Including: scala-library.jar
[...]
[info] Packaging /home/scala-2.10/vc/rhaag/scala/scala-benchmark/target/scala-2.10/microbenchmarks.jar ...

and the resulting microbenchmarks.jar contains everything required to run the benchmarks:

$ java -jar target/scala-2.10/microbenchmarks.jar -wi 3 -i 3 -f 1 app.benchmark.java.benchmark2.Benchmark2.run 

[...] 

Benchmark                  Mode   Samples         Mean   Mean error    Units 

a.b.j.b.Benchmark2.run    thrpt         3   607555,968    70243,275   ops/ms 

So far so good.

---

Scala benchmarks using sbt

From that base I tried to switch to .scala based benchmarks:

build.sbt

Replacing the Java archetype with the Scala one

libraryDependencies += "org.openjdk.jmh" % "jmh-scala-benchmark-archetype" % "0.5.5"

does not work, as the download fails.

This works:

libraryDependencies += "org.openjdk.jmh" % "jmh-scala-benchmark-archetype" % "0.5.5" from "http://repo1.maven.org/maven2/org/openjdk/jmh/jmh-scala-benchmark-archetype/0.5.5/jmh-scala-benchmark-archetype-0.5.5.jar"

Another simple benchmark:

src/main/scala/app/benchmark/scala/benchmark2/Benchmark2.scala

package app.benchmark.scala.benchmark2

import org.openjdk.jmh.annotations.GenerateMicroBenchmark
import org.openjdk.jmh.runner.Runner
import org.openjdk.jmh.runner.RunnerException
import org.openjdk.jmh.runner.options.Options
import org.openjdk.jmh.runner.options.OptionsBuilder

class Benchmark2 {
  @GenerateMicroBenchmark
  def run() = {
    Seq.range(0, 10).map(i => i * i).sum
  }
}

Now sbt assembly creates the jar file, but target/scala-2.10/microbenchmarks.jar#META-INF/MicroBenchmarks does not list the Scala benchmarks, and these are not shown by java -jar target/scala-2.10/microbenchmarks.jar -l either.

---

Resources:

• JMH with maven (there is a link to an ant based approach as well)
• A Japanese page where I got the inital sbt setup from
• JMH with gradle

How can I integrate the (bytecode based) JMH processor for Scala? Or from another perspective: Why is the (annotation based) JMH processor picked up automatically and produces Java based benchmarks?

Answer 1

I have implemented an sbt-jmh plugin that actually works: https://github.com/ktoso/sbt-jmh

Currently building benchmarks.jar is not supported, but you can simply type run -i 10 .*MyBenchmark.* and it will work as expected (doing all the multi-step compilation for you).

I hope this helps, cheers!