How do you specify the JMH microbenchmarks test to run WITHOUT using a resource file?

Question

jmh 0.6. I have jmh-core, jmh-generator-annprocess, jmh-generator-reflection as dependencies.

First, documentation is poor, unfortunately. For one, I use gradle, not maven, so using the maven archetype is not an option.

Second, I want to use the Java API, not the command line.

My very simple code is that:

public final class TestBenchmark
{
    private static final int COUNT = 100_000;
    private static final List<Integer> LIST = new ArrayList<>();

    static {
        for (int i = 0; i < COUNT; i++)
            LIST.add(i);
    }

    @GenerateMicroBenchmark
    public void foreachLoop()
    {
        int dummy;
        for (final int i: LIST)
            dummy = i;
    }

    @GenerateMicroBenchmark
    public void forLoop()
    {
        int dummy;
        final int size = LIST.size();
        for (int i = 0; i < size; i++)
            dummy = LIST.get(i);
    }

    public static void main(final String... args)
        throws RunnerException
    {
        final Options options = new OptionsBuilder()
            .forks(1)
            .warmupIterations(1)
            .measurementIterations(20)
            .verbosity(VerboseMode.EXTRA)
            .build();

        new Runner(options).run();
    }
}

As I have no .include(), it means .* as a regex therefore all benchmarks. This is the only class I have in my project.

But no: "no benchmarks found".

So, as a last resort I tried and created the META-INF/MicroBenchmarks file as suggested in other places; content, the name of the class:

com.github.parboiled1.grappa.TestBenchmark

but it doesn't work either:

Exception in thread "main" java.lang.IllegalStateException: Mismatched format for the line: com.github.parboiled1.grappa.TestBenchmark

and the format of this file is, of course, not documented.

But I don't want to use this file to start with; I want to specify the list of classes to run.

How do I do that?

Answer 1

Your problem is not about mismatched benchmarks, but rather the build misconfiguration which prevented JMH to generate and compile the synthetic benchmark code for you. You have to let annotation processors to run, and let the rest of the build to pack the generated source and resources into the complete bundle, even if you are to use API afterwards. The project produced by Maven archetype does just that. There is also the Ant sample which does specify exactly what should happen with other build systems. If you want to to that with Gradle, try this.

P.S. In JMH, META-INF/MicroBenchmarks is an internal API, and it may change without notice. No surprises you are not able to use it.

Answer 2

I am using Maven so this is a Maven solution but I am sure you can adapt it to Gradle if you follow the steps. I have to admit that the greatness of JMH is overshadowed by the lack of documentation.

I couldn't make it run properly at first so at the end I created a JUnit test cases that starts the benchmark. No main for me as it is easier for Jenkins and I don't have a jar project created.

1. You have execute the APT processor either by code or from a build script. In Maven it is working out of the box.
2. You can verify the success if JMH source code is generated.
3. Now the JMH source code needs to be compiled. Maven is doing out of the box. Not sure about Gradle.
4. Verify that your class files exist.
5. Now you can execute the main method.

This profile tells maven to add *Benchmark.java to surefire execution.

<profile>
  <id>benchmark</id>
  <build>
    <plugins>
      <plugin>
        <groupId>org.apache.maven.plugins</groupId>
        <artifactId>maven-surefire-plugin</artifactId>
        <configuration>
          <includes>
            <include>**/*Benchmark.java</include>
          </includes>
        </configuration>
      </plugin>
    </plugins>
  </build>
</profile>

when I run Maven with profile benchmark. Then Benchmarks are executed.