JMH: Using the same static object in all Benchmark tests

Question

I have a class that constructs some complicated data (imagine a large XML or JSON structure - that sort of thing). Constructing it takes time. So I want to construct it once, and then use that same data in all tests. Currently I basically have a public static object instance defined in a class that defines main, and then refer to it explicitly in the tests (code is a very simplified example):

public class Data 
{
    // This class constructs some complicated data 
}

public class TestSet 
{
    public static final Data PARSE_ME = new Data(...);

    public static void main(String[] args) throws RunnerException 
    {
        Options opt = new OptionsBuilder()
                .include(".*ParserTest") // several tests
                .forks(1)
                .build();

        new Runner(opt).run();
    }
}

@State(Scope.Thread)
public class SomeParserTest
{
    @Setup(Level.Iteration)
    public void setup()
    {
        Parser parser = new Parser(TestSet.PARSE_ME);
    }

    @Benchmark
    public void getId() 
    {
        parser.getId(123);
    }
}

And this is awful of course... An equally evil option would be creating a separate class just so that it can hold a single static object. It would be nice to use something like

Options opt = new OptionsBuilder()
    ...
    .param(/*my Data object comes here*/)

but param only accepts Strings, so not sure how would I pass an object (and more importantly: the same instance of the object!) to it.

So is there anything more elegant, than a global object, I described above?

Answer 1

Unfortunately, JMH provides no way to share data between the benchmarks.

For one thing, this breaks benchmark isolation, when one benchmark can silently modify the input data for another benchmark, rendering the comparison incorrect. This is why you are required to @Setup the @State object for every benchmark.

But more importantly, whatever trick you build to share the data between the benchmarks (e.g. static field accessible from both) would break in the default "forked" mode, when JMH will execute each test in its own VM. Notably, the thing you are suggesting with static final Data TestSet.PARSE_ME would actually execute for each @Benchmark, since every new VM instance would have to initialize TestSet anyhow ;) Granted, you may disable forking, but that introduces more problems than it solves.

Therefore, it may be a better idea to invest time into making the setup costs more tolerable, so that it is not excruciatingly painful. E.g., deserialize the data from disk instead of computing it. Or, just come up with a faster way to compute.