How to specify the command line when using Caliper?

Question

I find Google's micro benchmark project Caliper very interesting but the documentation is still (except some examples) quite non-existent.

I have two different cases where I need to influence the command line of the JVMs Caliper starts:

1. I need to set some fixed (ideally alternated between a few fixed values) -D parameters
2. I need to specify some fixed (ideally alternated between a few fixed values) JVM parameters

I saw some discussion about adding features like this but I could not conclude if it has been added or not and in that case what the syntax became?

Some example or pointers into Java doc (assuming this is at all documented somewhere) etc would be very much appreciated!

Answer 1

To fix a benchmark parameter with a command line argument, use -Dname=value. There is one special parameter named benchmark; it's values are the suffixes to your time methods. If you'd like to limit a parameter to multiple values, separate them by commas like this: -Dname=value1,value2.

To set JVM parameters, use -Jname=flag1,flag2.

For example consider this benchmark:

public class StringBuilderBenchmark extends SimpleBenchmark {

    @Param({"1", "10", "100"}) private int length;

    public void timeAppendBoolean(int reps) {
        for (int i = 0; i < reps; ++i) {
            StringBuilder sb = new StringBuilder();
            for (int j = 0; j < length; ++j) {
                sb.append(true);
            }
        }
    }

    public void timeAppendChar(int reps) {
        for (int i = 0; i < reps; ++i) {
            StringBuilder sb = new StringBuilder();
            for (int j = 0; j < length; ++j) {
                sb.append('c');
            }
        }
    }
}

To run this benchmark with lengths 5 and 6, and large and small heaps, use these parameters:

java -cp caliper-0.0.jar:build/classes/test \
    com.google.caliper.Runner examples.StringBuilderBenchmark \
    -Dlength=5,6 -Dbenchmark=AppendBoolean -Jmemory=-Xmx512M,-Xmx16M

This yields the following:

 0% Scenario{vm=java, trial=0, benchmark=AppendBoolean, length=5, memory=-Xmx512M} 81.79 ns; σ=0.31 ns @ 3 trials
25% Scenario{vm=java, trial=0, benchmark=AppendBoolean, length=6, memory=-Xmx512M} 89.72 ns; σ=2.19 ns @ 10 trials
50% Scenario{vm=java, trial=0, benchmark=AppendBoolean, length=5, memory=-Xmx16M} 111.44 ns; σ=6.01 ns @ 10 trials
75% Scenario{vm=java, trial=0, benchmark=AppendBoolean, length=6, memory=-Xmx16M} 120.23 ns; σ=4.59 ns @ 10 trials

  memory length    ns logarithmic runtime
-Xmx512M      5  81.8 =
-Xmx512M      6  89.7 =======
 -Xmx16M      5 111.4 ========================
 -Xmx16M      6 120.2 =============================

vm: java
trial: 0
benchmark: AppendBoolean