Caliper benchmark fails due to JIT compilation

Question

I got a Caliper benchmark which looks like this:

public Course timeCourseCreation( int i ) {

  return createCourse( i );
}

public Course createCourse( int t ) {

  Course course = new Course();

  for ( int i = 0 + t; i < this.students + t; i++ ) {
    Student student = new Student();
    student.setAge( ( int ) ( 100 * Math.random() ) + t );
    student.setName( UUID.randomUUID().toString() );
    course.getStudents().add( student );
  }

  for ( int i = 0 + t; i < this.courses + t; i++ ) {
    Topic topic = new Topic();
    topic.setDescription( UUID.randomUUID().toString() );
    topic.setDifficulty( ( int ) ( 10 * Math.random() ) + t );
    topic.setName( UUID.randomUUID().toString() );
    topic.setRating( ( int ) ( 10 * Math.random() ) + t );
    course.getTopics().add( topic );
  }

  return course;
}

So, I have a method which creates a data structure (createCourse(...)) and I want to measure how long it takes to create this structure.

However, when I run the benchmark, I get the following exception:

Failed to execute java -cp C:\Users\tuhrig\workspace\XmlVsJson\target\classes;C:\Users\tuhrig\.m2\repository\javax\xml\bind\jaxb-api\2.2.11\jaxb-api-2.2.11.jar;C:\Users\tuhrig\.m2\repository\com\google\code\gson\gson\2.2.4\gson-2.2.4.jar;C:\Users\tuhrig\.m2\repository\com\google\caliper\caliper\0.5-rc1\caliper-0.5-rc1.jar;C:\Users\tuhrig\.m2\repository\com\google\code\findbugs\jsr305\1.3.9\jsr305-1.3.9.jar;C:\Users\tuhrig\.m2\repository\com\google\guava\guava\11.0.1\guava-11.0.1.jar;C:\Users\tuhrig\.m2\repository\com\google\code\java-allocation-instrumenter\java-allocation-instrumenter\2.0\java-allocation-instrumenter-2.0.jar;C:\Users\tuhrig\.m2\repository\asm\asm\3.3.1\asm-3.3.1.jar;C:\Users\tuhrig\.m2\repository\asm\asm-analysis\3.3.1\asm-analysis-3.3.1.jar;C:\Users\tuhrig\.m2\repository\asm\asm-commons\3.3.1\asm-commons-3.3.1.jar;C:\Users\tuhrig\.m2\repository\asm\asm-tree\3.3.1\asm-tree-3.3.1.jar;C:\Users\tuhrig\.m2\repository\asm\asm-util\3.3.1\asm-util-3.3.1.jar;C:\Users\tuhrig\.m2\repository\asm\asm-xml\3.3.1\asm-xml-3.3.1.jar com.google.caliper.InProcessRunner --warmupMillis 3000 --runMillis 1000 --measurementType TIME --marker //ZxJ/ -Dbenchmark=CourseCreation de.tuhrig.Benchmark
starting Scenario{vm=java, trial=0, benchmark=CourseCreation}
[caliper] [starting warmup]
[caliper] [starting measured section]
[caliper] [done measured section]
[caliper] [starting measured section]
...
Error: Doing 2x as much work didn't take 2x as much time! Is the JIT optimizing away the body of your benchmark?
...
An exception was thrown from the benchmark code.
com.google.caliper.ConfigurationException: Failed to execute java -cp C:\Users\tuhrig\workspace\XmlVsJson\target\classes;C:\Users\tuhrig\.m2\repository\javax\xml\bind\jaxb-api\2.2.11\jaxb-api-2.2.11.jar;C:\Users\tuhrig\.m2\repository\com\google\code\gson\gson\2.2.4\gson-2.2.4.jar;C:\Users\tuhrig\.m2\repository\com\google\caliper\caliper\0.5-rc1\caliper-0.5-rc1.jar;C:\Users\tuhrig\.m2\repository\com\google\code\findbugs\jsr305\1.3.9\jsr305-1.3.9.jar;C:\Users\tuhrig\.m2\repository\com\google\guava\guava\11.0.1\guava-11.0.1.jar;C:\Users\tuhrig\.m2\repository\com\google\code\java-allocation-instrumenter\java-allocation-instrumenter\2.0\java-allocation-instrumenter-2.0.jar;C:\Users\tuhrig\.m2\repository\asm\asm\3.3.1\asm-3.3.1.jar;C:\Users\tuhrig\.m2\repository\asm\asm-analysis\3.3.1\asm-analysis-3.3.1.jar;C:\Users\tuhrig\.m2\repository\asm\asm-commons\3.3.1\asm-commons-3.3.1.jar;C:\Users\tuhrig\.m2\repository\asm\asm-tree\3.3.1\asm-tree-3.3.1.jar;C:\Users\tuhrig\.m2\repository\asm\asm-util\3.3.1\asm-util-3.3.1.jar;C:\Users\tuhrig\.m2\repository\asm\asm-xml\3.3.1\asm-xml-3.3.1.jar com.google.caliper.InProcessRunner --warmupMillis 3000 --runMillis 1000 --measurementType TIME --marker //ZxJ/ -Dbenchmark=CourseCreation de.tuhrig.Benchmark
    at com.google.caliper.Runner.measure(Runner.java:309)
    at com.google.caliper.Runner.runScenario(Runner.java:229)
    at com.google.caliper.Runner.runOutOfProcess(Runner.java:378)
    at com.google.caliper.Runner.run(Runner.java:97)
    at com.google.caliper.Runner.main(Runner.java:423)
    at de.tuhrig.CaliperRunner.main(CaliperRunner.java:22)

The exception tells me that the benchmark fails because of JIT compilation. So, how can I prevent the JIT compilation here?

I already tried to initialize the data structure with random values (as you can see in the code), I tried to return a value in my test (as you can see in the code) and I tried to use the course object, e.g. to call course.getSize() on it. But nothing works.

Answer 1

JIT is known to mess with your code in horrible ways, making all the measurements absolutely unreliable. People struggle a lot, trying to make things at least somewhat right, but end up writing things like JMH. Which is a great tool that helps measure performance and actually get results that make sense. It even enables you to avoid dead code elimination and has a great number of samples, looking at which would definitely help you solve your problem...

But this is the wrong way. It is usually kept a secret, but since you have figured out the right approach on your own, I will tell you. You can disable the JIT by using the -Xint option. There. No need to thank me.

Answer 2

You can't benchmark stuff that depends on UUID generation. Getting random information from most OS'es depends on the OS state, the state of the random pool, etc. I found out that this invokes unreliable behaviour.

Get rid of that to start with.