Empty methods noticeably slower in Java 11 than Java 8

Question

I was comparing the performance of JDK 8 and 11 using jmh 1.21 when I ran across some surprising numbers:

Java version: 1.8.0_192, vendor: Oracle Corporation  Benchmark                Mode  Cnt  Score   Error  Units MyBenchmark.emptyMethod  avgt   25  0.362 ± 0.001  ns/op   Java version: 9.0.4, vendor: Oracle Corporation  Benchmark                Mode  Cnt  Score    Error  Units MyBenchmark.emptyMethod  avgt   25  0.362 ±  0.001  ns/op   Java version: 10.0.2, vendor: Oracle Corporation  Benchmark                Mode  Cnt  Score   Error  Units MyBenchmark.emptyMethod  avgt   25  0.723 ± 0.001  ns/op   Java version: 11.0.1, vendor: Oracle Corporation  Benchmark                Mode  Cnt  Score   Error  Units MyBenchmark.emptyMethod  avgt   25  0.724 ± 0.002  ns/op 

OpenJDK 11 and 12 perform similar to OracleJDK 11. I have omitted their numbers for the sake of brevity.

I understand that microbenchmarks do not indicate the performance behavior of real-life applications. Still, I'm curious where this difference is coming from. Any ideas?

---

Here is the benchmark in its entirety:

pom.xml:

<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"          xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">     <modelVersion>4.0.0</modelVersion>      <groupId>jmh</groupId>     <artifactId>empty-method</artifactId>     <version>1.0-SNAPSHOT</version>     <packaging>jar</packaging>     <name>JMH benchmark sample: Java</name>      <dependencies>         <dependency>             <groupId>org.openjdk.jmh</groupId>             <artifactId>jmh-core</artifactId>             <version>${jmh.version}</version>         </dependency>         <dependency>             <groupId>org.openjdk.jmh</groupId>             <artifactId>jmh-generator-annprocess</artifactId>             <version>${jmh.version}</version>             <scope>provided</scope>         </dependency>     </dependencies>      <properties>         <project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>         <jmh.version>1.21</jmh.version>         <javac.target>1.8</javac.target>         <uberjar.name>benchmarks</uberjar.name>     </properties>      <build>         <plugins>             <plugin>                 <groupId>org.apache.maven.plugins</groupId>                 <artifactId>maven-enforcer-plugin</artifactId>                 <version>1.4.1</version>                 <executions>                     <execution>                         <id>enforce-versions</id>                         <goals>                             <goal>enforce</goal>                         </goals>                         <configuration>                             <rules>                                 <requireMavenVersion>                                     <version>3.0</version>                                 </requireMavenVersion>                             </rules>                         </configuration>                     </execution>                 </executions>             </plugin>             <plugin>                 <groupId>org.apache.maven.plugins</groupId>                 <artifactId>maven-compiler-plugin</artifactId>                 <version>3.8.0</version>                 <configuration>                     <compilerVersion>${javac.target}</compilerVersion>                     <source>${javac.target}</source>                     <target>${javac.target}</target>                 </configuration>             </plugin>             <plugin>                 <groupId>org.apache.maven.plugins</groupId>                 <artifactId>maven-shade-plugin</artifactId>                 <version>3.2.1</version>                 <executions>                     <execution>                         <phase>package</phase>                         <goals>                             <goal>shade</goal>                         </goals>                         <configuration>                             <finalName>${uberjar.name}</finalName>                             <transformers>                                 <transformer implementation="org.apache.maven.plugins.shade.resource.ManifestResourceTransformer">                                     <mainClass>org.openjdk.jmh.Main</mainClass>                                 </transformer>                             </transformers>                             <filters>                                 <filter>                                     <!--                                             Shading signed JARs will fail without this.                                             http://stackoverflow.com/questions/999489/invalid-signature-file-when-attempting-to-run-a-jar                                     -->                                     <artifact>*:*</artifact>                                     <excludes>                                         <exclude>META-INF/*.SF</exclude>                                         <exclude>META-INF/*.DSA</exclude>                                         <exclude>META-INF/*.RSA</exclude>                                     </excludes>                                 </filter>                             </filters>                         </configuration>                     </execution>                 </executions>             </plugin>         </plugins>         <pluginManagement>             <plugins>                 <plugin>                     <artifactId>maven-clean-plugin</artifactId>                     <version>2.6.1</version>                 </plugin>                 <plugin>                     <artifactId>maven-deploy-plugin</artifactId>                     <version>2.8.2</version>                 </plugin>                 <plugin>                     <artifactId>maven-install-plugin</artifactId>                     <version>2.5.2</version>                 </plugin>                 <plugin>                     <artifactId>maven-jar-plugin</artifactId>                     <version>3.1.0</version>                 </plugin>                 <plugin>                     <artifactId>maven-javadoc-plugin</artifactId>                     <version>3.0.0</version>                 </plugin>                 <plugin>                     <artifactId>maven-resources-plugin</artifactId>                     <version>3.1.0</version>                 </plugin>                 <plugin>                     <artifactId>maven-site-plugin</artifactId>                     <version>3.7.1</version>                 </plugin>                 <plugin>                     <artifactId>maven-source-plugin</artifactId>                     <version>3.0.1</version>                 </plugin>                 <plugin>                     <artifactId>maven-surefire-plugin</artifactId>                     <version>2.22.0</version>                 </plugin>             </plugins>         </pluginManagement>     </build> </project> 

src/main/java/jmh/MyBenchmark.java:

package jmh;  import org.openjdk.jmh.annotations.Benchmark; import org.openjdk.jmh.annotations.BenchmarkMode; import org.openjdk.jmh.annotations.Mode; import org.openjdk.jmh.annotations.OutputTimeUnit;  import java.util.concurrent.TimeUnit;  @BenchmarkMode(Mode.AverageTime) @OutputTimeUnit(TimeUnit.NANOSECONDS) public class MyBenchmark {     @Benchmark     public void emptyMethod()     {     } } 

Here is the Windows-specific script I use. It should be trivial to translate it to other platforms:

set JAVA_HOME=C:\Program Files\Java\jdk1.8.0_192 call mvn -V -Djavac.target=1.8 clean install "%JAVA_HOME%\bin\java" -jar target\benchmarks.jar  set JAVA_HOME=C:\Program Files\Java\jdk-9.0.4 call mvn -V -Djavac.target=9 clean install "%JAVA_HOME%\bin\java" -jar target\benchmarks.jar  set JAVA_HOME=C:\Program Files\Java\jdk-10.0.2 call mvn -V -Djavac.target=10 clean install "%JAVA_HOME%\bin\java" -jar target\benchmarks.jar  set JAVA_HOME=C:\Program Files\Java\oracle-11.0.1 call mvn -V -Djavac.target=11 clean install "%JAVA_HOME%\bin\java" -jar target\benchmarks.jar 

My runtime environment is:

Apache Maven 3.6.0 (97c98ec64a1fdfee7767ce5ffb20918da4f719f3; 2018-10-24T14:41:47-04:00) Maven home: C:\Program Files\apache-maven-3.6.0\bin\.. Default locale: en_CA, platform encoding: Cp1252 OS name: "windows 10", version: "10.0", arch: "amd64", family: "windows" 

More specifically, I am running Microsoft Windows [Version 10.0.17763.195].

Answer 1

You are measuring empty benchmarks, not empty methods. In other words, measuring the minimal infrastructure code that handles the benchmark itself. This is easy to dissect, because you'd expect only a few instructions on the hot path. JMH's -prof perfasm or -prof xperfasm would give you those hottest instructions in seconds.

I think the effect is due to Thread-Local Handshakes (JEP 312), see:

8u191: 0.389 ± 0.029 ns/op [so far so good]

  3.60%  ↗  ...a2: movzbl 0x94(%r8),%r10d   0.63%  │  ...aa: add    $0x1,%rbp  32.82%  │  ...ae: test   %eax,0x1765654c(%rip) ; global safepoint poll  58.14%  │  ...b4: test   %r10d,%r10d          ╰  ...b7: je     ...a2 

11.0.2: 0.585 ± 0.014 ns/op [oops, regression]

  0.31%  ↗  ...70: movzbl 0x94(%r9),%r10d       0.19%  │  ...78: mov    0x108(%r15),%r11  ; reading the thread-local poll addr  25.62%  │  ...7f: add    $0x1,%rbp            35.10%  │  ...83: test   %eax,(%r11)       ; thread-local safepoint poll  34.91%  │  ...86: test   %r10d,%r10d          ╰  ...89: je     ...70 

11.0.2, -XX:-ThreadLocalHandshakes: 0.399 ± 0.048 ns/op [back to 8u perf]

  5.64%  ↗  ...62: movzbl 0x94(%r8),%r10d       0.91%  │  ...6a: add    $0x1,%rbp            34.36%  │  ...6e: test   %eax,0x179be88c(%rip) ; global safepoint poll  54.79%  │  ...74: test   %r10d,%r10d          ╰  ...77: je     ...62 

I think this is largely visible mostly in tight loops like this one.

UPD: Hopefully, more details here.