HashMap performance Java 9 25% less than Java 8?

Question

Note: This is not about a performance issue. I only observe a difference in performance I cannot explain / understand.

While benchmarking some newly developed code, targeted for Java 9 I discovered something weird. A (very) simple benchmark of HashMap with 5 keys reveals that Java 9 is much slower than Java 8. Can this be explained or is my (benchmark) code just plain wrong?

Code:

@Fork(
    jvmArgsAppend = {"-Xmx512M", "-disablesystemassertions"}
)
public class JsonBenchmark {

    @State(Scope.Thread)
    public static class Data {

        final static Locale RUSSIAN = new Locale("ru");
        final static Locale DUTCH = new Locale("nl");

        final Map<Locale, String> hashmap = new HashMap<>();

        public Data() {
            hashmap.put(Locale.ENGLISH, "Flat flashing adjustable for flat angled roof with swivel");
            hashmap.put(Locale.FRENCH, "Solin pour toit plat inclinée");
            hashmap.put(Locale.GERMAN, "Flachdachkragen Flach Schrägdach");
            hashmap.put(DUTCH, "Plakplaat vlak/hellend dak inclusief glijschaal");
            hashmap.put(RUSSIAN, "Проход через плоскую кровлю регулир. для накл. кровли");
        }

    }

    @Benchmark
    public int bmHashMap(JsonBenchmark.Data data) {
        final Map<Locale, String> m = data.hashmap;
        int sum = 0;
        sum += m.get(Data.RUSSIAN).length();
        sum += m.get(Locale.FRENCH).length();
        sum += m.get(Data.DUTCH).length();
        sum += m.get(Locale.ENGLISH).length();
        sum += m.get(Locale.GERMAN).length();
        return sum;
    }    

}

Results:

• Java 8_151: JsonBenchmark.bmHashMap thrpt 40 47948546.439 ± 560763.711 ops/s
• Java 9_181: JsonBenchmark.bmHashMap thrpt 40 34962904.479 ± 276045.691 ops/s (-/- 27%!)

UPDATE

Thanks for the answers and great comments.

1. Suggestion by @Holger. My first reaction was: That must be the explanation. However, if I only benchmark the String#length() function, there is no significant difference in performance. And, when I only benchmark the HashMap#get() methods (as suggested by @Eugene) there is still a difference of about 10 - 12 %.

2. Suggestion by @Eugene. I changed the parameters (more warmup iterations, more memory), but I am not able to reproduce your outcome. I increased the heap to 4G however. But this cannot explain the difference, is not it?

3. Suggestion by @Alan Bateman. Yes, this improves the performance! However, still a difference of around 20%.

Answer 1

You are testing more than just HashMap. You are not only calling HashMap.get, you are implicitly calling Locale.hashCode and Locale.equals. Further, you are calling String.length.

Now, all four could have changed their performance characteristics, so you would need far more tests to reason about which method(s) exhibit(s) different performance.

But the hottest candidate is String.length. In Java 9, the String class does not use a char[] array anymore, but a byte[] array, to encode Latin 1 strings using only one byte per character, dramatically reducing the memory footprint of typical applications. This, however, implies that the length is not always identical to the array length anymore. So the complexity of this operation has changed.

But keep in mind that your result is about 77 ns difference in a microbenchmark. This is not enough to estimate the impact on a real application…

Answer 2

I had a hint this was about jmh set-up, more then it was about HashMap. As already noted you are measuring a lot more than simply HashMap::get here. But even so, I had doubts that java-9 would be that much slower, so I measured myself (latest jmh build from sources, java-8 and 9).

I haven't changed your code - just added way more heap (10GB) and way more warm-ups, thus reducing the "error" you see after ±

Using java-8:

Benchmark            Mode  Cnt   Score   Error  Units
SOExample.bmHashMap  avgt   25  22.059 ± 0.276  ns/op

Using java-9:

Benchmark            Mode  Cnt   Score   Error  Units
SOExample.bmHashMap  avgt   25  23.954 ± 0.383  ns/op

The results are on-par pretty much without a noticeable difference (these are nano-seconds after all) as you see it. Also if you really want to test just HashMap::get than your methods could simply return the invocation of that, like this:

@Benchmark
@Fork(5)
public int bmHashMap(SOExample.Data data) {
    return data.hashmap.get(data.key); // where key is a random generated possible key
}