JDK 11 vs JDK 13 performance

Question

UPDATE

Throughout comments it turned out that the approach for benchmark I taken was incorrect, therefore results were misleading. After correcting my approach (as in accepted answer) results are as one would expect - JDK 13 performance is just as good as with JDK 11. See the answer for more details.

Original question

I was doing some performance benchmarks on HashSet under Windows 10, using following test code for JMH:

@Benchmark
@BenchmarkMode(Mode.AverageTime)
@Fork(value = 1, warmups = 1)
public void init() {
    HashSet<String> s = new HashSet<>();
    for (int i = 0; i < 1000000; i++) {
        s.add(Math.random() + "");
    }
    s.size();
}

I compiled and run it under different JDK versions and here is what results I got:

I tested it with different heap sizes too (thus 3 different colors for each JDK). JDK 14 is of course a pre-release snapshot from today - just to see ZGC performing under Windows.

I wonder - what happened after JDK 11? (note, for JDK 12 it already starts growing, even though it's not present on the chart above)

Answer 1

Thank you all for suggestions in comments.

The answer was most likely Math.random() or HashSet, or missing Blackhole::consume or combination of all. I changed the test to simply do i + "aaaaaaaaa" and replaced HashSet with ArrayList pre-initialized with appropriate size to accommodate for all values to be populated. I also added Blackhole::consume at the end to exclude unwanted JIT optimizations.

After all of that, timing drops from JDK 8 to 11 gradually and then stays around the same among JDK 11-13. In JDK 14 it raises slightly, but well - it's not released yet.