Simple string concat example fast in Java 7, slow in Java 8

Question

I have a little class that demonstrates some timing characteristics of different ways to concatenate strings (StringBuilder vs. plain string contatenation). Irrespective of the relative timings of each strategy, I noticed particular performance behavior with Java 6, and then later Java 7, and now Java 8.

I had first written it with Java 6, and I was impressed with the large improvements I saw with Java 7. I had assumed that Java 8 would show similar performance. However, when I run this on my Win7/64-bit laptop, I find that the Java 8 numbers are even slower than my Java 6 numbers.

It's entirely possible I'm seeing issues with Java 6 and 7 where the JVM is removing the code I'm testing at runtime (a common obstacle to microbenchmarks), but I just don't know.

I'm going to show my perf numbers for the three runs, and then I'll include the source code. Note that the test has a hefty warmup pass, which hopefully should result in all classloading and JIT work finishing before the timing runs.

Also note that the relative timings in Java 6 and Java 8 are "consistent". For instance, I expect tests 2 and 3 to be the two fastest, 2 sometimes faster than 3, or vice versa. The Java 7 timings are all much lower than that, and mostly about the same, so it's possible I have an issue with the Java 7 JIT cutting out too much code.

Java 6:
stringBuilder0:499
stringBuilder1:260
stringBuilder2:262
stringBuilder3:180
stringBuilder4:184
stringBuilder5:210
stringBuilder6:286
stringBuilder7:287
stringBuilder8:203
stringBuilder9:248

Java 7:
stringBuilder0:66
stringBuilder1:269
stringBuilder2:57
stringBuilder3:57
stringBuilder4:57
stringBuilder5:57
stringBuilder6:299
stringBuilder7:57
stringBuilder8:57
stringBuilder9:57

Java 8:
stringBuilder0:714
stringBuilder1:367
stringBuilder2:368
stringBuilder3:279
stringBuilder4:271
stringBuilder5:318
stringBuilder6:415
stringBuilder7:406
stringBuilder8:308
stringBuilder9:349

Here are the two classes that generate this data.

StringBuilderTests:

package timings;

public class StringBuilderTests {
public static void main(String[] args) {
    StringBuilderTests  stringBuilderTests  = new StringBuilderTests(args);
    stringBuilderTests.go();
}

public StringBuilderTests(String[] args) {
}

private void go() {
    final String  str1    = "01234567890123456789012345678901234567890123456789012345678901234567890123456789";
    final String  str2    = "abcdefghiabcdefghiabcdefghiabcdefghiabcdefghi";
    final String  str3    = "xxxxxxxxxxxxxxxx";
    final String  str4    = "yyyyyyyyyyyyyyyy";
    final String  str5    = "zzzzzzzzzzzzzzzz";

    System.out.println("Running warmup pass.");

    for (int ctr = 0; ctr < 20000; ++ ctr) {
    testStringBuilder0(str1, str2, str3, str4, str5);
    testStringBuilder1(str1, str2, str3, str4, str5);
    testStringBuilder2(str1, str2, str3, str4, str5);
    testStringBuilder3(str1, str2, str3, str4, str5);
    testStringBuilder4(str1, str2, str3, str4, str5);
    testStringBuilder5(str1, str2, str3, str4, str5);
    testStringBuilder6(str1, str2, str3, str4, str5);
    testStringBuilder7(str1, str2, str3, str4, str5);
    testStringBuilder8(str1, str2, str3, str4, str5);
    testStringBuilder9(str1, str2, str3, str4, str5);
    }

    int     iters   = 10000000;

    System.out.println("Starting timing runs.");

    new TimingContainer(iters, "stringBuilder0", new TimingTest() {
    public void run() {
        testStringBuilder0(str1, str2, str3, str4, str5);
    }
    }).run();

    new TimingContainer(iters, "stringBuilder1", new TimingTest() {
    public void run() {
        testStringBuilder1(str1, str2, str3, str4, str5);
    }
    }).run();

    new TimingContainer(iters, "stringBuilder2", new TimingTest() {
    public void run() {
        testStringBuilder2(str1, str2, str3, str4, str5);
    }
    }).run();

    new TimingContainer(iters, "stringBuilder3", new TimingTest() {
    public void run() {
        testStringBuilder3(str1, str2, str3, str4, str5);
    }
    }).run();

    new TimingContainer(iters, "stringBuilder4", new TimingTest() {
    public void run() {
        testStringBuilder4(str1, str2, str3, str4, str5);
    }
    }).run();
    new TimingContainer(iters, "stringBuilder5", new TimingTest() {
    public void run() {
        testStringBuilder5(str1, str2, str3, str4, str5);
    }
    }).run();
    new TimingContainer(iters, "stringBuilder6", new TimingTest() {
    public void run() {
        testStringBuilder6(str1, str2, str3, str4, str5);
    }
    }).run();
    new TimingContainer(iters, "stringBuilder7", new TimingTest() {
    public void run() {
        testStringBuilder7(str1, str2, str3, str4, str5);
    }
    }).run();
    new TimingContainer(iters, "stringBuilder8", new TimingTest() {
    public void run() {
        testStringBuilder8(str1, str2, str3, str4, str5);
    }
    }).run();
    new TimingContainer(iters, "stringBuilder9", new TimingTest() {
    public void run() {
        testStringBuilder9(str1, str2, str3, str4, str5);
    }
    }).run();
}

private void useit(String str) {
    System.out.print(str.charAt(0));
}

public static String testStringBuilder0(String str1, String str2, String str3, String str4, String str5) {
    String  result  = null;
    result  = result + str1;
    result  = result + str2;
    result  = result + str3;
    result  = result + str4;
    result  = result + str5;
    return result;
}

public static String testStringBuilder1(String str1, String str2, String str3, String str4, String str5) {
    StringBuilder   sb  = new StringBuilder();
    sb.append(str1);
    sb.append("-");
    sb.append(str2);
    sb.append("-");
    sb.append(str3);
    sb.append("-");
    sb.append(str4);
    sb.append("-");
    sb.append(str5);
    return sb.toString();
}

public static String testStringBuilder2(String str1, String str2, String str3, String str4, String str5) {
    return new StringBuilder().
        append(str1).append("-").append(str2).append("-").append(str3).append("-").append(str4).append("-").append(str5).
        toString();
}

public static String testStringBuilder3(String str1, String str2, String str3, String str4, String str5) {
    return new StringBuilder(str1).append("-").append(str2).append("-").append(str3).append("-").append(str4).append("-").append(str5).toString();
}

public static String testStringBuilder4(String str1, String str2, String str3, String str4, String str5) {
    return str1 + "-" + str2 + "-" + str3 + "-" + str4 + "-" + str5;
}

public static String testStringBuilder5(String str1, String str2, String str3, String str4, String str5) {
    return new StringBuilder(160).
        append(str1).append("-").append(str2).append("-").append(str3).append("-").append(str4).append("-").append(str5).
        toString();
}

public static String testStringBuilder6(String str1, String str2, String str3, String str4, String str5) {
    StringBuffer   sb  = new StringBuffer();
    sb.append(str1);
    sb.append("-");
    sb.append(str2);
    sb.append("-");
    sb.append(str3);
    sb.append("-");
    sb.append(str4);
    sb.append("-");
    sb.append(str5);
    return sb.toString();
}

public static String testStringBuilder7(String str1, String str2, String str3, String str4, String str5) {
    return new StringBuffer().
        append(str1).append("-").append(str2).append("-").append(str3).append("-").append(str4).append("-").append(str5).
        toString();
}

public static String testStringBuilder8(String str1, String str2, String str3, String str4, String str5) {
    return new StringBuffer(str1).append("-").append(str2).append("-").append(str3).append("-").append(str4).append("-").append(str5).toString();
}

public static String testStringBuilder9(String str1, String str2, String str3, String str4, String str5) {
    return new StringBuffer(160).
        append(str1).append("-").append(str2).append("-").append(str3).append("-").append(str4).append("-").append(str5).
        toString();
}
}

TimingContainer:

package timings;

public class TimingContainer {
private int         iterations;
private String      label;
private TimingTest  timingTest;

public TimingContainer(int iterations, String label, TimingTest timingTest) {
    this.iterations = iterations;
    this.label      = label;
    this.timingTest = timingTest;
}

public void run() {
    long startTime  = System.nanoTime();
    for (int ctr = 0; ctr < iterations; ++ ctr) {
    timingTest.randomize();
    timingTest.run();
    }
    long    endTime = System.nanoTime();
    long    totalns = (endTime - startTime);
    System.out.println(label + ":" + (totalns / iterations));
}
}

Update:

TimingTest:

package timings;

public abstract class TimingTest {
public abstract void run();
public void randomize() {}
}

Answer 1

The discrepancy was that I was comparing 64-bit Java 7 with 32-bit Java 8 (The Eclipse in Spring Tool Suite 3.5.0 apparently won't work with the 64-bit Java 8, which is why I had the 32-bit Java 8 distro).

Once I ran the test with 64-bit Java 8, I got more reasonable results.

In addition, I learned about JMH, which is handy for writing and running microbenchmarks that need to be accurate.