StringBuilder vs. .concat vs. "+" Operator relative performance different in eclipse than command line?

Question

I was reading about how when possible the java compiler will compile strings concatenated with the "+" operator into instances of StringBuilder, and how this makes it better to use the simple "+" operator since they compile to the same code. (Except when you are building the string in a while loop, in which case it is clearly best to use a StringBuilder.)

I've also read that the .concat method on strings is the worst choice all the time (so much so that it was made into a bug by Findbugs!).

So I decided to test it myself writing a little java class in eclipse. My results surprised me a bit. What I found was that different methods were relatively faster or slower if I complied and ran them in eclipse vs. on the command line.

First my eclipse results were:

the total millis to concatenate with + was: 12154
the total millis to concatenate with .concat was: 8840
the total millis to concatenate with StringBuilder was: 11350
the total millis to concatenate with StringBuilder with a specified size was: 5611

So in eclipse StringBuilder with the size specified was fastest, followed by .concat (weird), then StringBuilder and "+" concatenation were pretty much the same.

My results on the command line, however, were:

the total millis to concatenate with + was: 4139
the total millis to concatenate with .concat was: 8590
the total millis to concatenate with StringBuilder was: 10888
the total millis to concatenate with StringBuilder with a specified size was: 6033

So when I compiled and ran from the commnad line the "+" operator was clearly the fastest, followed by String builder with size, then concat, and last was normal StringBuilder!

This doesn't make sense to me. Obviously all the stackoverflow answers I read saying that + operators compile into normal old StringBuilder instances must be outdated.

Does anyone know what's really going on here?

I'm using jdk1.7.0_07, and so far as I can tell both eclipse and my command line are referencing the exact same one. The only difference I know of is eclipse is using "javaw", but from what I've read, that shouldn't make a difference.

Here's my test class if you want to verify I'm not doing anything wrong, but I'm pretty sure it's solid.

public class Test {

    static final int LOOPS = 100000000;
    static final String FIRST_STRING = "This is such";
    static final String SECOND_STRING = " an awesomely cool ";
    static final String THIRD_STRING = "to write string.";

    /**
     * @param args
     */
    public static void main(String[] args) {

        Test.plusOperator();
        Test.dotConcat();
        Test.stringBuilder();
        Test.stringBuilderSizeSpecified();

    }

    public static void plusOperator() {
        String localOne = FIRST_STRING;
        String localTwo = SECOND_STRING;
        String localThree = THIRD_STRING;

        Calendar startTime = Calendar.getInstance();
        for (int x = 0; x < LOOPS; x++) {
            String toPrint = localOne + localTwo + localThree;
        }
        Calendar endTime = Calendar.getInstance();
        System.out.println("the total millis to concatenate with + was: " + 
                (endTime.getTimeInMillis() - startTime.getTimeInMillis()));
    }

    public static void stringBuilder() {
        String localOne = FIRST_STRING;
        String localTwo = SECOND_STRING;
        String localThree = THIRD_STRING;

        Calendar startTime = Calendar.getInstance();
        for (int x = 0; x < LOOPS; x++) {
            StringBuilder toBuild = new StringBuilder()
                .append(localOne)
                .append(localTwo)
                .append(localThree);
        }
        Calendar endTime = Calendar.getInstance();
        System.out.println("the total millis to concatenate with StringBuilder was: " + 
                (endTime.getTimeInMillis() - startTime.getTimeInMillis()));
    }

    public static void stringBuilderSizeSpecified() {
        String localOne = FIRST_STRING;
        String localTwo = SECOND_STRING;
        String localThree = THIRD_STRING;

        Calendar startTime = Calendar.getInstance();
        for (int x = 0; x < LOOPS; x++) {
            StringBuilder toBuild = new StringBuilder(50)
                .append(localOne)
                .append(localTwo)
                .append(localThree);
        }
        Calendar endTime = Calendar.getInstance();
        System.out.println("the total millis to concatenate with StringBuilder with a specified size was: " + 
                (endTime.getTimeInMillis() - startTime.getTimeInMillis()));
    }

    public static void dotConcat() {
        String localOne = FIRST_STRING;
        String localTwo = SECOND_STRING;
        String localThree = THIRD_STRING;

        Calendar startTime = Calendar.getInstance();
        for (int x = 0; x < LOOPS; x++) {
            String toPrint = localOne.concat(localTwo).concat(localThree);
        }
        Calendar endTime = Calendar.getInstance();
        System.out.println("the total millis to concatenate with .concat was: " + 
                (endTime.getTimeInMillis() - startTime.getTimeInMillis()));
    }

}

Answer 1

On Oracle JDK 1.7 (javac 1.7.0_17), the "+" operator is still implemented using StringBuilder, as shown by running javap -c on the class to get the bytecode (only showing the loops here):

public static void plusOperator();
Code:

  16: iload         4
  18: ldc           #10                 // int 100000000
  20: if_icmpge     53
  23: new           #11                 // class java/lang/StringBuilder
  26: dup           
  27: invokespecial #12                 // Method java/lang/StringBuilder."<init>":()V
  30: aload_0       
  31: invokevirtual #13                 // Method java/lang/StringBuilder.append:(Ljava/lang/String;)Ljava/lang/StringBuilder;
  34: aload_1       
  35: invokevirtual #13                 // Method java/lang/StringBuilder.append:(Ljava/lang/String;)Ljava/lang/StringBuilder;
  38: aload_2       
  39: invokevirtual #13                 // Method java/lang/StringBuilder.append:(Ljava/lang/String;)Ljava/lang/StringBuilder;
  42: invokevirtual #14                 // Method java/lang/StringBuilder.toString:()Ljava/lang/String;
  45: astore        5
  47: iinc          4, 1
  50: goto          16


public static void stringBuilder();
Code:

  16: iload         4
  18: ldc           #10                 // int 100000000
  20: if_icmpge     50
  23: new           #11                 // class java/lang/StringBuilder
  26: dup           
  27: invokespecial #12                 // Method java/lang/StringBuilder."<init>":()V
  30: aload_0       
  31: invokevirtual #13                 // Method java/lang/StringBuilder.append:(Ljava/lang/String;)Ljava/lang/StringBuilder;
  34: aload_1       
  35: invokevirtual #13                 // Method java/lang/StringBuilder.append:(Ljava/lang/String;)Ljava/lang/StringBuilder;
  38: aload_2       
  39: invokevirtual #13                 // Method java/lang/StringBuilder.append:(Ljava/lang/String;)Ljava/lang/StringBuilder;
  42: astore        5
  44: iinc          4, 1
  47: goto          16

The only difference between these two is that the version with "+" converts the StringBuilder to a String within the loop.

So the question becomes: why does your test show such different results for the same code. Or more completely, why is this not a valid micro-benchmark. Here are some possible reasons:

• You're counting wall-clock time. This means that you're actually measuring everything that the JVM is doing while running your test. Which includes garbage collection (which is important because you're creating a lot of garbage). You can mitigate this by getting the thread CPU time.
• You don't verify when or if HotSpot is compiling the methods. This is why you should do a warmup phase before any micro-benchmarks: basically, run your main() multiple times, before you run your actual test.

Answer 2

Try to place StringBuilder toBuild = new StringBuilder() above the loop. The same with String toPrint and do += for string and you will see the difference.
Don't create new String and StringBuilder in the loop.