Strange performance behavior

Question

I'm using Visual Studio 2010 SP1, Target framework is 2.0, Platform target: Any CPU, testing under Windows 7 x64 SP1.

I'm experiencing strange performance behavior.

Without an app.config, or with the following app.config, it makes my program run slowly (Stopwatch shows ~0.11 s)

<?xml version="1.0"?>
<configuration>
  <startup >
    <supportedRuntime version="v2.0.50727" />
  </startup>
</configuration>

The following app.config makes my program run x5 times faster (Stopwatch shows ~0.02 s)

<?xml version="1.0"?>
<configuration>
  <startup >
    <supportedRuntime version="v4.0.30319" sku=".NETFramework,Version=v4.0" />
  </startup>
</configuration>

This is the test program code:

using System;
using System.Collections.Generic;
using System.Text;
using System.Diagnostics;

class Program
{
    static void Main(string[] args)
    {
        Stopwatch sw = new Stopwatch();

        while (true)
        {
            sw.Reset();
            sw.Start();

            for (int i = 0; i < 1000000; i++ )
            {
                "blablabla".IndexOf("ngrhotbegmhroes", StringComparison.OrdinalIgnoreCase);
            }

            Console.WriteLine(sw.Elapsed);
        }
    }
}

I'm sitting for hours and can't figure out what is happening here. Have you any idea?

Answer 1

It sounds like you've just found a situation in which .NET 4 is a lot faster. By default, your app is running with the framework it was built to target. When you force it to use .NET 4, it's faster. That may be a JIT compiler improvement which happens to hit your situation, or it may be a framework improvement - but it shouldn't be too surprising that some things are faster in newer versions.

(For what it's worth, I'd increase the number of iterations you're timing over if I were you... on my box under .NET 4, each iteration is only 10ms, which isn't really a great measurement. I prefer to benchmark for at least a few seconds.)

(And like Mitch, I can confirm that I see the same effect.)

EDIT: I've just investigated this a bit further, and seen an interesting effect... I'll assume we're calling haystack.IndexOf(needle, StringComparison.OrdinalIgnoreCase):

• On .NET 2, the results are roughly the same however big the "needle" is
• On .NET 4:
  • If needle is bigger than haystack (as per your example) .NET 4 is much faster than .NET 2
  • If needle is the same size as haystack, .NET 4 is a little bit slower than .NET 2
  • If needle is smaller than haystack, .NET 4 is a lot slower than .NET 2

(This is keeping a test where the first character of needle never appears in haystack, btw.)

Answer 2

I just ran your benchmark with a few tweaks (which included more iterations and averaging), and can confirm that the .NET 4.0 targeted version is indeed 4-5 times faster.

So presumably IndexOf() was optimised in .NET 4.0

Answer 3

OK, some benchmarks with the new VS11

n = 1000000;
string haystack = "ngrhotbegmhroes";
string needle = "blablablablablablablablablangrhotbegmhrobla bla";

.NET 4.5 :  8 ms
.NET 4.0 :  8 ms
.NET 3.5 : 45 ms
.NET 2.0 : 45 ms

So these first results confirm your findings, the newer versions are faster.

It is however much more common to look for s short string inside a larger string:

n = 1000000; 
haystack = "blablablablablablablablablangrhotbegmhrobla bla";  
needle = "ngrhotbegmhroes";

.NET 4.5 : 1020 ms
.NET 4.0 : 1020 ms
.NET 3.5 :  155 ms
.NET 2.0 :  155 ms

And with a much longer haystack (~400 chars)

.NET 4.0 : 12100 ms
.NET 2.0 :  1700 ms

Which means things got worse for the most common use pattern...

---

All measurements in Release config, and Client Profile where available.
Running from VS 11 with Ctrl+F5
Win 7H, Core i7 2620M