How can C#'s string.IndexOf perform so fast, 10 times faster than ordinary for loop find?

Question

I have a very long string (60MB in size) in which I need to find how many pairs of '<' and '>' are in there.

---

I have first tried my own way:

        char pre = '!';
        int match1 = 0;
        for (int j = 0; j < html.Length; j++)
        {
            char c = html[j];
            if (pre == '<' && c == '>') //find a match
            {
                pre = '!';
                match1++;
            }
            else if (pre == '!' && c == '<')
                pre = '<';
        }

The above code runs on my string for roughly 1000 ms.

---

Then I tried using string.IndexOf

        int match2 = 0;
        int index = -1;
        do
        {
            index = html.IndexOf('<', index + 1);
            if (index != -1) // find a match
            {
                index = html.IndexOf('>', index + 1);
                if (index != -1)
                   match2++;
            }
        } while (index != -1);

The above code runs for only around 150 ms.

---

I am wondering what is the magic that makes string.IndexOf runs so fast?

Anyone can inspire me?

---

Edit

Ok, according to @BrokenGlass's answer. I modified my code in the way that they don't check the pairing, instead, they check how many '<' in the string.

---

        for (int j = 0; j < html.Length; j++)
        {
            char c = html[j];
            if (c == '>')
            {
                match1++;
            }
        }

the above code runs for around 760 ms.

---

Using IndexOf

        int index = -1;
        do
        {
            index = html.IndexOf('<', index + 1);
            if (index != -1)
            {
                match2++;
            }
        } while (index != -1);

The above code runs for about 132 ms. still very very fast.

---

Edit 2

After read @Jeffrey Sax comment, I realised that I was running in VS with Debug mode.

Then I built and ran in release mode, ok, IndexOf is still faster, but not that faster any more.

Here is the results:

For the pairing count: 207ms VS 144ms

For the normal one char count: 141ms VS 111ms.

My own codes' performance was really improved.

---

Lesson learned: when you do the benchmark stuff, do it in release mode!

Answer 1

Are you running your timings from within Visual Studio? If so, your code would run significantly slower for that reason alone.

Aside from that, you are, to some degree, comparing apples and oranges. The two algorithms work in a different way.

The IndexOf version alternates between looking for an opening bracket only and a closing bracket only. Your code goes through the whole string and keeps a status flag that indicates whether it is looking for an opening or a closing bracket. This takes more work and is expected to be slower.

Here's some code that does the comparison the same way as your IndexOf method.

int match3 = 0;
for (int j = 0; j < html.Length; j++) {
    if (html[j] == '<') {
        for (; j < html.Length; j++)
            if (html[j] == '>')
                match3++;
    }
}

In my tests this is actually about 3 times faster than the IndexOf method. The reason? Strings are actually not quite as simple as sequences of individual characters. There are markers, accents, etc. String.IndexOf handles all that extra complexity properly, but it comes at a cost.

Answer 2

The only thing that comes to my mind is actual implementation of IndexOf iniside string class, that call

callvirt    System.String.IndexOf

which, if we use a power of reflector (as much as it possible) ends up into the

CompareInfo.IndexOf(..)

call, which instead use super fast windows native function FindNLSString: