Fastest Way to Parse Large Strings (multi threaded)

Question

I am about to start a project which will be taking blocks of text, parsing a lot of data into them into some sort of object which can then be serialized, stored, and statistics / data gleaned from. This needs to be as fast as possible as I have > 10,000,000 blocks of text that I need to start on and will be getting 100,000's of thousands a day.

I am running this on a system with 12 xeon cores + hyper threading. I also have access / know a bit about CUDA programming but for string stuff think that its not appropriate. From each string I need to parse a lot of data and some of it I know the exact positions of, some I don't and need to use regex's / something smart.

So consider something like this:

object[] parseAll (string [] stringsToParse)
{
     parallel foreach 
          parse( string[n] )
}

object parse(string s)
{
     try to use exact positions / substring etc here instead of regex's
}

So my questions are:

• How much slower is using regex's to substr.
• Is .NET going to be significantly slower than other languages.
• What sort of optimizations (if any) can I do to maximize parallelism.
• Anything else I haven't considered?

Thanks for any help! Sorry if this is long winded.

Answer 1

How much slower is using regex's to substr.
If you are looking for an exact string, substr will be faster. Regular expressions however are highly optimized. They (or at least parts) are compiled to IL and you can even store these compiled versions in a separate assembly using Regex.CompileToAssembly. See http://msdn.microsoft.com/en-us/library/9ek5zak6.aspx for more information.

What you really need to do is do perform measurements. Using something like Stopwatch is by far the easiest way to verify whether one or the other code construct works faster.

What sort of optimizations (if any) can I do to maximize parallelism.
With Task.Factory.StartNew, you can schedule tasks to run on the thread pool. You may also have a look at the TPL (Task Parallel Library, of which Task is a part). This has lots of constructs that help you parallelize work and allows constructs like Parallel.ForEach() to execute an iteration on multiple threads. See http://msdn.microsoft.com/en-us/library/dd460717.aspx for more information.

Anything else I haven't considered?
One of the things that will hurt you with this volume of data is memory management. A few things to take into account:

• Limit memory allocation: try to re-use the same buffers for a single document instead of copying them when you only need a part. Say you need to work on a range starting at char 1000 to 2000, don't copy that range into a new buffer, but construct your code to work only in that range. This will make your code complexer, but it saves you memory allocations;

• StringBuilder is an important class. If you don't know of it yet, have a look.