C# Threading - Reading and hashing multiple files concurrently, easiest method?

Question

I've been trying to get what I believe to be the simplest possible form of threading to work in my application but I just can't do it.

What I want to do: I have a main form with a status strip and a progress bar on it. I have to read something between 3 and 99 files and add their hashes to a string[] which I want to add to a list of all files with their respective hashes. Afterwards I have to compare the items on that list to a database (which comes in text files). Once all that is done, I have to update a textbox in the main form and the progressbar to 33%; mostly I just don't want the main form to freeze during processing.

The files I'm working with always sum up to 1.2GB (+/- a few MB), meaning I should be able to read them into byte[]s and process them from there (I have to calculate CRC32, MD5 and SHA1 of each of those files so that should be faster than reading all of them from a HDD 3 times).

Also I should note that some files may be 1MB while another one may be 1GB. I initially wanted to create 99 threads for 99 files but that seems not wise, I suppose it would be best to reuse threads of small files while bigger file threads are still running. But that sounds pretty complicated to me so I'm not sure if that's wise either.

So far I've tried workerThreads and backgroundWorkers but neither seem to work too well for me; at least the backgroundWorkers worked SOME of the time, but I can't even figure out why they won't the other times... either way the main form still froze. Now I've read about the Task Parallel Library in .NET 4.0 but I thought I should better ask someone who knows what he's doing before wasting more time on this.

What I want to do looks something like this (without threading):

List<string[]> fileSpecifics = new List<string[]>();

int fileMaxNumber = 42; // something between 3 and 99, depending on file set

for (int i = 1; i <= fileMaxNumber; i++)
{
    string fileName = "C:\\path\\to\\file" + i.ToString("D2") + ".ext"; // file01.ext - file99.ext
    string fileSize = new FileInfo(fileName).Length.ToString();
    byte[] file = File.ReadAllBytes(fileName);
    // hash calculations (using SHA1CryptoServiceProvider() etc., no problems with that so I'll spare you that, return strings)
    file = null; // I didn't yet check if this made any actual difference but I figured it couldn't hurt
    fileSpecifics.Add(new string[] { fileName, fileSize, fileCRC, fileMD5, fileSHA1 });
}

// look for files in text database mentioned above, i.e. first check for "file bundles" with the same amount of files I have here; then compare file sizes, then hashes
// again, no problems with that so I'll spare you that; the database text files are pretty small so parsing them doesn't need to be done in an extra thread.

Would anybody be kind enough to point me in the right direction? I'm looking for the easiest way to read and hash those files quickly (I believe the hashing takes some time in which other files could already be read) and save the output to a string[], without the main form freezing, nothing more, nothing less.

I'm thankful for any input.

EDIT to clarify: by "backgroundWorkers working some of the time" I meant that (for the very same set of files), maybe the first and fourth execution of my code produces the correct output and the UI unfreezes within 5 seconds, for the second, third and fifth execution it freezes the form (and after 60 seconds I get an error message saying some thread didn't respond within that time frame) and I have to stop execution via VS.

Thanks for all your suggestions and pointers, as you all have correctly guessed I'm completely new to threading and will have to read up on the great links you guys posted. Then I'll give those methods a try and flag the answer that helped me the most. Thanks again!

Answer 1

With .NET Framework 4.X

1. Use Directory.EnumerateFiles Method for efficient/lazy files enumeration
2. Use Parallel.For() to delegate parallelism work to PLINQ framework or use TPL to delegate single Task per pipeline Stage
3. Use Pipelines pattern to pipeline following stages: calculating hashcodes, compare with pattern, update UI
4. To avoid UI freeze use appropriate techniques: for WPF use Dispatcher.BeginInvoke(), for WinForms use Invoke(), see this SO answer
5. Considering that all this stuff has UI it might be useful adding some cancellation feature to abandon long running operation if needed, take a look at the CreateLinkedTokenSource class which allows triggering CancellationToken from the "external scope" I can try adding an example but it's worth do it yourself so you would learn all this stuff rather than simply copy/paste - > got it working -> forgot about it.

PS: Must read - Pipelines paper at MSDN

---

TPL specific pipeline implementation

• Pipeline pattern implementation: three stages: calculate hash, match, update UI
• Three tasks, one per stage
• Two Blocking Queues

//

// 1) CalculateHashesImpl() should store all calculated hashes here
// 2) CompareMatchesImpl() should read input hashes from this queue
// Tuple.Item1 - hash, Typle.Item2 - file path
var calculatedHashes = new BlockingCollection<Tuple<string, string>>();


// 1) CompareMatchesImpl() should store all pattern matching results here
// 2) SyncUiImpl() method should read from this collection and update 
//    UI with available results
var comparedMatches = new BlockingCollection<string>();

var factory = new TaskFactory(TaskCreationOptions.LongRunning,
                              TaskContinuationOptions.None);


var calculateHashesWorker = factory.StartNew(() => CalculateHashesImpl(...));
var comparedMatchesWorker = factory.StartNew(() => CompareMatchesImpl(...));
var syncUiWorker= factory.StartNew(() => SyncUiImpl(...));

Task.WaitAll(calculateHashesWorker, comparedMatchesWorker, syncUiWorker);

CalculateHashesImpl():

private void CalculateHashesImpl(string directoryPath)
{
   foreach (var file in Directory.EnumerateFiles(directoryPath))
   {
       var hash = CalculateHashTODO(file);
       calculatedHashes.Add(new Tuple<string, string>(hash, file.Path));
   }
}

CompareMatchesImpl():

private void CompareMatchesImpl()
{
   foreach (var hashEntry in calculatedHashes.GetConsumingEnumerable())
   {
      // TODO: obviously return type is up to you
      string matchResult = GetMathResultTODO(hashEntry.Item1, hashEntry.Item2);
      comparedMatches.Add(matchResult);
   }
}

SyncUiImpl():

private void UpdateUiImpl()
{
    foreach (var matchResult in comparedMatches.GetConsumingEnumerable())
    {
        // TODO: track progress in UI using UI framework specific features
        // to do not freeze it
    }
}

TODO: Consider using CancellationToken as a parameter for all GetConsumingEnumerable() calls so you easily can stop a pipeline execution when needed.

Answer 2

First off, you should be using a higher level of abstraction to solve this problem. You have a bunch of tasks to complete, so use the "task" abstraction. You should be using the Task Parallel Library to do this sort of thing. Let the TPL deal with the question of how many worker threads to create -- the answer could be as low as one if the work is gated on I/O.

If you do want to do your own threading, some good advice:

• Do not ever block on the UI thread. That's is what is freezing your application. Come up with a protocol by which working threads can communicate with your UI thread, which then does nothing except for responding to UI events. Remember that methods of user interface controls like task completion bars must never be called by any other thread other than the UI thread.

• Do not create 99 threads to read 99 files. That's like getting 99 pieces of mail and hiring 99 assistants to write responses: an extraordinarily expensive solution to a simple problem. If your work is CPU intensive then there is no point in "hiring" more threads than you have CPUs to service them. (That's like hiring 99 assistants in an office that only has four desks. The assistants spend most of their time waiting for a desk to sit at instead of reading your mail.) If your work is disk-intensive then most of those threads are going to be idle most of the time waiting for the disk, which is an even bigger waste of resources.