Workaround for the WaitHandle.WaitAll 64 handle limit?

Question

My application spawns loads of different small worker threads via ThreadPool.QueueUserWorkItem which I keep track of via multiple ManualResetEvent instances. I use the WaitHandle.WaitAll method to block my application from closing until these threads have completed.

I have never had any issues before, however, as my application is coming under more load i.e. more threads being created, I am now beginning to get this exception:

WaitHandles must be less than or equal to 64 - missing documentation

What is the best alternative solution to this?

Code Snippet

List<AutoResetEvent> events = new List<AutoResetEvent>();

// multiple instances of...
var evt = new AutoResetEvent(false);
events.Add(evt);
ThreadPool.QueueUserWorkItem(delegate
{
    // do work
    evt.Set();
});

...
WaitHandle.WaitAll(events.ToArray());

Workaround

int threadCount = 0;
ManualResetEvent finished = new ManualResetEvent(false);

...
Interlocked.Increment(ref threadCount);
ThreadPool.QueueUserWorkItem(delegate
{
    try
    {
         // do work
    }
    finally
    {
        if (Interlocked.Decrement(ref threadCount) == 0)
        {
             finished.Set();
        }
    }
});

...
finished.WaitOne();

Answer 1

Create a variable that keeps track of the number of running tasks:

int numberOfTasks = 100;

Create a signal:

ManualResetEvent signal = new ManualResetEvent(false);

Decrement the number of tasks whenever a task is finished:

if (Interlocked.Decrement(ref numberOftasks) == 0)
{

If there is no task remaining, set the signal:

    signal.Set();
}

Meanwhile, somewhere else, wait for the signal to be set:

signal.WaitOne();

Answer 2

Starting with .NET 4.0, you have two more (and IMO, cleaner) options available to you.

The first is to use the CountdownEvent class. It prevents the need of having to handle the incrementing and decrementing on your own:

int tasks = <however many tasks you're performing>;

// Dispose when done.
using (var e = new CountdownEvent(tasks))
{
    // Queue work.
    ThreadPool.QueueUserWorkItem(() => {
        // Do work
        ...

        // Signal when done.
        e.Signal();
    });

    // Wait till the countdown reaches zero.
    e.Wait();
}

However, there's an even more robust solution, and that's to use the Task class, like so:

// The source of your work items, create a sequence of Task instances.
Task[] tasks = Enumerable.Range(0, 100).Select(i =>
    // Create task here.
    Task.Factory.StartNew(() => {
        // Do work.
    }

    // No signalling, no anything.
).ToArray();

// Wait on all the tasks.
Task.WaitAll(tasks);

Using the Task class and the call to WaitAll is much cleaner, IMO, as you're weaving less threading primitives throughout your code (notice, no wait handles); you don't have to set up a counter, handle incrementing/decrementing, you just set up your tasks and then wait on them. This lets the code be more expressive in the what of what you want to do and not the primitives of how (at least, in terms of managing the parallelization of it).

.NET 4.5 offers even more options, you can simplify the generation of the sequence of Task instances by calling the static Run method on the Task class:

// The source of your work items, create a sequence of Task instances.
Task[] tasks = Enumerable.Range(0, 100).Select(i =>
    // Create task here.
    Task.Run(() => {
        // Do work.
    })

    // No signalling, no anything.
).ToArray();

// Wait on all the tasks.
Tasks.WaitAll(tasks);

Or, you could take advantage of the TPL DataFlow library (it's in the System namespace, so it's official, even though it's a download from NuGet, like Entity Framework) and use an ActionBlock<TInput>, like so:

// Create the action block.  Since there's not a non-generic
// version, make it object, and pass null to signal, or
// make T the type that takes the input to the action
// and pass that.
var actionBlock = new ActionBlock<object>(o => {
    // Do work.
});

// Post 100 times.
foreach (int i in Enumerable.Range(0, 100)) actionBlock.Post(null);

// Signal complete, this doesn't actually stop
// the block, but says that everything is done when the currently
// posted items are completed.
actionBlock.Complete();

// Wait for everything to complete, the Completion property
// exposes a Task which can be waited on.
actionBlock.Completion.Wait();

Note that the ActionBlock<TInput> by default processes one item at a time, so if you want to have it process multiple actions at one time, you have to set the number of concurrent items you want to process in the constructor by passing a ExecutionDataflowBlockOptions instance and setting the MaxDegreeOfParallelism property:

var actionBlock = new ActionBlock<object>(o => {
    // Do work.
}, new ExecutionDataflowBlockOptions { MaxDegreeOfParallelism = 4 });

If your action is truly thread safe, then you can set the MaxDegreeOfParallelsim property to DataFlowBlockOptions.Unbounded:

var actionBlock = new ActionBlock<object>(o => {
    // Do work.
}, new ExecutionDataflowBlockOptions { 
    MaxDegreeOfParallelism = DataFlowBlockOptions.Unbounded
});

The point being, you have fine-grained control over how parallel you want your options to be.

Of course, if you have a sequence of items that you want passed into your ActionBlock<TInput> instance, then you can link an ISourceBlock<TOutput> implementation to feed the ActionBlock<TInput>, like so:

// The buffer block.
var buffer = new BufferBlock<int>();

// Create the action block.  Since there's not a non-generic
// version, make it object, and pass null to signal, or
// make T the type that takes the input to the action
// and pass that.
var actionBlock = new ActionBlock<int>(o => {
    // Do work.
});

// Link the action block to the buffer block.
// NOTE: An IDisposable is returned here, you might want to dispose
// of it, although not totally necessary if everything works, but
// still, good housekeeping.
using (link = buffer.LinkTo(actionBlock, 
    // Want to propagate completion state to the action block.
    new DataflowLinkOptions {
        PropagateCompletion = true,
    },
    // Can filter on items flowing through if you want.
    i => true)
{ 
    // Post 100 times to the *buffer*
    foreach (int i in Enumerable.Range(0, 100)) buffer.Post(i);

    // Signal complete, this doesn't actually stop
    // the block, but says that everything is done when the currently
    // posted items are completed.
    actionBlock.Complete();

    // Wait for everything to complete, the Completion property
    // exposes a Task which can be waited on.
    actionBlock.Completion.Wait();
}

Depending on what you need to do, the TPL Dataflow library becomes a much more attractive option, in that it handles the concurrency across all the tasks linked together, and it allows you to be very specific about just how parallel you want each piece to be, while maintaining proper separation of concerns for each block.

Answer 3

Your workaround is not correct. The reason is that the Set and WaitOne could race if the last work item causes the threadCount to go to zero before the queueing thread has had to chance to queue all work items. The fix is simple. Treat your queueing thread as if it were a work item itself. Initialize threadCount to 1 and do a decrement and signal when the queueing is complete.

int threadCount = 1;
ManualResetEvent finished = new ManualResetEvent(false);
...
Interlocked.Increment(ref threadCount); 
ThreadPool.QueueUserWorkItem(delegate 
{ 
    try 
    { 
         // do work 
    } 
    finally 
    { 
        if (Interlocked.Decrement(ref threadCount) == 0) 
        { 
             finished.Set(); 
        } 
    } 
}); 
... 
if (Interlocked.Decrement(ref threadCount) == 0)
{
  finished.Set();
}
finished.WaitOne(); 

As a personal preference I like using the CountdownEvent class to do the counting for me.

var finished = new CountdownEvent(1);
...
finished.AddCount();
ThreadPool.QueueUserWorkItem(delegate 
{ 
    try 
    { 
         // do work 
    } 
    finally 
    { 
      finished.Signal();
    } 
}); 
... 
finished.Signal();
finished.Wait();