C# Task thread pool - Running 100 tasks across only 10 threads [duplicate]

Question

I'm just wondering if someone can point me in the right direction about the async/await framework and thread pools?

Basically, what I'm trying to do is have x number of operations executed in a separate thread/async, but across a maximum of y threads.

For example, let's say that I have 100 database operations: await _repository.WriteData(someData);

What I'd like to do is have some method of running 10 of those operations at a time (ideally in a separate thread each, so 10 threads), and as each one is finished, the next is kicked off on the thread that then becomes available. We then wait for all operations to complete and all threads to finish...

Is this something that is readily achievable without too much effort or adding huge amounts of complexity?

Answer 1

I think you're missing the point by focusing on threads, especially for asynchronous operations that don't require threads to execute.

.NET has a great ThreadPool you can use. You don't know how many threads are in it and you don't care. It just works (until it doesn't and you need to configure it yourself, but that's very advance).

Running tasks on the ThreadPool is very simple. Either create a task for each operation and throttle them using a SemaphoreSlim or use the ready-made TPL Dataflow blocks. For example:

var block = new ActionBlock<SomeData>(
    _ => _repository.WriteDataAsync(_), // What to do on each item
    new ExecutionDataflowBlockOptions { MaxDegreeOfParallelism = 10 }); // How many items at the same time

foreach (var item in items)
{
    block.Post(item); // Post all items to the block
}

block.Complete(); // Signal completion
await block.Completion; // Asynchronously wait for completion.

If, however, you do plan on creating "dedicated" threads you can use Task.Factory.StartNew with the LongRunning option that creates a dedicated thread outside the ThreadPool. But keep in mind that async operations don't maintain the same thread throughout the operation as async operations don't need a thread. So starting on a dedicated thread may be pointless (more on that on my blog: LongRunning Is Useless For Task.Run With async-await)