Easy way to process queue with N threads in C#?

Question

I have a list of account numbers. Foreach account number I need to call method ProcessAccount. There will be approximately 150,000 accounts that need to be processed and each account could take between .5 to 2 seconds to process.

I'd like to setup threading somehow so I can be processing 4 accounts at a time for example.

Is there a simple pattern I can use for this?

What I'd like to be able to do is start 4 threads processing the first 4 accounts, and then as each individual thread is finished start another thread with the next account until all the accounts have been processed.

Answer 1

This is easy to handle with the TPL (Task Parallel Library). It would look something like

ParallelOptions options = new ParallelOptions() { MaxDegreeOfParallelism = 4 };
Parallel.ForEach(accounts, options, a =>
{
    ProcessAccount(a);
});

http://msdn.microsoft.com/en-us/library/dd782721.aspx

Note that the TPL might decide to run less than 4 concurrent threads, but will not run more than 4 based on the specified options. It might do that, for example, if it determines that the provided lamda (which calls ProcessAccount) is CPU bound and there are less than 4 CPU cores on the system. Generally, especially in .NET 4.5, the TPL makes very good decisions about the number of threads to use.

As @Servy notes in the comments, unless you have a very specific reason to limit the code to 4 threads, it is best to just let TPL sort out how many threads to use on its own. That way, if the same code is running on a 128 core processor in the year 2018, long after you move on to other things, it is free to use all 128 cores).

Answer 2

Use PLinq:

var accounts = //some 150,000 account numbers
accounts.AsParallel().ForAll(ProcessAccount);

or, if other arguments are required, use a lambda expression:

accounts.AsParallel().ForAll(account => ProcessAccount(account, argument2, argument3));