Using multithreading for loop

Question

I'm new to threading and want to do something similar to this question:

Speed up loop using multithreading in C# (Question)

However, I'm not sure if that solution is the best one for me as I want them to keep running and never finish. (I'm also using .net 3.5 rather than 2.0 as for that question.)

I want to do something like this:

foreach (Agent agent in AgentList)
{
    // I want to start a new thread for each of these
    agent.DoProcessLoop();
}

---

public void DoProcessLoop()
{
    while (true)
    {
        // do the processing

        // this is things like check folder for new files, update database
        // if new files found
    }
}

Would a ThreadPool be the best solution or is there something that suits this better?

Update: Thanks for all the great answers! I thought I'd explain the use case in more detail. A number of agents can upload files to a folder. Each agent has their own folder which they can upload assets to (csv files, images, pdfs). Our service (it's meant to be a windows service running on the server they upload their assets to, rest assured I'll be coming back with questions about windows services sometime soon :)) will keep checking every agent's folder if any new assets are there, and if there are, the database will be updated and for some of them static html pages created. As it could take a while for them to upload everything and we want them to be able to see their uploaded changes pretty much straight away, we thought a thread per agent would be a good idea as no agent then needs to wait for someone else to finish (and we have multiple processors so wanted to use their full capacity). Hope this explains it!

Thanks,

Annelie

Answer 1

Given the specific usage your describe (watching for files), I'd suggest you use a FileSystemWatcher to determine when there are new files and then fire off a thread with the threadpool to process the files until there are no more to process -- at which point the thread exits.

This should reduce i/o (since you're not constantly polling the disk), reduce CPU usage (since the constant looping of multiple threads polling the disk would use cycles), and reduce the number of threads you have running at any one time (assuming there aren't constant modifications being made to the file system).

You might want to open and read the files only on the main thread and pass the data to the worker threads (if possible), to limit i/o to a single thread.

Answer 2

I believe that the Parallels Extensions make this possible:

Parallel.Foreach

http://msdn.microsoft.com/en-us/library/system.threading.tasks.parallel.foreach.aspx http://blogs.msdn.com/pfxteam/

Answer 3

One issue with ThreadPool would be that if the pool happens to be smaller than the number of Agents you would like to have, the ones you try to start later may never execute. Some tasks may never begin to execute, and you could starve everything else in your app domain that uses the thread pool as well. You're probably better off not going down that route.