Thread Queue Process

Question

I'm building this program in visual studio 2010 using C# .Net4.0 The goal is to use thread and queue to improve performance.

I have a list of urls I need to process.

string[] urls = { url1, url2, url3, etc.} //up to 50 urls

I have a function that will take in each url and process them.

public void processUrl(string url) { 
    //some operation
}

Originally, I created a for-loop to go through each urls.

for (i = 0; i < urls.length; i++)
    processUrl(urls[i]);

The method works, but the program is slow as it was going through urls one after another.

So the idea is to use threading to reduce the time, but I'm not too sure how to approach that.

Say I want to create 5 threads to process at the same time.

When I start the program, it will start processing the first 5 urls. When one is done, the program start process the 6th url; when another one is done, the program starts processing the 7th url, and so on.

The problem is, I don't know how to actually create a 'queue' of urls and be able to go through the queue and process.

Can anyone help me with this?

-- EDIT at 1:42PM --

I ran into another issue when I was running 5 process at the same time.

The processUrl function involve writing to log file. And if multiple processes timeout at the same time, they are writing to the same log file at the same time and I think that's throwing an error.

I'm assuming that's the issue because the error message I got was "The process cannot access the file 'data.log' because it is being used by another process."

Answer 1

The simplest option would be to just use Parallel.ForEach. Provided processUrl is thread safe, you could write:

Parallel.ForEach(urls, processUrl);

I wouldn't suggest restricting to 5 threads (the scheduler will automatically scale normally), but this can be done via:

Parallel.ForEach(urls, new ParallelOptions { MaxDegreeOfParallelism = 5}, processUrl);

That being said, URL processing is, by its nature, typically IO bound, and not CPU bound. If you could use Visual Studio 2012, a better option would be to rework this to use the new async support in the language. This would require changing your method to something more like:

public async Task ProcessUrlAsync(string url)
{
    // Use await with async methods in the implementation...

You could then use the new async support in the loop:

// Create an enumerable to Tasks - this will start all async operations..
var tasks = urls.Select(url => ProcessUrlAsync(url));

await Task.WhenAll(tasks); // "Await" until they all complete