Parallel.ForEach questions

Question

I am using a Parallel.ForEach loop in C# / VS2010 to do processing and I have a couple of questions.

First of all I have a process that needs to extract information from a remote webservice and then needs to build images (GDI) on the fly.

I have a class that encapsulates all of the functionality into a single object with two main methods Load() and CreateImage() with all the GDI management / WebRequests "blackboxed" inside this object.

I then create a GenericList that contains all the objects that need to be processed and I iterate through the list using the following code:

try
        {
            Parallel.ForEach(MyLGenericList, ParallelOptions, (MyObject, loopState) =>
            {                                       

                    MyObject.DoLoad();
                    MyObject.CreateImage();
                    MyObject.Dispose();

                if (loopState.ShouldExitCurrentIteration || loopState.IsExceptional)
                    loopState.Stop();
            });
        }
        catch (OperationCanceledException ex)
        {
            // Cancel here
        }
        catch (Exception ex)
        {
            throw ex;
        }

Now my questions are:

1. Given that there could be ten thousand items in the list to parse, is the above code the best way to approach this? Any other ideas more then welcome
2. I have an issue whereby when I start the process the objects are created / loaded and images created very fast but after around six hundred objects the process starts to crawl. It doesn eventually finish, is this normal?

Thanks in advance :) Adam

Answer 1

I am not sure that downloading data in parallel is a good idea since it will block a lot of threads. Split your task into a producer and a consumer instead. Then you can parallelize each of them separately.

Here is an example of a single producer and multiple consumers.
(If the consumers are faster than the producer you can just use a normal foreach instead of parallel.ForEach)

var sources = BlockingCollection<SourceData>();
var producer = Task.Factory.CreateNew(
    () => {
        foreach (var item in MyGenericList) {
            var data = webservice.FetchData(item);
            sources.Add(data)
        }
        sources.CompleteAdding();
    }
)
Parallel.ForEach(sources.GetConsumingPartitioner(),
                 data => {
                     imageCreator.CreateImage(data);
                 });

(the GetConsumingPartitioner extension is part of the ParallelExtensionsExtras)

Edit A more complete example

var sources = BlockingCollection<SourceData>();

var producerOptions = new ParallelOptions { MaxDegreeOfParallelism = 5 };
var consumerOptions = new ParallelOptions { MaxDegreeOfParallelism = -1 };

var producers = Task.Factory.CreateNew(
    () => {
        Parallel.ForEach(MyLGenericList, producerOptions, 
            myObject => {
                myObject.DoLoad()
                sources.Add(myObject)
            });
        sources.CompleteAdding();
    });
Parallel.ForEach(sources.GetConsumingPartitioner(), consumerOptions,
    myObject => {
        myObject.CreateImage();
        myObject.Dispose();
    });

With this code you can optimize the amount of parallel downloads while keeping the cpu busy with the image processing.