Parallel.For freezes after around 1370 iterations, no idea why

Question

I'm running a Parallel.For loop on a little over 7500 objects. Inside that for loop, I'm doing a number of things to each of those objects, specifically calling two web services and two internal methods. The web services simply inspect the object, process and return a string that i then set as a property on the object. Same goes for the two internal methods.

I'm not writing anything out to disk or reading from disk.

I also update the UI in a winforms app with a label and progress bar to let the user know where it's at. Here's the code:

var task = Task.Factory.StartNew(() =>
{
  Parallel.For(0, upperLimit, (i, loopState) =>
  {
     if (cancellationToken.IsCancellationRequested)
        loopState.Stop();
     lblProgressBar.Invoke(
       (Action)
       (() => lblProgressBar.Text = string.Format("Processing record {0} of {1}.", (progressCounter++), upperLimit)));
     progByStep.Invoke(
       (Action)
       (() => progByStep.Value = (progressCounter - 1)));

      CallSvc1(entity[i]);
      Conversion1(entity[i]);
      CallSvc2(entity[i]);
      Conversion2(entity[i]);
  });
}, cancellationToken);

This is taking place on a Win7 32bit machine.

Any ideas as to why this suddenly freezes when the incrementer is around 1370 or so (It's been 1361, 1365 and 1371)?

Any ideas as to how I can debug this and see what's locking if anything?

EDIT:
Some answers to the comments below:
@BrokenGlass - No, no interop. I'll try the x86 compilation and let you know.

@chibacity - Because it's on a background task, it's not freezing the UI. Up until the time it freezes, the progress bar and label tick along at about 2 every second. When it freezes, it just stops moving. I can verify that the number that it stops at has been processed, but no more. CPU usage on a dual core 2.2GHz is minimal during the operation at 3-4% each and 1-2% once it freezes.

@Henk Holterman - It takes around 10-12 minutes to get to 1360 and yes, I can verify that all of those records have been processed but not the remaining records.

@CodeInChaos - Thanks, I'll try that! The code does work if I take out the parallel, it just takes forever and a day. I haven't tried restricting the number of threads, but will.

EDIT 2:
Some details as to what's going on with the webservices

Basically what's going on with the web services is that they pass in some data and receive data (an XmlNode). That node is then used in the Conversion1 process which in turn sets another property on the entity which is sent to the CallSvc2 method and so on. It looks like this:

private void CallSvc1(Entity entity)
{
    var svc = new MyWebService();
    var node = svc.CallMethod(entity.SomeProperty);
    entity.FieldToUpdate1.LoadXml(node.InnerXml);
}
private void Conversion1(Entity entity)
{
    // Do some xml inspection/conversion stuff
    if (entity.FieldToUpdate1.SelectSingleNode("SomeNode") == "something") {
        entity.FieldToUpdate2 = SomethingThatWasConverted;
    }
    else {
        // Do some more logic
    }
}
private void CallSvc2(Entity entity)
{
    var svc = new SomeOtherWebService();
    var xmlNode = svc.MethodToCall(entity.FieldToUpdate2.InnerXml);
    entity.AnotherXmlDocument.LoadXml(xmlNode.InnerXml);
}

As you can see, it's pretty straightforward stuff. There's a lot going on in some of the conversion methods, but none of it should be blocking. And as noted below, there are 1024 threads in "waiting" status that are all sitting on the webservice calls. I read here http://www.albahari.com/threading/ that the MaxThreads is defaulted to 1023 for .Net 4 on 32 bit machine.

How can I release those waiting threads given what I have here?

Answer 1

A possible explanation: you've got the process into a state where it can't create any more threads, which is preventing work from making progress, which is why everything's grinding to a halt.

Frankly, whether or not that hypothesis turns out to be correct, you need to take a completely different approach to this. Parallel.For is the wrong way to solve this. (Parallel is best suited to CPU-bound work. What you have here is IO-bound work.) If you truly need to have thousands of web service requests in progress, you need to move over to using asynchronous code, instead of multithreaded code. If you use async APIs, you'll be able to start thousands of requests simultaneously while using only a handful of threads.

Whether those requests will actually be able to execute simultaneously is another matter - whether you use your current "thread apocalypse" implementation or a more efficient async implementation, you may well be running into throttling. (.NET can sometimes limit the number of requests it'll actually make.) So you can ask to make as many requests as you like, but you might find that almost all of your requests are sat waiting for earlier ones to complete. E.g. I think WebRequest limits concurrent connections to any single domain to just 2... Firing up 1000+ threads (or 1000+ async requests) is just going to result in loads more requests sitting waiting to be one of the 2 current requests!

You should do your own throttling. You need to decide how many outstanding requests to have simultaneously, and make sure you only start that many requests at once. Just asking Parallel to launch as many as it can as quickly as it can will bog everything down.

Updated to add:

A quick fix might be to use the overload of Parallel.For that accepts a ParallelOptions object - you can set its MaxDegreeOfParallelism property to limit the number of concurrent requests. That would stop this thread-heavy implementation from actually running out of threads. But it remains an inefficient solution to the problem. (And for all I know, you do actually need to make thousands of concurrent requests. If you're writing a web crawler, for example, that's actually a reasonable thing to want to do. Parallel is not the right class for that job though. Use async operations. If the web service proxies you're using support the APM (BeginXxx, EndXxx), you can wrap that up in Task objects - Task.TaskFactory offers a FromAsync that'll provide a task representing an async operation in progress.

But if you are going to try to have thousands of requests in flight at once, you do need to think carefully about your throttling strategy. Just throwing requests out there as fast as possible is unlikely to be the optimal strategy.

Answer 2

Run the app in the VS debugger. When it seems to lock up, tell VS to Debug: Break All. Then go to Debug: Windows: Threads and look at the threads in your process. Some of them should be showing stack traces that are in your parallel for loop, and that will tell you what they were doing when the process was stopped by the debugger.