.NET cuts down on parallel threads on repeating runs

Question

Is there any good reason for .NET to cut down on number of parallel threads over time?

I am running calculations in many passes that take days to complete (each pass takes ~1 hour). The tasks are pure calculations of data in memory (read from disk). I'm using Parallel.For and Parallel.ForEach several places, both for main task and inside the task. Everything is repeated in many passes. The class instances are disposed (memory profiler shows no problems over time) properly for every pass and a new instance is created. It is repeating 100% same task in every pass with the exception of some numbers in the math being changed (equal number of iterations every time, same dataset).

The computer has six cores, and the app starts out by using all of them. After some time it uses 5, then 4, then 3, then 2. Looking at Parallel stacks (Debug->Window->Parallel stacks) it confirms that only that many are running.

How come .NET isn't maxing out number of threads on every pass? Does it regulate threads based on CPU usage?

Tips on how to debug? Can I force number of threads to be used?

Answer 1

I believe that this article on ThreadPool concurrency should provide some clues.

Tasks created by the Parallel methods will eventually end up executed by the ThreadPool. The ideal number of threads for the ThreadPool to use varies depending on the type of tasks being executed. If the tasks block a lot and don't have much contention, a higher number of threads will result in higher throughput. For tasks with not much blocking and high contention for limited resources, a lower number of threads will result in higher throughput.

Because of this property, the ThreadPool in .NET 4 implements a "hill climbing" algorithm where it tunes the number of threads the ThreadPool is running and reacts based on the measured throughput.

So one thing you could check is to see if the actual job throughput is decreasing with the lowered number of threads. It's also possible that as your app runs it's bottlenecked more and more on disk operations and the threads are being shut down simply from disuse.

As for forcing the ThreadPool to use a certain number of threads, I don't think that's possible. There's ThreadPool.SetMinThreads but that has never worked for me. Generally .NET will just ignore those values and use whatever number of threads it wants to.