ThreadPools vs Own Threads for long running processes

Question

We've got some long running business processes that are being initiated through WCF Services running in IIS (integrated mode) on WS 2008 R2. These business processes typically involve lots of interaction with our SQL Server backend. We have created a custom task queue implementation whereby the requests are queued via an initial service call and later executed based on priority. This execution can take long to complete (20-30 minutes extreme). Clients can then query the server for the progress of their own background tasks.

In our current implementation the tasks fires off on a separate thread to execute and not from the ThreadPool. This was done due to reading recommendations of not running long-running tasks using the ThreadPool as to prevent starving the ASP.NET requests from being served. We control the number of threads spawned by placing an upper limit on the number of background tasks that can be executed concurrently. This way we try to control the load on the CPU and prevent too much thread context switching. While all of this is happening we of course still need to serve the normal "on-line" requests for the application as well.

After reading this post by Thomas Marquardt I'm concerned about the fact that we are not using the ThreadPool as we won't get the benefit of the tuning heuristics built into it. We already solve the shutdown issue Thomas mentions by hooking into the ApplicationEnd event and cancelling the long running tasks. So my question is, should we switch over to using the ThreadPool? What about these threads being tied up for lengthy periods of time? If I understand Thomas correctly he is saying this doesn't matter as the ThreadPool will tune itself to create more requests to serve the normal on-line operations? I've also read through this StackOverflow question that covers the same grounds but I'm still unsure as to the way forward.

Answer 1

This is not exactly what you have asked - but why not execute your long running tasks in a separate process (say windows service) all together - you are anyway building a priority queue and clients are going to query back for results some time later. I would have gone with this approach where front facing WCF Services queues up requests and responds to status update queries while background service would keep serving requests. This would even allow worker process recycles etc w/o worrying about terminating currently executing tasks. Only issue that I see is overhead of out of process communication but then it should be insignificant compared to the time taken by task (as they are long running).

Answer 2

My suggestion is to avoid the ThreadPool as I am having exactly the thread starvation issues suggested in the links you have provided.

Our product allows user to fire off a call to a web service that carries out a series of quotes. This can be a long running process of up to 40 minutes, the quotation process is CPU intensive so, we use the thread pool on our quad core server to gain maximum CPU usage.

However, as ASP.NET looks as if it uses the thread pool to serve requests we have got an issue whereby any new jobs submitted by users never begin until the first job has completed due to the lack of available threads in the pool. To get around this I have been using a class i found on the web a while ago called ThreadPoolThrottle. This has allievated the problems a little bit, but as usage of the system is growing we are running into the same problems.

I am now considering VinayC's suggestion of running the quotes out of process in order to prevent thread starvation in my asp.net app.