Web Api Requests Queueing up forever on IIS (in state: ExecuteRequestHandler)

Question

I'm currently experiencing some hangs on production environment, and after some investigation I'm seeing a lot of request queued up in the worker process of the Application Pool. The common thing is that every request that is queued for a long time is a web api request, I'm using both MVC and Web API in the app.

The requests are being queued for about 3 hours, when the application pool is recycled they immediately start queueing up.

They are all in ExecuteRequestHandler state

Any ideas for where should I continue digging?

Answer 1

Your requests can be stalled for a number of reasons:

• they are waiting on I/O operation e.g database, web service call
• they are looping or performing operations on a large data set
• cpu intensive operations
• some combination of the above

In order to find out what your requests are doing, start by getting the urls of the requests taking a long time.

You can do this in the cmd line as follows

c:\windows\system32\inetsrv\appcmd list requests

If its not obvious from the urls and looking at the code, you need to do a process dump of the w3wp.exe on the server. Once you have a process dump you will need to load it into windbg in order to analyze what's taking up all the cpu cycles. Covering off windbg is pretty big, but here's briefly what you need to do:

• load the SOS dll (managed debug extension)
• call the !runaway command
• to get list of long running threads dive into a long running thread by selecting it and calling !clrstack command

There are many blogs on using windbg. Here is one example. A great resource on analyzing these types of issues is Tess Ferrandez's blog.