How to detect High CPU and long response time issues with Asp.NET Web Api

Question

We are running Asp.Net WebApi on 3 servers behind HAProxy. HAProxy simply randomly distributes requests among these 3 instances.

These instances connect to mongodb, redis and some windows services.

Normally, w3wp.exe uses about %30 cpu on each api server.

From time to time (a few times in an hour) one of the api servers decide to use high amount of cpu. In correlation with this behavior, we start to see increasing response times. The numbers keep raising until HAProxy sees 10000ms response times and decides to route requests to other two servers. All these occur in 10-20 seconds. After a while, this server goes back to its normal state and start taking requests again. After a few minutes, another server does exactly the same thing. This keeps going on and on.

We are using New Relic but since the application is a WebApi application, we do not get any useful info. We monitor all our servers (redis, mongo and windows services) for cpu usage, memory usage, network traffic and I/O but we do not see any significant load during aforementioned outages.

How can we detect the cause behind this application behavior?

Answer 1

A good option would be to take a mini-dump using something like Process Explorer and then inspect it with WinDBG or otherwise, to see what the threads are doing and so forth. I have a good blog post about how to do it here:

http://www.haneycodes.net/but-it-didnt-happen-in-dev-or-qa/