Windows Azure Role Unresponsive

Question

I have a worker role wherein a number of threads run to handle connections from clients. During load testing, we've noticed that the role becomes unresponsive, and stays in that state until we restart it. Looking at it in the visual studio debugger (by pausing the active process), we can see a number of threads waiting but when it's unpaused, no activity takes place. How do I know what's causing the role to remain in that state?

Answer 1

AS you mentioned that the problem started to happen with your Worker Role when you perform load test, it is safe to assume that the Role was working fine initially. This drives to the conclusion that problem is application specific rather then something platform specific. There are two ways you can get some diagnostics information from your Windows Azure VM:

• Option 1(Preferred): In order to check what is happening in your Azure VM, you really need to add Windows Azure Diagnostics with your role so you can get performance counter, memory details, role process health and other data shifted from Azure VM to your Azure Storage which you can analyze offline.

• Option 2: Enable RDP access to your Azure VM and then Log into your Azure VM and install, Perfmon, Process Explorer and other health monitoring tools in your Azure VM and monitor what is happening.

• Option 3 (Last Resort): RDP to your Azure VM and install WinDBG and debug the host process

In such condition when you have a worker role stopped working, this is what I really suggest to any one:

1. look for CPU, Memory pressure, list of threads and find out which threads are blocked and what could be reason.
2. Look for host worker process health, when it started last time, is it recycling due to some reason, how long it is running in the Azure VM
3. If you have a separate process which is set as "ProgramEntryPoint" in your worker role, check for its health
4. Check for event log at application an system level for any clue. In Worker role I dont see there will be something in application event log but it is worth to take a look.
5. I use Process Monitor to monitor a few process and collect the log to see when process die what was the last state, what it was accessing and was it being starved by other issues.

The bottom line is you really need to dig the root cause in Azure VM pretty much same way you will do in an on-premise machine.