Azure - how to troubleshoot failing deployments

Question

I have a worker role that fails to deploy (cycles between initializing and aborted) in the management console. It runs fine in the emulator.

The frustrating thing is not that the deployment fails, its that it seems virtually impossible to find out why.

I've checked all my connection strings, enabled diagnostics, checked all my assemblies are deployed, googled ALOT and lost some hair.

The point I'm at now is that I'm left adding code bit by bit, and redeploying to find the code that fails, a process that is ridiculously slow.

The worker itself connects to sql azure, and azure storage. I have it connecting to the live endpoints in the emulator without any problems.

It seems to fail as soon as I configure StructureMap (IoC). However, I'm using near identical code in my web role and this works fine.

So where can I go from here (apart from to the bottle)?

Answer 1

I'm going to start by re-iterating the feedback you've received so far. The biggest killer is the Run() process in the WorkerRole. If the WorkerRole is having trouble starting, you can wrap the code inside this method with a try/catch and log it.

If you choose to use the built in diagnostics, I'd recommend reading through Ryan Dunn's blog, as well as, smarx's blog. Both have tread this ground and done a great job documenting/sharing as they go. The MSDN site (sorry, first answer so only two links:)) has also improved quite a bit on this topic.

The part I'll add to this conversation, is in HOW you follow the advice. I don't use Intellitrace as I don't have access to it, and have resorted to configuring Remote Desktop (can be done from within Visual Studio) to my roles when hitting the wall. If you configure log4net, or something similar, (local to the role) you'll be able to log on via RDP and read through the logs.

Now, two of the things we find most frequently..

1. UseDevelopmentStorage=True - this is a default setting and can create problems when deployed. There's quite a bit written on this already.

2. Dependencies - There are many things devs have access to that are not in the Hosted Role. The easiest example of this, IMO, is ASP.NET MVC. You can either manage with the 'Stable Release' philosophy, or use something like the Web Platform Installer Command line (there's also the Azure Boostrapper on dunnry's blog) to prep the Role before startup.

For me, the key is the RDP as you can actually log on and see what's happening.

UPDATE - Can't believe I forgot this one as it kills me all the time, but, you may also need to configure the Firewall if using SQL Azure. In the dev process, we'll often destroy and redeploy our roles, instead of updating, and leads to occasional IP address changes. If these aren't configured in the Firewall where SQL Azure is involved, can be problematic.

Hope this helps man.