Increase/Decrease the number of Worker Role instances in Azure

Question

I can increase the number of Worker Role (WR) instances directly from Java using the ServiceManagementRest class in the Azure4Java package. See the tutorial Azure Management through Java.

My question is, when I decrease the number of WR instances, can I decide which WR instances shut down? Because, for the Cloud elasticity idea, I would stop the instances in a IDLE status and not the instances in EXECUTING status.

Regards, Fabrizio

Answer 1

You can't choose which instance(s) to shut down; you simply change the instance count, and the fabric controller takes care of shutting instances down. One reason is due to fault domains and SLA: if you had, say, 4 instances in 2 fault domains, and shut down two instances in fault domain 0, you'd now have 2 instances in fault domain 1. So now you have two instances in the same rack, perhaps, and that rack goes offline. Now you have zero instances running for a period of time.

Dealing with instance shutdown is a common scenario, and the typical pattern for working around this is by taking advantage of queues to buffer your workload, then have worker role instances consume work items from these queues. If you shut down an instance prior to its work being finished, the item eventually reappears on the queue and another instance can do the work.

This pattern requires idempotency, which is sometimes a challenge. With a recent update to Windows Azure queues, you can now modify queue messages, which makes this a bit easier - you can add information to your queue message as you complete various stages of your work item processing. Then, if your instance is shut down before work is completed, the next worker to pick it up can resume from a point other than "start."

One more detail: you should be able to handle the Stopping event, and tell the "instance being stopped" to stop reading from the queue (maybe set a flag). Then, override OnStop(), and wait for in-process operations to complete before returning. If the still-in-process operations will take more than 5 minutes, you might have to get creative...