Shutting down VM returns all VM states as unknown

Question

When using the methods below to shutdown and query the role instances. When I shutdown a VM all other role instances are returned with a status of ready state unknown. After about a couple of minutes I can query again and get the actual status. How can I get the actual status in real time, using Azure Management APIs. Or is this an issue with how the VMs are configured? They are configured with the same storage location and same virtual network

The code shown was based off the template for Deploy and Manage Virtual Machines in Visual Studio 2015.

The call to shutdown the VM:

var shutdownParams = new VirtualMachineShutdownParameters();

if (deallocate)//deallocate is true in this instance
    shutdownParams.PostShutdownAction = PostShutdownAction.StoppedDeallocated; // Fully deallocate resources and stop billing
else
    shutdownParams.PostShutdownAction = PostShutdownAction.Stopped; // Just put the machine in stopped state, keeping resources allocated

await _computeManagementClient.VirtualMachines.ShutdownAsync(_parameters.CloudServiceName, _parameters.CloudServiceName, vmName, shutdownParams);

The call to query for all role instances XXX_VirtualMachine is a class that holds the name and instance status:

internal List<XXX_VirtualMachine> GetAllVirtualMachines()
{
    List<XXX_VirtualMachine> vmList = new List<XXX_VirtualMachine>();
    try
    {
        DeploymentGetResponse deployment;

        deployment = _computeManagementClient.Deployments.GetByName(_parameters.CloudServiceName, _parameters.CloudServiceName);

        for (int i = 0; i < deployment.RoleInstances.Count; i++)
        {
            vmList.Add(new XXX_VirtualMachine(deployment.RoleInstances[i].InstanceName, deployment.RoleInstances[i]));
        }
    }
    catch (Exception e)
    {
        System.Windows.Forms.MessageBox.Show(e.Message);
    }
    return vmList;
}

Answer 1

So I finally got around to giving this a kick! (apologies for the delay, people kept expecting that work stuff - inconsiderate fools!)

Firstly, this isn't really an answer! just an explore of the problem, and you probably know all of this already, but maybe someone reading it will see something I've missed.

I've created three VMs, in a single Cloud Service, and lo-and-behold! it did exactly what you predicted when you shut one down.

Firstly both portals appear to be giving reliable answers, even when the .Net request is reporting RoleStatusUnknown.

Looking at the Xml that comes out of the request to

https://management.core.windows.net/{subscriptionid}/services/hostedservices/vm01-u3rzv2q6/deploymentslots/Production

we get

<RoleInstance>
  <RoleName>vm01</RoleName>
  <InstanceName>vm01</InstanceName>
  <InstanceStatus>RoleStateUnknown</InstanceStatus>
  <InstanceSize>Basic_A1</InstanceSize>
  <InstanceStateDetails />
  <PowerState>Started</PowerState>

I then fired up Powershell to see if that was doing the same, which it was (not unexpected since it calls the same REST point). with Get-AzureVm returning

ServiceName   Name Status          
-----------   ---- ------          
vm01-u3rzv2q6 vm01 CreatingVM      
vm01-u3rzv2q6 vm02 RoleStateUnknown
vm01-u3rzv2q6 vm03 RoleStateUnknown

At the appropriate times, which again, is as seen.

Wondering what the timing was, I then ran this

while ($true) { (get-azurevm -ServiceName vm01-u3rzv2q6 -Name vm01).InstanceStatus ; get-azurevm ; (date).DateTime } 

ReadyRole
vm01-u3rzv2q6 vm01 ReadyRole
vm01-u3rzv2q6 vm02 ReadyRole
vm01-u3rzv2q6 vm03 ReadyRole
07 March 2016 04:31:01

07 March 2016 04:31:36
StoppedDeallocated
vm01-u3rzv2q6 vm01 Stoppe...
vm01-u3rzv2q6 vm02 RoleSt...
vm01-u3rzv2q6 vm03 RoleSt...
07 March 2016 04:31:49

07 March 2016 04:33:44
StoppedDeallocated
vm01-u3rzv2q6 vm01 Stoppe...
vm01-u3rzv2q6 vm02 ReadyRole
vm01-u3rzv2q6 vm03 ReadyRole
07 March 2016 04:33:52

So it seems that the machine shuts down, then a process must begin to update the cloud service, which takes its ability to query its status for, what seems, exactly two minutes.

Somewhere in the API there must be a location that it is reported properly because the portals don't have this problem.

I spent a while down a blind alley looking for an 'InstanceView' for the VM, but it seems that doesn't exist for classic deployments.

My next thought is to put together a simple rest client that takes a management certificate and see if the URI can be hacked around a bit to give anything more interesting. (its got to be there somewhere!)

What may be useful, is that the PowerState isn't affected by this problem. So you could potentially have a secondary check for that while you have the RoleStateUnknown error, its far far from perfect, but depending on what you're looking to do it might work.

Failing that, I'd say it is clearly an bug in Azure, and could definitely have a support call raised for it.