No Instances found when trying to use AWS Console and EC2 Run Command for Windows

Question

I am trying to use the Amazon EC2 Run Command feature, and basically it says it can't find any instances.

I read all the documentation and followed all the steps that I could find on this relatively new feature, and I cant get it to work for the life of me.

Here is the info:

I go to "EC2 -> Commands -> Command History -> Run A Command" I click radio box for the "AWS-RunShellScript" command document I click the "Select Instances" drop down button, and it says "No instances found in this region" as shown in Figure 1 below.

I click the "Where are my instances?" hyperlink, which brings me to the AWS documentation show here:"Troubleshooting SSM Run Command"

The first suggestion for troubleshooting is to make sure that the prerequisites have been met, and provides another hyperlink shown here: "SSM Run Command Prerequisites"

Prerequisites:

1. Supported OS - Windows 2012 R2 - CHECK
2. Latest Agent Version - 3.17.1032 - CHECK (Shown In Figure 2)
3. Access to SSM Run Command - Assign EC2 Instance Role and IAM User Role - CHECK (Shown In Figures 3 and 4)
4. Internet Access - Outbound Internet Access - CHECK

I followed the AWS documentation for creating an Amazon EC2 Instance role for EC2 Run Command Access, and also to create an IAM User with proper Run Command Access. The documentation is shown here: Delegating Access to SSM Run Command

I used the Amazon CLI from my local machine to run the following command also shown in troubleshooting documentation:

aws ssm describe-instance-information --instance-information-filter-list key=InstanceIds,valueSet=instance-ID

The command returns no information:

{ "InstanceInformationList": [] }

UPDATE 1:

I did find the following errors and warnings in the following places, and have spent the last two hours trying to troubleshoot. I uinstalled and reinstalled the Ec2Config Service. I checked all the Ec2Config files. I Checked the 169.254 addresses, and the meta-data and dynamic data exists, and appears to be correct. Not sure what else to try here.

C:\Program Files\Amazon\Ec2ConfigService\Logs\Ec2ConfigLog - Error/Warning

2016-06-22T23:44:12.663Z: Warning: Unable to Publish to WMI. | System.Management.Instrumentation.WmiProviderInstallationException: Exception of type 'System.Management.Instrumentation.WMIInfraException' was thrown. at System.Management.Instrumentation.InstrumentationManager.Publish(Object value) 2016-06-22T23:44:16.263Z: Failed to fetch instance metadata http://169.254.169.254/latest/user-data with exception The remote server returned an error: (404) Not Found. 2016-06-22T23:44:16.263Z: Failed to get metadata/user-data The remote server returned an error: (404) Not Found.

Windows Event Viewer Error

2016-06-22 23:46:59,758 [_Worker-2] ERROR [aws:getDocument] - Failed to update instance information., RequestId=9cb8f2dd-38d3-11e6-bc83-19c0650ffecc,ErrorCode=AccessDeniedException,ErrorType=Unknown,StatusCode=BadRequest,Message=Caller
instance identity does not match the given instanceId

UPDATE 2:

Per @praetorian1 's linked stackoverflow article I updated the Ec2Config Services "config.xml" file to enable the "Ec2HandleUserData" setting, rebooted, and still no change.

Per @Kai 's suggestion I also made sure that the "role" attached to the instance matched the "AmazonEC2RoleforSSM" policy, and specifically permitted "ssm:UpdateInstanceInformation". It did.

I also looked under the AWS EC2 Management Console, at the "System Log" file for the specific instance, which did not contain any errors, and towards the end said that SSM Config: status:Active, iam:Yes. Also noteworthy, it contained: Info EC2Config configuration: status:2; region:us-east-1; iam:1; authz:1

2016/06/23 15:01:29Z: EC2ConfigMonitorState: 0 2016/06/23 15:01:29Z: Windows sysprep configuration complete. 2016/06/23 15:01:32Z: AMI Origin Version: 2016.01.13 2016/06/23 15:01:32Z: AMI Origin Name: Windows_Server-2012-R2_RTM-English-64Bit-SQL_2014_SP1_Web 2016/06/23 15:01:32Z: OS: Microsoft Windows NT 6.3.9600 2016/06/23 15:01:32Z: OsVersion: 6.3 2016/06/23 15:01:32Z: OsProductName: Windows Server 2012 R2 Standard 2016/06/23 15:01:32Z: OsBuildLabEx: 9600.18202.amd64fre.winblue_ltsb.160119-0600 2016/06/23 15:01:32Z: Language: en-US 2016/06/23 15:01:32Z: TimeZone: Eastern Standard Time 2016/06/23 15:01:32Z: Offset: UTC -04:00:00 2016/06/23 15:01:32Z: EC2 Agent: Ec2Config service v3.17.1032 2016/06/23 15:01:32Z: Driver: AWS PV Storage Host Adapter v7.3.2.0 2016/06/23 15:01:32Z: Driver: Intel(R) 82599 Virtual Function v1.0.15.3 2016/06/23 15:01:34Z: Message: Waiting for meta-data accessibility... 2016/06/23 15:01:34Z: Message: Meta-data is now available. 2016/06/23 15:01:37Z: Message: Windows is Ready to use 2016/06/23 15:01:40Z: Amazon EC2 Simple Systems Manager (SSM) is an optional service for custom configuration of instances. 2016/06/23 15:01:40Z: Info EC2Config configuration: status:2; region:us-east-1; iam:1; authz:1 2016/06/23 15:01:40Z: SSM Config: status:Active; iam:Yes

Figure 1:

Figure 2:

Figure 4:

Answer 1

Try checking the application logs in windows on the instance. Might be having issues talking to the SSM service for some reason.

Answer 2

As praetorian1 mentioned, Windows Event Viewer is the place to look for SSM - EC2 Run Command related log messages.

The message you posted indicates that the EC2 instance does not have required permission to call ssm:UpdateInstanceInformation API. Please double check that RunCommandInstance role has AmazonEC2RoleforSSM policy attached. (After attaching the policy, please restart EC2Config service or wait up to 15 min for EC2Config to start reporting instance information.)

For the IAM User in Fig. 4, we do not need policies other than AmazonSSMFullAccess for us to call SSM APIs with awscli.

To dive deeper:

EC2 Role and IAM Policy for the EC2 Instance

SSM on Windows requires an IAM EC2 Role attached to the EC2 instance at launch. Required permission for that Role is provided by an Amazon-managed IAM policy AmazonEC2RoleforSSM.

IAM Policy for IAM User

To make API calls to SSM service with awscli, an IAM user has to be setup with SSM permissions. For testing, Amazon provides a managed IAM policy AmazonSSMFullAccess. With the credentials of this IAM user, we will be able to make API call to describe instance information and also send command to the instance.

DescribeInstanceInformation API

Once the EC2 instance is setup with required permission and EC2Config starts reporting instance information back to SSM, we will be able to use DescribeInstanceInformation API to get a list of EC2 instances that are ready for SSM - EC2 Run Command.