ASG can't launch instances anymore with encrypted EBS root volume

Question

We have recently deployed a new application that uses an ASG configured to launch instances with encrypted EBS root volumes. We have plenty of existing applications that work using exactly this setup, but our new ASG refuses to launch instances. The instances don't even appear and we see an error in the ASG activity history: Client.InternalError: Client error on launch.

After experimenting, we've discovered that if we swap the AMI we are using for one that isn't encrypted it all starts working as expected. Confusingly we are using exactly the same AMI on a different ASG and it's all working as expected (formed from almost identical CloudFormation templates). Likewise we can launch an EC2 instance directly from the console using the same AMI and instance profile.

Has anyone seen this behaviour before?

I've found some solutions elsewhere (that led us to prove it was something to do with the encrypted volume) such as this from AWS but they don't seem to directly relate to our scenario.

Answer 1

We eventually found an AWS forum post that details Service Linked Roles (SLRs). It seems that sometime in the last couple of months they have changed the way that ASGs behave (for newly created ASGs only). Previously an ASG could use any KMS CMK, but this has been changed so that it can only access the default key. We're using a "customer managed" CMK so our newly created ASG can no longer access it by default.

Apparently this will be changed on existing ASGs towards the end of June.

In order to fix this we set about creating a new SLR with access to our key but later discovered that CloudFormation doesn't yet allow you to specify an SLR for an ASG.

In the meantime we've decided to essentially create the situation we had before by changing the policy on our CMK to allow access from the default SLR.

The CloudFormation we use to create our CMK now looks like this:

KmsKey:
  Type: AWS::KMS::Key
  DeletionPolicy: Retain
  Properties:
    Description: Used to encrypt AMIs
    EnableKeyRotation: True
    KeyPolicy:
      Version: 2012-10-17
      Id: ami-kms-key
      Statement:
      - Sid: Enable IAM User Permissions
        Effect: Allow
        Principal:
          AWS: !Sub arn:aws:iam::${AWS::AccountId}:root
        Action: kms:*
        Resource: "*"
      - Sid: Allow use of the key by the default service linked role
        Effect: Allow
        Principal:
          AWS:
          - !Sub arn:aws:iam::${AWS::AccountId}:role/aws-service-role/autoscaling.amazonaws.com/AWSServiceRoleForAutoScaling
        Action:
        - kms:Encrypt
        - kms:Decrypt
        - kms:ReEncrypt*
        - kms:GenerateDataKey*
        - kms:DescribeKey
        Resource: "*"
      - Sid: Allow attachment of persistent resources
        Effect: Allow
        Principal:
          AWS:
          - !Sub arn:aws:iam::${AWS::AccountId}:role/aws-service-role/autoscaling.amazonaws.com/AWSServiceRoleForAutoScaling
        Action:
        - kms:CreateGrant
        Resource: "*"
        Condition:
          Bool:
            kms:GrantIsForAWSResource: true