cfn-init fails to download S3 file even though "aws s3 cp" works

Question

I have a CloudFormation script that uses an AWS::CloudFormation::Init section to download a file from an S3 bucket which fails with Access Denied (403).

I have added an IAM role named s3access to the machine using an AWS::IAM::InstanceProfile. Downloading the file with aws s3 works:

[ec2-user@ip-172-31-26-26 ~]$ aws s3 cp s3://my-bucket/test-file 
.download: s3://my-bucket/test-file to ./test-file           

But cfn-init fails:

[ec2-user@ip-172-31-26-26 ~]$ sudo /opt/aws/bin/cfn-init -v --stack test --resource EC2 --region us-east-2
Error occurred during build: Failed to retrieve https://s3.us-east-2.amazonaws.com/my-bucket/test-file: HTTP Error 403 : 

I tried setting the IAM role explicitly but that fails too:

[ec2-user@ip-172-31-26-26 ~]$ sudo /opt/aws/bin/cfn-init -v --stack test --resource EC2 --region us-east-2 --role=s3access
AccessDenied: User: arn:aws:sts::196375698259:assumed-role/s3access/i-044499612c92b50f5 is not authorized to perform: cloudformation:DescribeStackResource on resource: arn:aws:cloudformation:us-east-2:196375698259:stack/test/*

I'm thinking of using aws s3 cp s3://my-bucket/test-file ./ directly from the user-data but I'd like to know why the AWS::CloudFormation::Init fails to assume the role assigned to the EC2 instance.

I found only one similar question - How can I access protected S3 files in a CFN script?, but the solution there is to apply an IAM role, which I have already done and cfn-init still fails.

Answer 1

I needed to add AWS::CloudFormation::Authentication to the Metadata of the EC2Instance. Details are in:

https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/aws-resource-authentication.html#cfn-cloudformation-authentication-rolename

For example:

EC2Instance:
    Type: 'AWS::EC2::Instance'    
        Metadata:  
            AWS::CloudFormation::Authentication:  
                 rolebased:  
                     type: S3  
                     roleName: !Ref EC2InstanceIAMRole  
                     buckets:  
                         - !Ref TemplateBucket  
            ...  

The Role has access to the relevant TemplateBucket.

The Role is also assigned to the EC2 instance.

Some details on the role are here:

https://docs.aws.amazon.com/IAM/latest/UserGuide/id_roles_use_switch-role-ec2.html

Before adding this I could download files from s3 via aws cli but the cfn-init did not work giving this error:

Error occurred during build: Failed to retrieve https://s3.amazonaws.com/FILE: HTTP Error 403 : <?xml version="1.0" encoding="UTF-8"?>
<Error><Code>AccessDenied</Code><Message>Access Denied</Message><RequestId>ID</RequestId><HostId>ID</HostId></Error>