InvalidClientTokenId when calling get-caller-identity on an AWS EC2 instance with instance profile

Question

We're having an issue where we're on a CentOS EC2 instance that is using a role through an attached instance profile. When we're on the console after SSHing in, we run the python awscli command line tool to get our identity:

$ aws sts get-caller-identity

we're getting

An error occurred (InvalidClientTokenId) when calling the GetCallerIdentity operation: The security token included in the request is invalid

other commands, such as aws ec2 describe-instances work and are allowed by the instance profile.

From reading the AWS documentation, no permissions should be required to get-caller-identity and there's no explicit deny set on the role associated with instance.

We checked and there's no .aws/credentials file and no env variables set, so access should be entirely managed through the metadata service on the EC2 instance.

Is there something missing in our setup or invocation of the awscli that might cause the permission to fail?

Answer 1

Just documenting the fix for anyone that runs into this issue.

All calls to the awscli should probably include a --region <region> parameter.

E.g.

$ aws sts get-caller-identity --region us-east-2

We were prompted for the region on our aws ec2 describe-instances call but on the aws sts get-caller-identity call, it just failed.

Additionally, we found that the AWS_REGION environment variable didn't seem to affect calls: we still needed to include the --region <region> parameter.