Terraform: Invalid Terraform AWS provider credentials when passing AWS system manager parameter store variables

Question

Background:

I'm using an AWS CodeBuild buildspec.yml to iterate through directories from a GitHub repo to apply IaC using Terraform. To access the credentials needed for the Terraform AWS provider, I used AWS system manager parameter store to retrieve the access and secret key within the buildspec.yml.

Problem:

The system manager parameter store masks the access and secret key env value so when they are inherited by the Terraform AWS provider, the provider outputs that the credentials are invalid:

Error: error configuring Terraform AWS Provider: error validating provider credentials: error calling sts:GetCallerIdentity: InvalidClientTokenId: The security token included in the request is invalid.
    status code: 403, request id: xxxx

To reproduce the problem:

• Create system manager parameter store variables (TF_VAR_AWS_ACCESS_KEY_ID=access, TF_AWS_SECRET_ACCESS_KEY=secret)

• Create AWS CodeBuild project with:

 "source": {
                "type": "NO_SOURCE",
}
"environment": {
                "type": "LINUX_CONTAINER",
                "image": "aws/codebuild/standard:4.0",
                "computeType": "BUILD_GENERAL1_SMALL"
}

buildspec.yml with the following: (modified to create .tf files instead of sourcing from github)

version: 0.2
env:
  shell: bash
  parameter-store:
    TF_VAR_AWS_ACCESS_KEY_ID: TF_AWS_ACCESS_KEY_ID
    TF_VAR_AWS_SECRET_ACCESS_KEY: TF_AWS_SECRET_ACCESS_KEY
phases:
  install:
    commands:
      - wget https://releases.hashicorp.com/terraform/0.12.28/terraform_0.12.28_linux_amd64.zip -q
      - unzip terraform_0.12.28_linux_amd64.zip && mv terraform /usr/local/bin/
      - printf "provider "aws" {\n\taccess_key = var.AWS_ACCESS_KEY_ID\n\tsecret_key = var.AWS_SECRET_ACCESS_KEY\n\tversion    = \"~> 3.2.0\"\n}" >> provider.tf
      - printf "variable "AWS_ACCESS_KEY_ID" {}\nvariable "AWS_SECRET_ACCESS_KEY" {}" > vars.tf
      - printf "resource \"aws_s3_bucket\" \"test\" {\n\tbucket = \"test\"\n\tacl = \"private\"\n}" >> s3.tf
      - terraform init
      - terraform plan

Attempts:

1. Passing creds through terraform -vars option:

terraform plan -var="AWS_ACCESS_KEY_ID=$TF_VAR_AWS_ACCESS_KEY_ID" -var="AWS_ACCESS_KEY_ID=$TF_VAR_AWS_SECRET_ACCESS_KEY"

but I get the same invalid credentials error

2. Export system manager parameter store credentials within buildspec.yml:

commands:
  - export AWS_ACCESS_KEY_ID=$TF_VAR_AWS_ACCESS_KEY_ID
  - export AWS_SECRET_ACCESS_KEY=$TF_VAR_AWS_SECRET_ACCESS_KEY

which results in duplicate masked variables and the same error above. printenv output within buildspec.yml:

AWS_ACCESS_KEY_ID=***
TF_VAR_AWS_ACCESS_KEY_ID=***
AWS_SECRET_ACCESS_KEY=***
TF_VAR_AWS_SECRET_ACCESS_KEY=***

Possible solution routes:

• Somehow pass the MASKED parameter store credential values into Terraform successfully (preferred)
• Pass sensitive credentials into the Terraform AWS provider using a different method e.g. AWS secret manager, IAM role, etc.
• Unmask the parameter store variables to pass into the aws provider (probably defeats the purpose of using aws system manager in the first place)

Answer 1

I experienced this same issue when working with Terraform on Ubuntu 20.04.

I had configured the AWS CLI using the aws configure command with an IAM credential for the terraform user I created on AWS.

However, when I run the command:

terraform plan

I get the error:

Error: error configuring Terraform AWS Provider: error validating provider credentials: error calling sts:GetCallerIdentity: InvalidClientTokenId: The security token included in the request is invalid. status code: 403, request id: 17268b96-6451-4527-8b17-0312f49eec51

Here's how I fixed it:

The issue was caused as a result of the misconfiguration of my AWS CLI using the aws configure command. I had inputted the AWS Access Key ID where I was to input AWS Secret Access Key and also inputted AWS Secret Access Key where I was to input AWS Access Key ID:

I had to run the command below to re-configure the AWS CLI correctly with an IAM credential for the terraform user I created on AWS:

aws configure

You can confirm that it is fine by running a simple was cli command:

aws s3 ls

If you get an error like the one below, then you know you're still not setup correctly yet:

An error occurred (InvalidAccessKeyId) when calling the ListBuckets operation: The AWS Access Key Id you provided does not exist in our records.

That's all.

I hope this helps

Answer 2

Pass sensitive credentials into the Terraform AWS provider using a different method e.g. AWS secret manager, IAM role, etc.

Generally you wouldn't need to hard-code AWS credentials for terraform to work. Instead CodeBuild IAM role should be enough for terraform, as explain in terraform docs.

Having this in mind, I verified that the following works and creates the bucket requested using terraform from CodeBuild project. The default CB role was modified with S3 permissions to allow creation of the bucket.

version: 0.2
phases:
  install:
    commands:
      - wget https://releases.hashicorp.com/terraform/0.12.28/terraform_0.12.28_linux_amd64.zip -q
      - unzip terraform_0.12.28_linux_amd64.zip && mv terraform /usr/local/bin/
      - printf "resource \"aws_s3_bucket\" \"test\" {\n\tbucket = \"test-43242-efdfdfd-4444334\"\n\tacl = \"private\"\n}" >> s3.tf
      - terraform init
      - terraform plan
      - terraform apply -auto-approve

Answer 3

Well my case was quite foolish but it might help:

So after downloading the .csv file I copy paste the keys with aws configure.

In the middle of the secret key there was a "+". In the editor I use the double click to copy, however will stop when meeting a non alphanumeric character, meaning that only the first part of the secret access key was copied.

Make sure that you had dutifully copied the full secret key.