Handling run time and build time secrets in AWS CodePipeline

Question

We are dealing with the problem of providing build time and run time secrets to our applications built using AWS CodePipeline and being deployed to ECS.

Ultimately, our vision is to create a generic pipeline for each of our applications that achieves the following goals:

• Complete separation of access
  • The services in the app-a-pipeline CANNOT access any of the credentials or use any of the keys used in the app-b-pipeline and visa-versa
• Secret management by assigned developers
  • Only developers responsible for app-a may read and write secrets for app-a

Here are the issues at hand:

• Some of our applications require access to private repositories for dependency resolution at build time For example, our java applications require access to a private maven repository to successfully build
• Some of our applications require database access credentials at runtime For example, the servlet container running our app requires an .xml configuration file containing credentials to find and access databases

Along with some caveats:

• Our codebase resides in a public repository. We do not want to expose secrets by putting either the plaintext or the cyphertext of the secret in our repository
• We do not want to bake runtime secrets into our Docker images created in CodeBuild even if ECR access is restricted
• The Cloudformation template for the ECS resources and its associated parameter file reside in the public repository in plaintext. This eliminates the possibility of passing runtime secrets to the ECS Cloudformation template through parameters (As far as I understand)

We have considered using tools like credstash to help with managing credentials. This solution requires that both CodeBuild and ECS task instances have the ability to use the AWS CLI. As to avoid shuffling around more credentials, we decided that it might be best to assign privileged roles to instances that require the use of AWS CLI. That way, the CLI can infer credentials from the role in the instances metadata

We have tried to devise a way to manage our secrets given these restrictions. For each app, we create a pipeline. Using a Cloudformation template, we create:

• 4 resources:

  • DynamoDB credential table
  • KMS credential key
  • ECR repo
  • CodePipeline (Build, deploy, etc)

• 3 roles:

  • CodeBuildRole Read access to DynamoDB credential table Decrypt permission with KMS key Write to ECR repo
  • ECSTaskRole Read access to DynamoDB credential table Decrypt permission with KMS key Read from ECR repo
  • DeveloperRole Read and write access to DynamoDB credential table Encrypt and decrypt permission with KMS key

The CodeBuild step of the CodePipeline assumes the CodeBuildRole to allow it to read build time secrets from the credential table. CodeBuild then builds the project and generates a Docker Image which it pushes to ECR. Eventually, the deploy step creates an ECS service using the Cloudformation template and the accompanying parameter file present in the projects public repository The ECS task definition includes assuming the ECSTaskRole to allow the tasks to read runtime secrets from the credential table and to pull the required image from ECR.

Here is a simple diagram of the AWS resources and their relationships as stated above

Our current proposed solution has the following issues:

• Role heavy
  • Creating roles is a privileged action in our organization. Not all developers who try to create the above pipeline will have permission to create the necessary roles
• Manual assumption of DeveloperRole:
  • As it stands, developers would need to manually assume the DeveloperRole. We toyed with the idea of passing in a list of developer user ARNs as a parameter to the pipeline Cloudformation template. Does Cloudformation have a mechanism to assign a role or policy to a specified user?

Is there a more well established way to pass secrets around in CodePipeline that we might be overlooking, or is this the best we can get?

Answer 1

Three thoughts:

• AWS Secret Manager
• AWS Parameter Store
• IAM roles for Amazon ECS tasks

AWS Secret ManagerAWS Secrets Manager helps you protect secrets to access applications, services, and IT resources. With you can rotate, manage, and retrieve database credentials, API keys, and other secrets throughout their lifecycle.

AWS Parameter Store can protect access keys with granular access. This access can be based on ServiceRoles.

ECS provides access to the ServiceRole via this pattern:

build:
    commands:
      - curl 169.254.170.2$AWS_CONTAINER_CREDENTIALS_RELATIVE_URI | jq 'to_entries | [ .[] | select(.key | (contains("Expiration") or contains("RoleArn"))  | not) ] |  map(if .key == "AccessKeyId" then . + {"key":"AWS_ACCESS_KEY_ID"} else . end) | map(if .key == "SecretAccessKey" then . + {"key":"AWS_SECRET_ACCESS_KEY"} else . end) | map(if .key == "Token" then . + {"key":"AWS_SESSION_TOKEN"} else . end) | map("export \(.key)=\(.value)") | .[]' -r > /tmp/aws_cred_export.txt
      - chmod +x /tmp/aws_cred_export.txt 
      - /aws_cred_export.txt && YOUR COMMAND HERE

If your ServiceRole provided to the CodeBuild task has access to use the Parameter store key you should be good to go.

Happy hunting and hope this helps