Securing GAE env variables by using gsutil builder to source app.yaml during build?

Question

I have the same problem as the one mentioned here: Securely storing environment variables in GAE with app.yaml - namely:

"I need to store API keys and other sensitive information in app.yaml as environment variables for deployment on GAE. The issue with this is that if I push app.yaml to GitHub, this information becomes public (not good)."

Additionally I'm looking to check the following boxes:

• Prevent vendor lock-in (as much as possible) & ability to take my dockerfile elsewhere.
• Ease of deployment with GitHub. I want a push to the master which triggers a build.
• Minimal setup, or a suitable effort and workflow for a solo-dev or small team.

My research yielded the following:

Securely storing environment variables in GAE with app.yaml

How to set environment variables/app secrets in Google App Engine

GAE : How to deploy various environments with secrets?

appengine and OS environment variables

How to pass environment variables to the app.yaml using cloud build

A lot of good information from GAE : How to deploy various environments with secrets? where the author listed the three workarounds and their reason to not be used:

• Use Google KMS - allows us to put encrypted secrets directly into the project, but it requires us to put custom code in our apps to decrypt them. It creates a different environment management between local, staging and production. It increases the risk of bugs due to the complexity.
• Store secrets in Google Datastore - I tried it, I created a helper that searches env vars in proccess.ENV, then in cache and ultimately in Datastore. But like KMS, it increases complexity a lot.
• Store secrets in a JSON file and put in on Google Cloud Storage : again, it requires to load env variables through an helper that checks env vars, then loads the file etc...

However the best solution for me came from How to pass environment variables to the app.yaml using cloud build

It allows me to have the following deployment flow using GAE flexible environment for nodejs:

1. A merge to my Github master branch triggers a cloud build
2. My first step in my cloudbuild.yaml sources my app.yaml file using the gsutil builder, since app.yaml is not in source control
3. My app.yaml points to my dockerfile for my runtime and has my env variables

This checks all my boxes and was a fairly easy solution but, this definitely doesn't seem to be a popular solution, so am I missing something here?

Most importantly are there any security concerns?

Answer 1

I am amazed at how you did your research, you actually collected all the possible ways to do achieve it.

As you mentioned there are many ways to pass the variables to the application but I believe that the solution you propose ( storing the variables in Google Cloud Storage and retrieving them with Google Cloud Build ) is optimal for your purposes. It doesn't require much code and it's elegant, I hope this post helps people to be aware of this solution. Regarding your security concerns, this solution includes a high degree of security as you can set the file in the bucket to only be accessible from Google Cloud Build and the owner of the project.

Answer 2

Another solution I've employed, is to store the env variables in the Cloud Build trigger substitution variables directly and use a custom Cloud Builder envsubt to render a templated app.yaml.

I could not find documentation on how the substitution variables are stored in the Cloud Build trigger (any reference here would be helpful). However, I think most data in Google Cloud is encrypted at rest and encrypted on use and transfer. The main drawback is that the values are show in plain text, so sensitive information like API keys are not obscured, and any one who has access to the trigger can see the sensitive information.

One benefit is that this keeps the templated app.yaml close to the code you'll be using it with, and can be reviewed in the same pull request. Also you don't need to use another service, like Google Storage.

Steps:

1. Add the envsubst Cloud builder to your project, see instructions here.
2. Create a templated app.yaml file, e.g.

runtime: <your runtime>

service: ${GAE_SERVICE}

env_variables:
  MY_VAR: ${MY_VAR}
  MY_VAR_2: ${MY_VAR_2}

3. Add an app.yaml template rendering step in cloudbuild.yaml

steps:
- id: "render-app-yaml"
  name: "gcr.io/${PROJECT_ID}/envsubst"
  env:
  - "GAE_SERVICE=${_GAE_SERVICE}"
  - "MY_VAR=${_MY_VAR}"
  - "MY_VAR_2=${_MY_VAR_2}"
  args: ["app.yaml"]

4. Add the substitution variables in the Cloud Build trigger, e.g. _GAE_SERVICE, _MY_VAR, and _MY_VAR_2. Note: user-defined variables in the trigger are prefixed with a _.

When I was doing my research, I couldn't find any solution like this one either. Any feedback is welcome.