GAE : How to deploy various environments with secrets?

Question

I have a staging and a production project on App Engine, with 6 services on each.

For the moment, we deploy from the developers computer, using gcloud app deploy app.staging.yaml --project staging-project

or gcloud app deploy app.production.yaml --project production-project

It works but it causes issues with environment variables, and especially with secrets.

Our apps gets their Api keys, database credentials and other things from the environment variables - that allows us to have the exact same app running locally, in a Docker Container or in App Engine without knowing where it is deployed.

If I follow the documentation way to deploy, our app.yaml files would look like this:

app.production.yaml

runtime: nodejs
env: flex

manual_scaling:
  instances: 1
env_variables:
  DATABASE_PASSWORD: "topsecret"
  MY_API_KEY: "ultrasecret"

I think everybody easily understands why it's a bad idea to store that on a Git repository.

For the moment, we have this shadow file that every developer have to fill before deployment

app.production.yaml.shadow

runtime: nodejs
env: flex

manual_scaling:
  instances: 1
env_variables:
  DATABASE_PASSWORD: "set me"
  MY_API_KEY: "set me"

But as the team grows and we want everybody to be able to deploy on staging, it becomes more and more difficult to have the right settings for each developpers and each service.

I found out 3 workarounds, and their reason to not be used:

• Use Google KMS - allows us to put encrypted secrets directly into the project, but it requires us to put custom code in our apps to decrypt them. It creates a different environment management between local, staging and production. It increases the risk of bugs due to the complexity.
• Store secrets in Google Datastore - I tried it, I created a helper that searches env vars in proccess.ENV, then in cache and ultimately in Datastore. But like KMS, it increases complexity a lot.
• Store secrets in a JSON file and put in on Google Cloud Storage : again, it requires to load env variables through an helper that checks env vars, then loads the file etc...

Ultimately, we are exploring the feasibility to use a deployment server that would be triggered by developers or Continuous Integration and handle all the secrets injection when deploying to App Engine. But these tools like Ansible, Salt, Puppet, Chef only have plugins for Compute Engine and do not support App Engine.

+-------------------------+    +-------------------+   +---------------------+
|                         |    |                   +--->                     |
| Developer workspace     |    |    Ansible        |   | App Engine STAGING  |
|                         +---->   (or other)      |   |                     |
+-------------------------+    |                   |   +---------------------+
                               |                   |
+-------------------------+    |                   |   +---------------------+
|                         +---->   Injects secrets |   |                     |
| Continous Integration        |                   |   | App Engine PROD.    |
|                         |    |                   +--->                     |
+-------------------------+    +-------------------+   +---------------------+

This leads me to 3 questions:

• Do you think using a deployment server with App Engine is a good idea ?
• How can I make sure that production and staging secrets keep synchronisation so that deployments from developers are always right ?
• Is there a way to use classical environment variables for secrets on App Engine ?

Answer 1

I would strongly encourage you to consider using a combination of KMS and Cloud Storage, as Google outlines here.

You're right that it can be a bit of a pain in the neck to set up, but once it's in place it's very easy to work with. We set up separate keyrings for each environment (dev, test, staging, prod) and then separate keys for each application and wrote a command line utility that makes it trivial for any developer on our team to write secrets into the secret store (while at the same time preventing them from reading production secrets back).

Ultimately, the management of secrets remains a hard problem. Managing secrets for multiple applications/environments as your team grows is tough. But I'd really encourage you to just pay the cost of investing in something like KMS up front, because it'll make your life easier essentially forever and really reduce the risk of you accidentally shooting yourself in the foot.

Answer 2

As you said, it is not possible to use either of those solutions (Ansible, Salt, etc), with App Engine. If you were to use them, you would have to move to GCE. Google has some official tutorials on how to start with either of those technologies.

If you stay on App Engine, you can use one of the mentioned solutions, KMS being the better oriented for this purpose.

Another possible solution would be migrating to a Custom Runtime. Based on your application, you can generate a Dockerfile with:

$ gcloud beta app gen-config --custom

Your Dockerfile will look like this:

FROM gcr.io/google_appengine/nodejs
RUN /usr/local/bin/install_node '>=4.3.2'
COPY . /app/
RUN npm install --unsafe-perm || \
  ((if [ -f npm-debug.log ]; then \
      cat npm-debug.log; \
    fi) && false)
CMD npm start

You can then modify it to include the secret keys, stored in your environment variables, with something like this example of how to input the keys in your file:

sed 's/topsecret/'"$KEY_1"'/g;s/ultrasecret/'"$KEY_2"'/g' app.production.yaml.shadow > app.production.yaml

Mind the use of singles and doubles quotes before and after both KEY_1 and KEY_2, as you need them to enter environment variables using sed.