How to handle secrets in Google App Engine?

Question

My application needs a bunch of secrets to run: database credentials, API credentials, etc. It's running in Google App Engine Standard Java 11. I need these secrets as environment variables or as arguments to my application, so that my framework can pick them up and establish the connections accordingly. My particular framework is Spring Boot, but I believe Django, Rails and many others use the same methods.

What's the best way of doing this?

One of the answers I get to this question is to use Google Cloud Key Management, which looks promising, but I can't figure out how to turn those values into environment variables in App Engine. Is it possible? I've read Setting Up Authentication for Server to Server Production Applications, but I don't see any indication there about how to turn the secrets into environment variables in App Engine (am I missing it?).

The other alternatives I've seen include hard-coding them in app.yaml or another file that is never committed and lives in my machine, which means I'm the only one who can deploy... I can't even deploy from another machine. This is problematic for me.

Another potential solution I've seen is to delegate the problem to Google Cloud Build, so that it fetches a value/file from CKM and pushes it to App Engine (1, 2). I'm not using GCB and I doubt I will, since it's so basic.

I really wish App Engine had a environment variables page like Heroku does.

Answer 1

[Update] (as of Feb 2020) GCP's Secret Manager is in beta, see:

https://cloud.google.com/secret-manager/docs/overview

For Java-specific implementation, see: https://cloud.google.com/secret-manager/docs/creating-and-accessing-secrets#secretmanager-access-secret-version-java

Your specific solution would depend how your app is set up, but you should be able to access the secret(s) and create environment variables with the values or otherwise pass them to your app.

You can use GCP IAM to create a service accounts to manage access or add a role like Secret Manager Secret Accessor to an existing member/service (e.g., in this case, I added that permision to the App Engine default service account).

I tried it out with Node.js on GAE standard, and it seems to work well; I didn't do any performance tests but it should be fine, particularly if you primarily need the secrets on app start or as part of a build process.

For local (non-GCP) development/testing, you can create a service account with appropriate secret manager permissions and get the json service key. You then set an environment variable named GOOGLE_APPLICATION_CREDENTIALS to the path of the file, e.g.:

export GOOGLE_APPLICATION_CREDENTIALS=/path/to/local_service_key.json

and the app running in that shell session should pick up the permissions without any additional auth code. See: https://cloud.google.com/docs/authentication/getting-started (You would want to exclude the key file from version control.)

Answer 2

At this date, App Engine Standard Standard does not have a Google provided solution for storing application secrets.

[UPDATE]

I noticed your comment on another answer that you require environment variables to be valid before you have application control. In that case, you have no options for App Engine today. I would deploy to a different service (Kubernetes) better suited for your system goals that can provided managed secrets.

[END UPDATE]

You have two choices for secrets for App Engine Standard:

1. Store the secrets as environment variables in app.yaml
2. Store the secrets someplace else.

For both options, you can add a layer of security by encrypting them. However, adding encryption adds another secret (decryption key) that you must somehow provide to your app. The chicken-or-egg situation.

App Engine Standard uses a Service Account. This service account can be used as an identity to control access to other resources. Examples of other resources are KMS and Cloud Storage. This means that you can securely access KMS or Cloud Storage without adding another secret to App Engine.

Let's assume that your company wants all application secrets encrypted. We can use the App Engine Service Account as the identity authorized to access KMS for a single key.

Note: The following examples use Windows syntax. Replace the line continuation ^ with \ for Linux/macOS.

Create the KMS Keyring. Keyrings cannot be deleted, so this is a one-time operation.

set GCP_KMS_KEYRING=app-keyring
set GCP_KMS_KEYNAME=app-keyname

gcloud kms keyrings create %GCP_KMS_KEYRING% --location global

Create the KMS Key.

gcloud kms keys create %GCP_KMS_KEYNAME% ^
--location global ^
--keyring %GCP_KMS_KEYRING% ^
--purpose encryption

Add the service account to the KMS policy for the keyring and key that we created.

This will allow App Engine to decrypt data without requiring secrets for KMS. The service account identity provides access control. No roles are required for KMS. You will need to provide the KMS Keyring and Keyname which can be included in app.yaml.

set GCP_SA=<replace with the app engine service acccount email adddress>
set GCP_KMS_ROLE=roles/cloudkms.cryptoKeyDecrypter

gcloud kms keys add-iam-policy-binding %GCP_KMS_KEYNAME% ^
--location global ^
--keyring %GCP_KMS_KEYRING% ^
--member serviceAccount:%GCP_SA% ^
--role %GCP_KMS_ROLE%

For this example, let's assume that you need to access a MySQL database. We will store the credentials in a JSON file and encrypt it. The file is named config.json.

{
        "DB_HOST": "127.0.0.1",
        "DB_PORT": "3306",
        "DB_USER": "Roberts",
        "DB_PASS": "Keep-This-Secret"
}

Encrypt config.json using Cloud KMS and store the encrypted results in config.enc:

call gcloud kms encrypt ^
--location=global ^
--keyring %GCP_KMS_KEYRING% ^
--key=%GCP_KMS_KEYNAME% ^
--plaintext-file=config.json ^
--ciphertext-file=config.enc

The encrypted file can be stored in Cloud Storage. Since it is encrypted, you could store the file with your build files, but I do not recommend that.

The final piece is to write the code in Java that is part of your program that uses KMS to decrypt the file config.enc using KMS. Google has a number of examples of KMS decryption:

Java KMS Decrypt

Java Samples