App Engine deploy: Permissions error fetching application

Question

We are using a service account to deploy our app to App Engine using Travis.

On every merged PR, Travis pulls the code from our GitHub repository, and pulls a Docker image which contains Google Cloud SDK and executes the gcloud app deploy command. We use a Service Account to perform the deployment with "Project Owner" role.

Everything used to work fine until I added a new service to the project which automates SSL certificate generation and renewal, along with a dispatch.yaml file to route traffic incoming from Let's Encrypt for domain verification. I needed to add more permissions to allow updating the SSL certificates we use for our custom domain. I removed the current service account, and created a new one with a new private key. I created a new role with the required permissions to update and view SSL certificates in addition to the previous permissions (all appengine.* permissions). I assigned the new role and the Project Owner role to the new account. After these changes, the deployment fails with the following error when executing the deploy command:

Permissions error fetching application [apps/hollowverse-c9cad]. Please make sure you are using the correct project ID and that you have permission to view applications on the project.

I used the same service account on my local machine with logging level set to debug. I got this error:

DEBUG: HttpError accessing <https://appengine.googleapis.com/v1/apps/hollowverse-c9cad?alt=json>: response: <{'status': '403', 'content-length': '335', 'x-xss-protection': '1; mode=block', 'x-content-type-options': 'nosniff', 'transfer-encoding': 'chunked', 'vary': 'Origin, X-Origin, Referer', 'server': 'ESF', '-content-encoding': 'gzip', 'cache-control': 'private', 'date': 'Wed, 02 Aug 2017 14:33:50 GMT', 'x-frame-options': 'SAMEORIGIN', 'alt-svc': 'quic=":443"; ma=2592000; v="39,38,37,36,35"', 'content-type': 'application/json; charset=UTF-8'}>, content <{
  "error": {
    "code": 403,
    "message": "Operation not allowed",
    "status": "PERMISSION_DENIED",
    "details": [
      {
        "@type": "type.googleapis.com/google.rpc.ResourceInfo",
        "resourceType": "gae.api",
        "description": "The \"appengine.applications.get\" permission is required."
      }
    ]
  }
}
>
DEBUG: (gcloud.beta.app.deploy) Permissions error fetching application [apps/hollowverse-c9cad]. Please make sure you are using the correct project ID and that you have permission to view applications on the project.

The description says that appengine.applications.get is required to perform the deployment. Looking at the permissions granted to the role assigned to the Travis account we use to deploy, appengine.applications.get is clearly granted:

I assigned every possible App Engine and Project role to the account, but deployment still fails with the same error. However, using the default service account, which is automatically created for every new project on GCP, seems to be working.

Answer 1

I removed the current service account, and created a new one with a new private key.

This is where it went wrong. The new account had the same ID as the previous one. Although I could not find this behavior documented anywhere, it looks like that once an ID is used for a service account, it cannot be used again for a new account, even if the previous one is removed.

We created a new account with a new ID (travis2@hollowverse-c9cad.iam.gserviceaccount.com) instead of [email protected]) and the issue is now resolved.