Google Cloud Tasks cannot authenticate to Cloud Run

Question

I am trying to invoke a Cloud Run service using Cloud Tasks as described in the docs here.

I have a running Cloud Run service. If I make the service publicly accessible, it behaves as expected.

I have created a cloud queue and I schedule the cloud task with a local script. This one is using my own account. The script looks like this

from google.cloud import tasks_v2

client = tasks_v2.CloudTasksClient()

project = 'my-project'
queue = 'my-queue'
location = 'europe-west1'
url = 'https://url_to_my_service'

parent = client.queue_path(project, location, queue)

task = {
        'http_request': {
            'http_method': 'GET',
            'url': url,
            'oidc_token': {
               'service_account_email': '[email protected]'
            }
        }
}

response = client.create_task(parent, task)
print('Created task {}'.format(response.name))

I see the task appear in the queue, but it fails and retries immediately. The reason for this (by checking the logs) is that the Cloud Run service returns a 401 response.

My own user has the roles "Service Account Token Creator" and "Service Account User". It doesn't have the "Cloud Tasks Enqueuer" explicitly, but since I am able to create the task in the queue, I guess I have inherited the required permissions. The service account "[email protected]" (which I use in the task to get the OIDC token) has - amongst others - the following roles:

• Cloud Tasks Enqueuer (Although I don't think it needs this one as I'm creating the task with my own account)
• Cloud Tasks Task Runner
• Cloud Tasks Viewer
• Service Account Token Creator (I'm not sure whether this should be added to my own account - the one who schedules the task - or to the service account that should perform the call to Cloud Run)
• Service Account User (same here)
• Cloud Run Invoker

So I did a dirty trick: I created a key file for the service account, downloaded it locally and impersonated locally by adding an account to my gcloud config with the key file. Next, I run

curl -H "Authorization: Bearer $(gcloud auth print-identity-token)" https://url_to_my_service

That works! (By the way, it also works when I switch back to my own account)

Final tests: if I remove the oidc_token from the task when creating the task, I get a 403 response from Cloud Run! Not a 401... If I remove the "Cloud Run Invoker" role from the service account and try again locally with curl, I also get a 403 instead of a 401.

If I finally make the Cloud Run service publicly accessible, everything works.

So, it seems that the Cloud Task fails to generate a token for the service account to authenticate properly at the Cloud Run service.

What am I missing?

Answer 1

I had the same issue here was my fix:

Diagnosis: Generating OIDC tokens currently does not support custom domains in the audience parameter. I was using a custom domain for my cloud run service (https://my-service.my-domain.com) instead of the cloud run generated url (found in the cloud run service dashboard) that looks like this: https://XXXXXX.run.app

Masking behavior: In the task being enqueued to Cloud Tasks, If the audience field for the oidc_token is not explicitly set then the target url from the task is used to set the audience in the request for the OIDC token.

In my case this meant that enqueueing a task to be sent to the target https://my-service.my-domain.com/resource the audience for the generating the OIDC token was set to my custom domain https://my-service.my-domain.com/resource. Since custom domains are not supported when generating OIDC tokens, I was receiving 401 not authorized responses from the target service.

My fix: Explicitly populate the audience with the Cloud Run generated URL, so that a valid token is issued. In my client I was able to globally set the audience for all tasks targeting a given service with the base url: 'audience' : 'https://XXXXXX.run.app'. This generated a valid token. I did not need to change the url of the target resource itself. The resource stayed the same: 'url' : 'https://my-service.my-domain.com/resource'

More Reading: I've run into this problem before when setting up service-to-service authentication: Google Cloud Run Authentication Service-to-Service