For Cloud Run triggered from PubSub, when is the right time to send ACK for the request message?

Question

I was building a service that runs on Cloud Run that is triggered by PubSub through EventArc. 'PubSub' guarantees delivery at least one time and it would retry for every acknowledgement deadline. This deadline is set in the queue subscription details.

We could send an acknowledgement back at two points when a service receives a pub-sub request (which is received as a POST request in the service).

1. At the beginning of the request as soon as the request was received. The service would then continue to process the request at its own pace. However, this article points out that

When an application running on Cloud Run finishes handling a request, the container instance's access to CPU will be disabled or severely limited. Therefore, you should not start background threads or routines that run outside the scope of the request handlers.

So sending a response at the beginning may not be an option

2. After the request has been processed by the service. So this would mean that, depending on what the service would do, we cannot always predict how long it would take to process the request. Hence we cannot set the Acknowledgement deadline correctly, resulting in PubSub retries and duplicate requests.

So what is the best practice here? Is there a better way to handle this?

Answer 1

Best practice is generally to ack a message once the processing is complete. In addition to the Cloud Run limitation you linked, consider that if the endpoint acked a message immediately upon receipt and then an error occurred in processing it, your application could lose that message.

To minimize duplicates, you can set the ack deadline to an upper bound of the processing time. (If your endpoint ends up processing messages faster than this, the ack deadline won’t rate-limit incoming messages.) If the 600s deadline is not sufficient, you could consider writing the message to some persistent storage and then acking it. Then, a separate worker can asynchronously process the messages from persistent storage.