Logo Questions Linux Laravel Mysql Ubuntu Git Menu
 

How to fix CloudRun error 'The request was aborted because there was no available instance'

Tags:

I'm using managed CloudRun to deploy a container with concurrency=1. Once deployed, I'm firing four long-running requests in parallel. Most of the time, all works fine -- But occasionally, I'm facing 500's from one of the nodes within a few seconds; logs only provide the error message provided in the subject.

Using retry with exponential back-off did not improve the situation; the retries also end up with 500s. StackDriver logs also do not provide further information.

Potentially relevant gcloud beta run deploy arguments:

--memory 2Gi --concurrency 1 --timeout 8m --platform managed

What does the error message mean exactly -- and how can I solve the issue?

like image 570
Jan Hacker Avatar asked Jul 12 '19 12:07

Jan Hacker


People also ask

What is Cloudrun?

Cloud Run is a managed compute platform that enables you to run containers that are invocable via requests or events. Cloud Run is serverless: it abstracts away all infrastructure management, so you can focus on what matters most — building great applications.

What is the Google Cloud run service agent?

Some Google Cloud services have Google-managed service accounts that allow the services to access your resources. These service accounts are known as service agents. You might see evidence of these service agents in several different places, including a project's allow policy and audit log entries for various services.


1 Answers

This error message can appear when the infrastructure didn't scale fast enough to catch up with the traffic spike. Infrastructure only keeps a request in the queue for a certain amount of time (about 10s) then aborts it.

This usually happens when:

  1. traffic suddenly largely increase
  2. cold start time is long
  3. request time is long
like image 196
Steren Avatar answered Sep 19 '22 00:09

Steren