In what cases does Google Cloud Run respond with "The request failed because the HTTP connection to the instance had an error."?

Question

We've been running Google Cloud Run for a little over a month now and noticed that we periodically have cloud run instances that simply fail with:

The request failed because the HTTP connection to the instance had an error.

This message is nearly always* proceeded by the following message (those are the only messages in the log):

This request caused a new container instance to be started and may thus take longer and use more CPU than a typical request.

* I cannot find, nor recall, a case where that isn't true, but I have not done an exhaustive search.

A few things that may be of importance:

• Our concurrency level is set to 1 because our requests can take up to the maximum amount of memory available, 2GB.
• We have received errors that we've exceeded the maximum memory, but we've dialed back our usage to obviate that issue.
• This message appears to occur shortly after 30 seconds (e.g., 32, 35) and our timeout is set to 75 seconds.

Answer 1

In my case, this error was always thrown after 120 seconds from receiving the request. I figured out the issue that Node 12 default request timeout is 120 seconds. So If you are using Node server you either can change the default timeout or update Node version to 13 as they removed the default timeout https://github.com/nodejs/node/pull/27558.

Answer 2

If your logs didn't catch anything useful, most probably the instance crashes because you run heavy CPU tasks. A mention about this can be found on the Google Issue Tracker:

A common cause for 503 errors on Cloud Run would be when requests use a lot of CPU and as the container is out of resources it is unable to process some requests