Is google.appengine.api.urlfetch deadline limited to 60s?

Question

I'm using python on google app engine, and keep getting google.appengine.api.urlfetch_errors.DeadlineExceededError on requests made from a machine which does some backend processing. The requests take approximately 60s, sometimes a little longer, so I've attempted to increase the deadline.

The requests are wrapped in a retry, and from the logs I can see that the time between retries is always ~60s. I assume this is either because I've configured things incorrectly, or misunderstand the limitations of the deadline.

The machine config is:

instance_class: B8
basic_scaling:
  max_instances: 1
  idle_timeout: 10m

The code I'm using is (redacted for simplicity):

from google.appengine.api import urlfetch
from retrying import retry

timeout = 600
retries = 10

@retry(
    stop_max_attempt_number=retries,
    wait_exponential_multiplier=1000,
    wait_exponential_max=1000*60*5
)
def fetch(url):
    """Fetch remote data, retrying as necessary"""
    urlfetch.set_default_fetch_deadline(timeout)
    result = urlfetch.fetch(url)
    if result.status_code != 200:
        raise IOError("Did not receive OK response from server")
    return result.content

data = fetch(config['url'])

I've tried setting the deadline explicitly as urlfetch.fetch(url, deadline=timeout) but setting the default seems to be the approach most people suggest.

Can anyone clarify whether there is a maximum value which can be set for deadline?

Answer 1

The Request Timer

The Google App Engine request timer (Java/Python/Go) ensures that requests have a finite lifespan and do not get caught in an infinite loop. Currently, the deadline for requests to frontend instances is 60 seconds. (Backend instances have no corresponding limit.) Every request, including warmup (request to /_ah/warmup) and loading requests ("loading_request=1" log header), is subject to this restriction.

If a request fails to return within 60 seconds and a DeadlineExceededError is thrown and not caught, the request is aborted and a 500 internal server error is returned. If the DeadlineExceededError is caught but a response is not produced quickly enough (you have less than a second), the request is aborted and a 500 internal server error is returned.

As far as i read the document i think the maximum request timeout is 60 seconds in the app engine. Here is the link to the documentation