How to set timeout for urlfetch in Google App Engine?

Question

I'm trying to have Django (on top of GAE) fetch data from another web service. I'm often hit with error like this:

ApplicationError: 2 timed out Request

Method: GET

Request URL:http://localhost:8080/

Exception Type: DownloadError

Exception Value: ApplicationError: 2 timed out

Exception Location: /google_appengine/google/appengine/api/urlfetch.py in _get_fetch_result, line 325

It feels as if it will time out only after 12 seconds (I'm not sure, but it's really short).

Question: how can I set a longer timeout?

Answer 1

Seeing as this is a Python question, I thought I'd provide a Python answer for anyone who comes across this problem.

Just import urlfetch and then define a deadline before doing anything else in your code:

from google.appengine.api import urlfetch

urlfetch.set_default_fetch_deadline(60)

Answer 2

You can set it using the deadline argument of the fetch function. From the docs:

The deadline can be up to a maximum of 60 seconds for request handlers and 10 minutes for tasks queue and cron job handlers. If deadline is None, the deadline is set to 5 seconds.

---

Edit: looks like this has changed now. From here:

You can set a deadline for a request, the most amount of time the service will wait for a response. By default, the deadline for a fetch is 5 seconds. You can adjust the default deadline for requests using the urlfetch.set_default_fetch_deadline() function.

And this page lists the default timeout values:

Currently, there are several errors named DeadlineExceededError for the Python runtime:

• google.appengine.runtime.DeadlineExceededError: raised if the overall request times out, typically after 60 seconds, or 10 minutes for task queue requests.
• google.appengine.runtime.apiproxy_errors.DeadlineExceededError: raised if an RPC exceeded its deadline. This is typically 5 seconds, but it is settable for some APIs using the 'deadline' option.
• google.appengine.api.urlfetch_errors.DeadlineExceededError: raised if the URLFetch times out.

Answer 3

For Go, you might want to try below code.

// createClient is urlfetch.Client with Deadline
func createClient(context appengine.Context, t time.Duration) *http.Client {
    return &http.Client{
        Transport: &urlfetch.Transport{
            Context:  context,
            Deadline: t,
        },
    }
}

Here is how to use it.

// urlfetch
client := createClient(c, time.Second*60)