Unable to handle DeadlineExceededError while using UrlFetch

Question

I have this basic utility class fetches (possibly) shortened URLs in parallel and returns a dictionary that has final URLs. It uses the wait_any functionality that was described in this blog post.

class UrlFetcher(object):

  @classmethod
  def fetch_urls(cls,url_list):
    rpcs = []
    for url in url_list:
      rpc = urlfetch.create_rpc(deadline=5.0)
      urlfetch.make_fetch_call(rpc, url,method = urlfetch.HEAD)
      rpcs.append(rpc)

    result = {}
    while len(rpcs) > 0:
      rpc = apiproxy_stub_map.UserRPC.wait_any(rpcs)
      rpcs.remove(rpc)
      request_url = rpc.request.url()
      try:
        final_url = rpc.get_result().final_url
      except AttributeError:
        final_url = request_url
      except DeadlineExceededError:
        logging.error('Handling DeadlineExceededError for url: %s' %request_url)
        final_url  = None
      except (DownloadError,InvalidURLError):
        final_url  = None        
      except UnicodeDecodeError: #Funky url with very evil characters
        final_url = unicode(rpc.get_result().final_url,'utf-8')

      result[request_url] = final_url

    logging.info('Returning results: %s' %result)
    return result

Even if I try to handle DeadlineExceededError, the application logs show otherwise.

2011-04-20 17:06:17.755
UrlFetchWorker started
E 2011-04-20 17:06:22.769
The API call urlfetch.Fetch() took too long to respond and was cancelled.
Traceback (most recent call last):
  File "/base/python_runtime/python_lib/versions/1/google/appengine/ext/webapp/__init__.py", line 636, in __call__
    handler.post(*groups)
  File "/base/data/home/apps/tweethitapp/1.349863151373877476/tweethit/handlers/taskworker.py", line 80, in post
    result_dict = UrlFetcher.fetch_urls(fetch_targets)
  File "/base/data/home/apps/tweethitapp/1.349863151373877476/tweethit/utils/rpc.py", line 98, in fetch_urls
    final_url = rpc.get_result().final_url
  File "/base/python_runtime/python_lib/versions/1/google/appengine/api/apiproxy_stub_map.py", line 592, in get_result
    return self.__get_result_hook(self)
  File "/base/python_runtime/python_lib/versions/1/google/appengine/api/urlfetch.py", line 345, in _get_fetch_result
    rpc.check_success()
  File "/base/python_runtime/python_lib/versions/1/google/appengine/api/apiproxy_stub_map.py", line 558, in check_success
    self.__rpc.CheckSuccess()
  File "/base/python_runtime/python_lib/versions/1/google/appengine/api/apiproxy_rpc.py", line 133, in CheckSuccess
    raise self.exception
DeadlineExceededError: The API call urlfetch.Fetch() took too long to respond and was cancelled.
W 2011-04-20 17:06:22.858
Found 1 RPC request(s) without matching response (presumably due to timeouts or other errors)

What am I missing here? Is there any other way to handle DeadlineExceededError?

Or are there different types of DeadlineExceededErrors and I'm importing the wrong one? I'm using: from google.appengine.runtime import DeadlineExceededError

Answer 1

Per the inline docs for google.appengine.runtime.DeadlineExceededError:

Exception raised when the request reaches its overall time limit.

Not to be confused with runtime.apiproxy_errors.DeadlineExceededError. That one is raised when individual API calls take too long.

This is a good demonstration of why you should use qualified imports (from google.appengine import runtime, then reference runtime.DeadlineExceededError), too!

Answer 2

As you guessed and others noted, you want a different DeadlineExceededError:

From https://developers.google.com/appengine/articles/deadlineexceedederrors, dated June 2012:

Currently, there are several errors named DeadlineExceededError for the Python runtime:>

google.appengine.runtime.DeadlineExceededError: raised if the overall request times out, typically after 60 seconds, or 10 minutes for task queue requests;

google.appengine.runtime.apiproxy_errors.DeadlineExceededError: raised if an RPC exceeded its deadline. This is typically 5 seconds, but it is settable for some APIs using the 'deadline' option;

google.appengine.api.urlfetch_errors.DeadlineExceededError: raised if the URLFetch times out.

Catching google.appengine.api.urlfetch_errors.DeadlineExceededError seems to do the trick for me. Also worth noting that (at least in dev app server 1.7.1), the urlfetch_errors.DeadlineExceededError is a subclass of DownloadError, which makes sense:

class DeadlineExceededError(DownloadError):
  """Raised when we could not fetch the URL because the deadline was exceeded.

  This can occur with either the client-supplied 'deadline' or the system
  default, if the client does not supply a 'deadline' parameter.
  """