Dealing with db.Timeout on Google App Engine

Question

I'm testing my application (on Google App Engine live servers) and the way I've written it I have about 40 db.GqlQuery() statements in my code (mostly part of classes).

I keep getting db.Timeout very often though.

How do I deal with this? I was going to surround all my queries with really brutal code like this:

  querySucceeded = False
  while not querySucceeded :
    try :
      result = db.GqlQuery( """xxx""" ).get()
      querySucceeded = True #only get here if above line doesn't raise exc
    except :
      querySucceeded = False

Is this ok? Do you agree? What's a better way to deal with db.Timeouts?

Edit:

I now use this for any get queries

""" Query gets single result """
def queryGet( gql ) :
  querySucceeded = False
  while not querySucceeded :
    try :
      result = db.GqlQuery( gql ).get()
      querySucceeded = True #only get here if above line doesn't raise
    except :
      querySucceeded = False
  
  return result

I have similar functions for fetch and count.

Answer 1

Here's a decorator to retry on db.Timeout, adapted from one from Kay framework:

import logging, time
from google.appengine.ext import db

def retry_on_timeout(retries=3, interval=1.0, exponent=2.0):
    """A decorator to retry a given function performing db operations."""
    def _decorator(func):
        def _wrapper(*args, **kwargs):
            count = 0
            while True:
                try:
                    return func(*args, **kwargs)
                except db.Timeout, e:
                    logging.debug(e)
                    if count >= retries:
                        raise e
                    else:
                        sleep_time = (exponent ** count) * interval
                        logging.warning("Retrying function %r in %d secs" %
                            (func, sleep_time))
                        time.sleep(sleep_time)
                        count += 1

        return _wrapper

    return _decorator

To use it, simply decorate any function that performs db operations and you'd like to do retries:

@retry_on_timeout()
def do_the_stuff(models):
    return db.put(models)