Read delay in App Engine Datastore after put()

Question

I write a code for a blog/news site. Main page has 10 most recent articles and also there is an archive section with all articles sorted by modification time descending. In archive section I use pagination based on cursors and I cache results starting from the second page as pages are changed only when new article is published or existing goes to drafts for some reason. Every page has 10 articles. So when a user hits an archive page with some number (not the first one) memcache is checked for that page number results first. If the page is not there, memcache is checked for the cursor for that page and then results are fetched from datastore using that cursor:

class archivePage:
    def GET(self, page):
        if not page:
            articles = memcache.get('archivePage')
            if not articles:
                articles = fetchArticles()
                memcache.set('archivePage', articles)
        else:
            if int(page) == 0 or int(page) == 1:
                raise web.seeother('/archive')
            articles = memcache.get('archivePage'+page)
            if not articles:
                pageCursor = memcache.get('ArchivePageMapping'+page)
                if not pageCursor:
                    pageMapping = ArchivePageMapping.query(ArchivePageMapping.page == int(page)).get()
                    pageCursor = pageMapping.cursor
                    memcache.set('ArchivePageMapping'+page, pageCursor)
                articles = fetchArticles(cursor=Cursor(urlsafe=pageCursor))
                memcache.set('archivePage'+page, articles)

Every time a new article is created or the status of an existing article is changed (draft/published) I refresh the cache for archive pages results and cursors. I do it after saving an article to the datastore:

class addArticlePage:     
    def POST(self):
        formData = web.input()
        if formData.title and formData.content:
            article = Article(title=formData.title,
                              content=formData.content,
                              status=int(formData.status))
            key = article.put()
            if int(formData.status) == 1:
                cacheArchivePages()
            raise web.seeother('/article/%s' % key.id())

def cacheArchivePages():
    articles, cursor, moreArticles = fetchArticlesPage()
    memcache.set('archivePage', articles)
    pageNumber=2
    while moreArticles:
        pageMapping = ArchivePageMapping.query(ArchivePageMapping.page == pageNumber).get()
        if pageMapping:
            pageMapping.cursor = cursor.urlsafe()
        else:
            pageMapping = ArchivePageMapping(page=pageNumber,
                                            cursor=cursor.urlsafe())
        pageMapping.put()
        memcache.set('ArchivePageMapping'+str(pageNumber), cursor.urlsafe())
        articles, cursor, moreArticles = fetchArticlesPage(cursor=cursor)
        memcache.set('archivePage'+str(pageNumber), articles)
        pageNumber+=1

And here comes the problem. Sometimes (there is no law, it happens randomly) after refreshing the cache I get the same results and cursors for archive pages as before the refresh. For example I add a new article. It is saved in the datastore and it appears on the front page and on the first page in the archive (the first page of the archive is not cached). But other archive pages are not updated. I've tested my cacheArchivePages() function and it works as expected. Could it be so that too little time has passed after I put() an update to the datastore and before I fetchArticlesPage() in cacheArchivePages() function? Maybe the write transaction didn't finish yet and so I get old results? I tried to use time.sleep() and wait a few seconds before calling cacheArchivePages() and in that case I was not able to reproduce that behaviour, but it seems to me that time.sleep() is not a good idea. Anyway I need to know the exact cause for that behaviour and how to deal with it.

Answer 1

You're most likely being hit by "eventually consistent queries". When using the HR datastore, queries may use slightly old data, and it takes a while for data written by put() to be visible to queries (there is no such delay for get() by key or id). The delay is typically measured in seconds but I don't think we guarantee an upper bound -- if you're hit by an unfortunate network partition it might be hours, I imagine.

There are all sorts of partial solutions, from cheating when the author of the most recent writes is viewing the query results to using ancestor queries (which have their own share of limitations). You might simply give your cache a limited lifetime and update it on read instead of write.

Good luck!