Twisted inlineCallbacks and remote generators

Question

I have used defer.inlineCallbacks in my code as I find it much easier to read and debug than using addCallbacks.

I am using PB and I have hit a problem when returning data to the client. The data is about 18Mb in size and I get a failed BananaError because of the length of the string being returned.

What I want to do is to write a generator so I can just keep calling the function and return some of the data each time the function is called.

How would I write that with inlineCallbacks already being used? Is it actually possible, If i return a value instead. Would something like the following work?

@defer.inlineCallbacks
def getLatestVersions(self):
    returnlist = []
    try:
        latest_versions = yield self.cur.runQuery("""SELECT id, filename,path,attributes ,MAX(version) ,deleted ,snapshot , modified, size, hash, 
                           chunk_table, added, isDir, isSymlink, enchash from files group by filename, path""")
    except:
        logger.exception("problem querying latest versions")

    for result in latest_versions:
        returnlist.append(result)
        if len(return_list) >= 10:
            yield return_list
            returnlist = [] 
    yield returnlist

Answer 1

A generator function decorated with inlineCallbacks returns a Deferred - not a generator. This is always the case. You can never return a generator from a function decorated with inlineCallbacks.

See the pager classes in twisted.spread.util for ideas about another approach you can take.