Tornado async call to a function

Question

I am making a web application using Python + Tornado which basically serves files to users. I have no database.

The files are either directly picked up and served if they are available, or generated on the fly if not.

I want the clients to be served in an async manner, because some files may already be available, while others need to be generated (thus they need to wait, and I don't want them to block other users).

I have a class that manages the picking or generation of files, and I just need to call it from Tornado.

What is the best way (most efficient on CPU and RAM) to achieve that? Should I use a thread? A sub process? A simple gen.Task like this one?

Also, I would like my implementation to work on Google App Engines (I think they do not allow sub processes to be spawned?).

I'm relatively new to the async web servicing, so any help is welcome.

Answer 1

I've found the answers to my questions: The genTask example is indeed the best way to implement an async call, and it is due to the fact that the example does use a Python coroutine, which I didn't understand at first glance because I thought yield was only used to return a value for generators.

Concrete example:

class MyHandler(tornado.web.RequestHandler):

    @asynchronous
    @gen.engine
    def get(self):
        response = yield gen.Task(self.dosomething, 'argument')

What is important here is the combination of two things:

• yield , which in fact spawns a coroutine (or pseudo-thread, which is very efficient and are done to be highly concurrent-friendly). http://www.python.org/dev/peps/pep-0342/

• gen.Task() which is a non-blocking (async) function, because if you spawn a coroutine on a blocking function, it won't be async. gen.Task() is provided by Tornado, specifically to work with the coroutine syntax of Python. More infos: http://www.tornadoweb.org/documentation/gen.html

So a canonical example of an async call in Python using coroutines:

response = yield non_blocking_func(**kwargs)