How does 'yield' work in tornado when making an asynchronous call?

Question

Recently, I was learning Introduction to Tornado, and I came across the following code:

class IndexHandler(tornado.web.RequestHandler):
    @tornado.web.asynchronous
    @tornado.gen.engine
    def get(self):
        query = self.get_argument('q')
        client = tornado.httpclient.AsyncHTTPClient()
        response = yield tornado.gen.Task(client.fetch,
                "http://search.twitter.com/search.json?" + \
                urllib.urlencode({"q": query, "result_type": "recent", "rpp": 100}))
        body = json.loads(response.body)

        [...omitted the following code...]

I used to learn that yield is the key word turning a common function into a generator, and when it used in the form other = yield foo means, "yield foo and, when a value is sent to me, set other to that value." So I tried the following code in ipython:

In [1]: result = 'init'     #set a global variable

In [2]: def test_yield():
   ...:     global result
   ...:     print 'start test...'
   ...:     result = yield 'foo'
   ...:     print 'end test...'
   ...:     

In [3]: t = test_yield()

In [4]: t.next()
start test...
Out[4]: 'foo'  #'foo' has been yield to the caller, and blocked

Now I printed the global varialbe result, and it still referred to string 'init':

In [5]: print result
init

Then I called the send() method, and sent a new string to yield:

In [6]: t.send('new message')
end test...
---------------------------------------------------------------------------
StopIteration                             Traceback (most recent call last)
/home/chiyu/<ipython-input-6-b86312ad7d0e> in <module>()
----> 1 t.send('new message')

StopIteration: 

As expected, a StopIteration was raised and output the string 'end test...', but now the global variable result has been changed:

In [7]: print result
new message

Apperantly, the yield statement accepted the string when we called the send() method, and assigned the new string to the variable result.

MY QUESTION IS:

Back to the code showed on the top, according to this logic,

response = yield tornado.gen.Task(client.fetch,
                    "http://search.twitter.com/search.json?" + \
                    urllib.urlencode({"q": query, "result_type": "recent", "rpp": 100}))

when method client.fetch returned, a Task instance would be created and yield to the caller, but the variable response on the left will recieve nothing because no send() method has been excuted. I got quite confused about this, and googled in vain.

I would be really appreciated for your explanations!

Answer 1

You are already understanding how Tornado uses generators to handle async calls.

I'm assuming here that client is an instance of tornado.httpclient.AsyncHTTPClient(); it's fetch() method takes a callback function.

The tornado.gen.Task object only takes a reference to the client.fetch method; you are not calling it at that point. You are constructing a Task() instance with that method reference and an argument, then yielding that.

Tornado will then run that Task; the Task will in turn call client.fetch() with the argument provided, plus a callback function. The client.fetch() then runs asynchronously, and calls the callback. Whatever is then passed to the callback is then recorded as the Task result.

That result is then sent to your IndexHandler.get() generator with send(), returned from the yield Task() expression and assigned to response.

In other words, Tornado does use .send() here and something is assigned to response.

Answer 2

It sort of means "I give up control to you; come back when you have the result of this Future", and is thus kind of like an asynchronous dereferencing operator. Tornado responds by sending back the result when it's ready.

See http://tornado.readthedocs.org/en/latest/gen.html for a more in-depth explanation.