how to understand appengine ndb.tasklet?

Question

From documentation:

An NDB tasklet is a piece of code that might run concurrently with other code. If you write a tasklet, your application can use it much like it uses an async NDB function: it calls the tasklet, which returns a Future; later, calling the Future's get_result() method gets the result.

The explanation and examples in the document really likes a magic for me. I can use it, but feel hard to understand it properly.

For example:

1. May I put any kind of code inside a function and decorate it as ndb.tasklet? Then used it as async function later. Or it must be appengine RPC?
2. Does this kind of decorator also works on my PC?
3. Is it the same as tasklet for pypy

Answer 1

If you look at the implementation of a Future, its very comparable to what a generator is in python. In fact, it uses the same yield keyword to achieve what it says it does. Read the intro comments on the tasklets.py for some clarification.

When you use the @tasklet decorator, it creates a Future and waits for a value on the wrapped function. If the value is a generator, it adds the Future to the event loop. When you yield on a Future, the event loop runs through ALL queued Futures until the Future you want is ready. The concurrency here is that each Future will execute its code until it either returns (using raise ndb.Return(...) or the function completes), an exception is thrown, or yield is used again. I guess technically, you can use yield in the code just to stop executing that function in favor of letting the event loop continue running other Futures, but I would assume this wouldn't help much unless you really have a clever use-case in mind.

To answer your questions:

1. Technically yes, but it will not run asynchronously. When you decorate a non-yielding function with @tasklet, its Future's value is computed and set when you call that function. That is, it runs through the entire function when you call it. If you want to achieve asynchronous operation, you must yield on something that does asynchronous work. Generally in GAE it will work its way down to an RPC call.

2. If by work on your PC you mean does the dev appserver implement tasklets/Futures like GAE, then yes, although this is more accurate with the devappserver2 (now the default in the newer SDK). I'm actually not 100% sure if local RPC calls will run in parallel when using Futures, but there is an eventloop going through Futures whether its local or in production. If you want to use Future's in your other, non-GAE code, then I think you would be better off using Python 3.2's built-in future (or find a backport here)

3. Kind of, its not really a simple comparison. Look at the documentation here. The idea is somewhat the same (the scheduler can be compared to the eventloop), but the low-level implementation greatly differs.