Python coroutines on builtin functions

Question

I'm currently using python 3.4, @coroutine decorator and yield keyword (in tornado) for asynchronous purposes. I wrote an ORM with a lot of objects instrospection which call a "slow" database and override built-in functions like __init__ or __contains__. My question is : For example, when my asynchronous code is in the __contains__ definition of an object, how can I call it implicitly/transparently when I use the "in" operator in my tornado controller? Implicit because I don't want the controller-side developper to change his code when he call built-in functions.

Answer 1

If I understand the question properly, the answer is that you can't; there's no way to write a magic method as an explicit coroutine and have it behave properly when called implicitly by Python. This is a well-known limitation of explicit coroutines.

So, if you have this:

class SomeObj:
    @coroutine
    def __contains__(self, obj):
        exists = yield self.somemethod(obj)
        return exists

This won't do what you want it to do:

o = SomeObj()
'x' in o # This won't work right

Python doesn't expect __contains__ to be a coroutine, and won't behave properly if it is - the core of the language doesn't know anything about tornado, or asyncio, or any other frameworks used to implement these coroutines, and won't integrate with them properly. The same applies for other implicitly-called magic methods like __init__, __getattr__, etc.

If you need to suspend, you have to explicitly call a method using yield or yield from (depending on the framework). Generally, this means using a function (or perhaps a @classmethod) to instantiate your SomeObj and then having that function call a method that does the slow, asynchronous call, rather than doing it all in __init__:

@coroutine
def create_someobj():
   s = SomeObj()
   yield s.slow_init()
   return s

And just calling a normal coroutine method called something like contains instead of relying on the in keyword. Not ideal, but that's the world we live in.

That said, there is some effort being made to improve this; PEP 492, in addition to introducing a new syntax for coroutines, adds support for asynchronous for-loops and context managers (using new magic methods specifically designed to be asynchronous). So starting with Python 3.5, you can this:

async def some_func():  # async is used instead of a coroutine decorator
    # Assume SomeObj implements __anext__, __aiter__, __aenter__, and __aexit__
    s = SomeObj()
    async for item in s:  # You can suspend while iterating over s using __anext__
       print(item)

    async with SomeObj() as s: # You can suspend on enter and exit of this context manager using __aenter__ and __aexit__
        await s.some_method() # await is used instead of yield from