Does __await__ need to be a generator?

Question

I want to implement an awaitable and noticed that __await__ 'needs' to be a generator.

From PEP-492:

An object with an __await__ method returning an iterator.

...

Objects with __await__ method are called Future-like objects in the rest of this PEP.

It is a TypeError if __await__ returns anything but an iterator.

In my experience, before await was a statement, yield from was used together with coroutines implemented as generators. Nowadays python (I'm using 3.5) has asynchronous methods using the async def syntax. I therefore consider the yield from syntax as old/deprecated.

So I broke out the interpreter to see how/if this works:

>>> class A:
...     def __await__(self):
...         yield from (asyncio.sleep(1).__await__())
...         return 'spam'
... 
>>> a = A()
>>> loop = asyncio.get_event_loop()
>>> loop.run_until_complete(a)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/lib64/python3.5/asyncio/base_events.py", line 467, in run_until_complete
    return future.result()
  File "/usr/lib64/python3.5/asyncio/futures.py", line 294, in result
    raise self._exception
  File "/usr/lib64/python3.5/asyncio/tasks.py", line 240, in _step
    result = coro.send(None)
  File "/usr/lib64/python3.5/asyncio/tasks.py", line 585, in _wrap_awaitable
    return (yield from awaitable.__await__())
  File "<stdin>", line 3, in __await__
AttributeError: 'generator' object has no attribute '__await__'

So it appears asyncio.sleep doesn't have the __await__ method. It also feels very awkward to use this yield from syntax.

So I decided to try with the async syntax, just to see if it would work:

>>> class A:
...     async def __await__(self):
...         await asyncio.sleep(1)
...         return 'spam'
... 
>>> a = A()
>>> 
>>> loop = asyncio.get_event_loop()
>>> loop.run_until_complete(a)
'spam'

It actually seems to work! So now I'm wondering, does the __await__ method really need to be a generator using the yield from syntax?

---

Edit: When adding a level of indirection, so the awaitable is used in an await statement the problem becomes apparent:

>>> async def test():
...     return await A()
... 
>>> loop.run_until_complete(test())
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/lib/python3.5/asyncio/base_events.py", line 387, in run_until_complete
    return future.result()
  File "/usr/lib/python3.5/asyncio/futures.py", line 274, in result
    raise self._exception
  File "/usr/lib/python3.5/asyncio/tasks.py", line 239, in _step
    result = coro.send(None)
  File "<stdin>", line 2, in test
TypeError: __await__() returned a coroutine

It actually thus needs to be returning a generator like so:

class A:
    def __await__(self):
        yield from asyncio.sleep(1)
        return 'spam'    

Answer 1

In order to work in an await expression, __await__ does not need to be a generator. However, certain operations are only available if the result of __await__ support the generator interface.

Namely, it is not possible to send values or throw exceptions into an iterator-__await__. Only None can be "sent" to an iterator-__await__, as if generator.__next__ were used.

---

Let's consider a simple Awaitable that returns an iterator from its __await__.

class Iter:
    """Basic iterator that yields the same value"""
    def __next__(self): return 1
    def __iter__(self): return self

class IterAwait:
    """Awaitable that uses an iterator for __await__"""
    def __await__(self):
        return Iter()

We can check that they implement the desired interfaces:

>>> from collections.abc import Awaitable, Iterator, Generator
>>> isinstance(IterAwait(), Awaitable)
True
>>> isinstance(IterAwait().__await__(), Iterator)
True
>>> isinstance(IterAwait().__await__(), Generator)
False

In order to see how this interacts with await, we wrap it in a coroutine:

async def iter_await():
    await IterAwait()

Every operation we perform on iter_await with the full coroutine/generator interface is forwarded by await to our iterator-__await__. This allows to study how the iterator-__await__ receives signals:

>>> test_iter = iter_await()
>>> test_iter.send(3)         # 0. does it appear like a coroutine?
TypeError: can`t send non-None value to a just-started coroutine
>>> test_iter.send(None)      # 1. must initialise just-started coroutine
1
>>> test_iter.send(None)      # 2. repeatedly use the underlying iterator
1
>>> next(test_iter)           # 3. do we expose the iterator?
TypeError: 'coroutine' object is not an iterator
>>> test_iter.send(3)         # 4. can we send non-None values?
AttributeError: 'Iter' object has no attribute 'send'
>>> test_iter = iter_await()  # we just broke the coroutine...
>>> test_iter.send(None)      # ...need a new one
1
>>> test_iter.throw(KeyError) # 4. can we throw Exceptions?
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "<stdin>", line 2, in iter_await
KeyError

As can be seen, await can handle an iterator-__await__, but does not forward all operations. However, some are translated and some are handled early.

• It is always possible to .send(None), which is translated to a bare __next__(). (1, 2)
• The coroutine does not magically expose .__next__ (3) and cannot translate .send with a value either (4).
• It is possible to .throw an exception, but await handles it early in the coroutine.

Note that await uses the throw and send method as available. If the result of __await__ implements send but not throw or vice versa, the functionality present is used. Only __next__ is mandatory.

Answer 2

So it appears asyncio.sleep doesn't have the __await__ method

True, but it doesn't have to have one to be awaitable. The documentation says that __await__, if present, needs to return an iterator, not that await will only work on objects that define __await__. In fact, it explicitly documents that the argument to await can be one of:

• A native coroutine object returned from a native coroutine function.

• A generator-based coroutine object returned from a function decorated with types.coroutine().

• An object with an __await__ method returning an iterator.

• An object defined in C providing the Python/C equivalent of the __await__ special method.

So now I'm wondering, does the __await__ method really need to be a generator using the yield from syntax?

If you actually have an __await__ method, it does need to return an iterator.