Is python3.6 new change 'async for' not compatible with enumerate

Question

I am trying to move my apps from python2.7 to python3.6,for the asyncio and relates libs,but I find some functions are not working as I excepted. I use motor to query from mongodb asyncly,like:

async def do_query():
    song_already_processed = set()
    song_table = db.song_table
    async for index, item in enumerate(song_table.find({'lang': 'English'},
                              {'id': 1, '_id': 0, 'title': 1, 'artist.name': 1})):
        if index > 100:
            break
        if item['id'] in song_already_processed:
            continue
        song_already_processed.add(item['id'])

    print(len(song_already_processed))

but it raised the error:

TypeError: 'AsyncIOMotorCursor' object is not iterable

I think the async iterator protocol is diffirent from normal interator protocol,so the enumerate didn't work well.Is there is an async enumerate I can use?

By the way, I know a lot of ways to just get 100 doc and stop the iterate, I just want to know how to use 'async for' properly

Answer 1

The asyncstdlib library (disclaimer: I maintain this package) provides async variants of standard library helpers. In specific, asyncstdlib.enumerate works like enumerate but takes and produces an async iterable.

import asyncstdlib as a

async for index, item in a.enumerate(song_table.find(...)):
    if index > 100:
        break
    ...

Note that it is generally not a good idea to break out of an async iteration -- the iterator may not cleanup at the end of iteration (see PEP 533 for details).

Since you are using index just to get the first 100 items, you can also use asyncstdlib.islice to safely limit the iteration directly:

import asyncstdlib as a

async for item in a.islice(song_table.find(...), 100):
    ...

---

Python3.6 added asynchronous generators, which allows to easily implement async enumeration:

async def aenumerate(asequence, start=0):
    """Asynchronously enumerate an async iterator from a given start value"""
    n = start
    async for elem in asequence:
        yield n, elem
        n += 1

For older versions of Python, one must reify the async generator by hand:

class AsyncEnumerate:
    """Asynchronously enumerate an async iterator from a given start value"""
    def __init__(self, asequence, start=0):
        self._asequence = asequence
        self._value = start

    async def __anext__(self):
        elem = await self._asequence.__anext__()
        value, self._value = self._value, self._value + 1
        return value, elem

    def __aiter__(self):
        return self