Logo Questions Linux Laravel Mysql Ubuntu Git Menu
 

How to use an async for loop to iterate over a list?

So I need to call an async function for all items in a list. This could be a list of URLs and an async function using aiohttp that gets a response back from every URL. Now obviously I cannot do the following:

async for url in ['www.google.com', 'www.youtube.com', 'www.aol.com']:

I can use a normal for loop but then my code will act synchronously and I lose the benefits and speed of having an async response fetching function.

Is there any way I can convert a list such that the above works? I just need to change the list's __iter__() to a __aiter__() method right? Can this be achieved by subclassing a list? Maybe encapsulating it in a class?

like image 753
Max Smith Avatar asked Jan 01 '18 18:01

Max Smith


2 Answers

Use asyncio.as_completed:

for future in asyncio.as_completed(map(fetch, urls)):
    result = await future

Or asyncio.gather:

results = await asyncio.gather(*map(fetch, urls))

EDIT: If you don't mind having an external dependency, you can use aiostream.stream.map:

from aiostream import stream, pipe

async def fetch_many(urls):
    xs = stream.iterate(urls) | pipe.map(fetch, ordered=True, task_limit=10)
    async for result in xs:
        print(result)

You can control the amount of fetch coroutine running concurrently using the task_limit argument, and choose whether to get the results in order, or as soon as possible.

See more examples in this demonstration and the documentation.

Disclaimer: I am the project maintainer.

like image 124
Vincent Avatar answered Nov 16 '22 14:11

Vincent


Please note, that Vincents answer has a partial problem:
You must have a splatter operator infront of the map function, otherwise asyncio.gather would try to use the list as whole. So do it like this:

results = await asyncio.gather(*map(fetch, url))
like image 11
quadronom Avatar answered Nov 16 '22 16:11

quadronom