I mean what do I get from using async for
. Here is the code I write with async for
, AIter(10)
could be replaced with get_range()
.
But the code runs like sync not async.
import asyncio async def get_range(): for i in range(10): print(f"start {i}") await asyncio.sleep(1) print(f"end {i}") yield i class AIter: def __init__(self, N): self.i = 0 self.N = N def __aiter__(self): return self async def __anext__(self): i = self.i print(f"start {i}") await asyncio.sleep(1) print(f"end {i}") if i >= self.N: raise StopAsyncIteration self.i += 1 return i async def main(): async for p in AIter(10): print(f"finally {p}") if __name__ == "__main__": asyncio.run(main())
The result I excepted should be :
start 1 start 2 start 3 ... end 1 end 2 ... finally 1 finally 2 ...
However, the real result is:
start 0 end 0 finally 0 start 1 end 1 finally 1 start 2 end 2
I know I could get the excepted result by using asyncio.gather
or asyncio.wait
.
But it is hard for me to understand what I got by use async for
here instead of simple for
.
What is the right way to use async for
if I want to loop over several Feature
object and use them as soon as one is finished. For example:
async for f in feature_objects: data = await f with open("file", "w") as fi: fi.write()
asyncio is used as a foundation for multiple Python asynchronous frameworks that provide high-performance network and web-servers, database connection libraries, distributed task queues, etc. asyncio is often a perfect fit for IO-bound and high-level structured network code.
To run an async function (coroutine) you have to call it using an Event Loop. Event Loops: You can think of Event Loop as functions to run asynchronous tasks and callbacks, perform network IO operations, and run subprocesses. Example 1: Event Loop example to run async Function to run a single async function: Python3.
When to use async in Python. A Python program is best suited for async when it has the following characteristics: It's trying to do something that is mostly bound by I/O or by waiting for some external process to complete, like a long-running network read.
To do so we have to create a new async function (main) and call all the async functions (which we want to run at the same time) in that new function (main). And then call the new (main) function using Event Loop. Note: . create_task() is used to run multiple async functions at a time.
But it is hard for me to understand what I got by use
async for
here instead of simplefor
.
The underlying misunderstanding is expecting async for
to automatically parallelize the iteration. It doesn't do that, it simply allows sequential iteration over an async source. For example, you can use async for
to iterate over lines coming from a TCP stream, messages from a websocket, or database records from an async DB driver.
None of the above would work with an ordinary for
, at least not without blocking the event loop. This is because for
calls __next__
as a blocking function and doesn't await its result. You cannot manually await
elements obtained by for
because for
expects __next__
to signal the end of iteration by raising StopIteration
. If __next__
is a coroutine, the StopIteration
exception won't be visible before awaiting it. This is why async for
was introduced, not just in Python, but also in other languages with async/await and generalized for
.
If you want to run the loop iterations in parallel, you need to start them as parallel coroutines and use asyncio.as_completed
or equivalent to retrieve their results as they come:
async def x(i): print(f"start {i}") await asyncio.sleep(1) print(f"end {i}") return i # run x(0)..x(10) concurrently and process results as they arrive for f in asyncio.as_completed([x(i) for i in range(10)]): result = await f # ... do something with the result ...
If you don't care about reacting to results immediately as they arrive, but you need them all, you can make it even simpler by using asyncio.gather
:
# run x(0)..x(10) concurrently and process results when all are done results = await asyncio.gather(*[x(i) for i in range(10)])
If you love us? You can donate to us via Paypal or buy me a coffee so we can maintain and grow! Thank you!
Donate Us With