I'd like to process coroutines in batches like:
import asyncio
async def test(i):
print(f"Start Task {i}")
await asyncio.sleep(0.1)
print(f"Finished Task {i}")
async def main():
for i in range(10):
await asyncio.gather(*[test(10*i+j) for j in range(10)])
asyncio.run(main())
Is there a way to do this with Python builtins or libraries so that I do not have to create the batches individually?
Unfortunately
async with asyncio.Semaphore(10):
await asyncio.gather(*[test(i) for i in range(100)])
isn't processing the coroutines as expected: the coroutines are created all at once. Only the excecutions are limited. I don't want to create all tasks at once. The tasks should be created in batches.
This is what I was looking for:
import asyncio
from aiodecorators import Semaphore
@Semaphore(10)
async def test(i):
print(f"Start Task {i}")
await asyncio.sleep(0.1)
print(f"Finished Task {i}")
async def main():
await asyncio.gather(*[test(i) for i in range(100)])
asyncio.run(main())
If you love us? You can donate to us via Paypal or buy me a coffee so we can maintain and grow! Thank you!
Donate Us With