Logo Questions Linux Laravel Mysql Ubuntu Git Menu
 

Python async coroutines in batches

I'd like to process coroutines in batches like:

import asyncio

async def test(i):
    print(f"Start Task {i}")
    await asyncio.sleep(0.1)
    print(f"Finished Task {i}")

async def main():
    for i in range(10):
        await asyncio.gather(*[test(10*i+j) for j in range(10)])


asyncio.run(main())

Is there a way to do this with Python builtins or libraries so that I do not have to create the batches individually?

Unfortunately

async with asyncio.Semaphore(10):
    await asyncio.gather(*[test(i) for i in range(100)])

isn't processing the coroutines as expected: the coroutines are created all at once. Only the excecutions are limited. I don't want to create all tasks at once. The tasks should be created in batches.

like image 532
McDizzy Avatar asked Jun 25 '21 21:06

McDizzy


1 Answers

This is what I was looking for:

import asyncio

from aiodecorators import Semaphore


@Semaphore(10)
async def test(i):
    print(f"Start Task {i}")
    await asyncio.sleep(0.1)
    print(f"Finished Task {i}")


async def main():
    await asyncio.gather(*[test(i) for i in range(100)])


asyncio.run(main())
like image 135
McDizzy Avatar answered Oct 22 '22 19:10

McDizzy