Making a tqdm progress bar for asyncio

Question

Am attempting a tqdm progress bar with asyncio tasks gathered.

Want the progress bar to be progressively updated upon completion of a task. Tried the code:

import asyncio
import tqdm
import random

async def factorial(name, number):
    f = 1
    for i in range(2, number+1):
        await asyncio.sleep(random.random())
        f *= i
    print(f"Task {name}: factorial {number} = {f}")

async def tq(flen):
    for _ in tqdm.tqdm(range(flen)):
        await asyncio.sleep(0.1)

async def main():
    # Schedule the three concurrently

    flist = [factorial("A", 2),
        factorial("B", 3),
        factorial("C", 4)]

    await asyncio.gather(*flist, tq(len(flist)))

asyncio.run(main())

...but this simply completes the tqdm bar and then processes factorials.

Is there a way to make the progress bar move upon completion of each asyncio task?

Answer 1

As of tqdm version 4.48.0, it is possible to use tqdm.asyncio.tqdm.as_completed()

import tqdm.asyncio
...
for f in tqdm.asyncio.tqdm.as_completed(flist):
    await f

Answer 2

Now, I'm not particularly familiar with asyncho, though I've used tqdm with some success for multiprocesses in python. The following change to your code seems to update the progress bar and print the result at the same time, which might be enough to get you started.

responses = [await f
                 for f in tqdm.tqdm(asyncio.as_completed(flist), total=len(flist))]

The above should replace await asyncio.gather(*flist, tq(len(flist))) in your main definition.

For more information, the above was inspired from asyncio aiohttp progress bar with tqdm

To only print the bar once and update it, I've done the following, which updates the description of the progress bar to include your message:

import asyncio
import tqdm


async def factorial(name, number):
    f = 1
    for i in range(2, number + 1):
        await asyncio.sleep(1)
        f *= i
    return f"Task {name}: factorial {number} = {f}"

async def tq(flen):
    for _ in tqdm.tqdm(range(flen)):
        await asyncio.sleep(0.1)


async def main():
    # Schedule the three concurrently

    flist = [factorial("A", 2),
             factorial("B", 3),
             factorial("C", 4)]

    pbar = tqdm.tqdm(total=len(flist))
    for f in asyncio.as_completed(flist):
        value = await f
        pbar.set_description(value)
        pbar.update()

if __name__ == '__main__':
    asyncio.run(main())

Answer 3

Made a couple of small changes to Dragos' code in pbar format and used tqdm.write() to get almost what I want, as follows:

import asyncio
import random

import tqdm


async def factorial(name, number):
    f = 1
    for i in range(2, number + 1):
        await asyncio.sleep(random.random())
        f *= i
    return f"Task {name}: factorial {number} = {f}"

async def tq(flen):
    for _ in tqdm.tqdm(range(flen)):
        await asyncio.sleep(0.1)


async def main():

    flist = [factorial("A", 2),
             factorial("B", 3),
             factorial("C", 4)]

    pbar = tqdm.tqdm(total=len(flist), position=0, ncols=90)
    for f in asyncio.as_completed(flist):
        value = await f
        pbar.set_description(desc=value, refresh=True)
        tqdm.tqdm.write(value)
        pbar.update()

if __name__ == '__main__':
    asyncio.run(main())