combining tqdm with delayed execution with dask in python

Question

tqdm and dask are both amazing packages for iterations in python. While tqdm implements the needed progress bar, dask implements the multi-thread platform and they both can make iteration process less frustrating. Yet - I'm having troubles to combine them both together.

For example, the following code implements a delayed execution in dask, with tqdm.trange progress bar. The thing is that since the delayed is performed quickly, the progress bar ends immediately, while the real computation-time effort is done during the compute part.

from dask import delayed,compute
from tqdm import trange
from time import sleep

ct = time()
result= []

def fun(x):
    sleep(x)
    return x

for i in trange(10):
    result.append(delayed(fun)(i))

print compute(result)

How can I attach the progress bar to the actual execution in compute command?

Answer 1

Consider Dask's progress bar

from dask.diagnostics import ProgressBar

with ProgressBar():
    compute(result)

Build a diagnostic of your own

You can use this plugin architecture to get a signal at the end of every task. http://dask.pydata.org/en/latest/diagnostics.html

Here is an example of someone doing exactly this: https://github.com/tqdm/tqdm/issues/278

Answer 2

Based on :

Dask Integration

from tqdm.dask import TqdmCallback

with TqdmCallback(desc="compute"):
    ...
    arr.compute()

# or use callback globally
cb = TqdmCallback(desc="global")
cb.register()
arr.compute()

Applied to the code in the question:

from dask import delayed,compute
from tqdm.auto import tqdm
# from tqdm import trange
from time import sleep

from tqdm.dask import TqdmCallback

# ct = time()
result= []

def fun(x):
    sleep(x)
    return x

for i in tqdm(range(10)):
    result.append(delayed(fun)(i))

with TqdmCallback(desc="compute"):
    print(compute(result))

screenshot of output in jupyter :