How to see progress of Dask compute task?

Question

I would like to see a progress bar on Jupyter notebook while I'm running a compute task using Dask, I'm counting all values of id column from a large csv file +4GB, so any ideas?

import dask.dataframe as dd  df = dd.read_csv('data/train.csv') df.id.count().compute() 

Answer 1

If you're using the single machine scheduler then do this:

from dask.diagnostics import ProgressBar ProgressBar().register() 

http://dask.pydata.org/en/latest/diagnostics-local.html

If you're using the distributed scheduler then do this:

from dask.distributed import progress  result = df.id.count.persist() progress(result) 

Or just use the dashboard

http://dask.pydata.org/en/latest/diagnostics-distributed.html