Dask reading CSV, setting partition as CSV length

Question

I'm trying to write code that will read from a set of CSVs named my_file_*.csv into a Dask dataframe.

Then I want to set the partitions based on the length of the CSV. I'm trying to map a function on each partition and in order to do that, each partition must be the whole CSV.

I've tried to reset the index, and then set partitions based on the length of each CSV but it looks like the index of the Dask dataframe is not unique.

Is there a better way to partition based on the length of each CSV?

Answer 1

So one partition should contain exactly one file? You cold do:

import dask.dataframe as dd
ddf = dd.read_csv(my_file_*.csv, blocksize = None)

Setting blocksize to None makes sure that files are not split up in several partitions. Therefore, ddf will be a dask dataframe containing one file per partition.

You might want to check out the documentation:

• general instructions how to generate dask dataframes from data
• details about read_csv