Logo Questions Linux Laravel Mysql Ubuntu Git Menu
 

Writing Dask partitions into single file

Tags:

python

dask

New to dask,I have a 1GB CSV file when I read it in dask dataframe it creates around 50 partitions after my changes in the file when I write, it creates as many files as partitions.
Is there a way to write all partitions to single CSV file and is there a way access partitions?
Thank you.

like image 906
rey Avatar asked Sep 19 '16 06:09

rey


People also ask

How many partitions can a Dask DataFrame have?

Repartition the DataFrame into 5 partitions.

Does Dask write to disk?

When Dask workers start to run out of memory they write extra data to disk. This is recorded in the status page as a disk-write- task.


1 Answers

Short answer

No, Dask.dataframe.to_csv only writes CSV files to different files, one file per partition. However, there are ways around this.

Concatenate Afterwards

Perhaps just concatenate the files after dask.dataframe writes them? This is likely to be near-optimal in terms of performance.

df.to_csv('/path/to/myfiles.*.csv') from glob import glob filenames = glob('/path/to/myfiles.*.csv') with open('outfile.csv', 'w') as out:     for fn in filenames:         with open(fn) as f:             out.write(f.read())  # maybe add endline here as well? 

Or use Dask.delayed

However, you can do this yourself using dask.delayed, by using dask.delayed alongside dataframes

This gives you a list of delayed values that you can use however you like:

list_of_delayed_values = df.to_delayed() 

It's then up to you to structure a computation to write these partitions sequentially to a single file. This isn't hard to do, but can cause a bit of backup on the scheduler.

Edit 1: (On October 23, 2019)

In Dask 2.6.x, there is a parameter as single_file. By default, It is False. You can set it True to get single file output without using df.compute().

For Example:

df.to_csv('/path/to/myfiles.csv', single_file = True) 

Reference: Documentation for to_csv

like image 99
MRocklin Avatar answered Sep 24 '22 19:09

MRocklin