dask groupby without aggregation

Question

I have this pure Pandas statement that works (on small dataset).

grouped_dfs = {key: group.drop(columns=['country']) for key, group in df.groupby('country')}

Now, to manage very large csv files, I am reading-in using Dask and trying to get a dict of grouped dfs like above. Is there a way to do exactly the same using Dask?

For now, I am converting the dask-dataframe to Pandas DF (df = df.compute()) and proceeding to run my tasks. Sounds crazy, but, currently, my dataset is small enough, so, I am able to get by.

I have read many posts and articles and couldn't find much. Dask seems to require some sort of aggregation after grouping.

Any help is appreciated.

Answer 1

You should check duckDB

Read your csv file as pandas df. Insert the df in duckdb and then you can perform queries like this:

query = F"SELECT * FROM your_df where country = ?;"

with duckdb.connect(local_file_path) as conn:
    result = conn.execute(query,('US',)).fetchdf()

or

countries = con.execute("SELECT DISTINCT country FROM my_df").fetchall()

grouped_dfs = {}
for country, in countries:
    query = f"SELECT value FROM my_table WHERE country = ?"
    grouped_dfs[country] = con.execute(query, (country)).df()