Logo Questions Linux Laravel Mysql Ubuntu Git Menu
 

pandas create multiple dataframes based on duplicate index dataframe

If I have a dataframe with duplicates in the index, how would I create a set of dataframes with no duplicates in the index?

More precisely, given the dataframe:

   a  b
1  1  6
1  2  7
2  3  8
2  4  9
2  5  0

I would want as output, a list of dataframes:

   a  b
1  1  6
2  3  8


   a  b
1  2  7
2  4  9


   a  b
2  5  0

This needs to be scalable to as many dataframes as needed based on the number of duplicates.

like image 638
Jim Eisenberg Avatar asked May 23 '19 08:05

Jim Eisenberg


3 Answers

df=df.reset_index()
dfs=[]
while not df.empty:
    dfs.append(df[~df.duplicated('index',keep='first')].set_index('index'))
    df=df[df.duplicated('index',keep='first')]

#dfs will have all your dataframes
like image 74
Vicky Avatar answered Oct 09 '22 12:10

Vicky


Use GroupBy.cumcount for custom groups and then convert groups to dictionaries:

df = dict(tuple(df.groupby(df.groupby(level=0).cumcount())))
print (df)
{0:    a  b
1  1  6
2  3  8, 1:    a  b
1  2  7
2  4  9, 2:    a  b
2  5  0}

print (dfs[0])
   a  b
1  1  6
2  3  8

Or convert to list of DataFrames:

dfs = [x for i, x in df.groupby(df.groupby(level=0).cumcount())]
print (dfs)
[   a  b
1  1  6
2  3  8,    a  b
1  2  7
2  4  9,    a  b
2  5  0]
like image 45
jezrael Avatar answered Oct 09 '22 12:10

jezrael


Another approach is to use pd.DataFrame.groupby.nth:

import numpy as np

g = df.groupby(df.index)
cnt = np.bincount(df.index).max()
dfs = [g.nth(i) for i in range(cnt)]

Output:

[  a  b
1  1  6
2  3  8,    
   a  b
1  2  7
2  4  9,
   a  b
2  5  0]
like image 24
Chris Avatar answered Oct 09 '22 14:10

Chris