Logo Questions Linux Laravel Mysql Ubuntu Git Menu
 

How to groupby with consecutive occurrence of duplicates in pandas

I have a dataframe which contains two columns [Name,In.cl]. I want to groupby Name but it based on continuous occurrence. For example consider below DataFrame,

Code to generate below DF:

df=pd.DataFrame({'Name':['A','B','B','A','A','B','C','C','C','B','C'],'In.Cl':[2,1,5,2,4,2,3,1,8,5,7]})

Input:

    In.Cl Name
0       2    A
1       1    B
2       5    B
3       2    A
4       4    A
5       2    B
6       3    C
7       1    C
8       8    C
9       5    B
10      7    C

I want to group the rows where it repeated consecutively. example group [B] (1,2), [A] (3,4), [C] (6,8) etc., and perform sum operation in In.cl column.

Expected Output:

    In.Cl Name col1   col2
0       2    A   A(1)    2
1       1    B   B(2)    6
2       5    B   B(2)    6
3       2    A   A(2)    6
4       4    A   A(2)    6
5       2    B   B(1)    2
6       3    C   C(3)   12
7       1    C   C(3)   12
8       8    C   C(3)   12
9       5    B   B(1)    5
10      7    C   C(1)    7

So far i tried combination of duplicate and groupby, it didn't work as i expected. I think I need some thing groupby + consecutive. but i don't have an idea to solve this problem.

Any help would be appreciated.

like image 510
Mohamed Thasin ah Avatar asked Jun 18 '18 09:06

Mohamed Thasin ah


People also ask

How do I get rid of consecutive duplicates in pandas?

To drop consecutive duplicates with Python Pandas, we can use shift . to check if the last column isn't equal the current one with a. shift(-1) !=

How do I get repeated values in pandas?

The pandas. DataFrame. duplicated() method is used to find duplicate rows in a DataFrame. It returns a boolean series which identifies whether a row is duplicate or unique.


2 Answers

In [37]: g = df.groupby((df.Name != df.Name.shift()).cumsum())

In [38]: df['col1'] = df['Name'] + '(' + g['In.Cl'].transform('size').astype(str) + ')'

In [39]: df['col2'] = g['In.Cl'].transform('sum')

In [40]: df
Out[40]:
   Name  In.Cl  col1  col2
0     A      2  A(1)     2
1     B      1  B(2)     6
2     B      5  B(2)     6
3     A      2  A(2)     6
4     A      4  A(2)     6
5     B      2  B(1)     2
6     C      3  C(3)    12
7     C      1  C(3)    12
8     C      8  C(3)    12
9     B      5  B(1)     5
10    C      7  C(1)     7
like image 190
MaxU - stop WAR against UA Avatar answered Oct 14 '22 03:10

MaxU - stop WAR against UA


Slightly long-winded answer utilizing itertools.groupby.

For greater than ~1000 rows, use @MaxU's solution - it's faster.

from itertools import groupby, chain
from operator import itemgetter

chainer = chain.from_iterable

def sumfunc(x):
    return (sum(map(itemgetter(1), x)), len(x))

grouper = groupby(zip(df['Name'], df['In.Cl']), key=itemgetter(0))
summer = [sumfunc(list(j)) for _, j in grouper]

df['Name'] += pd.Series(list(chainer(repeat(j, j) for i, j in summer))).astype(str)
df['col2'] = list(chainer(repeat(i, j) for i, j in summer))

print(df)

    In.Cl Name  col2
0       2   A1     2
1       1   B2     6
2       5   B2     6
3       2   A2     6
4       4   A2     6
5       2   B1     2
6       3   C3    12
7       1   C3    12
8       8   C3    12
9       5   B1     5
10      7   C1     7
like image 24
jpp Avatar answered Oct 14 '22 05:10

jpp