Logo Questions Linux Laravel Mysql Ubuntu Git Menu
 

How do I drop duplicates and keep the last timestamp on pandas

I want to drop duplicates and keep the last timestamp. The duplicates that want to be dropped is customer_id and var_name .Here's my data

    customer_id  value   var_name     timestamp
    1            1       apple        2018-03-22 00:00:00.000        
    2            3       apple        2018-03-23 08:00:00.000
    2            4       apple        2018-03-24 08:00:00.000
    1            1       orange       2018-03-22 08:00:00.000
    2            3       orange       2018-03-24 08:00:00.000
    2            5       orange       2018-03-23 08:00:00.000

So the result will be

    customer_id  value   var_name     timestamp
    1            1       apple        2018-03-22 00:00:00.000        
    2            4       apple        2018-03-24 08:00:00.000
    1            1       orange       2018-03-22 08:00:00.000
    2            3       orange       2018-03-24 08:00:00.000
like image 235
Nabih Bawazir Avatar asked Mar 22 '18 10:03

Nabih Bawazir


Video Answer


1 Answers

I think need sort_values with drop_duplicates:

df = df.sort_values('timestamp').drop_duplicates(['customer_id','var_name'], keep='last')
print (df)
   customer_id  value var_name                timestamp
0            1      1    apple  2018-03-22 00:00:00.000
3            1      1   orange  2018-03-22 08:00:00.000
2            2      4    apple  2018-03-24 08:00:00.000
4            2      3   orange  2018-03-24 08:00:00.000

If dont need sorting - order is important:

df = df.loc[df.groupby(['customer_id','var_name'], sort=False)['timestamp'].idxmax()]
print (df)
   customer_id  value var_name           timestamp
0            1      1    apple 2018-03-22 00:00:00
2            2      4    apple 2018-03-24 08:00:00
3            1      1   orange 2018-03-22 08:00:00
4            2      3   orange 2018-03-24 08:00:00
like image 198
jezrael Avatar answered Oct 17 '22 05:10

jezrael