Logo Questions Linux Laravel Mysql Ubuntu Git Menu
 

how do I remove rows with duplicate values of columns in pandas data frame?

Tags:

python

pandas

I have a pandas data frame which looks like this.

  Column1  Column2 Column3 0     cat        1       C 1     dog        1       A 2     cat        1       B 

I want to identify that cat and bat are same values which have been repeated and hence want to remove one record and preserve only the first record. The resulting data frame should only have.

  Column1  Column2 Column3 0     cat        1       C 1     dog        1       A 
like image 374
Sayonti Avatar asked Jun 16 '18 04:06

Sayonti


People also ask

How do I delete rows in pandas DataFrame based on condition?

Use pandas. DataFrame. drop() method to delete/remove rows with condition(s).

Which pandas function using to removed duplicate rows?

Pandas drop_duplicates() Function Syntax If 'first', duplicate rows except the first one is deleted. If 'last', duplicate rows except the last one is deleted.


2 Answers

Using drop_duplicates with subset with list of columns to check for duplicates on and keep='first' to keep first of duplicates.

If dataframe is:

df = pd.DataFrame({'Column1': ["'cat'", "'toy'", "'cat'"],                    'Column2': ["'bat'", "'flower'", "'bat'"],                    'Column3': ["'xyz'", "'abc'", "'lmn'"]}) print(df) 

Result:

  Column1   Column2 Column3 0   'cat'     'bat'   'xyz' 1   'toy'  'flower'   'abc' 2   'cat'     'bat'   'lmn' 

Then:

result_df = df.drop_duplicates(subset=['Column1', 'Column2'], keep='first') print(result_df) 

Result:

  Column1   Column2 Column3 0   'cat'     'bat'   'xyz' 1   'toy'  'flower'   'abc' 
like image 163
student Avatar answered Sep 23 '22 02:09

student


import pandas as pd  df = pd.DataFrame({"Column1":["cat", "dog", "cat"],                     "Column2":[1,1,1],                     "Column3":["C","A","B"]})  df = df.drop_duplicates(subset=['Column1'], keep='first') print(df) 
like image 45
zafrin Avatar answered Sep 23 '22 02:09

zafrin