Dropping duplicates in Pandas excluding one column

Question

This seems simple, but I can not find any information on it on the internet.

I have a dataframe like below:

City    State Zip           Date        Description        Earlham IA    50072-1036    2014-10-10  Postmarket Assurance: Devices Earlham IA    50072-1036    2014-10-10  Compliance: Devices Madrid  IA    50156-1748    2014-09-10  Drug Quality Assurance 

How can I eliminate duplicates that match 4 of 5 columns? The column not matching being Description.

The result would be

City    State Zip           Date        Description        Earlham IA    50072-1036    2014-10-10  Postmarket Assurance: Devices Madrid  IA    50156-1748    2014-09-10  Drug Quality Assurance 

I found online that drop_duplicates with the subset parameter could work, but I am unsure of how I can apply it to multiple columns.

Answer 1

You've actually found the solution. For multiple columns, subset will be a list.

df.drop_duplicates(subset=['City', 'State', 'Zip', 'Date'])  

Or, just by stating the column to be ignored:

df.drop_duplicates(subset=df.columns.difference(['Description']))