How to remove duplicates in a csv file based on two columns?

Question

I have a csv file like this :

column1    column2

john       kerry
adam       stephenson
ashley     hudson
john       kerry
etc..

I want to remove duplicates from this file, to get only :

column1    column2

john       kerry
adam       stephenson
ashley     hudson

I wrote this script that removes duplicates based on lastnames, but I need to remove duplicates based on lastnames AND firstname.

import csv

reader=csv.reader(open('myfilewithduplicates.csv', 'r'), delimiter=',')
writer=csv.writer(open('myfilewithoutduplicates.csv', 'w'), delimiter=',')

lastnames = set()
for row in reader:
    if row[1] not in lastnames:
        writer.writerow(row)
        lastnames.add( row[1] )

Answer 1

You can now use the .drop_duplicates method in pandas. I would do the following:

import pandas as pd
toclean = pd.read_csv('myfilewithduplicates.csv')
deduped = toclean.drop_duplicates([col1,col2])
deduped.to_csv('myfilewithoutduplicates.csv')