set difference for pandas

Question

A simple pandas question:

Is there a drop_duplicates() functionality to drop every row involved in the duplication?

An equivalent question is the following: Does pandas have a set difference for dataframes?

For example:

In [5]: df1 = pd.DataFrame({'col1':[1,2,3], 'col2':[2,3,4]})  In [6]: df2 = pd.DataFrame({'col1':[4,2,5], 'col2':[6,3,5]})  In [7]: df1 Out[7]:     col1  col2 0     1     2 1     2     3 2     3     4  In [8]: df2 Out[8]:     col1  col2 0     4     6 1     2     3 2     5     5 

so maybe something like df2.set_diff(df1) will produce this:

   col1  col2 0     4     6 2     5     5 

However, I don't want to rely on indexes because in my case, I have to deal with dataframes that have distinct indexes.

By the way, I initially thought about an extension of the current drop_duplicates() method, but now I realize that the second approach using properties of set theory would be far more useful in general. Both approaches solve my current problem, though.

Thanks!

Answer 1

Bit convoluted but if you want to totally ignore the index data. Convert the contents of the dataframes to sets of tuples containing the columns:

ds1 = set(map(tuple, df1.values)) ds2 = set(map(tuple, df2.values)) 

This step will get rid of any duplicates in the dataframes as well (index ignored)

set([(1, 2), (3, 4), (2, 3)])   # ds1 

can then use set methods to find anything. Eg to find differences:

ds1.difference(ds2) 

gives: set([(1, 2), (3, 4)])

can take that back to dataframe if needed. Note have to transform set to list 1st as set cannot be used to construct dataframe:

pd.DataFrame(list(ds1.difference(ds2))) 

Answer 2

Here's another answer that keeps the index and does not require identical indexes in two data frames. (EDIT: make sure there is no duplicates in df2 beforehand)

pd.concat([df2, df1, df1]).drop_duplicates(keep=False) 

It is fast and the result is

   col1  col2 0     4     6 2     5     5