df.unique() on whole DataFrame based on a column

Question

I have a DataFrame df filled with rows and columns where there are duplicate Id's:

Index   Id   Type 0       a1   A 1       a2   A 2       b1   B 3       b3   B 4       a1   A ... 

When I use:

uniqueId = df["Id"].unique()  

I get a list of unique IDs.

How can I however apply this filtering on the whole DataFrame such that it keeps the structure but that the duplicates (based on "Id") are removed?

Answer 1

It seems you need DataFrame.drop_duplicates with parameter subset which specify where are test duplicates:

#keep first duplicate value df = df.drop_duplicates(subset=['Id']) print (df)        Id Type Index          0      a1    A 1      a2    A 2      b1    B 3      b3    B 

---

#keep last duplicate value df = df.drop_duplicates(subset=['Id'], keep='last') print (df)        Id Type Index          1      a2    A 2      b1    B 3      b3    B 4      a1    A 

---

#remove all duplicate values df = df.drop_duplicates(subset=['Id'], keep=False) print (df)        Id Type Index          1      a2    A 2      b1    B 3      b3    B