check for duplicates in Pyspark Dataframe

Question

Is there a simple and efficient way to check a python dataframe just for duplicates (not drop them) based on column(s)?

I want to check if a dataframe has dups based on a combination of columns and if it does, fail the process.

TIA.

Answer 1

The easiest way would be to check if the number of rows in the dataframe equals the number of rows after dropping duplicates.

if df.count() > df.dropDuplicates([listOfColumns]).count():
    raise ValueError('Data has duplicates')

Answer 2

If you also want to actually inspect the duplicates, you can do

df \
    .groupby(['column1', 'column2']) \
    .count() \
    .where('count > 1') \
    .sort('count', ascending=False) \
    .show()