pytest assert for pyspark dataframe comparison

Question

I have 2 pyspark dataframe as shown in file attached. expected_df and actual_df

In my unit test I am trying to check if both are equal or not.

for which my code is

expected = map(lambda row: row.asDict(), expected_df.collect()) 
actual = map(lambda row: row.asDict(), actaual_df.collect()) 
assert expected = actual 

Since both dfs are same but row order is different so assert fails here. What is best way to compare such dfs.

Answer 1

You can try pyspark-test

https://pypi.org/project/pyspark-test/

This is inspired by the panadas testing module build for pyspark.

Usage is simple

from pyspark_test import assert_pyspark_df_equal

assert_pyspark_df_equal(df_1, df_2)

Also apart from just comparing dataframe, just like the pandas testing module it also accepts many optional params that you can check in the documentation.

Note:

1. The datatypes in pandas and pysaprk are bit different, thats why directly converting to .toPandas and using panadas testing module might not be the right approach.
2. This package is for unit/integration testing, so meant to be used with small size dfs

Answer 2

This is done in some of the pyspark documentation:

assert sorted(expected_df.collect()) == sorted(actaual_df.collect())