Dataframe filtering with condition applied to list of columns

Question

I want to filter a pyspark dataframe if any of the string columns in a list are empty.

df = df.where(all([col(x)!='' for x in col_list]))

ValueError: Cannot convert column into bool: please use '&' for 'and', '|' for 'or', '~' for 'not' when building DataFrame boolean expressions.

Answer 1

You can use reduce from functools to simulate all like this

from functools import reduce

spark_df.where(reduce(lambda x, y: x & y,  (F.col(x) != '' for x in col_list))).show()