How to check if a Spark data frame struct Array contains a specific value

Question

I have a data frame with following schema

My requirement is to filter the rows that matches given field like city in any of the address array elements.I can access individual fields like loyaltyMember.address[0].city, but i have to check all address array elements to see if any match exists. How can i achieve that in spark sql, i couldn't use array_contains function since the array is of complex type

root
 |-- loyaltyMember: struct (nullable = true)
 |    |-- Name: string (nullable = true)
 |    |-- address: array (nullable = true)
 |    |    |-- element: struct (containsNull = true)
 |    |    |    |-- addressType: string (nullable = true)
 |    |    |    |-- city: string (nullable = true)
 |    |    |    |-- countryCode: string (nullable = true)
 |    |    |    |-- postalCode: string (nullable = true)
 |    |    |    |-- street: string (nullable = true)

Answer 1

I believe you can still use array_contains as follows (in PySpark):

from pyspark.sql.functions import col, array_contains

df.filter(array_contains(col('loyaltyMember.address.city'), 'Prague'))

This will filter all rows that have in the array column city element 'Prague'.