Get index of item in array that is a column in a Spark dataframe

Question

I am able to filter a Spark dataframe (in PySpark) based on if a particular value exists within an array field by doing the following:

from pyspark.sql.functions import array_contains
spark_df.filter(array_contains(spark_df.array_column_name, "value that I want")).show() 

Is there a way to get the index of where in the array the item was found? It seems like that should exist, but I am not finding it. Thank you.

Answer 1

In spark 2.4+, there's the array_position function:

df = spark.createDataFrame([(["c", "b", "a"],), ([],)], ['data'])
df.show()
#+---------+
#|     data|
#+---------+
#|[c, b, a]|
#|       []|
#+---------+

from pyspark.sql.functions import array_position
df.select(df.data, array_position(df.data, "a").alias('a_pos')).show()
#+---------+-----+
#|     data|a_pos|
#+---------+-----+
#|[c, b, a]|    3|
#|       []|    0|
#+---------+-----+

Notes from the docs:

1. Locates the position of only the first occurrence of the given value in the given array;

2. The position is not zero based, but 1 based index. Returns 0 if the given value could not be found in the array.