How to use array_contains with 2 columns in spark scala?

Question

I have an issue , I want to check if an array of string contains string present in another column . I am currently using below code which is giving an error.

.withColumn("is_designer_present", when(array_contains(col("list_of_designers"),$"dept_resp"),1).otherwise(0))

error:

java.lang.RuntimeException: Unsupported literal type class org.apache.spark.sql.ColumnName dept_resp
  at org.apache.spark.sql.catalyst.expressions.Literal$.apply(literals.scala:77)

Answer 1

you can write a udf function to get your job done

import org.apache.spark.sql.functions._
def stringContains = udf((array: collection.mutable.WrappedArray[String], str: String) => array.contains(str))
df.withColumn("is_designer_present", when(stringContains(col("list_of_designers"), $"dept_resp"),1).otherwise(0))

You can return appropriate value from udf function itself so that you don't have to use when function

import org.apache.spark.sql.functions._
def stringContains = udf((array: collection.mutable.WrappedArray[String], str: String) => if (array.contains(str)) 1 else 0)
df.withColumn("is_designer_present", stringContains(col("list_of_designers"), $"dept_resp"))

Answer 2

With Spark 1.6 you can wrap your array_contains() as a string into the expr() function:

import org.apache.spark.sql.functions.expr

.withColumn("is_designer_present",
    when(expr("array_contains(list_of_designers, dept_resp)"), 1).otherwise(0))

This form of array_contains inside the expr can accept a column as the second argument.