split a Spark column of Array[String] into columns of String

Question

If I have a Dataframe containing a column of Array[String]:

scala> y.show
+---+----------+
|uid|event_comb|
+---+----------+
|  c|  [xx, zz]|
|  b|  [xx, xx]|
|  b|  [xx, yy]|
|  b|  [xx, zz]|
|  b|  [xx, yy]|
|  b|  [xx, zz]|
|  b|  [yy, zz]|
|  a|  [xx, yy]|
+---+----------+

How can I split the column "event_comb" into two columns (e.g. "event1" and "event2")?

Answer 1

If your column type is list or Map you can use getItem function to get the value

getItem(Object key)

An expression that gets an item at position ordinal out of an array, or gets a value by key key in a MapType.

val data = Seq(
    ("c", List("xx", "zz")),
  ("b", List("xx", "xx")),
  ("b", List("xx", "yy")),
  ("b", List("xx", "zz")),
  ("b", List("xx", "yy")),
  ("b", List("xx", "zz")),
  ("b", List("yy", "zz")),
  ("a", List("xx", "yy"))
  ).toDF("uid", "event_comb")

  data.withColumn("event1", $"event_comb".getItem(0))
      .withColumn("event2", $"event_comb".getItem(1))
      .show(false)

Output:

+---+----------+------+------+
|uid|event_comb|event1|event2|
+---+----------+------+------+
|c  |[xx, zz]  |xx    |zz    |
|b  |[xx, xx]  |xx    |xx    |
|b  |[xx, yy]  |xx    |yy    |
|b  |[xx, zz]  |xx    |zz    |
|b  |[xx, yy]  |xx    |yy    |
|b  |[xx, zz]  |xx    |zz    |
|b  |[yy, zz]  |yy    |zz    |
|a  |[xx, yy]  |xx    |yy    |
+---+----------+------+------+