Explode array data into rows in spark [duplicate]

Question

I have a dataset in the following way:

FieldA    FieldB    ArrayField 1         A         {1,2,3} 2         B         {3,5} 

I would like to explode the data on ArrayField so the output will look in the following way:

FieldA    FieldB    ExplodedField 1         A         1 1         A         2 1         A         3 2         B         3 2         B         5 

I mean I want to generate an output line for each item in the array the in ArrayField while keeping the values of the other fields.

How would you implement it in Spark. Notice that the input dataset is very large.

Answer 1

The explode function should get that done.

pyspark version:

>>> df = spark.createDataFrame([(1, "A", [1,2,3]), (2, "B", [3,5])],["col1", "col2", "col3"]) >>> from pyspark.sql.functions import explode >>> df.withColumn("col3", explode(df.col3)).show() +----+----+----+ |col1|col2|col3| +----+----+----+ |   1|   A|   1| |   1|   A|   2| |   1|   A|   3| |   2|   B|   3| |   2|   B|   5| +----+----+----+ 

Scala version

scala> val df = Seq((1, "A", Seq(1,2,3)), (2, "B", Seq(3,5))).toDF("col1", "col2", "col3") df: org.apache.spark.sql.DataFrame = [col1: int, col2: string ... 1 more field]  scala> df.withColumn("col3", explode($"col3")).show() +----+----+----+ |col1|col2|col3| +----+----+----+ |   1|   A|   1| |   1|   A|   2| |   1|   A|   3| |   2|   B|   3| |   2|   B|   5| +----+----+----+