How to transpose/pivot the rows data to column in Spark Scala? [duplicate]

Question

I am new to Spark-SQL. I have information in Spark Dataframe like this

Company Type Status
A       X    done
A       Y    done
A       Z    done
C       X    done
C       Y    done
B       Y    done

I am want to be displayed like the following

Company X-type Y-type Z-type
A       done    done    done
B       pending done    pending
C       done    done    pending

I am not able to acheive this is Spark-SQL

Please Help

Answer 1

You can groupby Company and then use pivot function on column Type

Here is the simple example

import org.apache.spark.sql.functions._

val df = spark.sparkContext.parallelize(Seq(
        ("A", "X", "done"),
        ("A", "Y", "done"),
        ("A", "Z", "done"),
        ("C", "X", "done"),
        ("C", "Y", "done"),
        ("B", "Y", "done")
      )).toDF("Company", "Type", "Status")

val result = df.groupBy("Company")
    .pivot("Type")
    .agg(expr("coalesce(first(Status), \"pending\")"))

result.show()

Output:

+-------+-------+----+-------+
|Company|      X|   Y|      Z|
+-------+-------+----+-------+
|      B|pending|done|pending|
|      C|   done|done|pending|
|      A|   done|done|   done|
+-------+-------+----+-------+

You can rename the column later.

Hope this helps!