spark collect as Array[T] and not as Array[Row] from data frame

Question

I can collect a column like this using the RDD API.

df.map(r => r.getAs[String]("column")).collect

However, as I am initially using a Dataset I rather would like to not switch the API level. A simple df.select("column).collect returns an Array[Row] where the .flatten operator no longer works. How can I collect to Array[T e.g. String] directly?

Answer 1

With Datasets ( Spark version >= 2.0.0 ), you just need to convert the dataframe to dataset and then collect it.

df.select("column").as[String].collect()

would return you an Array[String]