Converting Scala DataFrame column to Seq[Int]

Question

I have a Scala Dataframe df that looks like this:

+-----+--------------------+
|id   |      measured_value|
+-----+--------------------+
|    0|             1999298|
|    1|              854791|
|    2|             1032910|
|    3|              310905|
|    4|              515442|
|    5|             4176270|
|    6|              807807|
+-----+--------------------+

I want to get the column named measured_value into a sequence of integers (Seq[Int]) and I tried using df.select("measured_value").rdd.map(r=>r(0)).collect(). But this gives me Array[Any]. How can I convert this into Seq[Int]?

Answer 1

Try this:

df.select("measured_value").map(_.getInt(0)).collect.toSeq 

Some useful examples related to the topic can be found here.
Please also remember that collect leads to collecting all the data on the Spark driver, so in case of a big dataset this may be expensive from the resources perspective.