Spark Scala: How to transform a column in a DF

Question

I have a dataframe in Spark with many columns and a udf that I defined. I want the same dataframe back, except with one column transformed. Furthermore, my udf takes in a string and returns a timestamp. Is there an easy way to do this? I tried

val test = myDF.select("my_column").rdd.map(r => getTimestamp(r))  

but this returns an RDD and just with the transformed column.

Answer 1

If you really need to use your function, I can suggest two options:

1) Using map / toDF:

import org.apache.spark.sql.Row import sqlContext.implicits._  def getTimestamp: (String => java.sql.Timestamp) = // your function here  val test = myDF.select("my_column").rdd.map {   case Row(string_val: String) => (string_val, getTimestamp(string_val)) }.toDF("my_column", "new_column") 

2) Using UDFs (UserDefinedFunction):

import org.apache.spark.sql.functions._  def getTimestamp: (String => java.sql.Timestamp) = // your function here  val newCol = udf(getTimestamp).apply(col("my_column")) // creates the new column val test = myDF.withColumn("new_column", newCol) // adds the new column to original DF 

There's more detail about Spark SQL UDFs in this nice article by Bill Chambers .

---

Alternatively,

If you just want to transform a StringType column into a TimestampType column you can use the unix_timestamp column function available since Spark SQL 1.5:

val test = myDF   .withColumn("new_column", unix_timestamp(col("my_column"), "yyyy-MM-dd HH:mm").cast("timestamp")) 

Note: For spark 1.5.x, it is necessary to multiply the result of unix_timestamp by 1000 before casting to timestamp (issue SPARK-11724). The resulting code would be:

val test = myDF   .withColumn("new_column", (unix_timestamp(col("my_column"), "yyyy-MM-dd HH:mm") *1000L).cast("timestamp")) 

Edit: Added udf option