Scala: How can I replace value in Dataframes using scala

Question

For example I want to replace all numbers equal to 0.2 in a column to 0. How can I do that in Scala? Thanks

Edit:

|year| make|model| comment            |blank| |2012|Tesla| S   | No comment         |     |  |1997| Ford| E350|Go get one now th...|     |  |2015|Chevy| Volt| null               | null|  

This is my Dataframe I'm trying to change Tesla in make column to S

Answer 1

Spark 1.6.2, Java code (sorry), this will change every instance of Tesla to S for the entire dataframe without passing through an RDD:

dataframe.withColumn("make", when(col("make").equalTo("Tesla"), "S")                              .otherwise(col("make")                      ); 

Edited to add @marshall245 "otherwise" to ensure non-Tesla columns aren't converted to NULL.

Answer 2

Building off of the solution from @Azeroth2b. If you want to replace only a couple of items and leave the rest unchanged. Do the following. Without using the otherwise(...) method, the remainder of the column becomes null.

import org.apache.spark.sql.functions._  val newsdf =   sdf.withColumn(     "make",     when(col("make") === "Tesla", "S").otherwise(col("make"))   ); 

Old DataFrame

+-----+-----+  | make|model|  +-----+-----+  |Tesla|    S|  | Ford| E350|  |Chevy| Volt|  +-----+-----+  

New Datarame

+-----+-----+ | make|model| +-----+-----+ |    S|    S| | Ford| E350| |Chevy| Volt| +-----+-----+