Logo Questions Linux Laravel Mysql Ubuntu Git Menu
 

Spark SQL - JAVA syntax of CASE-THEN?

I noticed I can use CASE-THEN with Spark if I use an SQLContext and the .sql() function. Is there a way to use this in a JAVA syntax, directly on dataframes, too? How? Now, I write:

SparkConf sparkConf = new SparkConf();
JavaSparkContext ctx = new JavaSparkContext(sparkConf);
SQLContext sqlContext = new SQLContext(ctx);
DataFrame df = //some imported data 
df.registerTempTable("df");
sqlContext.sql("SELECT *use case-then in here* FROM df");

I'm looking for something like

df.select(case("this").then("that"));
like image 304
lte__ Avatar asked Jul 18 '16 10:07

lte__


1 Answers

Just import org.apache.spark.sql.functions then use when(Column col, Object obj).

import org.apache.spark.sql.functions;

df.select(functions.when(df.col("colName").equalTo("this"), "that").otherwise("something"));
like image 196
Yuan JI Avatar answered Oct 02 '22 00:10

Yuan JI