Logo Questions Linux Laravel Mysql Ubuntu Git Menu
 

How to add column with constant in Spark-java data frame

I have imported

import org.apache.spark.sql.Column;
import org.apache.spark.sql.functions;

in my Java-Spark driver

But

DataFrame inputDFTwo = hiveContext.sql("select * from sourcing_src_tbl");
inputDFTwo.withColumn("asofdate", lit("2016-10-2"));

here "lit" is still showing error in eclipse(windows).Which library should I include to make it work.

like image 995
user2895589 Avatar asked Sep 22 '16 22:09

user2895589


1 Answers

Either import object like you do know and use it to access method:

import org.apache.spark.sql.functions;

df.withColumn("foo", functions.lit(1));

or use import static and call method directly:

import static org.apache.spark.sql.functions.lit;

df.withColumn("foo", lit(1));
like image 125
zero323 Avatar answered Oct 17 '22 20:10

zero323