How is it possible to add new column to existing Dataframe in Spark SQL

Question

I use DataFrame API.

I have existing DataFrame and a List object (can also use Array). How is it possible to add this List to existing DataFrame as a new column? Should I use the class Column for this?

Answer 1

You should probably convert your List to a single Column RDD and apply join on critetia pickeg by you. Simple DataFrame conversion:

 val df1 = sparkContext.makeRDD(yourList).toDF("newColumn")

If you need to create additional column to perform join on you can add more columns, mapping your list:

val df1 = sparkContext.makeRDD(yourList).map(i => (i, fun(i)).toDF("newColumn", "joinOnThisColumn")

I am not familiar with Java version, but you should try using JavaSparkContext.parallelize(yourList) and apply similar mapping operations based on this doc.

Answer 2

Sorry, It was my fault, I already found the function withColumn(String colName, Column col) which should solve my problem