Logo Questions Linux Laravel Mysql Ubuntu Git Menu
 

How to add a Spark Dataframe to the bottom of another dataframe?

I can use withcolumnto add new columns to a Dataframe. But in scala how can I add new rows to a DataFrame?

I'm trying to add a dataframe to the bottom of another one. So either how to add rows in scala or how to add a DataFrame to the bottom of another one will help. Thanks

like image 793
Gavin Niu Avatar asked Nov 05 '15 17:11

Gavin Niu


People also ask

How do I append data in Pyspark DataFrame?

To append row to dataframe one can use collect method also. collect() function converts dataframe to list and you can directly append data to list and again convert list to dataframe.


1 Answers

If they have the same schema, simply use union for spark 2+:

val dfUnion = df1.union(df2) 

Or unionAll for spark 1+:

val dfUnion = df1.unionAll(df2) 
like image 175
Jean Logeart Avatar answered Nov 11 '22 21:11

Jean Logeart