Logo Questions Linux Laravel Mysql Ubuntu Git Menu
 

How can I make (Spark1.6) saveAsTextFile to append existing file?

In SparkSQL,I use DF.wirte.mode(SaveMode.Append).json(xxxx),but this method get these files likeenter image description here

the filename is too complex and random,I can't use api to get.So I want to use saveAstextfile ,beacuse filename is not complex and regular, but I don't know how to append file in same diretory?Appreciate for your time.

like image 525
yjxyjx Avatar asked Mar 13 '23 10:03

yjxyjx


1 Answers

worked on Spark 1.5 , I think this is right usage..

dataframe.write().mode(SaveMode.Append).format(FILE_FORMAT).**partitionBy**("parameter1", "parameter2").save(path);
like image 71
Ram Ghadiyaram Avatar answered Mar 15 '23 12:03

Ram Ghadiyaram