Logo Questions Linux Laravel Mysql Ubuntu Git Menu
 

how to insert data to existing collection in mongodb with mongodb-spark connector

I'm using the mongo-spark connector to connect between spark and MongoDB. and I could not insert data to MongoDB because of spark default save mode is to error if the "table" (collection) exists and the I try this.

MongoSpark.write(centenarians).option("collection", "hundredClub")*.option("mode","append")*.save();

but this is not an effect. how can I solve this problem please help me thanks!

This is error message:

Exception in thread "main" java.lang.UnsupportedOperationException: MongoCollection already exists
like image 970
min heo Avatar asked Sep 18 '25 00:09

min heo


2 Answers

I solved this problem by myself... I share this solution (I'm using spark language by Scala):

centenarians
  .write
  .format("com.mongodb.spark.sql.DefaultSource")
  .option("collection", "hundredClub")
  .mode("append")
  .save()
like image 146
min heo Avatar answered Sep 23 '25 06:09

min heo


I'm usign mongo-spark-connector_2.11. If you have your spark data in form of Dataset or dataframe , you can save it using different modes of save method :

For example , (Java Code)

MongoSpark.write(yourDataSet).mode("append").option("replaceDocument","false").save();

It will append new data in your existing document matching it with _id on mongo collection.

like image 34
j ' Avatar answered Sep 23 '25 05:09

j '