How can I upsert multiple objects with MongoDB & Node.js?

Question

Let's say I have an array of Movie genres like so:

[ 
    { id: 28, name: 'Action' },
    { id: 12, name: 'Adventure' },
    { id: 16, name: 'Animation' },
    { id: 35, name: 'Comedy' },
    { id: 80, name: 'Crime' },
    { id: 99, name: 'Documentary' },
    { id: 18, name: 'Drama' },
    { id: 10751, name: 'Family' },
    { id: 14, name: 'Fantasy' },
    { id: 10769, name: 'Foreign' },
    { id: 36, name: 'History' },
    { id: 27, name: 'Horror' },
    { id: 10402, name: 'Music' },
    { id: 9648, name: 'Mystery' },
    { id: 10749, name: 'Romance' },
    { id: 878, name: 'Science Fiction' },
    { id: 10770, name: 'TV Movie' },
    { id: 53, name: 'Thriller' },
    { id: 10752, name: 'War' },
    { id: 37, name: 'Western' }
]

and I have a connection to a MongoDB (v3.2) instance: db, and I'm using the standard mongodb Node.js driver (const mongodb = require('mongodb').MongoClient).

What I want to be able to do is one bulk upsert operation onto a collection, say genres, where the _id field maps to the id field of our genre objects.

Now, I know I could loop through each item in the array, and do a simple upsert:

for (let i = 0; i < genres.length; i++) {
  await db.collection('genres').update(
    { _id: genres[i].id },
    genres[i],
    { upsert: true }
  );
}

But this feels wasteful and wrong.

Is there an easier way to do what should be a relatively simple task?

Thanks

Answer 1

Use the bulkWrite API to carry out the updates:

var bulkUpdateOps = genres.map(function(doc) {
    return {
        "updateOne": {
            "filter": { "_id": doc.id },
            "update": { "$set": { "name": doc.name } },
            "upsert": true
        }
    };
});

db.collection('genres').bulkWrite(bulkUpdateOps, function(err, r) {
    // do something with result
})

---

If you're dealing with larger arrays i.e. > 1000 then consider sending the writes to the server in batches of 500 which gives you a better performance as you are not sending every request to the server, just once in every 500 requests:

var bulkUpdateOps = [],
    counter = 0;

genres.forEach(function(doc) {
    bulkUpdateOps.push({
        "updateOne": {
            "filter": { "_id": doc.id },
            "update": { "$set": { "name": doc.name } },
            "upsert": true
        }
    });
    counter++;

    if (counter % 500 == 0) {
        db.collection('genres').bulkWrite(bulkUpdateOps, function(err, r) {
            // do something with result
        });
        bulkUpdateOps = [];
    }
})

if (counter % 500 != 0) {
    db.collection('genres').bulkWrite(bulkUpdateOps, function(err, r) {
        // do something with the result
    });
}

Answer 2

I would try:

db.collection('genres').update(genres, {upsert: true, multi: true});

Note: untested code...

UPDATE: to remap id field to _id:

var _genres = genres.map(function(genre) {
   return { _id: genre.id, name: genre.name };
});

db.collection('genres').update(_genres, {upsert: true, multi: true});