Trying to do a bulk upsert with Mongoose. What's the cleanest way to do this?

Question

I have a collection that holds documents that contains three fields: first_name, last_name, and age. I'm trying to figure out what query in Mongoose I can use to do a bulk upsert. My app is occasionally receiving a new array of objects with those same three fields. I want the query to check if the first AND last name already exist within a document, and if they do - update the age if it's different. Otherwise, if the first and last name don't exist, insert a new document.

Currently, I'm only doing the import - and haven't yet built out the logic for this upsert piece.

app.post('/users/import', function(req, res) {   let data = req.body;   let dataArray = [];   data.forEach(datum => {     dataArray.push({         first: datum.first,         last: datum.last,         age: datum.age     }) })  User.insertMany(dataArray, answer => {     console.log(`Data Inserted:`,answer) }) 

`

And my User model looks like this:

const mongoose = require('mongoose');  const Schema = mongoose.Schema;  const userSchema = new Schema({   first: String,   last: String,   age: Number,   created_at: { type: Date, default: Date.now } });  var User = mongoose.model('User', userSchema); module.exports = User; 

Answer 1

([email protected], [email protected])

TL;DR

await GasStation.collection.bulkWrite([ // <<==== use the model name   {     'updateOne': {       'filter': { 'id': '<some id>' },       'update': { '$set': { /* properties to update */ } },       'upsert': true,  // <<==== upsert in every document     }   },   /* other operations here... */ ]); 

Long story:

After struggling with Mongoose API poor documentation, I solved the bulk upsert tweaking updateOne:{} operation in the bulkWrite() method.

A couple of undocumented things to consider:

// suppose: var GasStation = mongoose.model('gasstation', gasStationsSchema); var bulkOps = [ ];  // for ( ... each gasStation to upsert ...) {   let gasStation = { country:'a', localId:'b', xyz:'c' };   // [populate gasStation as needed]   // Each document should look like this: (note the 'upsert': true)   let upsertDoc = {     'updateOne': {       'filter': { 'country': gasStation.country, 'localId': gasStation.localId },       'update': gasStation,       'upsert': true   }};   bulkOps.push(upsertDoc); // end for loop  // now bulkWrite (note the use of 'Model.collection') GasStation.collection.bulkWrite(bulkOps)   .then( bulkWriteOpResult => {     console.log('BULK update OK');     console.log(JSON.stringify(bulkWriteOpResult, null, 2));   })   .catch( err => {     console.log('BULK update error');     console.log(JSON.stringify(err, null, 2));   }); 

The two key things here are incomplete API documentation issues (at the time of writing, at least):

• 'upsert': true in each document. This is not documented in Mongoose API (), which often refers to node-mongodb-native driver. Looking at updateOne in this driver, you could think to add 'options':{'upsert': true}, but, no... that won't do. I also tried to add both cases to the bulkWrite(,[options],) argument, with no effect either.
• GasStation.collection.bulkWrite(). Although Mongoose bulkWrite() method claims it should be called Model.bulkWrite() (in this case, GasStation.bulkWrite()), that will trigger MongoError: Unknown modifier: $__. So, Model.collection.bulkWrite() must be used.

Additionally, note:

• You don't need to use the $set mongo operator in the updateOne.update field, since mongoose handles it in case of upsert (see bulkWrite() comments in example).
• Note that my unique index in the schema (needed for upsert to work properly) is defined as:

gasStationsSchema.index({ country: 1, localId: 1 }, { unique: true });

Hope it helps.

==> EDIT: (Mongoose 5?)

As noticed by @JustinSmith, the $set operator added by Mongoose doesn't seem to be working anymore. Maybe it's because of Mongoose 5?

In any case, using $set explicitly should do:

'update': { '$set': gasStation },