How can I use a cursor.forEach() in MongoDB using Node.js?

Question

I have a huge collection of documents in my DB and I'm wondering how can I run through all the documents and update them, each document with a different value.

Answer 1

The answer depends on the driver you're using. All MongoDB drivers I know have cursor.forEach() implemented one way or another.

Here are some examples:

node-mongodb-native

collection.find(query).forEach(function(doc) {   // handle }, function(err) {   // done or error }); 

mongojs

db.collection.find(query).forEach(function(err, doc) {   // handle }); 

monk

collection.find(query, { stream: true })   .each(function(doc){     // handle doc   })   .error(function(err){     // handle error   })   .success(function(){     // final callback   }); 

mongoose

collection.find(query).stream()   .on('data', function(doc){     // handle doc   })   .on('error', function(err){     // handle error   })   .on('end', function(){     // final callback   }); 

Updating documents inside of .forEach callback

The only problem with updating documents inside of .forEach callback is that you have no idea when all documents are updated.

To solve this problem you should use some asynchronous control flow solution. Here are some options:

• async
• promises (when.js, bluebird)

Here is an example of using async, using its queue feature:

var q = async.queue(function (doc, callback) {   // code for your update   collection.update({     _id: doc._id   }, {     $set: {hi: 'there'}   }, {     w: 1   }, callback); }, Infinity);  var cursor = collection.find(query); cursor.each(function(err, doc) {   if (err) throw err;   if (doc) q.push(doc); // dispatching doc to async.queue });  q.drain = function() {   if (cursor.isClosed()) {     console.log('all items have been processed');     db.close();   } } 

Answer 2

Using the mongodb driver, and modern NodeJS with async/await, a good solution is to use next():

const collection = db.collection('things') const cursor = collection.find({   bla: 42 // find all things where bla is 42 }); let document; while ((document = await cursor.next())) {   await collection.findOneAndUpdate({     _id: document._id   }, {     $set: {       blu: 43     }   }); } 

This results in only one document at a time being required in memory, as opposed to e.g. the accepted answer, where many documents get sucked into memory, before processing of the documents starts. In cases of "huge collections" (as per the question) this may be important.

If documents are large, this can be improved further by using a projection, so that only those fields of documents that are required are fetched from the database.