Mongodb queries from multiple processes; how to implement atomicity?

Question

I have a mongodb database where multiple node processes read and write documents. I would like to know how can I make that so only one process can work on a document at a time. (Some sort of locking) that is freed after the process finished updating that entry.

My application should do the following:

• Walk through each entry one by one with a cursor.

• (Lock the entry so no other processes can work with it)

• Fetch information from a thirdparty site.

• Calculate new information and update the entry.
• (Unlock the document)

Also after unlocking the document there will be no need for other processes to update it for a few hours.

Later on I would like to set up multiple mongodb clusters so that I can reduce the load on the databases. So the solution should apply to both single and multiple database servers. Or at least using multiple mongo servers.

Answer 1

An elegant solution that doesn't involve locks is:

• Add a version property to your document.

• When updating the document, increment the version property.

• When updating the document, include the last read version in the find query. If your document has been updated elsewhere, the find query will yield no results and your update will fail.

• If your update fails, you can retry the operation.

I have used this pattern with great success in the past.

Example

Imagine you have a document {_id: 123, version: 1}.

Imagine now you have 3 Mongo clients doing db.collection.findAndModify({ query: {_id: 123, version: 1}, update: { $inc: 1 }});, concurrently.

The first update will apply, the remaining will fail. Why? because now version is 2, and the query included version: 1.

Answer 2

Per MongoDB documentation: isolated: Prevents a write operation that affects multiple documents from yielding to other reads or writes once the first document is written... $isolated operator causes write operations to acquire an exclusive lock on the collection...will make WiredTiger single-threaded for the duration of the operation. So if you are updating multiple documents, you could first get the data from the third-party API, parse the info into an array for example, and then use something like this in Mongo shell:

db.foo.update(
    { status : "A" , $isolated : 1 },
    { $set: { < your key >: < your info >}}, //use the info in your array
    { multi: true }
)

Or if you have to update the document one by one, you could use findAndModify() or updateOne() of the Node Driver for MongoDB. Please note that per MongoDB documentation 'When modifying a single document, both findAndModify() and the update() method atomically update the document.'

An example of updating one by one: first you connect to the Mongod with the NodeJS driver, then connect to the third part API using NodeJS's Request module, for example, get and parse the data, before using the data to modify your documents, something like below:

var request = require('request');

var MongoClient = require('mongodb').MongoClient,
    test = require('assert');

MongoClient.connect('mongodb://localhost:27017/test', function(err, db) {
    var collection = db.collection('simple_query');
    collection.find().forEach(
        function(doc) {
            request('http://www.google.com', function(error, response, body) {
                console.log('body:', body); // parse body for your info
                collection.findAndModify({
                    <query based on your doc>
                }, {
                    $set: { < your key >: < your info >
                    }
                })
            });
        }, function(err) {
        });
    });