How to continue insertion after duplicate key error using PyMongo

Question

If I need to insert a document in MongoDB if it does not exist yet

db_stock.update_one(document, {'$set': document}, upsert=True)

.will do the job (feel free to correct me if I am wrong)

But if I have a list of documents and want to insert them all what would be a best way of doing it?

There is a single-record version of this question but I need an en mass version of it, so it's different.

Let me reword my question. I have millions of documents, few of which can be already stored. How do I store remaining ones in MongoDB in a matter of seconds, not minutes/hours?

Answer 1

You need to use insert_many method and set the ordered option to False.

db_stock.insert_many(<list of documents>)

As mentioned in the ordered option documentation:

ordered (optional): If True (the default) documents will be inserted on the server serially, in the order provided. If an error occurs all remaining inserts are aborted. If False, documents will be inserted on the server in arbitrary order, possibly in parallel, and all document inserts will be attempted.

Which means that insertion will continue even if there is duplicate key error.

Demo:

>>> c.insert_many([{'_id': 2}, {'_id': 3}])
<pymongo.results.InsertManyResult object at 0x7f5ca669ef30>
>>> list(c.find())
[{'_id': 2}, {'_id': 3}]
>>> try:
...     c.insert_many([{'_id': 2}, {'_id': 3}, {'_id': 4}, {'_id': 5}], ordered=False)
... except pymongo.errors.BulkWriteError:
...     list(c.find())
... 
[{'_id': 2}, {'_id': 3}, {'_id': 4}, {'_id': 5}]

As you can see document with _id 4, 5 were inserted into the collection.

---

It worth noting that this is also possible in the shell using the insertMany method. All you need is set the undocumented option ordered to false.

db.collection.insertMany(
    [ 
        { '_id': 2 }, 
        { '_id': 3 },
        { '_id': 4 }, 
        { '_id': 5 }
    ],
    { 'ordered': false }
)