How to handle pymongo AutoReconnect exception with insert_many method

Question

I have a MongoDB replica set with 3 members and a Python application which stores data in it.

I can handle pymongo's AutoReconnect exception when using single document inserts with a wrapper as follows:

def safe_mongo_call(method, num_retries, *args, **kwargs):
    while True:
        try:
            return method(*args, **kwargs)
        except (pymongo.errors.AutoReconnect,
                pymongo.errors.ServerSelectionTimeoutError) as e:
            if num_retries > 0:
                logger.debug('Retrying MongoDB operation: %s', str(e))
                num_retries -= 1
            else:
                raise

I'm not sure though how to handle these exceptions when using bulk writes, e.g. insert_many method. According to the documentation, bulk writes are not atomic, so even if one of the exceptions occurs, there might be already some documents written to the database successfully. Thus, I can't simply reuse the wrapper method as above.

What's the best way how to handle these situations?

Answer 1

For this case there’s BulkWriteError that must provide details on what’s been done https://api.mongodb.com/python/current/examples/bulk.html#ordered-bulk-write-operations

But in case of connection loss AutoReconnect is sent instead and the info on the operation progress appears to be lost (tested for pymongo==3.5.1)

In any case you'd need to reconstruct what's been written and what not and retry the operation for the remaining items. In the latter case it'd be a bit harder as you don't have prior info on what's been actually written but still doable

As a sketch solution: each document to be inserted is assigned an ObjectId unless _id is already present. You might handle this yourself - iterate over the documents, manually assign _id for those missing it and save the IDs in a temporary variable. Once you hit an exception find the last _id successfully inserted leveraging i.e a binary search-like approach to have at worst ~O(logN) queries, and maybe also use the fact bulk operations are split into smaller batches (https://api.mongodb.com/python/current/examples/bulk.html#bulk-insert). But of course applicability of this approach is up to the load profile you have on your mongod instances and whether additional queries bursts are admittable. If BulkWriteError were thrown as expected, you could just grab documents that are not inserted and retry the operation for these docs only.

Back to the AutoReconnect problem, I'd personally open a ticket in mongo-python-driver issues tracker, chances are high its either a bug or being done like that on purpose