ReplaceOne throws duplicate key exception

Question

My app receives data from a remote server and calls ReplaceOne to either insert new or replace existing document with a given key with Upsert = true. (the key is made anonymous with *) The code only runs in a single thread.

However, occasionally, the app crashes with the following error:

Unhandled Exception: MongoDB.Driver.MongoWriteException: A write operation resulted in an error.                             
  E11000 duplicate key error collection: ****.orders index: _id_ dup key: { : "****-********-********-************" } ---> MongoDB.Driver.MongoBulkWriteException`1[MongoDB.Bson.BsonDocument]: A bulk write operation resulted in one or more errors.                                                                                                              
  E11000 duplicate key error collection: ****.orders index: _id_ dup key: { : "****-********-********-************" }                                                                                                                  
   at MongoDB.Driver.MongoCollectionImpl`1.BulkWrite(IEnumerable`1 requests, BulkWriteOptions options, CancellationToken cancellationToken)                                                                                                               
   at MongoDB.Driver.MongoCollectionBase`1.ReplaceOne(FilterDefinition`1 filter, TDocument replacement, UpdateOptions options, CancellationToken cancellationToken)                                                                                       
   --- End of inner exception stack trace ---                                                                                
   at MongoDB.Driver.MongoCollectionBase`1.ReplaceOne(FilterDefinition`1 filter, TDocument replacement, UpdateOptions options, CancellationToken cancellationToken)                                                                                       
   at Dashboard.Backend.AccountMonitor.ProcessOrder(OrderField& order)                                                       
   at Dashboard.Backend.AccountMonitor.OnRtnOrder(Object sender, OrderField& order)                                          
   at XAPI.Callback.XApi._OnRtnOrder(IntPtr ptr1, Int32 size1)                                                               
   at XAPI.Callback.XApi.OnRespone(Byte type, IntPtr pApi1, IntPtr pApi2, Double double1, Double double2, IntPtr ptr1, Int32 size1, IntPtr ptr2, Int32 size2, IntPtr ptr3, Int32 size3)                                                                   
Aborted (core dumped) 

My question is, why is it possible to have dup key when I use ReplaceOne with Upsert = true options?

The app is working in the following environment and runtime:

.NET Command Line Tools (1.0.0-preview2-003121)

Product Information:
 Version:            1.0.0-preview2-003121
 Commit SHA-1 hash:  1e9d529bc5

Runtime Environment:
 OS Name:     ubuntu
 OS Version:  16.04
 OS Platform: Linux
 RID:         ubuntu.16.04-x64

And MongoDB.Driver 2.3.0-rc1.

Answer 1

Upsert works based on the filter query. If the filter query doesn't match, it will try to insert the document.

If the filter query finds the document, it will replace the document.

In your case, it could have gone in either way i.e. insert/update. Please check the data to analyze the scenario.

Insert scenario:-

The actual _id is created automatically by upsert if _id is not present in filter criteria. So, _id shouldn't create uniqueness issue. If some other fields are part of unique index, it would create uniqueness issue.

Replace scenario:-

The field that you are trying to update should have unique index defined on it. Please check the indexes on the collection and its attributes.

Optional. When true, replaceOne() either: Inserts the document from the replacement parameter if no document matches the filter. Replaces the document that matches the filter with the replacement document.

To avoid multiple upserts, ensure that the query fields are uniquely indexed.

Defaults to false.

MongoDB will add the _id field to the replacement document if it is not specified in either the filter or replacement documents. If _id is present in both, the values must be equal.

Answer 2

I could not get IsUpsert = true to work correctly due to a unique index on the same field used for the filter, leading to this error: E11000 duplicate key error collection A retry, as suggested in this Jira ticket, is not a great workaround.

What did seem to work was a Try/Catch block with InsertOne and then ReplaceOne without any options.

try
{
    // insert into MongoDB
    BsonDocument document = BsonDocument.Parse(obj.ToString());
    collection.InsertOne(document);
}
catch
{
    BsonDocument document = BsonDocument.Parse(obj.ToString());
    var filter = Builders<BsonDocument>.Filter.Eq("data.order_no", obj.data.order_no);
    collection.ReplaceOne(filter, document);
}