MongoDB insert raises duplicate key error

Question

I am getting the following error when trying to do a bulk insert into an empty mongodb collection.

pymongo.errors.DuplicateKeyError: E11000 duplicate key error index: cmdDistros.locDistro.$id dup key: { : ObjectId('51dac9d0c74cd81acd85c0fd') }

I am not specifying an _id when I create any of the documents, so mongodb should create the unique index correct? Here is the code I used:

#Populate database with uniform distribution
            entries = []
            for coor in freeIndices:
                for theta in range(360):
                    entry = {"x" : coor[0], "y" : coor[1], "heading" : theta}
                    for i in range(numData):
                            entry["data" + str(i)] = 1./numData
                    entries.append(entry)
            print "Entries created, loading into database..."

            locDistro.insert(entries)

Taking fate out of mongoDB's hands, I tried creating my own index using:

#Populate database with uniform distribution
            entries = []
            idNum = 0
            for coor in freeIndices:
                for theta in range(360):
                    print idNum
                    entry = {"_id" : idNum, "x" : coor[0], "y" : coor[1], "heading" : theta}
                    idNum += 1
                    for i in range(numData):
                            entry["data" + str(i)] = 1./numData
                    entries.append(entry)
            print "Entries created, loading into database..."

            locDistro.insert(entries, manipulate = False)

The print statement showed each idnum as the documents were created, and they were all unique and incremented just as expected. However on insert, I received the error:

pymongo.errors.DuplicateKeyError: E11000 duplicate key error index: cmdDistros.locDistro.$id dup key: { : 0 }

and only one document was inserted into my database.

I am completely stumped, anyone have an answer as to why this might be happening?

Answer 1

You need to understand that your entries list has a bunch of references to one entry dict. So when PyMongo sets entries[0]['_id'], all the other entries get the same _id. (In fact, PyMongo will iterate through the list setting each entry's _id, so all the entries will have the final _id at the end.) A quick fix would be:

entries.append(entry.copy())

This is merely a shallow copy, but in the code you shared I believe this is enough to fix your problem.

Answer 2

Delete the key "_id":

for i in xrange(2): 
    doc['i'] = i 
    if '_id' in doc: 
        del doc['_id'] 
    collection.insert(doc)

Or manually create a new one:

from bson.objectid import ObjectId

for i in xrange(2): 
    doc['i'] = i 
    doc['_id'] = ObjectId() 
    collection.insert(doc)

Getting "err" : "E11000 duplicate key error when inserting into mongo using the Java driver

Answer 3

I had the same error using insert_one() and also insert_many()

My solution is, to use update_one() with upsert=True

  doc = {a: 1, b:2, x:{xx:"hello",yy:"world"}}
  db.collection.update_one(doc,{'$set':doc},upsert=True)

This works for me :-)