MongoDB Error While Deleting Duplicates

Question

My duplicates deleting code (in MongoDB shell) is like this:

db.<collection_name>.aggregate([
    {
        $group: {
            _id: { <duplicated_keys>: "$<duplicated_keys>" },
            dups: { $addToSet: "$_id" },
            count: { $sum: 1 }
        }
    },
    {
        $match: {
            count: { $gt: 1 }
        }
    }
], { allowDiskUse: true })
.forEach(function(doc) {
    doc.dups.shift();
    db.<collection_name>.remove({ _id: { $in: doc.dups } });
});

And I've got an error like this:

[thread1] Error: getMore command failed: {
        "ok" : 0,
        "errmsg" : "Cursor not found, cursor id: 144931661890",
        "code" : 43
}

What is the cause of this error? And how could I solve this?

UPDATE

• MongoDB version is 3.2

• Before forEach, the result is:

  { "_id" : { <duplicated_keys>: <dupkey_values> }, "dups" : [ ObjectId("56f8e4d37a88ea2aa938414d"), ObjectId("56f63ab87a88ea141ca33856") ], "count" : 2 }
  

  And if I find with ObjectId("56f63ab87a88ea141ca33856"), it is the duplicated document.

• The data amount is relatively large (30+ GB), could this be the problem?
• While running the query, there are insertions to the same collection.

Answer 1

At last figured out the solution. The cursor in MongoDB has a life time, by default it is 10 minute. Once this time exceeds, shell can not find the next cursor any more.

To avoid this, set the cursor's life time with noCursorTimeout(). For example:

db.<collection_name>.aggregate([
    {
        $group: {
            _id: { <duplicated_keys>: "$<duplicated_keys>" },
            dups: { $addToSet: "$_id" },
            count: { $sum: 1 }
        }
    },
    {
        $match: {
            count: { $gt: 1 }
        }
    },
    {
        $out: "tempCollection"
    }
], { allowDiskUse: true });

db.tempCollection.find().noCursorTimeout().forEach(...);

Or use less batch size. For example:

db.<collection_name>.aggregate([
    {
        $group: {
            _id: { <duplicated_keys>: "$<duplicated_keys>" },
            dups: { $addToSet: "$_id" },
            count: { $sum: 1 }
        }
    },
    {
        $match: {
            count: { $gt: 1 }
        }
    },
    {
        $out: "tempCollection"
    }
],
{
    allowDiskUse: true,
    cursor: { batchSize: 0 }
});

db.tempCollection.find().forEach(...);

Answer 2

Refer to this issue https://jira.mongodb.org/browse/SERVER-13358, this error Error: getMore command failed: caused by cursor failed in the aggregation. Refer to the same issue https://jira.mongodb.org/browse/SERVER-6036, it is planned to fix but not scheduled