Overflow sort stage buffered data usage

Question

We have a mongoDB 2.6.4 replica set running and are trying to diagnose this behavior. We are getting the Runner error: Overflow sort stage buffered data usage of 33598393 bytes exceeds internal limit of 33554432 bytes when we expect that we would not. The collection has millions of records and has a compound index that includes the key that is being sorted. As an example

index looks like this

{ from: 1, time : -1, otherA : 1, otherB : 1}

our find is

 find.collection({ from : { $in : ["a", "b"] }, time : { $gte : timestamp }, 
       otherA : {$in:[...]}, otherB : {$in:[...]}})
       .sort( time : -1 )   

mongoDB parallels (clauses) this query like this:

{ from : a }, { time : { $gte : timestamp }, ... }
{ from : b }, { time : { $gte : timestamp }, ... }

In the explain each stage reports that scanAndOrder : false, which implies that the index was used to return the results. This all seems fine, however the mongoDB client gets the Runner error: Overflow sort stage buffered data usage error. This seems to imply that the sort was done in memory. Is this because it is doing an in-memory merge sort of the clauses? Or is there some other reason that this error could occur?

Answer 1

I was also facing the same problem of memory overflow. I am using PHP with MongoDB to manipulate Documents. When I am accessing a collection which is probably having large set of documents it is throwing an error. As per the following link, it can sort only upto 32MB data at a time. http://docs.mongodb.org/manual/reference/limits/#Sorted-Documents .

So as per the description given in MongoDocs about sorting, I sorted the array converted from MongoCursor object with PHP rather than Mongo's sort() method.

Hope it'll help you. Thanks.