MongoDB 2.6 server throwing 'BSONObj size is invalid' error on queries below the limit

Question

I have been trying to upgrade from MongoDB 2.4 to 2.6, and the only thing holding me back is this very strange behavior. When querying a mongod 2.4 with a fairly large query ~6MB, I have no problem - the query completes just fine (even though the data below is generated and fake, I have tested with valid data and the query completes). When querying against a mongod 2.6 instance with the same data, I get the error:

error: {
     "$err" : "BSONObj size: 16828982 (0x100CA36) is invalid. Size must be between 0 and 16793600(16MB)",
     "code" : 10334
}

However, my incoming queries are nowhere near 16MB and I can test different sizes of outgoing results - no change. Note that this only happens when querying against a field of type ObjectId.

References:

• Discussion of the 16MB limit
• Where the error is thrown in the server code

Why does 2.6 incorrectly judge the size of an incoming query, and what can I do about it?

One theory I have is that there is some difference in how the shell and the server see ObjectIds, and so on the server the same query is larger...

---

Things that don't matter:

• monogdb client version
• 2.6.X version - I've tested 2.6.1 - 2.6.3
• if the query returns data or not (i.e. I can try to match _id against fake ids or real ids, no difference)
• if we, in the shell, use ObjectId or not (see big_20_with_obj.json, created with the echo line: echo "ObjectId(\"12345123451234512$i\"),")

Things that do matter:

• mongod version is 2.6.X; 2.4.10 and below are not affected
• if we are querying against an ObjectId field
• if the collection exists (aka db.randomfakecollection.find({'_id': {'$in': big}}) does not throw the error)
• if we use the $in operator or not - $eq does not throw

How to replicate:

How to create the large file:

echo 'var big = [' >> big_20.json
for i in {300000..520000}; do
  echo "\"123451234512345123$i\"," >> big_20.json
done;
echo "];" >> big_20.json

Size of raw files:

$ ls -lh
-rw-rw-r--  1 ubuntu ubuntu 5.8M Jul  2 17:35 big_15.json
-rw-rw-r--  1 ubuntu ubuntu 5.9M Jul  2 17:35 big_20.json
-rw-rw-r--  1 ubuntu ubuntu 8.0M Jul  2 18:18 big_20_with_obj.json

Running the file:

> load('./big_15.json')
true
> Object.bsonsize(big)
7843932
> big.length
215001
> db.validcollection.find({'_id': {'$in': big}})

> load('./big_20.json')
true
> Object.bsonsize(big)
8028932
> big.length
220001
> db.validcollection.find({'_id': {'$in': big}})
error: {
     "$err" : "BSONObj size: 16828982 (0x100CA36) is invalid. Size must be between 0 and 16793600(16MB)",
     "code" : 10334
}

> load('./big_20_with_obj.json')
true
> Object.bsonsize(big)
4288915
> big.length
220001
> db.validcollection.find({'_id': {'$in': big}})
error: {
    "$err" : "BSONObj size: 17160614 (0x105D9A6) is invalid. Size must be between 0 and 16793600(16MB) First element: type: \"FETCH\"",
    "code" : 10334
}
> db.validcollection.find({'_id': {'$eq': big}})
>

Answer 1

You have run into https://jira.mongodb.org/browse/SERVER-14123 which describes the case where the query may be well under 16MB limit, but internal query runner plan generated exceeds the 16MB limit.

The fix should be in the next 2.6 release (likely 2.6.4 some time later this month July 2014).