How do I avoid a cursor timeout on a long-running mapreduce operation?

Question

I'm running a duplicate-detection mapreduce operation on a large collection against a mongos instance on a sharded cluster and I expect the operation to take longer than 10 minutes:

m = function () {
    emit(this.fieldForDupCheck, 1);
}
r = function (k, vals) {
    return Array.sum(vals);
}
res = db.Collection.mapReduce(m, r, { out : "dups" });

Running this gives me the following error after about 10 minutes of processing:

uncaught exception: map reduce failed:{
"ok" : 0,
"errmsg" : "MR post processing failed: { result: "dups", errmsg: "exception: getMore: cursor didn't exist on server, possible restart or timeout?", code: 13127, ok: 0.0 }"
}

I tried adding a noTimeout option by using .addOption(DBQuery.Option.noTimeout) on the mapReduce call but this results in a JS error in the shell Object [object Object] has no method 'addOption'

How do I avoid a cursor timeout on a long-running mapreduce operation?

Answer 1

You haven't mentioned what release of MongoDB you're using, but the solution will be similar to what is presented here anyway. I'll demonstrate on top of 2.2.4, which is what comes with Ubuntu 13.04.

The problem of doing this is indeed injecting the option into the cursor. That's where the addOption lives:

> var cursor = db.test.find()
> cursor.addOption
function (option) {
    this._options |= option;
    return this;
}

Let's look at how mapReduce is defined:

> db.test.mapReduce
function (map, reduce, optionsOrOutString) {
    var c = {mapreduce:this._shortName, map:map, reduce:reduce};
    ...
    var raw = this._db.runCommand(c);
    ...
    return new MapReduceResult(this._db, raw);
}

So it builds a document to run the command via runCommand. Let's look further into it:

> db.runCommand
function (obj) {
    if (typeof obj == "string") {
        var n = {};
        n[obj] = 1;
        obj = n;
    }
    return this.getCollection("$cmd").findOne(obj);
}

So the command is run via findOne. Let's look into it:

> db.test.findOne
function (query, fields, options) {
    var cursor = this._mongo.find(this._fullName, this._massageObject(query) || {}, fields, -1, 0, 0, options || this.getQueryOptions());
    if (!cursor.hasNext()) {
        return null;
    }
    var ret = cursor.next();
    ...
    return ret;
}

Ah, there's something interesting here. The cursor is being initialized with flags coming from the parameter options, which unfortunately doesn't help your case because runCommand leaves it unset, but it ORs it with getQueryOptions(), which comes from the collection. Let's look into it:

> db.collection.getQueryOptions
function () {
    var options = 0;
    if (this.getSlaveOk()) {
        options |= 4;
    }
    return options;
}

Oops.. that's no good. So we don't have access to the cursor, nor any way to inject query options into the executed command via non-hackish means.

Well, but we've learned a good deal about how map reduce commands are in fact delivered to the server through that process. It's just a document that is queried against a specific collection in the database. That means we can build the same query, and run it ourselves, but providing any necessary flags.

I won't go over the trouble of building the whole MongoDB command and setting up the result for you, but I'll just show you it actually works by doing it with the isMaster command.

This is the command running without any flags:

> db.getCollection("$cmd").findOne({isMaster: 1}).ismaster
true

To see the difference in effect, we'll tcpdump the communication with the database. We can see in the wire protocol documentation that the relevant flags live right before the collection name, in a 32 bits integer, so it's easy to spot the relevant piece of the dump:

    .                  vvvvvvvvv
    0x0040:  d407 0000 0000 0000 7465 7374 2e24 636d  ........test.$cm
    0x0050:  6400 0000 0000 ffff ffff 1700 0000 0169  d..............i

Good. We can see four bytes zeroed, right before the collection name.

Now, let's do the same while providing some flags. We've learned from the above debugging section that the query flags can be provided as the third option of findOne, so let's do that:

> db.getCollection("$cmd").findOne({isMaster: 1}, undefined, 0xBEEF).ismaster
true

and see the dump:

    .                  vvvvvvvvv
    0x0040:  d407 0000 efbe 0000 7465 7374 2e24 636d  ........test.$cm
    0x0050:  6400 0000 0000 ffff ffff 1700 0000 0169  d..............i

Hey, our flags got delivered where they should, and we can also see it got inverted, meaning the bytes are encoded as little-endian, matching the docs.

So, that means you can provide the flag DBQuery.Option.noTimeout as the third option of findOne, and hand-code the map-reduce command as described in the documentation in a similar way to what we did with isMaster, and you'll get what you want.