MongoDB getmore on a collection is very slow

Question

I am trying to debug a high CPU issue on MongoDB instance. We have two shard r3.large AWS instances. There are not many page faults compared to ops count.

System profile shows ton of getmore entries like below. Please help me in finding out what causing the getmore to be very slow.

    {
        "op" : "getmore",
        "ns" : "mydb.mycollection",
        "cursorid" : 74493486271,
        "ntoreturn" : 0,
        "keyUpdates" : 0,
        "numYield" : 7,
        "lockStats" : {
            "timeLockedMicros" : {
                "r" : NumberLong(16140),
                "w" : NumberLong(0)
            },
            "timeAcquiringMicros" : {
                "r" : NumberLong(6458801),
                "w" : NumberLong(294321)
            }
        },
        "nreturned" : 120,
        "responseLength" : 13100,
        "millis" : 6304,
        "execStats" : {

        },
        "ts" : ISODate("2015-06-16T14:20:39.886Z"),
        "client" : "1.5.1.3",
        "allUsers" : [ ],
        "user" : ""
    }

Answer 1

Answering my own question it may help some one else.

• Enabled more logging with when CPU is high db.adminCommand( { setParameter: 1, logLevel: 1 } ) , after did reset to logLevel: 0 (default).
• Then log showed up a aggregate query with 0ms, but right after that, getmore entry with 5 to 6 secs.

• Aggregate query has cursor: { batchSize: 0 } with initial batch size as zero. So, the query returns quickly. But when application started iterating thru cursor, then the getmore is logged and that entry doesn't have any query details.

  Fixing aggregating query $match to use indexes solved problem