Mongodb read locks

Question

I have a mongodb collection with custom _id and 500M+ documents. Size of the _id's index is ≈25Gb and whole collection is ≈125 Gb. Server has 96 Gb RAM. Read activity is only range queries by _id. Explain() shows that queries use the index. Mongo works rather fast some time after load tests start and slows down after a while. I can see in a log a lot of entries like this:

[conn116] getmore csdb.archive query: { _id: { $gt: 2812719756651008, $lt: 2812720361451008 } } cursorid:444942282445272280 ntoreturn:0 keyUpdates:0 numYields: 748 locks(micros) r:7885031 nreturned:40302 reslen:1047872 10329ms

A piece of db.currentOp():

"waitingForLock" : false,
                        "numYields" : 193,
                        "lockStats" : {
                                "timeLockedMicros" : {
                                        "r" : NumberLong(869051),
                                        "w" : NumberLong(0)
                                },
                                "timeAcquiringMicros" : {
                                        "r" : NumberLong(1369404),
                                        "w" : NumberLong(0)
                                }
                        }

What is locks(micros) r? What can I do to cut it down?

Answer 1

What is locks(micros) r?

The amount of time that read locks was held (in microseconds).

• R - Global read lock
• W - Global write lock
• r - Database specific read lock
• w - Database specific write lock

---

What can I do to cut it down?

• How does sharding affect concurrency?

  Sharding improves concurrency by distributing collections over multiple mongod instances, allowing shard servers (i.e. mongos processes) to perform any number of operations concurrently to the various downstream mongod instances.

• Diagnosing Performance Issues (Locks)

  MongoDB uses a locking system to ensure data set consistency. However, if certain operations are long-running, or a queue forms, performance will slow as requests and operations wait for the lock. Lock-related slowdowns can be intermittent. To see if the lock has been affecting your performance, look to the data in the globalLock section of the serverStatus output. If globalLock.currentQueue.total is consistently high, then there is a chance that a large number of requests are waiting for a lock. This indicates a possible concurrency issue that may be affecting performance.

  If globalLock.totalTime is high relative to uptime, the database has existed in a lock state for a significant amount of time. If globalLock.ratio is also high, MongoDB has likely been processing a large number of long running queries. Long queries are often the result of a number of factors: ineffective use of indexes, non-optimal schema design, poor query structure, system architecture issues, or insufficient RAM resulting in page faults and disk reads.

• How We Scale MongoDB (Vertically)

  Sadly, MongoDB itself will usually become a bottleneck before the capacity of a server is exhausted. Write lock is almost always the biggest problem (though there are practical limits to how much IO capacity a single MongoDB process can take advantage of).