MongoDB - Different query execution times after restarting server

Question

Say that I start a mongo db server:

mongo --dbpath=/some/path --port=12345 --storageEngine wiredTiger

I then run a the same query 10 times (disregarding the first one, so that they are all on warm cache) as follows:

mongo query1.js

The times that I get are (as verified via time and also the mongodb logger):

8137ms 8145ms 8193ms 8091ms 8152ms 8110ms 8182ms 8142ms 8133ms 8098ms

Great -- pretty consistent. All are within ~100ms of eachother, which makes sense.

I then shutdown the server, in any of the following fashions:

pkill mongod mongod --dbpath=/some/path --shutdown mongo shutdown.js

Where shutdown.js contains:

db.getSiblingDB('admin').shutdownServer();

I then restart it, using the exact same command, and I get the following times:

8531ms 8492ms 8613ms 8555ms 8538ms 8512ms 8551ms 8511ms 8608ms 8522ms

Again, they are consistently within ~100ms, but they are all at a different baseline.

If I do this again, it might be around 8.3, 8.6, 8.9, or anywhere in between really. No other user processes are open (except those needed to ssh into the machine).

I ran an experiment as follows:

while True: run the query 25 times and record the minimum such runtime shutdown the server and restart it, wait for it to listen

This ran for two days over the weekend while I did not interact with the machine, collecting 223 data points, and the minimum runtimes ranged from 7.9s to 8.9s. If I did not shutdown the server in between, this does not happen, but again, I might get a baseline of 7.9s or I might get one of 8.9s.

The standard deviation of one data point (the runtimes of the 25 queries) was always really low (around 0.06), but between all queries, was really high.

Does anybody have intuition on why this is happening and how I can prevent it? I am trying to figure out if one query is faster than another, but I can't get a good baseline to test against. Restarting the server is not absolutely necessary, but it would make my life easier, since I do not always have the server running.

Answer 1

MongoDB uses cache for serving some queries as you go. When you restart server, some cache must be clears out. MongoDB keeps all of the most recently used data in RAM. If you have created indexes for your queries and your working data set fits in RAM, MongoDB serves all queries from memory.

Query plans are saved in cache which clears out on mongo restart. So it takes time when you run query first time. See explain("executionStats").

With WiredTiger, MongoDB utilizes both the filesystem cache and WiredTiger cache. By default, starting in MongoDB 3.2, the WiredTiger cache will use 60% of RAM minus 1 GB or it will use 1 GB, whichever is larger. For systems with up to 10 GB of RAM, this is less than or equal to the 3.0 setting. For systems with more than 10 GB of RAM, the configuration is greater than the 3.0 setting.

In MongoDB 3.0, the WiredTiger cache, by default, uses either 1 GB or half of the installed physical RAM, whichever is larger.

MongoDB also automatically uses all free memory on the machine via the filesystem cache (data in the filesystem cache is compressed).

See for MongoDB Fundamentals