How can I count the number of accesses/queries to database through Mongoid?

Question

I'm using the Mongoid in a Rails project. To improve the performance of large queries, I'm using the includes method to eager load the relationships.

I would like to know if there is an easy way to count the real number of queries performed by a block of code so that I can check if my includes really reduced the number of DB accesses as expected. Something like:

# It will perform a large query to gather data from companies and their relationships
count = Mongoid.count_queries do
  Company.to_csv
end

puts count # Number of DB access

I want to use this feature to add Rspec tests to prove that my query remains efficient after changes (e.g; when adding data from a new relationship). In python's Django framework, for instance, one may use the assertNumQueries method to this end.

Answer 1

Checking on rubygems.org didn't yield anything that seems to do what you want. You might be better off looking into app performance tools like New Relic, Scout, or DataDog. You may be able to get some out of the gate benchmarking specs with

https://github.com/piotrmurach/rspec-benchmark

Answer 2

I just implemented this feature to count mongo queries in my rspec suite in a small module using mongo Command Monitoring.

It can be used like this:

expect { code }.to change { finds("users") }.by(3)
expect { code }.to change { updates("contents") }.by(1)
expect { code }.not_to change { inserts }

Or:

MongoSpy.flush
# ..code..
expect(MongoSpy.queries).to match(
  "find" => { "users" => 1, "contents" => 1 },
  "update" => { "users" => 1 }
)

Here is the Gist (ready to copy) for the last up-to-date version: https://gist.github.com/jarthod/ab712e8a31798799841c5677cea3d1a0

And here is the current version:

module MongoSpy
  module Helpers
    %w(find delete insert update).each do |op|
      define_method(op.pluralize) { |ns = nil|
        ns ? MongoSpy.queries[op][ns] : MongoSpy.queries[op].values.sum
      }
    end
  end

  class << self
    def queries
      @queries ||= Hash.new { |h, k| h[k] = Hash.new(0) }
    end

    def flush
      @queries = nil
    end

    def started(event)
      op = event.command.keys.first # find, update, delete, createIndexes, etc.
      ns = event.command[op] # collection name
      return unless ns.is_a?(String)
      queries[op][ns] += 1
    end

    def succeeded(_); end
    def failed(_); end
  end
end

Mongo::Monitoring::Global.subscribe(Mongo::Monitoring::COMMAND, MongoSpy)

RSpec.configure do |config|
  config.include MongoSpy::Helpers
end

Answer 3

What you're looking for is command monitoring. With Mongoid and the Ruby Driver, you can create a custom command monitoring class that you can use to subscribe to all commands made to the server.

I've adapted this from the Command Monitoring Guide for the Mongo Ruby Driver.

For this particular example, make sure that your Rails app has the log level set to debug. You can read more about the Rails logger here.

The first thing you want to do is define a subscriber class. This is the class that tells your application what to do when the Mongo::Client performs commands against the database. Here is the example class from the documentation:

class CommandLogSubscriber
  include Mongo::Loggable

  # called when a command is started
  def started(event)
    log_debug("#{prefix(event)} | STARTED | #{format_command(event.command)}")
  end

  # called when a command finishes successfully
  def succeeded(event)
    log_debug("#{prefix(event)} | SUCCEEDED | #{event.duration}s")
  end

  # called when a command terminates with a failure
  def failed(event)
    log_debug("#{prefix(event)} | FAILED | #{event.message} | #{event.duration}s")
  end

  private

  def logger
    Mongo::Logger.logger
  end

  def format_command(args)
    begin
      args.inspect
    rescue Exception
      '<Unable to inspect arguments>'
    end
  end

  def format_message(message)
    format("COMMAND | %s".freeze, message)
  end

  def prefix(event)
    "#{event.address.to_s} | #{event.database_name}.#{event.command_name}"
  end
end

(Make sure this class is auto-loaded in your Rails application.)

Next, you want to attach this subscriber to the client you use to perform commands.

subscriber = CommandLogSubscriber.new

Mongo::Monitoring::Global.subscribe(Mongo::Monitoring::COMMAND, subscriber)

# This is the name of the default client, but it's possible you've defined
#   a client with a custom name in config/mongoid.yml
client = Mongoid::Clients.from_name('default')
client.subscribe( Mongo::Monitoring::COMMAND, subscriber)

Now, when Mongoid executes any commands against the database, those commands will be logged to your console.

# For example, if you have a model called Book
Book.create(title: "Narnia")
# => D, [2020-03-27T10:29:07.426209 #43656] DEBUG -- : COMMAND | localhost:27017 | mongoid_test_development.insert | STARTED | {"insert"=>"books", "ordered"=>true, "documents"=>[{"_id"=>BSON::ObjectId('5e7e0db3f8f498aa88b26e5d'), "title"=>"Narnia", "updated_at"=>2020-03-27 14:29:07.42239 UTC, "created_at"=>2020-03-27 14:29:07.42239 UTC}], "lsid"=>{"id"=><BSON::Binary:0x10600 type=uuid data=0xfff8a93b6c964acb...>}}
# => ...

You can modify the CommandLogSubscriber class to do something other than logging (such as incrementing a global counter).