How do You monitor sidekiq processes?

Question

I'm working on a production app that has multiple rails servers behind nginx loadbalancer. We are monitoring sidekiq processes with monit, and it works just fine - when sidekiq proces dies monit starts it right back.

However recently encountered a situation where one of these processes was running and visible to monit, but for some reason not visible to sidekiq. That resulted in many failed jobs and took us some time to notice that we're missing one process in sidekiq Web UI, since monit was telling us everything was fine and all processes were running. Simple restart fixed the problem.

And that bring me to my question: how do you monitor your sidekiq processes? I know i can use something like rollbar to notify me when jobs fail, but i'd like to know if there is a way to monitor process count and preferably send mail when one dies. Any suggestions?

Something that would ping sidekiq/stats and verify response.

Answer 1

My super simple solution to a similar problem looks like this:

# sidekiq_check.rb
namespace :sidekiq_check do
  task rerun: :environment do
    if Sidekiq::ProcessSet.new.size == 0
      exec 'bundle exec sidekiq -d -L log/sidekiq.log -C config/sidekiq.yml -e production'
    end
  end
end

and then using cron/whenever

# schedule.rb
every 5.minutes do 
  rake 'sidekiq_check:rerun'
end