How to assure the same job doesn't run twice at the same time?

Question

I have a simple sidekiq job that I've set to run every 5 minutes using clockwork. What I'm worried about is that, if the job takes longer than 5 minutes, it will run again, which could cause some issues if it's running at the same time as the original job.

Is there a way to lock the clockwork so that it only runs if the previous running has completed?

Answer 1

I've used the Sidekiq Unique Jobs gem to accomplish this. It has various strategies for preventing duplicate jobs.

Answer 2

Sidekiq Enterprise includes support for unique jobs. Unique jobs are defined with an option in the worker class:

# app/workers/my_worker.rb
class MyWorker
  include Sidekiq::Worker
  sidekiq_options unique_for: 10.minutes

  def perform(...)
  end
end

They must be enabled in the Sidekiq initializer as well:

# config/initializers/sidekiq.rb
Sidekiq::Enterprise.unique! unless Rails.env.test?

The time window only applies while the job is running, e.g., if it is unique for 10 minutes and the job completes in 6 minutes then Sidekiq will immediately allow another job to be enqueued; you don't have to wait 4 more minutes for the uniqueness lock to be dropped.

The uniqueness constraint is enforced across all Sidekiq Enterprise nodes.

You may need to adjust your retries options because a failed job will continue to hold the lock for as long as it is in a retry state. (even if it is not actively executing)