Recurring job is triggered on several workers at the same time

Question

I am using the latest stable hangfire (1.6.17.0).

I have 4-6 workers running in parallel at all times.

When new workers start up, they are verifying that all jobs are well created - and adding/updating all the jobs that should run, using the following code:

var jobId = HangfireConsts.FormatReccurringJobStore(storeId); 

RecurringJob.AddOrUpdate(
    jobId,
    () => ExecuteRefreshStore(storeId),
    Cron.HourInterval(8),
    null,
    HangfireConsts.QueueStores);

I figured this should be harmless since I'm using AddOrUpdate() - so worst case it should just update the existing. The locking system would make sure each of the jobs is added once (based on the jobId). This allows me to start even of a fresh new clean schema for the scheduler, and get persistent results - as all recurring jobs will be re-created.

Yet, I can see that some of the jobs start, with the exact same jobId, are triggered at the exact same time on multiple workers. Sometimes on 2 workers in parallel, sometimes on 3 workers in parallel, sometimes on 4 or 5 (out of 6).

How is that possible? Is this a bug?

Worker 1

Worker 2

Worker 3

Answer 1

The following is based on comments from Performing recurrent tasks documentation:

• One approach is to not create a scheduled job but to create single shot delayed job which creates a new delayed job when it is finished processing.

  The main problem with this is that if you start two processors, you now have to decide which one of them schedules the first job.

• Another one is to use the [DisableConcurrentExecution] attribute with 0 timeout on the job method together with AutomaticRetry to avoid retries (and warn logs):

  [DisableConcurrentExecution(0)]
  [AutomaticRetry(Attempts = 0, LogEvents = false, OnAttemptsExceeded = AttemptsExceededAction.Delete)]