Stop Hangfire starting a recurring job if it's already running

Question

I have a list of recurring jobs set up in Hangfire, all with an interval of 20 minutes.

They all call the same method, but with different parameters. e.g.

Job ID test_1 => MyTestMethod(1)
Job ID test_50 => MyTestMethod(50)
Job ID test_37 => MyTestMethod(37)

Sometimes, a job might take longer than its interval, i.e. longer than 20 minutes, and I want to make sure that job doesn't get run again whilst another instance of it is already running.

The DisableConcurrentExecutionAttribute is not suitable here, because the method CAN be run concurrently, just not with the same parameter(s).

I attempted to have a static Dictionary<string, DateTime> that tracked whether a job was already running so that it could block concurrent method calls, but the problem there is that if the application restarts for any reason, then it "forgets" what jobs are currently running (Hangfire of course will continue to run in background).

UPDATE
I also tried adding a JobState table to my database to track which jobs are running and then checking that table when MyTestMethod starts, but this relies on MyTestMethod setting running=1 when it starts and running=0 when it ends and if the thread crashes halfway through that job (for whatever reason) that might not happen, thus preventing the job running again at all.

I am sure I can solve this by simply querying Hangfire job status by job ID - I just can't find how to do this?

Answer 1

I had a similar requirement, basically I wanted to use the DisableConcurrentExecutionAttribute but have it take into account the parameters. That way if a job gets queued with the same parameter it will still run just not in parallel. I took the example DisableMultipleQueuedItemsFilter which actually removes jobs and modified the DisableConcurrentExecutionAttribute to use the parameters. The difference is the jobs will be queued if they have the same parameters list they won't run in parallel.

A full example can be seen here with both attributes: https://gist.github.com/sbosell/3831f5bb893b20e82c72467baf8aefea

The relevant code for the attribute:

 public class DisableConcurrentExecutionWithParametersAttribute : JobFilterAttribute, IServerFilter
    {
        private readonly int _timeoutInSeconds;

        public DisableConcurrentExecutionWithParametersAttribute (int timeoutInSeconds)
        {
            if (timeoutInSeconds < 0) throw new ArgumentException("Timeout argument value should be greater that zero.");

            _timeoutInSeconds = timeoutInSeconds;
        }

        public void OnPerforming(PerformingContext filterContext)
        {
            var resource = GetResource(filterContext.BackgroundJob.Job);

            var timeout = TimeSpan.FromSeconds(_timeoutInSeconds);

            var distributedLock = filterContext.Connection.AcquireDistributedLock(resource, timeout);
            filterContext.Items["DistributedLock"] = distributedLock;
        }

        public void OnPerformed(PerformedContext filterContext)
        {
            if (!filterContext.Items.ContainsKey("DistributedLock"))
            {
                throw new InvalidOperationException("Can not release a distributed lock: it was not acquired.");
            }

            var distributedLock = (IDisposable)filterContext.Items["DistributedLock"];
            distributedLock.Dispose();
        }

        private static string GetFingerprint(Job job)
        {
            var parameters = string.Empty;
            if (job?.Arguments != null)
            {
                parameters = string.Join(".", job.Arguments);
            }
            if (job?.Type == null || job.Method == null)
            {
                return string.Empty;
            }
            var payload = $"{job.Type.FullName}.{job.Method.Name}.{parameters}";
            var hash = SHA256.Create().ComputeHash(System.Text.Encoding.UTF8.GetBytes(payload));
            var fingerprint = Convert.ToBase64String(hash);
            return fingerprint;
        }
        private static string GetResource(Job job)
        {
            return GetFingerprint(job);
        }
    }

Answer 2

There's an attribute you can use to prevent concurrent execution (for a time):

[DisableConcurrentExecution(3600)] // argument in seconds, e.g., an hour
public void MyTestMethod(int id) {
  // ...
}