Hangfire Custom State Expiration

Question

I have implemented a custom state "blocked" that moves into the enqueued state after certain external requirements have been fulfilled.

Sometimes these external requirements are never fulfilled which causes the job to be stuck in the blocked state. What I'd like to have is for jobs in this state to automatically expire after some configurable time.

Is there any support for such a requirement? There is the ExpirationDate field, but from looking at the code it seems to be only used for final states.

The state is as simple as can be:

    internal sealed class BlockedState : IState
    {
        internal const string STATE_NAME = "Blocked";

        public Dictionary<string, string> SerializeData()
        {
            return new Dictionary<string, string>();
        }

        public string Name => STATE_NAME;

        public string Reason => "Waiting for external resource";

        public bool IsFinal => false;

        public bool IgnoreJobLoadException => false;
    }

and is used simply as _hangfireBackgroundJobClient.Create(() => Console.WriteLine("hello world"), new BlockedState());

At a later stage it is then moved forward via _hangfireBackgroundJobClient.ChangeState(jobId, new EnqueuedState(), BlockedState.STATE_NAME)

Answer 1

I would go for a custom implementation IBackgroundProcess taking example from DelayedJobScheduler which picks up delayed jobs on a regular basis to enqueue it.

In this custom implementation I would use a JobStorageConnection.GetAllItemsFromSet("blocked") to get all the blocked job ids (where the DelayedJobScheduler uses JobStorageConnection.GetFirstByLowestScoreFromSet)

Then I would get each blocked job data with JobStorageConnection.GetJobData(jobId). For each of them, depending on its CreatedAt field, I would do nothing if the job is not expired, or change its state to another state (Failed ?) if it is expired.

The custom job process can be declared like this :

       app.UseHangfireServer(storage, options, 
             new IBackgroundProcess[] { 
                        new MyCustomJobProcess(
                                myTimeSpanForExpiration, 
                                (IBackgroundJobStateChanger) new BackgroundJobStateChanger(filterProvider)) });

A difficulty here is to obtain an IBackgroundJobStateChanger as the server does not seem to expose its own. If you use a custom FilterProvider as option for your server pass its value as filterProvider, else use (IJobFilterProvider) JobFilterProviders.Providers