Can hang-fire throttle number of background jobs running in parallel

Question

I have IO intensive method that i am running as background job using hangfire.

public IHttpActionResult Request(int[] ids)
{
    foreach(var id in ids)
    {
       BackgroundJob.Enqueue<IMyService>(x => x.DoWork(id));
    }
}

So for each id i Enqueue a background job and hangfire immediately invoke DoWork() as expected. However DoWork is IO intensive. So if i have 100+ ids it takes lot of CPU power and bandwidth

Is there anyway to throttle number of background jobs in Hangfire

Answer 1

You can make use of hangfire queues and set the number of workers for that queue.

In your hangfire startup configuration set up the following queues:

var options = new BackgroundJobServerOptions
{
    Queues = new[] { "default" }, // You can have multiple queues and multiple worker counts.. check hangfire documentation
    WorkerCount = 5 // this is up to you
};

app.UseHangfireServer(options);

See code example below:

public IHttpActionResult Request(int[] ids)
{
    foreach(var id in ids)
    {
       BackgroundJob.Enqueue<IMyService>(x => x.DoWork(id));
    }
}


[Queue("default")]// add this attribute on your function
public void DoWork() { 
    //Magical IO code here
}

I recommend you look at the following hangfire documentation if you need any futher info: http://docs.hangfire.io/en/latest/background-processing/configuring-degree-of-parallelism.html https://discuss.hangfire.io/t/different-queues-having-different-worker-counts/114

Hope this helps.