.Net framework to manage background running processess on seperate machines

Question

I am having an asp.mvc application which resides on a server.From this application, I want to start a process which is a bit long-running operation and will be resource intensive operation.

So what I want to do is I want to have some user agent like 3 which will be there on 3 machines and this user agent will use resources of their respective machines only.

Like in Hadoop we have master nodes and cluster in which tasks are run on the individual cluster and there is 1 master node keeping track of all those clusters.

In Azure, we have virtual machines on which tasks are run and if require Azure can automatically scale horizontally by spinning up the new instance in order to speed up the task.

So I want to create infrastructure like this where I can submit my task to 3 user agents from the mvc application and my application will keep track of this agents like which agent is free, which is occupied, which is not working something like this.

I would like to receive progress from each of this user agent and show on my MVC application.

Is there any framework in .net from which I can manage this background running operations(tracking, start, stop etc..) or what should be the approach for this?

Update : I don't want to put loads of server for this long running operations and moreover I want to keep track of this long running process too like what they are doing, where is error etc.

Following are the approach which I am thinking and I don't know which will make more sense:

1) Install Windows Service in the form of agents of 2-3 computer on premises to take advantage of resp resources and open a tcp/ip connection with this agents unless and until the long running process is complete.

2) Use hangfire to run this long running process outside of IIS thread but I guess this will put load on server.

I would like to know possible problems of above approaches and if there are any better approaches than this.

Answer 1

Hangfire is really a great solution for processing background tasks, and we have used used it extensively in our projects.

We have setup our MVC application on separate IIS servers which is also a hangfire client and just enqueues the jobs needs to be executed by hangfire server. Then we have two hangfire server instances, which are windows service application. So effectively there is no load on the MVC app server to process the background jobs, as it is being processed by separate hangfire servers.

One of the extremely helpful feature of hangfire is its out of the box dashboard, that allows you to monitor and control any aspect of background job processing, including statistics, background job history etc.

Configure the hangfire in application as well as in hangfire servers

public void Configuration(IAppBuilder app)
{
    GlobalConfiguration.Configuration.UseSqlServerStorage("<connection string or its name>");

    app.UseHangfireDashboard();
    app.UseHangfireServer();
}

Please note that you use the same connection string across. Use app.UseHangfireServer() only if you want to use the instance as hangfire server, so in your case you would like to omit this line from application server configuration and use only in the hangfire servers. Also use app.UseHangfireDashboard() in instance which will serve your hangfire dashboard, which would be probably your MVC application.

At that time we have done it using Windows Service, but if had to do it now, I would like to go with Azure worker role or even better now Azure Web Jobs to host my hangfire server, and manage things like auto scaling easily.

Do refer hangfire overview and documentation for more details.