Long running operations (threads) in a web (asp.net) environment

Question

I have an asp.net (mvc) web site. As the part of the functions I will have to support some long running operations, for example:

Initiated from user: User can upload (xml) file to the server. On the server I need to extract file, do some manipulation (insert into the db) etc... This can take from one minute to ten minutes (or even more - depends on file size). Of course I don't want to block the request when the import is running , but I want to redirect user to some progress page where he will have a chance to watch the status, errors or even cancel the import.

This operation will not be frequently used, but it may happen that two users at the same time will try to import the data. It would be nice to run the imports in parallel. At the beginning I was thinking to create a new thread in the iis (controller action) and run the import in a new thread. But I am not sure if this is a good idea (to create working threads on a web server). Should I use windows services or any other approach?

Initiated from system: - I will have to periodically update lucene index with the new data. - I will have to send mass emails (in the future).

Should I implement this as a job in the site and run the job via Quartz.net or should I also create a windows service or something?

What are the best practices when it comes to running site "jobs"?

Thanks!

Answer 1

I would implement stand-alone windows service for long running tasks. Web application delegates long-running tasks to this service via queue approach. It's up to you how you will organize tasks queue. Will queued task have priority, maximum execution time or not. Queue can be implemented as ordinary table(s) in DBMS, that contains job execution status info properties (pretty straightforward approach).

So common scenario may look the following:

• Client sends all required information to web-server

• Web-server delegates task to service and notifies client - task was successfully queued (task id is sent to client also)

• External service starts task processing, updating progress information.

• Client starts polling web server with short execution requests about job (with id received earlier) status and progress.

You can choose different technologies (Windows Service + DB / WCF Service) and different communication approaches (polling, pushing, callbacks) but I advise to delegate long-running tasks to external service (not perform them within web app).

Executing long running tasks brings thread-per-request model (in multithreaded programming terms). This model has poor scalability and thread pool maximum number of threads limitations. It's not your case though :)

Answer 2

In my opinion long running tasks should generally always be delegated to non UI based operations. I would suggest perhaps a WF or Window service.

Answer 3

I've successfully implemented a scenario similar to this using jQuery. basically, i use the beforeSend: function to display the 'please wait' type page. here's the basic code in play (altso i don't, you could also use the AsyncController base class to make the action async):

<script type="text/javascript">
    $(document).ready(function() {
        $('#create').bind('click', function() {
            saveFundProperty();
        });
    });

    // main functions
    function saveFundProperty() {
        var url = '<%= Url.Action("Create", "FundProperty") %>';
        var params = { fundId: $("#FundID").val(), propertyId: $("#PropertyID").val() };
        SendAjax(url, params, beforeQuery, saveFundPropertyResponse);
    }

    function beforeQuery() {
        var url = '<%= Url.Action("Wait", "FundProperty") %>';
        $("#statusMsg").load(url);
    }

    function saveFundPropertyResponse(data) {
        if (data.length != 0) {
            if (data.indexOf("ERROR:") >= 0) {
                $("#statusMsg").html(data).css('backgroundColor','#eeaa00');
            }
            else {
                $("#statusMsg").html(data);
            }
        }
    }
</script>

hope this helps.

the SendAjax method is purely a wrapper function to make things a little more consistant. here it is in full:

<script type="text/javascript">
function SendAjax(urlMethod, jsonData, beforeSendFunction, returnFunction, dataType, contentType) {
    $.ajaxSetup({ cache: false });
    dataType = dataType || "text"; // default return type
    contentType = contentType || "application/x-www-form-urlencoded"; // default input type
    $.ajax({
        type: "POST",
        url: urlMethod,
        data: jsonData,
        dataType: dataType,
        contentType: contentType,
        beforeSend: function() {
            if(beforeSendFunction!==null)
                beforeSendFunction();
        },
        success: function(data) {
            // Do something interesting here.
            if (data != null && returnFunction!==null) {
                returnFunction(data);
            }
        },
        error: function(xhr, status, error) {
            // Boil the ASP.NET AJAX error down to JSON.
            var err = eval("(" + xhr.responseText + ")");

            // Display the specific error raised by the server
            alert(err.Message);
        }
    });
}
</script>

[edit] - not sure what's happening with the SendAjax formatting. hope it's easy to copy/paste out...