Upload and Process CSV File in ASP.NET MVC 4 Architectural Considerations

Question

I am working on an ASP.NET MVC 4 Application that imports and processes a CSV file. I am using a standard form and controller for the upload. Here is an overview of what I am doing currently:

Controller Logic

public ActionResult ImportRecords(HttpPostedFileBase importFile){

    var fp = Path.Combine(HttpContext.Server.MapPath("~/ImportUploads"), Path.GetFileName(uploadFile.FileName));
    uploadFile.SaveAs(fp);

    var fileIn = new FileInfo(fp);
    var reader = fileIn.OpenText();
     var tfp = new TextFieldParser(reader) {TextFieldType = FieldType.Delimited, Delimiters = new[] {","}};
    while(!tfp.EndOfData){
        //Parse records into domain object and save to database
    }
    ...
}

HTML

@using (Html.BeginForm("ImportRecords", "Import", FormMethod.Post, new { @id = "upldFrm", @enctype = "multipart/form-data" }))
{
    <input id="uploadFile" name="uploadFile" type="file" />
    <input id="subButton" type="submit" value="UploadFile" title="Upload File" />
}

The import file can contain a large number of records (average 40K+) and can take quite some time to complete. I'd rather not have a user sitting at the import screen for 5+ minutes for each file processed. I have considered adding a console application to watch the uploads folder for new files, and process when something new is added, but would like to see what input I receive from the community before starting my journey down this path.

Is there a more efficient way to handle this operation?

Is there a way to perform this action, allowing the user to continue about his/her merry way, and then notify the user when processing is done?

Answer 1

The solution to the issue I was having is a bit complex, but works similar to the IFrame fix. The result is a pop-up window that handles the processing, allowing the user to continue navigation throughout the site.

The file is submitted to the server (UploadCSV controller), a Success page is returned with a bit of JavaScript to handle the initial kick-off of the processing. When the user clicks "Begin Processing", a new window is opened (ImportProcessing/Index) that loads the initial status (kicking off an interval loop that retrieves status updates) and then makes a call to the "StartProcessing" action, kicking off the processing process.

The "FileProcessor" class that I am using is housed in a static dictionairy variable within the ImportProcessing controller; allowing for status results based on the key. The FileProcessor is promptly removed after the operation is complete or an error is encountered.

Upload Controller:

 [AcceptVerbs(HttpVerbs.Post)]
        public ActionResult UploadCSV(HttpPostedFileBase uploadFile)
        {
            var filePath = string.Empty;
            if (uploadFile.ContentLength <= 0)
            {
                return View();
            }
                filePath  = Path.Combine(Server.MapPath(this.UploadPath), "DeptartmentName",Path.GetFileName(uploadFile.FileName));
            if (new FileInfo(filePath).Exists)
            {
                ViewBag.ErrorMessage =
                    "The file currently exists on the server.  Please rename the file you are trying to upload, delete the file from the server," +
                    "or contact IT if you are unsure of what to do.";
                return View();
            }
            else
            {
                uploadFile.SaveAs(filePath);
                return RedirectToAction("UploadSuccess", new {fileName = uploadFile.FileName, processType = "sonar"});
            }
        }

 [HttpGet]
        public ActionResult UploadSuccess(string fileName, string processType)
        {
            ViewBag.FileName = fileName;
            ViewBag.PType = processType;
            return View();
        }

Upload Success HTML:

@{
    ViewBag.Title = "UploadSuccess";
}

<h2>File was uploaded successfully</h2>
<p>Your file was uploaded to the server and is now ready to be processed.  To begin processing this file, click the "Process File" button below.
</p>
<button id="beginProcess" >Process File</button>
<script type="text/javascript">
    $(function () {
        $("#beginProcess").click(BeginProcess);
        function BeginProcess() {
            window.open("/SomeController/ImportProcessing/[email protected]&[email protected]", "ProcessStatusWin", "width=400, height=250, status=0, toolbar=0,  scrollbars=0, resizable=0");
            window.location = "/Department/Import/Index";
        }
    });
</script>

Once this new window is opened up, the file processing begins. Updates are retrieved from a custom FileProcessing class.

ImportProcessing Controller:

  public ActionResult Index(string fileName, string type)
        {
            ViewBag.File = fileName;
            ViewBag.PType = type;
            switch (type)
            {
                case "somematch":
                    if (!_fileProcessors.ContainsKey(fileName)) _fileProcessors.Add(fileName, new SonarCsvProcessor(Path.Combine(Server.MapPath(this.UploadPath), "DepartmentName", fileName), true));
                    break;
                default:
                    break;
            }
            return PartialView();
        }

ImportProcessing Index:

@{
    ViewBag.Title = "File Processing Status";
}
@Scripts.Render("~/Scripts/jquery-1.8.2.js")

<div id="StatusWrapper">
    <div id="statusWrap"></div>
</div>
<script type="text/javascript">
    $(function () {
        $.ajax({
            url: "GetStatusPage",
            data: { fileName: "@ViewBag.File" },
            type: "GET",
            success: StartStatusProcess,
            error: function () {
                $("#statusWrap").html("<h3>Unable to load status checker</h3>");
            }
        });
        function StartStatusProcess(result) {
            $("#statusWrap").html(result);
            $.ajax({
                url: "StartProcessing",
                data: { fileName: "@ViewBag.File" },
                type: "GET",
                success: function (data) {
                    var messag = 'Processing complete!\n Added ' + data.CurrentRecord + ' of ' + data.TotalRecords + " records in " + data.ElapsedTime + " seconds";
                    $("#statusWrap #message").html(messag);
                    $("#statusWrap #progressBar").attr({ value: 100, max: 100 });
                    setTimeout(function () {
                        window.close();
                    }, 5000);
                },
                error: function (xhr, status) {
                    alert("Error processing file");
                }
            });
        }
    });
</script>

Finally the Status Checker html:

@{
    ViewBag.Title = "GetStatusPage";
}
<h2>Current Processing Status</h2>
    <h5>Processing: @ViewBag.File</h5>
    <h5>Updated: <span id="processUpdated"></span></h5>
    <span id="message"></span>
    <br />
    <progress id="progressBar"></progress>
<script type="text/javascript">
    $(function () {
        var checker = undefined;
        GetStatus();
        function GetStatus() {
            if (checker == undefined) {
                checker = setInterval(GetStatus, 3000);
            }
            $.ajax({
                url: "[email protected]",
                type: "GET",
                success: function (result) {
                    result = result || {
                        Available: false,
                        Status: {
                            TotalRecords: -1,
                            CurrentRecord: -1,
                            ElapsedTime: -1,
                            Message: "No status data returned"
                        }
                    };
                    if (result.Available == true) {
                        $("#progressBar").attr({ max: result.Status.TotalRecords, value: result.Status.CurrentRecord });
                        $("#processUpdated").text(result.Status.Updated);
                        $("#message").text(result.Status.Message);
                    } else {
                        clearInterval(checker);
                    }

                },
                error: function () {
                    $("#statusWrap").html("<h3>Unable to load status checker</h3>");
                    clearInterval(checker);
                }
            });
        }
    });
</script>