keep load balancer from timing out during long operation

Question

I'm using ASP.NET MVC 4 to accept a video upload of up to 30 mb in size. Once the video is uploaded to the site, the video is then posted to a 3rd party web service using a HttpWebRequest. This upload to the 3rd party service must complete and return a response before my site can return a response to the browser. On Rackspace Cloud Sites, the load balancer has a 30 second timeout. If no response is returned from my site through the load balancer for 30 seconds, then load balancer returns a timeout error to the browser and the request is terminated.

So what I've been told needs to happen is that my site needs to keep the connection alive by sending data back to the browser while my site is sending the video to the 3rd party system, so that the load balancer doesn't timeout. How would you accomplish this task?

I'm looking for 2 things - What do I send back as data to the browser while the upload is happening? My actual response is JSON, so if I can keep my regular JSON response at the end, that would be ideal.

And finally, how do I execute my HttpWebRequest upload to the 3rd party while sending the keep-alive data at the same time?

-

For reference, this is the load balancer connection timeout doc from rackspace cloud: http://www.rackspace.com/knowledge_center/article/connection-timed-out-error-message-1

They don't offer up any concrete solutions for this problem.

Answer 1

Here's how I solved my problem. This solution should be applicable to any .net hosting that uses a load balancer with an unreasonable timeout. Rackspace Cloud Sites have a 30 second timeout on their load balancer. Amazon and Azure have longer timeouts, but this still might be a useful solution if you happen to have a long-running operation.

In asp.net mvc 4 you can change your controller to inherit from AsyncController. This allows you to spin off async tasks and then await their completion. This solution creates a "keep alive" task which sends data back through the asp.net Response every 10 seconds so that the load balancer detects activity and doesn't timeout. This task runs until it sees that the uploadFinished variable is set to true.

The other task performs the long-running upload, or whatever task needs to be completed before the asp.net mvc controller ends the web request it is handling. After the long-running operation is completed it sets the uploadFinished variable to true.

Finally, I'm manually constructing a container json object around the actual json response so that I can have the "keep alive" data be part of valid json that gets sent back to the browser. A slightly ugly hack, but it works!

using System.Linq;
using System.Threading;
using System.Threading.Tasks;
using System.Web;
using System.Web.Mvc;
using System.Web.Script.Serialization;

namespace MvcApplication4.Controllers
{
    public class ExampleController : AsyncController
    {

        [HttpPost]
        public string Upload(UploadModel model, HttpPostedFileBase videoFile)
        {
            // when using jquery ajax form for upload
            // IE requires text/plain content type
            // otherwise it will try to open the response as a file
            // instead of pass the response back to your javascript
            if (Request.AcceptTypes.Contains("application/json"))
            {
                Response.ContentType = "application/json";
            }
            else
            {
                Response.ContentType = "text/plain";
            }

            // start json object
            Response.Write("{\"keepalive\":\"");
            Response.Flush();

            Task[] tasks = new Task[2];
            tasks[0] = Task.Factory.StartNew(() => DoKeepAlive());
            tasks[1] = Task.Factory.StartNew(() => DoUpload(model, videoFile));
            Task.WaitAll(tasks);

            // end keepalive json property
            Response.Write("\",\"data\":");

            // insert actual response data
            JavaScriptSerializer json = new JavaScriptSerializer();
            Response.Write(json.Serialize(uploadResponse));

            // end json object
            Response.Write("}");

            return "";
        }

        public bool uploadFinished = false;
        public UploadResponseModel uploadResponse = null;

        public void DoUpload(UploadModel model, HttpPostedFileBase videoFile)
        {
            // do upload to 3rd party
            MyServiceClient c = new MyServiceClient();
            uploadResponse = c.UploadVideo(model, videoFile);
            uploadFinished = true;
        }

        public void DoKeepAlive()
        {
            // send . every 10 seconds
            while (!uploadFinished)
            {
                Response.Write(".");
                Response.Flush();
                Thread.Sleep(10000);
            }
        }
    }
}

This solution depends on .Net 4 to do the async stuff. If you have .Net 4.5 there are newer ways to do async in your mvc controllers that don't depend on the AsyncController class.

This site is helpful and explains how to do async operations in asp.net mvc in various versions of the .Net framework:

http://dotnet.dzone.com/news/net-zone-evolution