Streaming Large File Uploads in ASP.NET

Question

I have an ASP.NET MVC application with a page that allows users to upload files. The files will be several hundred megabytes.

I am using FineUploader on the client side, which will use FileAPI/XHR if the browser supports it, otherwise will fallback to Iframe/form with enctype="multipart whatever".

So on the server side I need to evaluate Request.Files.Count > 1. If true, this is an old school upload and I save the file like Request.Files[0].InputStream.CopyTo(myFileStream) otherwise I do Request.InputStreawm.CopyTo(myFileStream).

Here's some of the actual code I've written that does this stuff: https://github.com/ronnieoverby/file-uploader/blob/master/server/ASP.NET%20MVC%20C%23/FineUpload.cs

This all works fine, but in my testing I've noticed that neither an ASP.NET MVC controller action nor an HttpHandler will begin processing until the entire file is uploaded, which is bad if the file very large because that means it's occupying a lot of the web server's RAM.

I found this: Streaming large file uploads to ASP.NET MVC which sounds promising, but I really don't have an idea of where the code resides in his application.

So, the question is: how to stream uploaded files to disk while the upload is still taking place in ASP.NET?

Update

I just saw a key detail that didn't sink in before. From the HttpPostedFile documentation:

By default, all requests, including form fields and uploaded files, larger than 256 KB are buffered to disk, rather than held in server memory.

Ok, that addresses the concern that the web server's RAM utilization could spike during a large upload. But, there's still a problem: After the file is completely transferred to the web server, the server has to spend time moving it to it's final destination. If the file system operation is a copy (guaranteed if the destination is on another physical disk), then the response is delayed unnecessarily.

Honestly, I could probably live with this by increasing response timeout for the upload handler/action. But, it would be nice to stream the bytes directly to their destination.

Answer 1

You can handle uploads in a completely customized way without buffering using HttpRequest.GetBufferlessInputStream method. Basically you are getting access to the raw incoming data and free to do whatever you want with it.

I've just created small sample which saves raw request content to a file:

1. Create handler:

   public class UploadHandler : IHttpHandler
   {
       public void ProcessRequest(HttpContext context)
       {
           using (var stream = context.Request.GetBufferlessInputStream())
           using (var fileStream = File.Create("c:\\tempfile.txt"))
           {
               stream.CopyTo(fileStream);
           }
       }
       public bool IsReusable { get {  return true; } }
   }
   

2. Register in Web.config:

   <system.webServer>
       <modules runAllManagedModulesForAllRequests="true"/>
       <handlers>
           <add name="UploadHandler" verb="POST"
               path="/upload"
               type="UploadHandler"
           resourceType="Unspecified"/>
       </handlers>
   </system.webServer>
   

3. Create a page with a form:

<form action="/upload" method="post" enctype='multipart/form-data'>
     <input type="file" name="aa" id="aa"/>
     <input type="submit"/>
</form>

Answer 2

If the uploading and streaming is using up valuable server resources then you might wanna take a look at hosting your media files on a cloud of some sort. It's possible with ASP.NET to use a Rackspace, Amazon Cloud API have your users upload the files directly to a CDN network and then serve the content that way, I know this isn't answering your question but many people will or already have and thought I'd get my 2 cents in. Many people still not opting to use the cloud amazes me! once you go CDN you never go back. Furthermore with most CDN's you will also be given a streaming URL for your upload container where it supports lots of different movie types, and its lighting fast, not only for your users to upload too but also your never have slow speeds on your website as a result.