Best way to handle very large http uploads and downloads on the server-side

Question

I've an upcoming project where I will need to handle very large uploads from browsers (either the classic input type="file" or a Java Applet), and looking for the best tool to do the job on the server.

Theses are the things I need :

• low memory consumption on the server
• ability to save the file in its final destination on the server (no copying the file around)
• no blocking of other important tasks done by the webserver
• good handling of files up to 2 gb
• authorization of files (permissions would be given in the app)

I still have some latitude on what technology to use so I would like to have some advice in order to be able to choose the best technology on the server to handle this task :

• ASP.NET ?
• Java ?
• Amazon S3 ?
• Other choices ?

I'm more used to the Microsoft Stack, but willing to change if necessary : as told above, I'm just looking for the best tool for the job.

Thanks !

Update : The server side is the part I'm really interested in for this question, not the client side.

It looks like it may be trivial, but when you start to digg a bit you see 4 Mb limitations with .NET, downloads that use a lot of memory, that CAN block other threads (when you have a limit on the number of threads, and if a thread can execute for the duration of 2 Gb file upload/download over the internet : well this ain't gonna scale very well, will it ?), etc.

Answer 1

You'll need:

• Client-side code (Java Applet, Silverlight, etc) to break files in small chunks
• Server-side code (ASP.NET, Java, doesn't matter) to build those files back

I just finished an application exactly like that; I'd use Silverlight (WebRequest async), ASP.NET (IHttpHandler/IHttpAsyncHandler) and SQL Server 2005 (UPDATETEXT/READTEXT) for file storage.

UPDATE: About ASP.NET server-side code:

ASP.NET default config will allow 100 threads per processor; IHttpAsyncHandler won't block your process and there you can write your file content directly to context.Response.OutputStream.

For upload, you'll also send several data chunks, but in multiple HTTP connections; while this can bring some overheat due HTTP headers, works very well in my tests.