How to deliver big files in ASP.NET Response?

Question

I am not looking for any alternative of streaming file contents from database, indeed I am looking for root of the problem, this was running file till IIS 6 where we ran our app in classic mode, now we upgraded our IIS to 7 and we are running app pool in pipeline mode and this problem started.

I have an handler, where I have to deliver big files to client request. And I face following problems,

Files are of average size 4 to 100 MB, so lets consider 80MB file download case.

Buffering On, Slow Start

Response.BufferOutput = True; 

This results in very slow start of file, as user downloads and even progress bar does not appear till few seconds, typically 3 to 20 seconds, reason behind is, IIS reads entire file first, determines the content-length and then begin the file transfer. File is being played in video player, and it runs very very slow, however iPad only downloads fraction of file first so it works fast.

Buffering Off, No Content-Length, Fast Start, No Progress

Reponse.BufferOutput = False; 

This results in immediate start, however end client (typical browser like Chrome) does not know Content-Length as IIS does not know either, so it does not display progress, instead it says X KB downloaded.

Buffering Off, Manual Content-Length, Fast Start, Progress and Protocol Violation

Response.BufferOutput = False; Response.AddHeader("Content-Length", file.Length); 

This results in correct immediate file download in Chrome etc, however in some cases IIS handler results in "Remote Client Closed Connection" error (this is very frequent) and other WebClient results in protocol violation. This happens 5 to 10% of all requests, not every requests.

I guess what is happening is, IIS does not send anything called 100 continue when we dont do buffering and client might disconnect not expecting any output. However, reading files from source may take longer time, but at client side I have increased timeout but seems like IIS timesout and have no control.

Is there anyway I can force Response to send 100 continue and not let anyone close the connection?

UPDATE

I found following headers in Firefox/Chrome, nothing seems unusual here for Protocol Violation or Bad Header.

Access-Control-Allow-Headers:* Access-Control-Allow-Methods:POST, GET, OPTIONS Access-Control-Allow-Origin:* Access-Control-Max-Age:1728000 Cache-Control:private Content-Disposition:attachment; filename="24.jpg" Content-Length:22355 Content-Type:image/pjpeg Date:Wed, 07 Mar 2012 13:40:26 GMT Server:Microsoft-IIS/7.5 X-AspNet-Version:4.0.30319 X-Powered-By:ASP.NET 

UPDATE 2

Turning Recycling still did not offer much but I have increased my MaxWorkerProcess to 8 and I now get less number of errors then before.

But on an average, out of 200 requests in one second, 2 to 10 requests fail.., and this happens on almost every alternate seconds.

UPDATE 3

Continuing 5% of requests failing with "The server committed a protocol violation. Section=ResponseStatusLine", I have another program that downloads content from the webserver which uses WebClient, and which gives this error 4-5 times a second, on an average I have 5% of requests failing. Is there anyway to trace WebClient failure?

Problems Redefined

Zero Byte File Received

IIS closes connection for some reason, on client side in WebConfig, I receive 0 bytes for the file which is not zero bytes, We do SHA1 hash check, this told us that in IIS web server, no error is recorded.

This was my mistake, and its resolved as we are using Entity Framework, it was reading dirty (uncommitted rows) as read was not in transaction scope, putting it in transaction scope has resolved this issue.

Protocol Violation Exception Raised

WebClient throws WebException saying "The server committed a protocol violation. Section=ResponseStatusLine.

I know I can enable unsafe header parsing but that is not the point, when it is my HTTP Handler that is sending proper headers, dont know why IIS is sending anything extra (checked on firefox and chrome, nothing unusual), this happens only 2% of times.

UPDATE 4

Found sc-win32 64 error and I read somewhere that WebLimits for MinBytesPerSecond must be changed from 240 to 0, still everything is same. However I have noticed that whenever IIS logs 64 sc-win32 error, IIS records HTTP Status as 200 but there was some error. Now I cant turn on Failed Trace Logging for 200 because it will result in massive files.

Both of above problems were solved by increasing MinBytesPerSecond and as well as disabling Sessions, I have added detailed answer summarizing every point.

Answer 1

Although correct way to deliver the big files in IIS is the following option,

1. Set MinBytesPerSecond to Zero in WebLimits (This will certainly help in improving performance, as IIS chooses to close clients holding KeepAlive connections with smaller size transfers)
2. Allocate More Worker Process to Application Pool, I have set to 8, now this should be done only if your server is distributing larger files. This will certainly cause other sites to perform slower, but this will ensure better deliveries. We have set to 8 as this server has only one website and it just delivers huge files.
3. Turn off App Pool Recycling
4. Turn off Sessions
5. Leave Buffering On
6. Before each of following steps, check if Response.IsClientConnected is true, else give up and dont send anything.
7. Set Content-Length before sending the file
8. Flush the Response
9. Write to Output Stream, and Flush in regular intervals