OutOfMemoryException when send big file 500MB using FileStream ASPNET

Question

I'm using Filestream for read big file (> 500 MB) and I get the OutOfMemoryException.

I use Asp.net , .net 3.5, win2003, iis 6.0

I want this in my app:

Read DATA from Oracle

Uncompress file using FileStream and BZip2

Read file uncompressed and send it to asp.net page for download.

When I read file from disk, Fails !!! and get OutOfMemory...

. My Code is:

using (var fs3 = new FileStream(filePath2, FileMode.Open, FileAccess.Read)) 
        { 
          byte[] b2 = ReadFully(fs3, 1024); 
        } 

 // http://www.yoda.arachsys.com/csharp/readbinary.html
 public static byte[] ReadFully(Stream stream, int initialLength) 
  { 
    // If we've been passed an unhelpful initial length, just 
    // use 32K. 
    if (initialLength < 1) 
    { 
      initialLength = 32768; 
    } 

    byte[] buffer = new byte[initialLength]; 
    int read = 0; 

    int chunk; 
    while ((chunk = stream.Read(buffer, read, buffer.Length - read)) > 0) 
    { 
      read += chunk; 

      // If we've reached the end of our buffer, check to see if there's 
      // any more information 
      if (read == buffer.Length) 
      { 
        int nextByte = stream.ReadByte(); 

        // End of stream? If so, we're done 
        if (nextByte == -1) 
        { 
          return buffer; 
        } 

        // Nope. Resize the buffer, put in the byte we've just 
        // read, and continue 
        byte[] newBuffer = new byte[buffer.Length * 2]; 
        Array.Copy(buffer, newBuffer, buffer.Length); 
        newBuffer[read] = (byte)nextByte; 
        buffer = newBuffer; 
        read++; 
      } 
    } 
    // Buffer is now too big. Shrink it. 
    byte[] ret = new byte[read]; 
    Array.Copy(buffer, ret, read); 
    return ret; 
  } 

Now, I specify my issue better.

Uncompress file using FileStream and BZip2 is OK, all is right.

The Problem is the following:

Read fat big file in disk (> 500 MB) in byte[] and send bytes to Response (asp.net) for download it.

When use

http://www.yoda.arachsys.com/csharp/readbinary.html

public static byte[] ReadFully

I get the error: OutOfMemoryException...

If better BufferedStream than Stream (FileStream, MemoryStream, ...) ??

Using BufferedStream , Can I read big file of 700 MB ?? (any sample code source using BufferedStream for download big file)

I think, this is the question: Not "how to read a 500mb file into memory?" , But "how to send a large file to the ASPNET Response stream?"

I found this code by Cheeso:

using (var fs = new FileStream(filePath, FileMode.Open, FileAccess.Read))  
{  
   Response.BufferOutput= false;   // to prevent buffering 
   byte[] buffer = new byte[1024]; 
   int bytesRead = 0; 
   while ((bytesRead = fs.Read(buffer, 0, buffer.Length)) > 0)  
   { 
       Response.OutputStream.Write(buffer, 0, bytesRead); 
   } 
}

Is it good code ?? any improvements for high performance ??

A collegue say me, use

Response.TransmitFile(filePath);

Now, another question, better TransmitFile or code by Cheeso ??

Many years ago, in msdn magazine appears great article about it but I cannot access http://msdn.microsoft.com/msdnmag/issues/06/09/WebDownloads/,

Update: You can access using webarchive in the link: https://web.archive.org/web/20070627063111/http://msdn.microsoft.com/msdnmag/issues/06/09/WebDownloads/

Any suggestions, comments, sample code source??

Answer 1

I came across this question in my search to return a FileStreamResult from a controller, as I kept running into issues while working with large streams due to .Net trying to build the entire response all at once. Pavel Morshenyuk's answer was a huge help, but I figured I'd share the BufferedFileStreamResult that I ended up with.

/// <summary>Based upon https://stackoverflow.com/a/3363015/595473 </summary>
public class BufferedFileStreamResult : System.Web.Mvc.FileStreamResult
{
    public BufferedFileStreamResult(System.IO.Stream stream, string contentType, string fileDownloadName)
        : base(stream, contentType)
    {
        FileDownloadName = fileDownloadName;
    }

    public int BufferSize { get; set; } = 16 * 1024 * 1024;//--16MiB

    protected override void WriteFile(System.Web.HttpResponseBase response)
    {
        try
        {
            response.Clear();
            response.Headers.Set("Content-Disposition", $"attachment; filename={FileDownloadName}");
            response.Headers.Set("Content-Length", FileStream.Length.ToString());

            byte[] buffer;
            int bytesRead;

            while (response.IsClientConnected)//--Prevent infinite loop if user disconnects
            {
                buffer = new byte[BufferSize];

                //--Read the data in buffer
                if ((bytesRead = FileStream.Read(buffer, 0, BufferSize)) == 0)
                {
                    break;//--Stop writing if there's nothing left to write
                }

                //--Write the data to the current output stream
                response.OutputStream.Write(buffer, 0, bytesRead);

                //--Flush the data to the output
                response.Flush();
            }
        }
        finally
        {
            FileStream?.Close();
            response.Close();
        }
    }
}

Now, in my controller, I can just

return new BufferedFileStreamResult(stream, contentType, fileDownloadName);

Answer 2

I've created download page which allows user to download up to 4gb (may be more) few months ago. Here is my working snippet:

  private void TransmitFile(string fullPath, string outFileName)
    {
        System.IO.Stream iStream = null;

        // Buffer to read 10K bytes in chunk:
        byte[] buffer = new Byte[10000];

        // Length of the file:
        int length;

        // Total bytes to read:
        long dataToRead;

        // Identify the file to download including its path.
        string filepath = fullPath;

        // Identify the file name.
        string filename = System.IO.Path.GetFileName(filepath);

        try
        {
            // Open the file.
            iStream = new System.IO.FileStream(filepath, System.IO.FileMode.Open,
                        System.IO.FileAccess.Read, System.IO.FileShare.Read);


            // Total bytes to read:
            dataToRead = iStream.Length;

            Response.Clear();
            Response.ContentType = "application/octet-stream";
            Response.AddHeader("Content-Disposition", "attachment; filename=" + outFileName);
            Response.AddHeader("Content-Length", iStream.Length.ToString());

            // Read the bytes.
            while (dataToRead > 0)
            {
                // Verify that the client is connected.
                if (Response.IsClientConnected)
                {
                    // Read the data in buffer.
                    length = iStream.Read(buffer, 0, 10000);

                    // Write the data to the current output stream.
                    Response.OutputStream.Write(buffer, 0, length);

                    // Flush the data to the output.
                    Response.Flush();

                    buffer = new Byte[10000];
                    dataToRead = dataToRead - length;
                }
                else
                {
                    //prevent infinite loop if user disconnects
                    dataToRead = -1;
                }
            }
        }
        catch (Exception ex)
        {
            throw new ApplicationException(ex.Message);
        }
        finally
        {
            if (iStream != null)
            {
                //Close the file.
                iStream.Close();
            }
            Response.Close();
        }
    }

Answer 3

You do not need to hold the whole file in memory just read it and write to the response stream in a loop.