Uploading file to server throws out of memory exception

Question

I am trying to implement a file upload system with asp.net web api and I am running into a problem. I am trying to get the multipart form data into a memory stream so it can be written to either disk or blob storage depending on the service layer implementation. The problem is it works fine for small files but I am trying to upload a file of 291 MB and it is throwing an out of memory exception. Here is the code:

if (!Request.Content.IsMimeMultipartContent())
{
    Request.CreateErrorResponse(HttpStatusCode.UnsupportedMediaType, "Request must be multipart.");
}

var provider = new MultipartMemoryStreamProvider();

try
{
    await Request.Content.ReadAsMultipartAsync(provider);

    var infoPart = provider.Contents.Where(x => x.Headers.ContentDisposition.Name.Replace("\"", string.Empty) == "fileInfo").SingleOrDefault();
    var filePart = provider.Contents.Where(x => x.Headers.ContentDisposition.Name.Replace("\"", string.Empty) == "filePart" && x.Headers.ContentDisposition.FileName != null).Single();
    byte[] file = null;

    using (Stream stream = filePart.ReadAsStreamAsync().Result)
    {
        using (MemoryStream memory = new MemoryStream())
        {
            stream.CopyTo(memory);
            file = memory.ToArray();
        }
    }

    string fileContentType = filePart.Headers.ContentType.MediaType;

    FileDto result = _fileService.AddFileToResource(Variables);
    string uri = Url.Link("DefaultGet", new { id = result.ID });
    return Request.CreateResponse(HttpStatusCode.OK);

The part that throws the error is on the

await Request.Content.ReadAsMultipartAsync(provider);

The exact error is

Error writing MIME multipart body part to output stream.

with inner exception of

Exception of type 'System.OutOfMemoryException' was thrown.

I have tried creating a custom BufferPolicySelector as shown in the second answer of this post and many other places but that doesn't seem to help at all.

I have also added to my web.config:

<httpRuntime targetFramework="4.5" maxRequestLength="307200"/>

and

<security>
  <requestFiltering>
    <requestLimits maxAllowedContentLength="367001600"/>
  </requestFiltering>
</security>

Answer 1

One solution would be to use MultipartFormDataStreamProvider instead of the MultipartMemoryStreamProvider to avoid the out of memory exception during the call

Request.Content.ReadAsMultipartAsync(..)

I was facing a similar problem while trying to use a MemoryStreamProvider while reading the MultiPart file contents for a large file (> 100 MB). The work around that worked for me was to use MultipartFormDataStreamProvider. The file is written to the disk during the ReadAsMultipartAsync call and can be later loaded back in if you need it in memory.

Here is an example taken from:

Sending HTML Form Data in Web API: File Upload and Multipart MIME

    string root = HttpContext.Current.Server.MapPath("~/App_Data");
    var provider = new MultipartFormDataStreamProvider(root);

    try
    {
        // Read the form data.
        await Request.Content.ReadAsMultipartAsync(provider);

        // This illustrates how to get the file names.
        foreach (MultipartFileData file in provider.FileData)
        {
            Trace.WriteLine(file.Headers.ContentDisposition.FileName);
            Trace.WriteLine("Server file path: " + file.LocalFileName);
        }
        return Request.CreateResponse(HttpStatusCode.OK);
    }
    catch(...)