Logo Questions Linux Laravel Mysql Ubuntu Git Menu
 

Dealing with large file uploads on ASP.NET Core 1.0

When I'm uploading large files to my web api in ASP.NET Core, the runtime will load the file into memory before my function for processing and storing the upload is fired. With large uploads this becomes an issue as it is both slow and requires more memory. For previous versions of ASP.NET there are some articles on how to disable the buffering of requests, but I'm not able to find any information on how to do this with ASP.NET Core. Is it possible to disable the buffering of requests so I don't run out of memory on my server all the time?

like image 474
Martin Avatar asked Apr 05 '16 21:04

Martin


People also ask

How do I handle a large file upload?

Possible solutions: 1) Configure maximum upload file size and memory limits for your server. 2) Upload large files in chunks. 3) Apply resumable file uploads. Chunking is the most commonly used method to avoid errors and increase speed.

What is difference between FromForm and FromBody?

[FromForm] - Gets values from posted form fields. [FromBody] - Gets values from the request body.

What is MultipartReader?

MultipartReader because it... can parse any stream [with] minimal buffering. It gives you the headers and body of each section one at a time and then you do what you want with the body of that section (buffer, discard, write to disk, etc.).


1 Answers

Use the Microsoft.AspNetCore.WebUtilities.MultipartReader because it...

can parse any stream [with] minimal buffering. It gives you the headers and body of each section one at a time and then you do what you want with the body of that section (buffer, discard, write to disk, etc.).

Here is a middleware example.

app.Use(async (context, next) => {     if (!IsMultipartContentType(context.Request.ContentType))     {         await next();         return;     }      var boundary = GetBoundary(context.Request.ContentType);     var reader = new MultipartReader(boundary, context.Request.Body);     var section = await reader.ReadNextSectionAsync();      while (section != null)     {         // process each image         const int chunkSize = 1024;         var buffer = new byte[chunkSize];         var bytesRead = 0;         var fileName = GetFileName(section.ContentDisposition);          using (var stream = new FileStream(fileName, FileMode.Append))         {             do             {                 bytesRead = await section.Body.ReadAsync(buffer, 0, buffer.Length);                 stream.Write(buffer, 0, bytesRead);              } while (bytesRead > 0);         }          section = await reader.ReadNextSectionAsync();     }      context.Response.WriteAsync("Done."); }); 

Here are the helpers.

private static bool IsMultipartContentType(string contentType) {     return          !string.IsNullOrEmpty(contentType) &&         contentType.IndexOf("multipart/", StringComparison.OrdinalIgnoreCase) >= 0; }  private static string GetBoundary(string contentType) {     var elements = contentType.Split(' ');     var element = elements.Where(entry => entry.StartsWith("boundary=")).First();     var boundary = element.Substring("boundary=".Length);     // Remove quotes     if (boundary.Length >= 2 && boundary[0] == '"' &&          boundary[boundary.Length - 1] == '"')     {         boundary = boundary.Substring(1, boundary.Length - 2);     }     return boundary; }  private string GetFileName(string contentDisposition) {     return contentDisposition         .Split(';')         .SingleOrDefault(part => part.Contains("filename"))         .Split('=')         .Last()         .Trim('"'); } 

External References

  • https://github.com/aspnet/HttpAbstractions/pull/146
  • https://github.com/aspnet/HttpAbstractions
like image 72
Shaun Luttin Avatar answered Sep 28 '22 10:09

Shaun Luttin