InvalidDataException: Multipart body length limit 16384 exceeded

Question

What I am trying to do:

I am attempting to upload a multipart/form-data with a file and a JSON blob using Postman to an ASP.NET Core 2.2 APIController and stream the file to a temporary file on the disk instead of fully into memory due to the files likely being potentially large in size (20MB - 2GB). I've followed both examples from https://docs.microsoft.com/en-us/aspnet/core/mvc/models/file-uploads?view=aspnetcore-2.2 starting with the large file example, but I've also tried testing the small file example with the same error, similar but different stack traces. The server uses Kestrel.

Large file example's stack-trace (Caught in debugger):

Exception has occurred: CLR/System.IO.InvalidDataException
Exception thrown: 'System.IO.InvalidDataException' in System.Private.CoreLib.dll: 'Multipart body length limit 16384 exceeded.'
   at Microsoft.AspNetCore.WebUtilities.MultipartReaderStream.UpdatePosition(Int32 read)
   at Microsoft.AspNetCore.WebUtilities.MultipartReaderStream.<ReadAsync>d__36.MoveNext()
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter`1.GetResult()
   at Microsoft.AspNetCore.WebUtilities.StreamHelperExtensions.<DrainAsync>d__3.MoveNext()
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.AspNetCore.WebUtilities.MultipartReader.<ReadNextSectionAsync>d__20.MoveNext()
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter`1.GetResult()
   at LookupServiceAPI.Helpers.FileStreamingHelper.<StreamFile>d__1.MoveNext() in <hidden-path-to-project>\Helpers\FileStreamingHelper.cs:line 35

Small file example's stack-trace (returned in response, doesn't hit any breakpoints or debugger exception catches):

System.IO.InvalidDataException: Multipart body length limit 16384 exceeded.
   at Microsoft.AspNetCore.WebUtilities.MultipartReaderStream.UpdatePosition(Int32 read)
   at Microsoft.AspNetCore.WebUtilities.MultipartReaderStream.ReadAsync(Byte[] buffer, Int32 offset, Int32 count, CancellationToken cancellationToken)
   at Microsoft.AspNetCore.WebUtilities.StreamHelperExtensions.DrainAsync(Stream stream, ArrayPool`1 bytePool, Nullable`1 limit, CancellationToken cancellationToken)
   at Microsoft.AspNetCore.WebUtilities.MultipartReader.ReadNextSectionAsync(CancellationToken cancellationToken)
   at Microsoft.AspNetCore.Http.Features.FormFeature.InnerReadFormAsync(CancellationToken cancellationToken)
   at Microsoft.AspNetCore.Mvc.ModelBinding.FormValueProviderFactory.AddValueProviderAsync(ValueProviderFactoryContext context)
   at Microsoft.AspNetCore.Mvc.ModelBinding.CompositeValueProvider.CreateAsync(ActionContext actionContext, IList`1 factories)
   at Microsoft.AspNetCore.Mvc.ModelBinding.CompositeValueProvider.CreateAsync(ControllerContext controllerContext)
   at Microsoft.AspNetCore.Mvc.Internal.ControllerBinderDelegateProvider.<>c__DisplayClass0_0.<<CreateBinderDelegate>g__Bind|0>d.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at Microsoft.AspNetCore.Mvc.Internal.ControllerActionInvoker.InvokeInnerFilterAsync()
   at Microsoft.AspNetCore.Mvc.Internal.ResourceInvoker.InvokeNextResourceFilter()
   at Microsoft.AspNetCore.Mvc.Internal.ResourceInvoker.Rethrow(ResourceExecutedContext context)
   at Microsoft.AspNetCore.Mvc.Internal.ResourceInvoker.Next(State& next, Scope& scope, Object& state, Boolean& isCompleted)
   at Microsoft.AspNetCore.Mvc.Internal.ResourceInvoker.InvokeFilterPipelineAsync()
   at Microsoft.AspNetCore.Mvc.Internal.ResourceInvoker.InvokeAsync()
   at Microsoft.AspNetCore.Routing.EndpointMiddleware.Invoke(HttpContext httpContext)
   at Microsoft.AspNetCore.Routing.EndpointRoutingMiddleware.Invoke(HttpContext httpContext)
   at Microsoft.AspNetCore.Diagnostics.DeveloperExceptionPageMiddleware.Invoke(HttpContext context)

Here is my basic controller code and helper classes for the large file example:

FileStreamingHelper.cs

using System;
using System.Globalization;
using System.IO;
using System.Text;
using System.Threading.Tasks;
using Microsoft.AspNetCore.Http;
using Microsoft.AspNetCore.Http.Features;
using Microsoft.AspNetCore.Http.Extensions;
using Microsoft.AspNetCore.Mvc.ModelBinding;
using Microsoft.AspNetCore.WebUtilities;
using Microsoft.Net.Http.Headers;

namespace LookupServiceAPI.Helpers
{
    public static class FileStreamingHelper
    {
        private static readonly FormOptions _defaultFormOptions = new FormOptions();

        public static async Task<FormValueProvider> StreamFile(this HttpRequest request, Stream targetStream)
        {
            if (!MultipartRequestHelper.IsMultipartContentType(request.ContentType))
            {
                throw new Exception($"Expected a multipart request, but got {request.ContentType}");
            }

            // Used to accumulate all the form url encoded key value pairs in the 
            // request.
            var formAccumulator = new KeyValueAccumulator();

            var boundary = request.GetMultipartBoundary();
            var reader = new MultipartReader(boundary, request.Body);
            reader.BodyLengthLimit = Int32.MaxValue;
            reader.HeadersLengthLimit = Int32.MaxValue;
            var section = await reader.ReadNextSectionAsync(); //EXCEPTION HERE
            while (section != null)
            {
                ContentDispositionHeaderValue contentDisposition;
                var hasContentDispositionHeader = ContentDispositionHeaderValue.TryParse(section.ContentDisposition, out contentDisposition);

                if (hasContentDispositionHeader)
                {
                    if (MultipartRequestHelper.HasFileContentDisposition(contentDisposition))
                    {
                        await section.Body.CopyToAsync(targetStream);
                    }
                    else if (MultipartRequestHelper.HasFormDataContentDisposition(contentDisposition))
                    {
                        // Content-Disposition: form-data; name="key"
                        //
                        // value

                        // Do not limit the key name length here because the 
                        // multipart headers length limit is already in effect.
                        var key = HeaderUtilities.RemoveQuotes(contentDisposition.Name);
                        var encoding = GetEncoding(section);
                        using (var streamReader = new StreamReader(
                            section.Body,
                            encoding,
                            detectEncodingFromByteOrderMarks: true,
                            bufferSize: 1024,
                            leaveOpen: true))
                        {
                            // The value length limit is enforced by MultipartBodyLengthLimit
                            var value = await streamReader.ReadToEndAsync();
                            if (String.Equals(value, "undefined", StringComparison.OrdinalIgnoreCase))
                            {
                                value = String.Empty;
                            }
                            formAccumulator.Append(key.Value, value); // For .NET Core <2.0 remove ".Value" from key

                            if (formAccumulator.ValueCount > _defaultFormOptions.ValueCountLimit)
                            {
                                throw new InvalidDataException($"Form key count limit {_defaultFormOptions.ValueCountLimit} exceeded.");
                            }
                        }
                    }
                }

                // Drains any remaining section body that has not been consumed and
                // reads the headers for the next section.
                section = await reader.ReadNextSectionAsync();
            }

            // Bind form data to a model
            var formValueProvider = new FormValueProvider(
                BindingSource.Form,
                new FormCollection(formAccumulator.GetResults()),
                CultureInfo.CurrentCulture);

            return formValueProvider;
        }

        private static Encoding GetEncoding(MultipartSection section)
        {
            MediaTypeHeaderValue mediaType;
            var hasMediaTypeHeader = MediaTypeHeaderValue.TryParse(section.ContentType, out mediaType);
            // UTF-7 is insecure and should not be honored. UTF-8 will succeed in 
            // most cases.
            if (!hasMediaTypeHeader || Encoding.UTF7.Equals(mediaType.Encoding) || mediaType.Encoding == null)
            {
                return Encoding.UTF8;
            }
            return mediaType.Encoding;
        }
    }
}

MultipartRequestHelper.cs

using System;
using System.IO;
using Microsoft.Net.Http.Headers;

namespace LookupServiceAPI.Helpers
{
    public static class MultipartRequestHelper
    {
        public static bool IsMultipartContentType(string contentType)
        {
            return !string.IsNullOrEmpty(contentType)
                   && contentType.IndexOf("multipart/", StringComparison.OrdinalIgnoreCase) >= 0;
        }

        public static bool HasFormDataContentDisposition(ContentDispositionHeaderValue contentDisposition)
        {
            // Content-Disposition: form-data; name="key";
            return contentDisposition != null
                   && contentDisposition.DispositionType.Equals("form-data")
                   && string.IsNullOrEmpty(contentDisposition.FileName.Value)
                   && string.IsNullOrEmpty(contentDisposition.FileNameStar.Value);
        }

        public static bool HasFileContentDisposition(ContentDispositionHeaderValue contentDisposition)
        {
            // Content-Disposition: form-data; name="myfile1"; filename="Misc 002.jpg"
            return contentDisposition != null
                   && contentDisposition.DispositionType.Equals("form-data")
                   && (!string.IsNullOrEmpty(contentDisposition.FileName.ToString())
                       || !string.IsNullOrEmpty(contentDisposition.FileNameStar.ToString()));
        }
    }
}

Minimal Controller:

[Route("api/v0.1/data/excel")]
[ApiController]
public class DataExcelController : ControllerBase
{
    [HttpPost, DisableRequestSizeLimit]
    public async Task<IActionResult> ImportExcel()
    {
        var processID = Guid.NewGuid();
        FormValueProvider multipartContent;
        string tempFilePath = Path.GetTempPath() + processID;
        using(var tempStream = System.IO.File.OpenWrite(tempFilePath))
        {
            multipartContent = await Request.StreamFile(tempStream);
        }
        /** Other unnecessary code **/
        return Ok();
    }
}

Startup.cs

namespace LookupServiceAPI
{
    public class Startup
    {
        public Startup(IConfiguration configuration)
        {
            Configuration = configuration;
        }

        public IConfiguration Configuration { get; }

        // This method gets called by the runtime. Use this method to add services to the container.
        public void ConfigureServices(IServiceCollection services)
        {
            services.AddMvc().SetCompatibilityVersion(CompatibilityVersion.Version_2_2);

            services.Configure<FormOptions>(x =>
            {
                x.MultipartHeadersLengthLimit = Int32.MaxValue;
                x.MultipartBoundaryLengthLimit = Int32.MaxValue;
                x.MultipartBodyLengthLimit = Int64.MaxValue;
                x.ValueLengthLimit = Int32.MaxValue;
                x.BufferBodyLengthLimit = Int64.MaxValue;
                x.MemoryBufferThreshold = Int32.MaxValue;
            });
        }

        // This method gets called by the runtime. Use this method to configure the HTTP request pipeline.
        public void Configure(IApplicationBuilder app, IHostingEnvironment env)
        {
            if (env.IsDevelopment())
            {
                app.UseDeveloperExceptionPage();
            }
            else
            {
                // The default HSTS value is 30 days. You may want to change this for production scenarios, see https://aka.ms/aspnetcore-hsts.
                app.UseHsts();
            }

            app.UseHttpsRedirection();
            app.UseMvc();
        }
    }
}

Postman configuration image (only values set are in the image, no values set in Headers):

Postman Console output:

Things I've tried:

From Multipart body length limit 16384 exceeded:

• Set MemoryBufferThreshold
• Set MultipartBodyLengthLimit
• Ensured the Postman configured Headers does not have Content-Type manually set to multipart/form-data

From Multipart body length limit exceeded exception:

• Set ValueLengthLimit
• Used [DisableRequestSizeLimit]

Where I think the problem is, but I'm unsure of a workaround or what's causing the issue: https://github.com/aspnet/AspNetCore/blob/master/src/Http/WebUtilities/src/MultipartReader.cs#L48-L50

It seems the preamble for my request is beyond the size limit of 1024 * 16 (16384) set for DefaultHeadersLengthLimit, but I have no clue why that could be the case. Or, if the preamble should be greater than that, how to work around it without re-implementing the whole set of classes or waiting for Microsoft to put out a fix which doesn't seem to be coming down the pipeline: https://github.com/aspnet/Mvc/issues/7019 https://github.com/aspnet/HttpAbstractions/issues/736

Apparently someone fixed their issue which was very similar to mine (https://github.com/aspnet/Mvc/issues/5128#issuecomment-307675219) here: https://github.com/aspnet/Mvc/issues/5128#issuecomment-307962922 but I can't seem to figure out how to figure out if it even applies.

Hopefully this is enough information. If not, please let me know what you need and I'd be happy to provide it or test out any suggestions. I've been stuck researching this and trying everything I can find for 6+ hours now.

Answer 1

I have similar issue and i found problem is related to debug breakpoint in visual studio , because i think visual studio try to read the stream and after that try it break stream,

so try to skip view Request.From in Watch or debug quick view

Answer 2

I've resolved my question. Turns out it was the URL I was using.

To fix my issue, I realized that I was sending to the http endpoint instead of the https endpoint, causing a redirect. I changed my url from http://localhost:5000/ to https://localhost:5001/ and everything started working immediately.

Interestingly, this caused an issue in cURL as well with a log looking like this:

== Info: Connected to localhost (::1) port 5000 (#0)
=> Send header, 257 bytes (0x101)
0000: POST /api/v0.1/data/excel HTTP/1.1
0024: Host: localhost:5000
003a: User-Agent: curl/7.64.0
0053: Accept: */*
0060: cache-control: no-cache
0079: Content-Length: 13286446
0093: Content-Type: multipart/form-data; boundary=--------------------
00d3: ----7b12fc7773ed7878
00e9: Expect: 100-continue
00ff: 
== Info: Expire in 1000 ms for 0 (transfer 0xa6aa80)
<= Recv header, 33 bytes (0x21)
0000: HTTP/1.1 307 Temporary Redirect
<= Recv header, 37 bytes (0x25)
0000: Date: Tue, 09 Apr 2019 18:04:24 GMT
<= Recv header, 17 bytes (0x11)
0000: Server: Kestrel
<= Recv header, 19 bytes (0x13)
0000: Content-Length: 0
<= Recv header, 54 bytes (0x36)
0000: Location: https://localhost:5001/api/v0.1/data/excel
== Info: HTTP error before end of send, stop sending
<= Recv header, 2 bytes (0x2)
0000: 
== Info: Closing connection 0

Changing the endpoint fixed that as well.

No idea why a multipart/form-data upload breaks due to that redirect. If anyone has any ideas why, I'd be happy to learn.