Large File upload to ASP.NET Core 3.0 Web API fails due to Request Body to Large

Question

I have an ASP.NET Core 3.0 Web API endpoint that I have set up to allow me to post large audio files. I have followed the following directions from MS docs to set up the endpoint.

https://learn.microsoft.com/en-us/aspnet/core/mvc/models/file-uploads?view=aspnetcore-3.0#kestrel-maximum-request-body-size

When an audio file is uploaded to the endpoint, it is streamed to an Azure Blob Storage container.

My code works as expected locally.

When I push it to my production server in Azure App Service on Linux, the code does not work and errors with

Unhandled exception in request pipeline: System.Net.Http.HttpRequestException: An error occurred while sending the request. ---> Microsoft.AspNetCore.Server.Kestrel.Core.BadHttpRequestException: Request body too large.

Per advice from the above article, I have configured incrementally updated Kesterl with the following:

.ConfigureWebHostDefaults(webBuilder =>
{
     webBuilder.UseKestrel((ctx, options) =>
     {
        var config = ctx.Configuration;

        options.Limits.MaxRequestBodySize = 6000000000;
        options.Limits.MinRequestBodyDataRate =
            new MinDataRate(bytesPerSecond: 100,
                gracePeriod: TimeSpan.FromSeconds(10));
        options.Limits.MinResponseDataRate =
            new MinDataRate(bytesPerSecond: 100,
                gracePeriod: TimeSpan.FromSeconds(10));
        options.Limits.RequestHeadersTimeout =
                TimeSpan.FromMinutes(2);
}).UseStartup<Startup>();

Also configured FormOptions to accept files up to 6000000000

services.Configure<FormOptions>(options =>
{
   options.MultipartBodyLengthLimit = 6000000000;
});

And also set up the API controller with the following attributes, per advice from the article

[HttpPost("audio", Name="UploadAudio")]
[DisableFormValueModelBinding]
[GenerateAntiforgeryTokenCookie]
[RequestSizeLimit(6000000000)]
[RequestFormLimits(MultipartBodyLengthLimit = 6000000000)]

Finally, here is the action itself. This giant block of code is not indicative of how I want the code to be written but I have merged it into one method as part of the debugging exercise.

public async Task<IActionResult> Audio()
{
    if (!MultipartRequestHelper.IsMultipartContentType(Request.ContentType))
    {
        throw new ArgumentException("The media file could not be processed.");
    }

    string mediaId = string.Empty;
    string instructorId = string.Empty;
    try
    {
        // process file first
        KeyValueAccumulator formAccumulator = new KeyValueAccumulator();

        var streamedFileContent = new byte[0];

        var boundary = MultipartRequestHelper.GetBoundary(
            MediaTypeHeaderValue.Parse(Request.ContentType),
            _defaultFormOptions.MultipartBoundaryLengthLimit
            );
        var reader = new MultipartReader(boundary, Request.Body);
        var section = await reader.ReadNextSectionAsync();

        while (section != null)
        {
            var hasContentDispositionHeader = ContentDispositionHeaderValue.TryParse(
                section.ContentDisposition, out var contentDisposition);

            if (hasContentDispositionHeader)
            {
                if (MultipartRequestHelper
                    .HasFileContentDisposition(contentDisposition))
                {
                    streamedFileContent =
                        await FileHelpers.ProcessStreamedFile(section, contentDisposition,
                            _permittedExtensions, _fileSizeLimit);

                }
                else if (MultipartRequestHelper
                    .HasFormDataContentDisposition(contentDisposition))
                {
                    var key = HeaderUtilities.RemoveQuotes(contentDisposition.Name).Value;
                    var encoding = FileHelpers.GetEncoding(section);

                    if (encoding == null)
                    {
                        return BadRequest($"The request could not be processed: Bad Encoding");
                    }

                    using (var streamReader = new StreamReader(
                        section.Body,
                        encoding,
                        detectEncodingFromByteOrderMarks: true,
                        bufferSize: 1024,
                        leaveOpen: true))
                    {
                        // The value length limit is enforced by 
                        // MultipartBodyLengthLimit
                        var value = await streamReader.ReadToEndAsync();

                        if (string.Equals(value, "undefined",
                            StringComparison.OrdinalIgnoreCase))
                        {
                            value = string.Empty;
                        }

                        formAccumulator.Append(key, value);

                        if (formAccumulator.ValueCount >
                            _defaultFormOptions.ValueCountLimit)
                        {
                            return BadRequest($"The request could not be processed: Key Count limit exceeded.");
                        }
                    }
                }
            }

            // Drain any remaining section body that hasn't been consumed and
            // read the headers for the next section.
            section = await reader.ReadNextSectionAsync();
        }

        var form = formAccumulator;
        var file = streamedFileContent;

        var results = form.GetResults();

        instructorId = results["instructorId"];
        string title = results["title"];
        string firstName = results["firstName"];
        string lastName = results["lastName"];
        string durationInMinutes = results["durationInMinutes"];

        //mediaId = await AddInstructorAudioMedia(instructorId, firstName, lastName, title, Convert.ToInt32(duration), DateTime.UtcNow, DateTime.UtcNow, file);

        string fileExtension = "m4a";

        // Generate Container Name - InstructorSpecific
        string containerName = $"{firstName[0].ToString().ToLower()}{lastName.ToLower()}-{instructorId}";

        string contentType = "audio/mp4";
        FileType fileType = FileType.audio;

        string authorName = $"{firstName} {lastName}";
        string authorShortName = $"{firstName[0]}{lastName}";
        string description = $"{authorShortName} - {title}";

        long duration = (Convert.ToInt32(durationInMinutes) * 60000);

        // Generate new filename
        string fileName = $"{firstName[0].ToString().ToLower()}{lastName.ToLower()}-{Guid.NewGuid()}";

        DateTime recordingDate = DateTime.UtcNow;
        DateTime uploadDate = DateTime.UtcNow;
        long blobSize = long.MinValue;
        try
        {
            // Update file properties in storage
            Dictionary<string, string> fileProperties = new Dictionary<string, string>();
            fileProperties.Add("ContentType", contentType);

            // update file metadata in storage
            Dictionary<string, string> metadata = new Dictionary<string, string>();
            metadata.Add("author", authorShortName);
            metadata.Add("tite", title);
            metadata.Add("description", description);
            metadata.Add("duration", duration.ToString());
            metadata.Add("recordingDate", recordingDate.ToString());
            metadata.Add("uploadDate", uploadDate.ToString());

            var fileNameWExt = $"{fileName}.{fileExtension}";

            var blobContainer = await _cloudStorageService.CreateBlob(containerName, fileNameWExt, "audio");

            try
            {
                MemoryStream fileContent = new MemoryStream(streamedFileContent);
                fileContent.Position = 0;
                using (fileContent)
                {
                    await blobContainer.UploadFromStreamAsync(fileContent);
                }

            }
            catch (StorageException e)
            {
                if (e.RequestInformation.HttpStatusCode == 403)
                {
                    return BadRequest(e.Message);
                }
                else
                {
                    return BadRequest(e.Message);
                }
            }

            try
            {
                foreach (var key in metadata.Keys.ToList())
                {
                    blobContainer.Metadata.Add(key, metadata[key]);
                }

                await blobContainer.SetMetadataAsync();

            }
            catch (StorageException e)
            {
                return BadRequest(e.Message);
            }

            blobSize = await StorageUtils.GetBlobSize(blobContainer);
        }
        catch (StorageException e)
        {
            return BadRequest(e.Message);
        }

        Media media = Media.Create(string.Empty, instructorId, authorName, fileName, fileType, fileExtension, recordingDate, uploadDate, ContentDetails.Create(title, description, duration, blobSize, 0, new List<string>()), StateDetails.Create(StatusType.STAGED, DateTime.MinValue, DateTime.UtcNow, DateTime.MaxValue), Manifest.Create(new Dictionary<string, string>()));

        // upload to MongoDB
        if (media != null)
        {
            var mapper = new Mapper(_mapperConfiguration);

            var dao = mapper.Map<ContentDAO>(media);

            try
            {
                await _db.Content.InsertOneAsync(dao);
            }
            catch (Exception)
            {
                mediaId = string.Empty;
            }

            mediaId = dao.Id.ToString();
        }
        else
        {
            // metadata wasn't stored, remove blob
            await _cloudStorageService.DeleteBlob(containerName, fileName, "audio");
            return BadRequest($"An issue occurred during media upload: rolling back storage change");
        }

        if (string.IsNullOrEmpty(mediaId))
        {
            return BadRequest($"Could not add instructor media");
        }

    }
    catch (Exception ex)
    {
        return BadRequest(ex.Message);
    }

    var result = new { MediaId = mediaId, InstructorId = instructorId };

    return Ok(result);
}

I reiterate, this all works great locally. I do not run it in IISExpress, I run it as a console app.

I submit large audio files via my SPA app and Postman and it works perfectly.

I am deploying this code to an Azure App Service on Linux (as a Basic B1).

Since the code works in my local development environment, I am at a loss of what my next steps are. I have refactored this code a few times but I suspect that it's environment related.

I cannot find anywhere that mentions that the level of App Service Plan is the culprit so before I go out spending more money I wanted to see if anyone here had encountered this challenge and could provide advice.

UPDATE: I attempted upgrading to a Production App Service Plan to see if there was an undocumented gate for incoming traffic. Upgrading didn't work either.

Thanks in advance.

-A

Answer 1

Currently, as of 11/2019, there is a limitation with the Azure App Service for Linux. It's CORS functionality is enabled by default and cannot be disabled AND it has a file size limitation that doesn't appear to get overridden by any of the published Kestrel configurations. The solution is to move the Web API app to a Azure App Service for Windows and it works as expected.

I am sure there is some way to get around it if you know the magic combination of configurations, server settings, and CLI commands but I need to move on with development.