UPDATE [2] - ASP.NET Core Web API upload and download file - Stream exception

Question

This question is related to: ASP.NET Core Web API upload and download file

First, I would like to thank Powel Gerr who helped me to understand some nuances with his post (http://weblogs.thinktecture.com/pawel/2017/03/aspnet-core-webapi-performance.html). I still have a problem to solve.

My scenario Suppose we have a .NET Core console application:

private static void Main(string[] args)
{
    Console.WriteLine($"Test starts at {DateTime.Now.ToString("o")}");
    FileStream fileStream = new FileStream(@"C:\Windows10Upgrade\Windows10UpgraderApp.exe", FileMode.Open);

    MyFile vFile = new MyFile()
    {
        Lenght = 0,
        Path = "https://c2calrsbackup.blob.core.windows.net/containername/Windows10UpgraderApp.exe",
        RelativePath = "Windows10UpgraderApp.exe"
    };
    Stream uploadStream = GetUploadStream(vFile).GetAwaiter().GetResult();

    fileStream.CopyTo(uploadStream);

    Console.WriteLine($"Test ends at {DateTime.Now.ToString("o")}");
    Console.Write("Press any key to exit...");
    Console.ReadKey();
}

private static async Task<Stream> GetUploadStream(MyFile myFile)
{
    Stream stream = null;

    try
    {
        using (HttpClient httpClient = new HttpClient())
        {
            httpClient.BaseAddress = new Uri("https://localhost:5000");
            using (MultipartFormDataContent multipartFormDataContent = new MultipartFormDataContent())
            {
                multipartFormDataContent.Add(new StringContent(JsonConvert.SerializeObject(myFile), Encoding.UTF8, "application/json"), nameof(MyFile));

                HttpResponseMessage httpResult = await httpClient.PostAsync("api/uploaddownload/upload", multipartFormDataContent).ConfigureAwait(false);

                httpResult.EnsureSuccessStatusCode();
                stream = await httpResult.Content.ReadAsStreamAsync().ConfigureAwait(false);
            }
        }
    }
    catch (Exception e)
    {
        Console.WriteLine(e.Message);
    }
    return stream;
}

As you can see, MyFile is a support class that contains some information. On the other hand, the controller is as follows:

[HttpPost("upload")]
public async Task<IActionResult> GetUploadStream()
{
    const string contentType = "application/octet-stream";
    string boundary = GetBoundary(Request.ContentType);
    MultipartReader reader = new MultipartReader(boundary, Request.Body, 80 * 1024);
    Dictionary<string, string> sectionDictionary = new Dictionary<string, string>();
    FileMultipartSection fileMultipartSection;
    MultipartSection section;

    while ((section = await reader.ReadNextSectionAsync().ConfigureAwait(false)) != null)
    {
        ContentDispositionHeaderValue contentDispositionHeaderValue = section.GetContentDispositionHeader();

        if (contentDispositionHeaderValue.IsFormDisposition())
        {
            FormMultipartSection formMultipartSection = section.AsFormDataSection();
            string value = await formMultipartSection.GetValueAsync().ConfigureAwait(false);

            sectionDictionary.Add(formMultipartSection.Name, value);
        }
        else if (contentDispositionHeaderValue.IsFileDisposition())
        {
            fileMultipartSection = section.AsFileSection();
        }
    }

    CloudStorageAccount.TryParse(connectionString, out CloudStorageAccount cloudStorageAccount);
    CloudBlobClient cloudBlobClient = cloudStorageAccount.CreateCloudBlobClient();
    CloudBlobContainer cloudBlobContainer = cloudBlobClient.GetContainerReference(containerName);

    if (await cloudBlobContainer.CreateIfNotExistsAsync().ConfigureAwait(false))
    {
        BlobContainerPermissions blobContainerPermission = new BlobContainerPermissions()
        {
            PublicAccess = BlobContainerPublicAccessType.Container
        };

        await cloudBlobContainer.SetPermissionsAsync(blobContainerPermission).ConfigureAwait(false);
    }

    MyFile myFile = JsonConvert.DeserializeObject<MyFile>(sectionDictionary.GetValueOrDefault(nameof(MyFile)));
    CloudBlockBlob cloudBlockBlob = cloudBlobContainer.GetBlockBlobReference(myFile.RelativePath);
    Stream streamResult = await cloudBlockBlob.OpenWriteAsync().ConfigureAwait(false);

    return new FileStreamResult(streamResult, contentType);
}

The problem

When the controller returns with the instruction return new FileStreamResult(streamResult, contentType); the following exception is generated in the controller itself (not in the calling console app):

Microsoft.AspNetCore.Diagnostics.DeveloperExceptionPageMiddleware:Error: An unhandled exception has occurred while executing the request.

System.NotSupportedException: Stream does not support reading. at System.IO.Stream.BeginReadInternal(Byte[] buffer, Int32 offset, Int32 count, AsyncCallback callback, Object state, Boolean serializeAsynchronously, Boolean apm) at System.IO.Stream.BeginEndReadAsync(Byte[] buffer, Int32 offset, Int32 count) at System.IO.Stream.ReadAsync(Byte[] buffer, Int32 offset, Int32 count, CancellationToken cancellationToken) at Microsoft.AspNetCore.Http.Extensions.StreamCopyOperation.CopyToAsync(Stream source, Stream destination, Nullable`1 count, Int32 bufferSize, CancellationToken cancel) at Microsoft.AspNetCore.Mvc.Infrastructure.FileResultExecutorBase.WriteFileAsync(HttpContext context, Stream fileStream, RangeItemHeaderValue range, Int64 rangeLength) at Microsoft.AspNetCore.Mvc.Infrastructure.FileStreamResultExecutor.ExecuteAsync(ActionContext context, FileStreamResult result) at Microsoft.AspNetCore.Mvc.Internal.ResourceInvoker.InvokeResultAsync(IActionResult result) at Microsoft.AspNetCore.Mvc.Internal.ResourceInvoker.InvokeNextResultFilterAsyncTFilter,TFilterAsync at Microsoft.AspNetCore.Mvc.Internal.ResourceInvoker.Rethrow(ResultExecutedContext context) at Microsoft.AspNetCore.Mvc.Internal.ResourceInvoker.ResultNext[TFilter,TFilterAsync](State& next, Scope& scope, Object& state, Boolean& isCompleted) at Microsoft.AspNetCore.Mvc.Internal.ResourceInvoker.InvokeResultFilters() at Microsoft.AspNetCore.Mvc.Internal.ResourceInvoker.InvokeNextResourceFilter() at Microsoft.AspNetCore.Mvc.Internal.ResourceInvoker.Rethrow(ResourceExecutedContext context) at Microsoft.AspNetCore.Mvc.Internal.ResourceInvoker.Next(State& next, Scope& scope, Object& state, Boolean& isCompleted) at Microsoft.AspNetCore.Mvc.Internal.ResourceInvoker.InvokeFilterPipelineAsync() at Microsoft.AspNetCore.Mvc.Internal.ResourceInvoker.InvokeAsync() at Microsoft.AspNetCore.Builder.RouterMiddleware.Invoke(HttpContext httpContext) at Microsoft.AspNetCore.Diagnostics.DeveloperExceptionPageMiddleware.Invoke(HttpContext context)

Please note that it says Stream does not support reading and this is ok because I ask to create the stream with the following command: cloudBlockBlob.OpenWriteAsync(), but as you can see, I'm not doing any reading operation and I'm only returning the stream to the console app.

Questions

• What do you think it can be? Is there any hidden reading operation that I'm not aware of?
• How to solve the problem?

Thank you,

Attilio

UPDATE

Hi all,

finally we wrote the following:

Controller

public static class Program
{
    public static void Main(string[] args)
    {
        CreateWebHostBuilder(args).Build().Run();
    }

    public static IWebHostBuilder CreateWebHostBuilder(string[] args) =>
        WebHost.CreateDefaultBuilder(args)
            .UseStartup<Startup>()
            .UseKestrel();
}

[Route("api/[controller]")]
[ApiController]
public class ValuesController : Controller
{
    [HttpPost("upload")]
    public async Task<IActionResult> Upload()
    {
        try
        {
            CloudBlobContainer vCloudBlobContainer = await GetCloudBlobContainer().ConfigureAwait(false);
            string boundary = GetBoundary(Request.ContentType);

            MultipartReader reader = new MultipartReader(boundary, Request.Body, 80 * 1024);
            Dictionary<string, string> sectionDictionary = new Dictionary<string, string>();
            MultipartSection section;
            MyFile myFile;

            while ((section = await reader.ReadNextSectionAsync().ConfigureAwait(false)) != null)
            {
                ContentDispositionHeaderValue contentDispositionHeaderValue = section.GetContentDispositionHeader();

                if (contentDispositionHeaderValue.IsFormDisposition())
                {
                    FormMultipartSection formMultipartSection = section.AsFormDataSection();
                    string value = await formMultipartSection.GetValueAsync().ConfigureAwait(false);

                    sectionDictionary.Add(formMultipartSection.Name, value);
                }
                else if (contentDispositionHeaderValue.IsFileDisposition())
                {
                    myFile = JsonConvert.DeserializeObject<MyFile>(sectionDictionary.GetValueOrDefault(nameof(MyFile)));
                    if (myFile == default(object))
                    {
                        throw new InvalidOperationException();
                    }

                    CloudBlockBlob cloudBlockBlob = vCloudBlobContainer.GetBlockBlobReference(myFile.RelativePath);
                    Stream stream = await cloudBlockBlob.OpenWriteAsync().ConfigureAwait(false);
                    FileMultipartSection fileMultipartSection = section.AsFileSection();

                    await cloudBlockBlob.UploadFromStreamAsync(fileMultipartSection.FileStream).ConfigureAwait(false);
                }
            }
            return Ok();
        }
        catch (Exception e)
        {
            throw e;
        }
    }

    private string GetBoundary(string contentType)
    {
        if (contentType == null)
        {
            throw new ArgumentNullException(nameof(contentType));
        }

        string[] elements = contentType.Split(' ');
        string element = elements.First(entry => entry.StartsWith("boundary="));
        string boundary = element.Substring("boundary=".Length);

        return HeaderUtilities.RemoveQuotes(boundary).Value;
    }

    private async Task<CloudBlobContainer> GetCloudBlobContainer()
    {
        const string connectionString = "[Your connection string]";
        const string containerName = "container";
        try
        {
            CloudStorageAccount.TryParse(connectionString, out CloudStorageAccount cloudStorageAccount);
            CloudBlobClient cloudBlobClient = cloudStorageAccount.CreateCloudBlobClient();
            CloudBlobContainer cloudBlobContainer = cloudBlobClient.GetContainerReference(containerName);

            if (await cloudBlobContainer.CreateIfNotExistsAsync().ConfigureAwait(false))
            {
                BlobContainerPermissions blobContainerPermission = new BlobContainerPermissions()
                {
                    PublicAccess = BlobContainerPublicAccessType.Container
                };

                await cloudBlobContainer.SetPermissionsAsync(blobContainerPermission).ConfigureAwait(false);
            }
            return cloudBlobContainer;
        }
        catch (Exception)
        {
           throw;
        }
    }
}

Client

internal static class Program
{
    private const string filePath = @"D:\Test.txt";
    private const string baseAddress = "http://localhost:5000";

    private static async Task Main(string[] args)
    {
        Console.WriteLine($"Test starts at {DateTime.Now.ToString("o")}");
        FileStream fileStream = new FileStream(filePath, FileMode.Open);
        MyFile vFile = new MyFile()
        {
            Lenght = 0,
            RelativePath = "Test.txt"
        };

        await UploadStream(vFile, fileStream).ConfigureAwait(false);

        Console.WriteLine($"Test ends at {DateTime.Now.ToString("o")}");
        Console.Write("Press any key to exit...");
        Console.ReadKey();
    }

    private static async Task UploadStream(MyFile myFile, Stream stream)
    {
        try
        {
            using (HttpClient httpClient = new HttpClient())
            {
                httpClient.BaseAddress = new Uri(baseAddress);
                using (MultipartFormDataContent multipartFormDataContent = new MultipartFormDataContent())
                {
                    multipartFormDataContent.Add(new StringContent(JsonConvert.SerializeObject(myFile), Encoding.UTF8, "application/json"), nameof(MyFile));
                    multipartFormDataContent.Add(new StreamContent(stream), "stream", nameof(MyFile));

                    HttpResponseMessage httpResult = await httpClient.PostAsync("api/values/upload", multipartFormDataContent).ConfigureAwait(false);
                    httpResult.EnsureSuccessStatusCode();
                }
            }
        }
        catch (Exception e)
        {
            Console.WriteLine(e.Message);
        }
    }
}

We'll manage big files, so we have a problem...

Tests

We did the following tests:

• we tried to upload "small" files (less than 30000000 bytes) and all worked fine.
• we tried to upload "big" files (more than 30000000 bytes) and the controller returned a Microsoft.AspNetCore.Server.Kestrel.Core.BadHttpRequestException ("Request body too large.").
• we changed the line of code .UseKestrel() with .UseKestrel(options => options.Limits.MaxRequestBodySize = null) and tried to upload the same file. This code change should have to solve the problem but the client returned a System.NET.Sockets.SocketException ("The I/O operation has been aborted because of either a thread exit or an application request") while no exception was thrown in the controller.

Any idea?

Answer 1

The stream you get in the client is not the same the one you return in your api. The mvc framework need a readable stream to be able to get the content and send it as byte[] throught the network top the client.

You need to send all requiered data to your api to be able to write into the azure blob stream.

Client side

private static async Task Main(string[] args) // async main available in c# 7.1 
{
    Console.WriteLine($"Test starts at {DateTime.Now.ToString("o")}");
    FileStream fileStream = new FileStream(@"C:\Windows10Upgrade\Windows10UpgraderApp.exe", FileMode.Open);
    MyFile vFile = new MyFile()
    {
        Lenght = 0,
        Path = "https://c2calrsbackup.blob.core.windows.net/containername/Windows10UpgraderApp.exe",
        RelativePath = "Windows10UpgraderApp.exe"
    };

    await UploadStream(vFile, fileStream);

    Console.WriteLine($"Test ends at {DateTime.Now.ToString("o")}");
    Console.Write("Press any key to exit...");
    Console.ReadKey();
}

private static async Task UploadStream(MyFile myFile, Stream stream)
{
    try
    {
        using (HttpClient httpClient = new HttpClient()) // instance should be shared
        {
            httpClient.BaseAddress = new Uri("https://localhost:5000");
            using (MultipartFormDataContent multipartFormDataContent = new MultipartFormDataContent())
            {
                multipartFormDataContent.Add(new StringContent(JsonConvert.SerializeObject(myFile), Encoding.UTF8, "application/json"), nameof(MyFile));
                // Here we add the file to the multipart content.
                // The tird parameter is required to match the `IsFileDisposition()` but could be anything
                multipartFormDataContent.Add(new StreamContent(stream), "stream", "myfile");

                HttpResponseMessage httpResult = await httpClient.PostAsync("api/uploaddownload/upload", multipartFormDataContent).ConfigureAwait(false);
                httpResult.EnsureSuccessStatusCode();
                // We don't need any result stream anymore
            }
        }
    }
    catch (Exception e)
    {
        Console.WriteLine(e.Message);
    }
}

Api side :

[HttpPost("upload")]
public async Task<IActionResult> GetUploadStream()
{
    const string contentType = "application/octet-stream";
    string boundary = GetBoundary(Request.ContentType);
    MultipartReader reader = new MultipartReader(boundary, Request.Body, 80 * 1024);
    Dictionary<string, string> sectionDictionary = new Dictionary<string, string>();
    var memoryStream = new MemoryStream();
    MultipartSection section;

    while ((section = await reader.ReadNextSectionAsync()) != null)
    {
        ContentDispositionHeaderValue contentDispositionHeaderValue = section.GetContentDispositionHeader();

        if (contentDispositionHeaderValue.IsFormDisposition())
        {
            FormMultipartSection formMultipartSection = section.AsFormDataSection();
            string value = await formMultipartSection.GetValueAsync();

            sectionDictionary.Add(formMultipartSection.Name, value);
        }
        else if (contentDispositionHeaderValue.IsFileDisposition())
        {
            // we save the file in a temporary stream
            var fileMultipartSection = section.AsFileSection();
            await fileMultipartSection.FileStream.CopyToAsync(memoryStream);
            memoryStream.Position = 0;
        }
    }

    CloudStorageAccount.TryParse(connectionString, out CloudStorageAccount cloudStorageAccount);
    CloudBlobClient cloudBlobClient = cloudStorageAccount.CreateCloudBlobClient();
    CloudBlobContainer cloudBlobContainer = cloudBlobClient.GetContainerReference(containerName);

    if (await cloudBlobContainer.CreateIfNotExistsAsync())
    {
        BlobContainerPermissions blobContainerPermission = new BlobContainerPermissions()
        {
            PublicAccess = BlobContainerPublicAccessType.Container
        };

        await cloudBlobContainer.SetPermissionsAsync(blobContainerPermission);
    }

    MyFile myFile = JsonConvert.DeserializeObject<MyFile>(sectionDictionary.GetValueOrDefault(nameof(MyFile)));
    CloudBlockBlob cloudBlockBlob = cloudBlobContainer.GetBlockBlobReference(myFile.RelativePath);
    using(Stream blobStream = await cloudBlockBlob.OpenWriteAsync())
    {
        // Finally copy the file into the blob writable stream
        await memoryStream.CopyToAsync(blobStream);
    }

    // you can replace OpenWriteAsync by 
    // await cloudBlockBlob.UploadFromStreamAsync(memoryStream);

    return Ok(); // return httpcode 200
}

See https://learn.microsoft.com/en-us/aspnet/core/mvc/models/file-uploads#uploading-large-files-with-streaming for documentation

You can avoid the temporary memory stream if you move azureblog code inside the else if block. But you need to ensure the order of FormData. (Metadata then file)