Uploading large files to Controller in chunks using HttpClient, IFormFile always empty

Question

I am trying to create a .Net Standard "Client" class for uploading (sometimes very large) files to a Controller. I want to do this by breaking the file into chunks and uploading them one at a time. The intent is for other applications to use this instead of communicating directly to the Web Api.

I already have the Controller working. I've verified that it works using a Kendo-ui control which supports chunk-saving.

The issue I am having is that the IEnumerable<IFormFile> files parameter for my controller is always empty when posted from my client class

Controller

[Route("api/Upload")]
public ActionResult ChunkSave(IEnumerable<IFormFile> files, string metaData, Guid id)
{
    MemoryStream ms = new MemoryStream(Encoding.UTF8.GetBytes(metaData));
    var serializer = new DataContractJsonSerializer(typeof(ChunkMetaData));
    ChunkMetaData somemetaData = serializer.ReadObject(ms) as ChunkMetaData;

    // The Name of the Upload component is "files"
    if (files != null)
    {
        // If this is the first chunk, try to delete the file so that we don't accidently
        // and up appending new bytes to the old file.
        if (somemetaData.ChunkIndex == 0)
        {
            _io.DeleteFile(id, Path.GetFileName(somemetaData.FileName));
        }

        foreach (var file in files)
        {
            // Some browsers send file names with full path. This needs to be stripped.
             _io.AppendToFile(id, Path.GetFileName(somemetaData.FileName), file.OpenReadStream());
        }
    }

    FileResult fileBlob = new FileResult();
    fileBlob.uploaded = somemetaData.TotalChunks - 1 <= somemetaData.ChunkIndex;
    fileBlob.fileUid = somemetaData.UploadUid;
    return new JsonResult(fileBlob);
}

Client:

public class FileTransferClient
{
    HttpClient Client { get; set; } 

    public FileTransferClient(Uri apiUrl)
    {
        this.Client = new HttpClient(new HttpClientHandler() { UseDefaultCredentials = true })
        {
            BaseAddress = apiUrl
        };
        this.Client.DefaultRequestHeaders.Accept.Add(
            new MediaTypeWithQualityHeaderValue("application/json"));
    }

    public async Task<bool> UploadFile(Guid id, Stream file, string name, string contentType)
    {
        bool ret = true;
        int chunckSize = 2097152; //2MB
        int totalChunks = (int)(file.Length / chunckSize);
        if (file.Length % chunckSize != 0)
        {
            totalChunks++;
        }

        for (int i = 0; i < totalChunks; i++)
        {
            long position = (i * (long)chunckSize);
            int toRead = (int)Math.Min(file.Length - position + 1, chunckSize);
            byte[] buffer = new byte[toRead];
            await file.ReadAsync(buffer, 0, toRead);

            MultipartFormDataContent content = new MultipartFormDataContent();
            content.Add(new StringContent(id.ToString()), "id");
            var meta = JsonConvert.SerializeObject(new ChunkMetaData
            {
                UploadUid = id.ToString(),
                FileName = name,
                ChunkIndex = i,
                TotalChunks = totalChunks,
                TotalFileSize = file.Length,
                ContentType = contentType
            });
            content.Add(new StringContent(meta), "metaData");
            using (var ms = new MemoryStream(buffer))
            {
                content.Add(new StreamContent(ms),"files");
                var response = await Client.PostAsync("/api/Upload", content).ConfigureAwait(false);
                if (!response.IsSuccessStatusCode)
                {
                    ret = false;
                    break;
                }
            }
        }
        return ret;
    }
}

Answer 1

Your param is empty, because you're not sending an array of files, but rather just one file. Therefore, the binding fails, and you get a null. The act of chunking (which you aren't actually even doing) does not equate to an IEnumerable<IFormFile>; it's still just an IFormFile.

While you need to send as multipart/form-data because you're sending both a file upload and some other post data, I think you're misunderstanding what this actually does. It simply means the request body contains multiple different mime-types, it does not mean that it's uploading the file in multiple parts, which seems to be what you're thinking it does.

The actual act of streaming the upload occurs on the server-side. It's about how the server chooses to handle the file being uploaded, and not so much about how the user is uploading it. More specifically, any sort of modelbinding, particular to an IFormFile will cause the file to be spooled to disk first, and then passed into your action. In other words, if you're accepting an IFormFile, you've already lost the battle. It's already been fully transferred from the client to your server.

The ASP.NET Core docs show you how to actually stream the upload, and unsurprisingly there's a fair bit of code involved, none of which you have currently. You basically have to turn modelbinding off entirely on the action and manually parse the request body yourself, being careful to actually chunk the reads from the stream and not do something that will force the entirely thing into memory at once.