What is the best strategy to upload large file using HttpClient in a low memory windows phone device?

Question

I am trying to upload files using similar approach HttpClient: How to upload multiple files at once in windows phone.

using (var content = new MultipartFormDataContent())
{
    content.Add(CreateFileContent(imageStream, "image.jpg", "image/jpeg"));
    content.Add(CreateFileContent(signatureStream, "image.jpg.sig", "application/octet-stream"));

    var response = await httpClient.PostAsync(_profileImageUploadUri, content);
    response.EnsureSuccessStatusCode();
}

private StreamContent CreateFileContent(Stream stream, string fileName, string contentType)
{
    var fileContent = new StreamContent(stream);
    fileContent.Headers.ContentDisposition = new ContentDispositionHeaderValue("form-data") 
    { 
        Name = "\"files\"", 
        FileName = "\"" + fileName + "\""
    }; // the extra quotes are key here
    fileContent.Headers.ContentType = new MediaTypeHeaderValue(contentType);            
    return fileContent;
}

This works fine while uploading small files. If I tried to upload larger file(say > 50mb) in a low end device(512Mb memory), it throws System.OutOfMemoryException. I used the diagnostic tools to monitor the memory consumption and noticed that memory grows exponentially during PostAsync call. Seems like it is copying the entire content to the memory. Right now we don't have chunking support in the api.

What is the best strategy to upload large file using HttpClient in a low memory windows phone device?

Answer 1

I'm not really an expert on MultipartFormDataContent (and it might split the content underwater) but a tip could be that you divide the data you want to send.

Then send the other blocks and reconstruct it on the receiving end.

e.g. divide the images in smaller blocks (for example 10mb or less depending on the memory usage) and send these

So it might result in a for loop to traverse the blocks.

foreach (byte[] block in dividedContent)
{
    using (var content = new MultipartFormDataContent())
    {
        content.Add(block);

        var response = await httpClient.PostAsync(_profileImageUploadUri, content);
        response.EnsureSuccessStatusCode();
    }
}

perhaps something like that would solve your problem :)

Answer 2

Do multipart POST manually - without help from MultipartFormDataContent

If you must send it multi-part, then you could POST it more manually, reading from the source file in 4k buffer blocks.

You don't necessarily need to do so with async methods. The solution is "manual control over 4k buffering". But async would be ideal, being most thread/CPU efficient.

Here's another recommended link to understand how to code multipart posts. And another for understanding the protocol, here's an example of what gets sent out over the stream, illustrating the boundary markers

Also, architecturally, I tend to prefer uploading files separately to any (form) data. This avoids multipart posting completely, making your APIs atomic and simple. You may have a service which simple stores an uploaded file and returns the URL or ID. That URL or ID can then be referenced with your data and posted subsequently.