Why does HttpClient.PostAsync buffer response?

Question

Recently I encountered performance issues when using HttpClients.PostAsync on .Net 4.5.1. Originally server (Owin + WebApi) was buffering response before sending. This was causing huge memory usage overhead (serialized response size is > 1Gb). After I turned on response streaming on the server client literally stopped working. It turned out that reason was buffer re-allocation on the client when reading response from server. I checked HttpClient implementation and found this interesting part in HttpClient.SendAsync method:

if (result.Content == null || completionOption == HttpCompletionOption.ResponseHeadersRead)
{
    this.SetTaskCompleted(request, linkedCts, tcs, result);
}
else
{
    this.StartContentBuffering(request, linkedCts, tcs, result);
}

So when completionOption is not ResponseHeadersRead response is always buffered. According to SendAsync documentation implementation of SendAsync is consistent with the intent.

Now, as PostAsync is implemented sending ResponseContentRead, response stream is always buffered on POSTs. So question is why does PostAsync have to wait (and buffer) for whole response to arrive before processing continues?

Answer 1

There's the obvious part - if you don't specify HttpCompletionOption.ResponseHeadersRead, the Task will only complete when the whole response is read; you have to store the response data somewhere in the meantime.

Why doesn't PostAsync allow you to specify HttpCompletionOption.ResponseHeadersRead? Probably because it's not really all that useful, most of the time. POST is for posting data, not for retrieving it - that's GET's job. HttpClient was designed around WebAPIs and "REST" services, with proper use of HTTP verbs.

If you need to use POST to retrieve such large amounts of data, you have two basic options:

• Use SendAsync
• Don't use HttpClient (HttpWebRequest is a bit more complicated, but gives you way more control)