Logo Questions Linux Laravel Mysql Ubuntu Git Menu
 

HttpClient buffer size limit exceeded

Tags:

I am using my client to get some info about a certain file stored in my Swift Object Storage which can be accessed throught REST Api. In Swift the HEAD method and url leading to specified object returns it's metadata (hash, timestamp, etc) contained in HTML response's (has NO CONTENT) headers.

My code works perfectly when file size is < 2GB. I get HttpResponseMessage and I am able to parse it for required data, but when I ask for file > 2GB I get exception : "Cannot write more bytes to the buffer than the configured maximum buffer size: 2147483647".

I understand, that HttpClient property MaxResponseContentBufferSize cannot be set to value > 2GB, but I don't want to get it's content. Is this some bug or is there some better way to solve this?

public HttpResponseMessage FileCheckResponse(string objectName)
   {
        //create url which will be appended to HttpClient (m_client)
        string requestUrl = RequestUrlBuilder(m_containerName, objectName);
        //create request message with Head method
        HttpRequestMessage request = new HttpRequestMessage(HttpMethod.Head, requestUrl);
        //exception thrown here... 
        HttpResponseMessage response = m_client.SendAsync(request).Result;            

        return response;
    }

When trying to perform same action using Dev HTTP Client (Chrome extension) I have no problem. It seems that Content-Length header makes it unfeasible. Here is the output from Dev HTTP Client:

Content-Length: 3900762112
Accept-Ranges: bytes
Last-Modified: Fri, 06 Sep 2013 16:24:30 GMT
Etag: da4392bdb5c90edf31c14d008570fb95
X-Timestamp: 1378484670.87557
Content-Type: application/octet-stream
Date: Tue, 10 Sep 2013 13:25:27 GMT
Connection: keep-alive

I will be glad for any ideas! Thank you

Solution

First - thanks to Darrel Mirrel, who solved my whole day problem in few seconds :) I just needed to edit one line in code by adding HttpCompletitionOption where response is obtained:

HttpResponseMessage response = m_client.SendAsync(request, HttpCompletionOption.ResponseHeadersRead).Result;

The option ResponseHeaderRead tells the client to finish operation as soon as Headers are read withnout reading content of the message.

like image 995
Milanec Avatar asked Sep 10 '13 13:09

Milanec


1 Answers

Use the SendAsync overload that allows you to specify the HttpCompletionOptions. With this you can tell HttpClient not to create a buffer for the response contents.

like image 55
Darrel Miller Avatar answered Sep 18 '22 11:09

Darrel Miller