Limit size of response read by rest-client

Question

I'm using the Ruby gem rest-client (1.6.7) to retrieve data using HTTP GET requests. However, sometimes the responses are bigger than I want to handle, so I would like some way to have the RestClient stop reading once it exceeds a size limit I set. The documentation says

For cases not covered by the general API, you can use the RestClient::Request class which provide a lower-level API.

but I do not see how that helps me. I do not see anything that looks like a hook into processing the incoming data stream, only operations I could perform after the whole thing is read. I don't want to waste time and memory reading a huge response into a buffer only to discard it.

How can I set a limit on the amount of data read by RestClient in a GET request? Or is there a different client I can use that makes it easy to set such a limit?

Answer 1

rest-client uses ruby's Net::HTTP underneath: https://github.com/rest-client/rest-client/blob/master/lib/restclient/request.rb#L303

Unfortunately, it doesn't seem like Net::HTTP will let you abandon response based on its length as it uses, after all, this method to issue all requests: http://docs.ruby-lang.org/en/2.0.0/Net/HTTP.html#method-i-transport_request

As you can see, it uses HTTPResponse to read an HTTP response from server: http://ruby-doc.org/stdlib-2.0.0/libdoc/net/http/rdoc/Net/HTTPResponse.html#method-i-read_body

HTTPResponse seems like the place where you could control whether to read all response and store it into memory, or read and throw away. I you don't want even to read the response, I guess you'll need to close the socket.

I don't know whether there are rest-clients with functionality you need. I guess you'll need to write your own little rest-client if you want to have such a fine-grained control.