Does Ruby's 'open_uri' reliably close sockets after read or on fail?

Question

I have been using open_uri to pull down an ftp path as a data source for some time, but suddenly found that I'm getting nearly continual "530 Sorry, the maximum number of allowed clients (95) are already connected."

I am not sure if my code is faulty or if it is someone else who's accessing the server and unfortunately there's no way for me to really seemingly know for sure who's at fault.

Essentially I am reading FTP URI's with:

  def self.read_uri(uri)
    begin
      uri = open(uri).read
      uri == "Error" ? nil : uri
    rescue OpenURI::HTTPError
      nil
    end
  end

I'm guessing that I need to add some additional error handling code in here... I want to be sure that I take every precaution to close down all connections so that my connections are not the problem in question, however I thought that open_uri + read would take this precaution vs using the Net::FTP methods.

The bottom line is I've got to be 100% sure that these connections are being closed and I don't somehow have a bunch open connections laying around.

Can someone please advise as to correctly using read_uri to pull in ftp with a guarantee that it's closing the connection? Or should I shift the logic over to Net::FTP which could yield more control over the situation if open_uri is not robust enough?

If I do need to use the Net::FTP methods instead, is there a read method that I should be familiar with vs pulling it down to a tmp location and then reading it (as I'd much prefer to keep it in a buffer vs the fs if possible)?

Answer 1

I suspect you are not closing the handles. OpenURI's docs start with this comment:

It is possible to open http/https/ftp URL as usual like opening a file:

open("http://www.ruby-lang.org/") {|f|
  f.each_line {|line| p line}
}

I looked at the source and the open_uri method does close the stream if you pass a block, so, tweaking the above example to fit your code:

uri = ''
open("http://www.ruby-lang.org/") {|f|
  uri = f.read
}

Should get you close to what you want.

---

Here's one way to handle exceptions:

# The list of URLs to pass in to check if one times out or is refused.
urls = %w[
  http://www.ruby-lang.org/
  http://www2.ruby-lang.org/
]

# the method
def self.read_uri(urls)

  content = ''

  open(urls.shift) { |f| content = f.read }
  content == "Error" ? nil : content

  rescue OpenURI::HTTPError
    retry if (urls.any?)
    nil
end

Answer 2

Try using a block:

data = open(uri){|f| f.read}