How to visit a URL with Ruby via http and read the output?

Question

So far I have been able to stitch this together :)

 begin
  open("http://www.somemain.com/" + path + "/" + blah)
 rescue OpenURI::HTTPError
   @failure += painting.permalink
 else
   @success += painting.permalink
 end

But how do I read the output of the service that I would be calling?

Answer 1

Open-URI extends open, so you'll get a type of IO stream returned:

open('http://www.example.com') #=> #<StringIO:0x00000100977420>

You have to read that to get content:

open('http://www.example.com').read[0 .. 10] #=> "<!DOCTYPE h"

A lot of times a method will let you pass different types as a parameter. They check to see what it is and either use the contents directly, in the case of a string, or read the handle if it's a stream.

For HTML and XML, such as RSS feeds, we'll typically pass the handle to a parser and let it grab the content, parse it, and return an object suitable for searching further:

require 'nokogiri'
doc = Nokogiri::HTML(open('http://www.example.com'))
doc.class #=> Nokogiri::HTML::Document
doc.to_html[0 .. 10] #=> "<!DOCTYPE h"
doc.at('h1').text #=> "Example Domains"

Answer 2

doc = open("http://etc..")
content = doc.read

More often people want to be able to parse the returned document, for this use something like hpricot or nokogiri