What is the correct way to get google search results?

Question

I want to get all the search results for a particular keyword search on google. I've seen suggestions of scraping, but this seems like a bad idea. I've seen Gems (I plan on using ruby) that do scraping and use the API. I've also seen suggestions of using the API.

Does anyone know the best way to do this right now? The API Is no longer supported and I've seen people report they get unusable data back. Do the Gems help solve this or no?

Thanks in advance.

Answer 1

I also go for the scrape option, its quicker than asking google for a key and plus, and you are not limited to 100 search queries per day. Google´s TOS is an issue though, as Richard points out. Here´s an example i´ve done that works for me - it´s also useful if you want to connect through a proxy:

require 'rubygems'
require 'mechanize'

agent = Mechanize.new
agent.set_proxy '78.186.178.153', 8080
page = agent.get('http://www.google.com/')

google_form = page.form('f')
google_form.q = 'new york city council'

page = agent.submit(google_form, google_form.buttons.first)

page.links.each do |link|
    if link.href.to_s =~/url.q/
        str=link.href.to_s
        strList=str.split(%r{=|&}) 
        url=strList[1] 
        puts url
    end 
end

Answer 2

According to http://code.google.com/apis/websearch/ , the Search API has been deprecated -- but there's a replacement, the Custom Search API. Will that do what you want?

If so, a quick Web search turned up https://github.com/alexreisner/google_custom_search , among other gems.