What can I do about Mechanize waiting on an unresponsive web site?

Question

I noticed that when I fetch a site that is not responding using Mechanize, it just keeps on waiting.

How can I overcome this problem?

Answer 1

There's a couple ways to deal with it.

Open-Uri, and Net::HTTP have ways of passing in timeout values, which then tell the underlying networking stack how long you are willing to wait. For instance, Mechanize lets you get at its settings when you initialize an instance, something like:

mech = Mechanize.new { |agent|
  agent.open_timeout   = 5
  agent.read_timeout   = 5
}

It's all in the docs for new but you'll have to view the source to see what instance variables you can get at.

Or you can use Ruby's timeout module:

require 'timeout'
status = Timeout::timeout(5) {
  # Something that should be interrupted if it takes too much time...
}

Answer 2

http://mechanize.rubyforge.org/mechanize/Mechanize.html on this page there are 2 undocumented attributes open_timeout and read_timeout, try using them.

agent = Mechanize.new { |a| a.log = Logger.new("mech.log") }
agent.keep_alive=false
agent.open_timeout=15
agent.read_timeout=15

HTH