Ruby Mechanize Zlib::BufError

Question

Not sure why I'm getting this error now with the Mechanize gem - been using it for a while now with no issues.

My script will randomly stop and throw the following error:

/Users/username/.rvm/gems/ruby-1.9.3-p194/gems/mechanize-2.5.1/lib/mechanize/http/agent.rb:798:in `rescue in response_content_encoding': error handling content-encoding gzip: buffer error (Zlib::BufError) (Mechanize::Error)

Any ideas?

Answer 1

It's possible that you're hitting a URL that points to a load-balancer. One of the hosts behind that load-balancer is mis-configured, or perhaps it's configured differently than its peers, and is returning a gzipped version of the content, where others aren't. I've seen that problem in the past.

I've also see situations where the server said it was returning gzipped content, but sent it uncompressed. Or it could be sending zipped, not gzipped. The combinations are many.

The fix is to be sure your code is capable of sensing whether the returned content is compressed. Make sure you're sending the correct acceptable-content HTTP headers for your code to their server too. You have to program defensively and look at the actual content you get back, and then branch to do the right decompression, then pass that on for parsing.