Zlib::BufError when using progressbar/ruby-progressbar gem

Question

I use the following Ruby snippet to download a 8.9MB file.

require 'open-uri'
require 'net/http'
require 'uri'

def http_download_no_progress_bar(uri, filename)
  uri.open(read_timeout: 500) do |file|
    open filename, 'w' do |io|
      file.each_line do |line|
        io.write line
      end
    end
  end
end

I want to add the progressbar gem to visualize the download process:

require 'open-uri'
require 'progressbar'
require 'net/http'
require 'uri'

def http_download_with_progressbar(uri, filename)
  progressbar = nil
  uri.open(
    read_timeout: 500,
    content_length_proc: lambda { |total|
    if total && 0 < total.to_i
      progressbar = ProgressBar.new("...", total)
      progressbar.file_transfer_mode
    end
    },
    progress_proc: lambda { |step|
       progressbar.set step if progressbar
    }
  ) do |file|
    open filename, 'w' do |io|
      file.each_line do |line|
        io.write line
      end
    end
  end
end

However, it now fails with the following error:

/home/user/.rvm/rubies/ruby-2.1.1/lib/ruby/2.1.0/net/http/response.rb:357:in `finish': 
buffer error (Zlib::BufError)oooooo  |   8.0MB   8.6MB/s ETA:   0:00:00

    from /home/user/.rvm/rubies/ruby-2.1.1/lib/ruby/2.1.0/net/http/response.rb:357:in `finish'
    from /home/user/.rvm/rubies/ruby-2.1.1/lib/ruby/2.1.0/net/http/response.rb:262:in `ensure in inflater'
    from /home/user/.rvm/rubies/ruby-2.1.1/lib/ruby/2.1.0/net/http/response.rb:262:in `inflater'
    from /home/user/.rvm/rubies/ruby-2.1.1/lib/ruby/2.1.0/net/http/response.rb:274:in `read_body_0'
    from /home/user/.rvm/rubies/ruby-2.1.1/lib/ruby/2.1.0/net/http/response.rb:201:in `read_body'
    from /home/user/.rvm/rubies/ruby-2.1.1/lib/ruby/2.1.0/open-uri.rb:328:in `block (2 levels) in open_http'
    from /home/user/.rvm/rubies/ruby-2.1.1/lib/ruby/2.1.0/net/http.rb:1415:in `block (2 levels) in transport_request'
    from /home/user/.rvm/rubies/ruby-2.1.1/lib/ruby/2.1.0/net/http/response.rb:162:in `reading_body'
    from /home/user/.rvm/rubies/ruby-2.1.1/lib/ruby/2.1.0/net/http.rb:1414:in `block in transport_request'
    from /home/user/.rvm/rubies/ruby-2.1.1/lib/ruby/2.1.0/net/http.rb:1405:in `catch'
    from /home/user/.rvm/rubies/ruby-2.1.1/lib/ruby/2.1.0/net/http.rb:1405:in `transport_request'
    from /home/user/.rvm/rubies/ruby-2.1.1/lib/ruby/2.1.0/net/http.rb:1378:in `request'
    from /home/user/.rvm/rubies/ruby-2.1.1/lib/ruby/2.1.0/open-uri.rb:319:in `block in open_http'
    from /home/user/.rvm/rubies/ruby-2.1.1/lib/ruby/2.1.0/net/http.rb:853:in `start'
    from /home/user/.rvm/rubies/ruby-2.1.1/lib/ruby/2.1.0/open-uri.rb:313:in `open_http'
    from /home/user/.rvm/rubies/ruby-2.1.1/lib/ruby/2.1.0/open-uri.rb:724:in `buffer_open'
    from /home/user/.rvm/rubies/ruby-2.1.1/lib/ruby/2.1.0/open-uri.rb:210:in `block in open_loop'
    from /home/user/.rvm/rubies/ruby-2.1.1/lib/ruby/2.1.0/open-uri.rb:208:in `catch'
    from /home/user/.rvm/rubies/ruby-2.1.1/lib/ruby/2.1.0/open-uri.rb:208:in `open_loop'
    from /home/user/.rvm/rubies/ruby-2.1.1/lib/ruby/2.1.0/open-uri.rb:149:in `open_uri'
    from /home/user/.rvm/rubies/ruby-2.1.1/lib/ruby/2.1.0/open-uri.rb:704:in `open'

Meanwhile I also tried the ruby-progressbar gem:

require 'open-uri'
require 'ruby-progressbar'
require 'net/http'
require 'uri'

def http_download_with_ruby_progressbar(uri, filename)
  progressbar = nil
  uri.open(
    read_timeout: 500,
    content_length_proc: lambda { |total|
      if total && 0 < total.to_i
        progressbar = ProgressBar.create(title: filename, total: total)
      end
      },
      progress_proc: lambda { |step|
        progressbar.progress = step if progressbar
      }
  ) do |file|
    open filename, 'w' do |io|
      file.each_line do |line|
        io.write line
      end
    end
  end
end

It fails with the same error. Here is the associated issue for the problem.

Answer 1

The problem is the file you are trying to download as every method works with this file: https://androidnetworktester.googlecode.com/files/1mb.txt.

The problem is that your file is larger than it says it is. The content_length_proc says that it is 8549968 bytes (8.15MB) whereas it is 101187668 bytes (96.5MB) (check with ls after downloading the file). Now I have an alternative that does not crash and gives you a progressbar:

def http_download_with_words(uri, filename)
  bytes_total = nil
  uri.open(
           read_timeout: 500,
           :content_length_proc => lambda{|content_length|
             bytes_total = content_length},
           :progress_proc => lambda{|bytes_transferred|
             if bytes_total
               # Print progress
               print("\r#{bytes_transferred}/#{bytes_total}")
             else
               # We don’t know how much we get, so just print number
               # of transferred bytes
               print("\r#{bytes_transferred} (total size unknown)")
             end
             }
           ) do |file|
    open filename, 'w' do |io|
      file.each_line do |line|
        io.write line
      end
    end
  end
end

http_download_with_words(URI( 'http://data.wien.gv.at/daten/geo?service=WFS&request=GetFeature&version=1.1.0&typeName=ogdwien%3aBAUMOGD&srsName=EPSG:4326' ), 'temp.txt')

which is pretty self-explanatory, (seen here.)

Now the part I haven't been able to figure out is how exactly the progressbar gem is interfering with the ZLib. Most things seem to work fine inside the procs (e.g. having them print random stuff) so I assume both of these progressbars do something odd on completion that somehow messes with the transfer. I'd be very interested if anyone can figure out why that is?

Answer 2

In my testing when this occurred it was due to the raise in #set. As for why it results in an error in Zlib, that's not clear. Perhaps some strange exception handling in there. In my case I did "progbar.set(count) rescue nil" to get rid of the issue.