Curl::Err::PartialFileError: Transferred a partial file error

Question

I am trying to execute following code in rails console:

ce = Curl::Easy.new("http://www.homestolove.com.au/bathroom-profile-fresh-approach-2391")
ce.verbose=true
ce.perform

But I am getting following error. Can anyone suggest how to fix this ? It looks like this url is quite slow in returning the response. Can we do anything to stop the connection from terminating until file transfer is complete ?

* Adding handle: conn: 0x95f3210
* Adding handle: send: 0
* Adding handle: recv: 0
* Curl_addHandleToPipeline: length: 1
* - Conn 0 (0x95f3210) send_pipe: 1, recv_pipe: 0
* About to connect() to www.homestolove.com.au port 80 (#0)
*   Trying 52.85.77.138...
* Connected to www.homestolove.com.au (52.85.77.138) port 80 (#0)
> GET /bathroom-profile-fresh-approach-2391 HTTP/1.1
Host: www.homestolove.com.au
Accept: */*

< HTTP/1.1 200 OK
< Content-Type: text/html; charset=utf-8
< Content-Length: 717180
< Cache-Control: public, max-age=1800
< Cache-Control: no-cache="set-cookie"
< Date: Sat, 29 Oct 2016 06:22:49 GMT
< ETag: W/"af17c-wej92PccZYKK1ePeKXoQ1Q"
* Server nginx is not blacklisted
< Server: nginx
< Strict-Transport-Security: max-age=31536000; includeSubdomains
< X-Frame-Options: SAMEORIGIN
< X-Powered-By: Express
< X-XSS-Protection: 1; mode=block
< Vary: Accept-Encoding,Accept-Encoding
< X-Cache: Miss from cloudfront
< Via: 1.1 94812f867206232855f09570ec52b557.cloudfront.net (CloudFront), 1.1 wsg.Lehi05
< X-Amz-Cf-Id: qiwFsn3d4lSmNZTSC_BVojcZmWjU1Toba7XagEiFCe-zlywo6GlkVw==
< Age: 2
< 
* transfer closed with 431831 bytes remaining to read
* Closing connection 0
Curl::Err::PartialFileError: Transferred a partial file
    from /opt/rails/test/shared/bundle/ruby/1.9.1/gems/curb-0.9.3/lib/curl/easy.rb:73:in `perform'
    from (irb):5
    from /opt/rails/test/shared/bundle/ruby/1.9.1/bundler/gems/rails-efa67857350e/railties/lib/rails/commands/console.rb:47:in `start'
    from /opt/rails/test/shared/bundle/ruby/1.9.1/bundler/gems/rails-efa67857350e/railties/lib/rails/commands/console.rb:8:in `start'
    from /opt/rails/test/shared/bundle/ruby/1.9.1/bundler/gems/rails-efa67857350e/railties/lib/rails/commands.rb:41:in `<top (required)>'
    from script/rails:6:in `require'
    from script/rails:6:in `<main>'

Answer 1

It seems that it works with encoding set to gzip

ce = Curl::Easy.new("http://www.homestolove.com.au/bathroom-profile-fresh-approach-2391")
ce.verbose = true
ce.encoding = 'gzip'
ce.perform

output:

*   Trying 216.137.61.96...
* Connected to www.homestolove.com.au (216.137.61.96) port 80 (#0)
> GET /bathroom-profile-fresh-approach-2391 HTTP/1.1
Host: www.homestolove.com.au
Accept: */*
Accept-Encoding: gzip

< HTTP/1.1 200 OK
< Content-Type: text/html; charset=utf-8
< Transfer-Encoding: chunked
< Connection: keep-alive
< Cache-Control: public, max-age=1800
< Cache-Control: no-cache="set-cookie"
< Content-Encoding: gzip
< Date: Sun, 06 Nov 2016 19:49:27 GMT
< ETag: W/"a7c31-QuyLkXEk3HhTu/KMM5QDNQ"
< Server: nginx
< Strict-Transport-Security: max-age=31536000; includeSubdomains
< X-Frame-Options: SAMEORIGIN
< X-Powered-By: Express
< X-XSS-Protection: 1; mode=block
< Vary: Accept-Encoding
< Age: 162
< X-Cache: Hit from cloudfront
< Via: 1.1 e24fef4a7b03bd84e1e8d57f2471a84d.cloudfront.net (CloudFront)
< X-Amz-Cf-Id: 96AZ5p0ts5UZdPXTEpCk8S3wEQJk9a4vZD7vhY-8sXHX_NTyGSUouA==
<

Answer 2

The site has wrong "Content-Length" header. Thus curl can't parse it properly.

So let's just skip the header, and let curl to take it on own

ce = Curl::Easy.new("http://www.homestolove.com.au/bathroom-profile-fresh-approach-2391")
ce.verbose = true
ce.ignore_content_length = true # add the line
ce.perform

And result is here:

ce.body

I hope it helped you