`write': "\xCF" from ASCII-8BIT to UTF-8 (Encoding::UndefinedConversionError) while writing to file from url

Question

I am getting error:

write': "\xCF" from ASCII-8BIT to UTF-8 (Encoding::UndefinedConversionError)

from line:

open(uri) {|url_file| tempfile.write(url_file.read)}

relevant code is:

require 'tempfile'
require 'open-uri'
require 'uri'
..
uri = URI.parse(@download_link)
tempfile = Tempfile.create(file_name)
open(uri) {|url_file| tempfile.write(url_file.read)}`
..

It runs completely fine if I run it like ruby lib/file.rb, but gives error when I run it in rails environment: rails runner lib/file.rb.

Most questions with this error refer to gem installation scenarios. My guess that I have to include/update some gems, but have no idea which.

Answer 1

This should solve the problem.

data = URI.parse(@download_link).read
tempfile = Tempfile.create(file_name)
tempfile.binmode # This will help deal encoding problem with download files from the internet
tempfile.write(data)

binmode is binary mode

Answer 2

Use force_encoding:

open(uri) {|url_file| tempfile.write(url_file.read.force_encoding("UTF-8"))

Answer 3

Accepted answer is fine, but I think it is worth mentioning that You can also set encoding when creating/opening Tempfile, for example:

Tempfile.new("file.pdf", encoding: 'ascii-8bit') # or 'utf-8'