How to change the encoding during CSV parsing in Rails

Question

I would like to know how can I change the encoding of my CSV file when I import it and parse it. I have this code:

csv = CSV.parse(output, :headers => true, :col_sep => ";")
csv.each do |row|
  row = row.to_hash.with_indifferent_access
  insert_data_method(row)
end

When I read my file, I get this error:

Encoding::CompatibilityError in FileImportingController#load_file
incompatible character encodings: ASCII-8BIT and UTF-8

I read about row.force_encoding('utf-8') but it does not work:

NoMethodError in FileImportingController#load_file
undefined method `force_encoding' for #<ActiveSupport::HashWithIndifferentAccess:0x2905ad0>

Thanks.

Answer 1

Hey I wrote a little blog post about what I did, but it's slightly more verbose than what's already been posted. For whatever reason, I couldn't get those solutions to work and this did.

This gist is that I simply replace (or in my case, remove) the invalid/undefined characters in my file then rewrite it. I used this method to convert the files:

def convert_to_utf8_encoding(original_file)  
  original_string = original_file.read
  final_string = original_string.encode(invalid: :replace, undef: :replace, replace: '') #If you'd rather invalid characters be replaced with something else, do so here.
  final_file = Tempfile.new('import') #No need to save a real File
  final_file.write(final_string)
  final_file.close #Don't forget me
  final_file
end 

Hope this helps.

Edit: No destination encoding is specified here because encode assumes that you're encoding to your default encoding which for most Rails applications is UTF-8 (I believe)

Answer 2

force_encoding is meant to be run on a string, but it looks like you're calling it on a hash. You could say:

output.force_encoding('utf-8')
csv = CSV.parse(output, :headers => true, :col_sep => ";")
...