Rails View Encoding Issues

Question

I'm using Ruby 2.0 and Rails 3.2.14. My view is littered several UTF-8 characters, mainly currency symbols like บาท and د.إ etc. I noticed some

(ActionView::Template::Error) "incompatible character encodings: ASCII-8BIT and UTF-8

in our production code and promptly tried visiting the page url on my browser without any issues. On digging in, I realised the error was actually caused by BingBot and few spiders. So when I tried to curl the same url, I was able to reproduce the issue. So, if I try

curl http://localhost:3000/?x=✓

I get the error where UTF-8 symbols are used in the view code. I also realised that if use HTML encoded strings in place of the symbols, this does not occur. However, I prefer using the actual symbols.

And I have already tried setting Encoding.default_external = Encoding::UTF_8 in environment.rb adding #encoding: utf-8 magic comment to top of file and it does not help.

So, why does this error occur? What is the difference between hitting this url on browser and on CURL besides cookies? And how do I go about fixing this issue and allow BingBot to index our site? Thanks.

Answer 1

The culprit that was leaking non UTF-8 characters in my template was an innocuous meta tag for Facebook Open Graph

%meta{property: "og:url", content: request.url}

And when the request is non-standard, this causes the encoding issue. Changing it to

%meta{property: "og:url", content: request.url.force_encoding('UTF-8')}

made the trick.