Logo Questions Linux Laravel Mysql Ubuntu Git Menu
 

How do I convert special UTF-8 chars to their iso-8859-1 equivalent using javascript?

I'm making a javascript app which retrieves .json files with jquery and injects data into the webpage it is embedded in.

The .json files are encoded with UTF-8 and contains accented chars like é, ö and å.

The problem is that I don't control the charset on the pages that are going to use the app.

Some will be using UTF-8, but others will be using the iso-8859-1 charset. This will of course garble the special chars from the .json files.

How do I convert special UTF-8 chars to their iso-8859-1 equivalent using javascript?

like image 515
Hobhouse Avatar asked Mar 22 '11 19:03

Hobhouse


People also ask

How do I convert UTF-8 to ISO-8859-1?

byte[] utf8 = ... byte[] latin1 = new String(utf8, "UTF-8"). getBytes("ISO-8859-1"); You can exercise more control by using the lower-level Charset APIs. For example, you can raise an exception when an un-encodable character is found, or use a different character for replacement text.

What is the difference between UTF-8 and ISO-8859-1?

UTF-8 is a multibyte encoding that can represent any Unicode character. ISO 8859-1 is a single-byte encoding that can represent the first 256 Unicode characters. Both encode ASCII exactly the same way.

Is UTF-8 a subset of ISO-8859-1?

ISO-8859-1 contains a subset of UTF-8 Unicode, which substantially overlaps with ASCII. All ASCII is UTF-8 Unicode. All the ISO 8859-1 (ISO Latin 1) characters below codes 7f hex are ASCII compatible and UTF-8 compatible in one byte.

How do I change my UTF-8 character set?

Click Tools, then select Web options. Go to the Encoding tab. In the dropdown for Save this document as: choose Unicode (UTF-8). Click Ok.


2 Answers

Actually, everything is typically stored as Unicode of some kind internally, but lets not go into that. I'm assuming you're getting the iconic "åäö" type strings because you're using an ISO-8859 as your character encoding. There's a trick you can do to convert those characters. The escape and unescape functions used for encoding and decoding query strings are defined for ISO characters, whereas the newer encodeURIComponent and decodeURIComponent which do the same thing, are defined for UTF8 characters.

escape encodes extended ISO-8859-1 characters (UTF code points U+0080-U+00ff) as %xx (two-digit hex) whereas it encodes UTF codepoints U+0100 and above as %uxxxx (%u followed by four-digit hex.) For example, escape("å") == "%E5" and escape("あ") == "%u3042".

encodeURIComponent percent-encodes extended characters as a UTF8 byte sequence. For example, encodeURIComponent("å") == "%C3%A5" and encodeURIComponent("あ") == "%E3%81%82".

So you can do:

fixedstring = decodeURIComponent(escape(utfstring)); 

For example, an incorrectly encoded character "å" becomes "Ã¥". The command does escape("Ã¥") == "%C3%A5" which is the two incorrect ISO characters encoded as single bytes. Then decodeURIComponent("%C3%A5") == "å", where the two percent-encoded bytes are being interpreted as a UTF8 sequence.

If you'd need to do the reverse for some reason, that works too:

utfstring = unescape(encodeURIComponent(originalstring)); 

Is there a way to differentiate between bad UTF8 strings and ISO strings? Turns out there is. The decodeURIComponent function used above will throw an error if given a malformed encoded sequence. We can use this to detect with a great probability whether our string is UTF8 or ISO.

var fixedstring;  try{     // If the string is UTF-8, this will work and not throw an error.     fixedstring=decodeURIComponent(escape(badstring)); }catch(e){     // If it isn't, an error will be thrown, and we can assume that we have an ISO string.     fixedstring=badstring; } 
like image 153
nitro2k01 Avatar answered Sep 22 '22 10:09

nitro2k01


The problem is that once the page is served up, the content is going to be in the encoding described in the content-type meta tag. The content in "wrong" encoding is already garbled.

You're best to do this on the server before serving up the page. Or as I have been know to say: UTF-8 end-to-end or die.

like image 27
Diodeus - James MacFarlane Avatar answered Sep 18 '22 10:09

Diodeus - James MacFarlane