Changing encoding in idea intellij doesn't work

Question

I have a .java file with a string String s="P�rsh�ndetje bot�!";.

When I open this file in Notepad++ and change encoding to ISO-8859-1 it shows appropriate string: "Përshëndetje botë!", but if i open the file in idea intellij and change encoding to ISO-8859-1, it gives me a warning of how some symbols can't be converted and then replaces those symbols with ? mark: "P?rsh?ndetje bot?!".

Why is this happening? Why Notepad++ is able to convert the file, while idea is not?

Answer 1

I'm not sure, but it is possible that when you first opened the file it was read as UTF-8 and the invalid byte sequences were turned into the Unicode replacement character, then when you try to convert to ISO-8859-1 it is trying to convert the Unicode replacement character but there is no value for that in ISO-8859-1 so it is converted to ? instead.

(Even though text like "ërs" can be represented in Unicode and thus UTF-8, the ISO-8859-1 encoding of "ërs" is EB 72 73 which is the start byte of a three-byte UTF-8 sequence, but the next two bytes are not continuation bytes, so a program treating it as UTF-8 would think those accented characters are invalid.)

I think you need to get IntelliJ to open the file as ISO-8859-1, rather than opening it first as UTF-8 and then trying to convert to ISO-8859-1.

(When you switch the encoding in Notepad++ it must be going back to the original bytes of the file and interpreting them as ISO-8859-1, rather than trying to convert content that it has already altered by changing invalid bytes to the replacement character.)

Note that ë is a perfectly valid Unicode character. It can be represented as either U+00EB, Latin small letter e with diaeresis, or as two code points, U+0065 and U+0308, Latin small letter e combined with Combining diaeresis. But U+00EB would be encoded in UTF-8 as the two-byte sequence C3 AB, and for U+0065 U+0308 the "e" would be encoded as itself, 65, and U+0308 would be encoded as CC 88.

So "ë" in UTF-8 must be either C3 AB or 65 CC 88. It can't be EB.

Answer 2

I believe there is some bug here in IDEA (where the default encoding is UTF-8) in that when you convert the file containing valid ISO-8859-1 encoded characters and change the file encoding to ISO-8859-1, it messes it up. The particular codepoint that it messes up is ë. For some reason, it replaces it with \ufffd whereas its correct codepoint is \u00eb. This is the character that shows up as � in your editor.

My suggestion is to just use UTF-8 and not change it to ISO-8859-1. UTF-8 is backward compatible with ISO-8859-1 and you could write this string using the IME on your OS (which appears to be Windows). I am not sure how to do it on Windows, but on a Mac, I use the U+ keyboard

and then add this character as 00eb while keeping the ALT key pressed. Then it shows up correctly: