How to convert UTF8 to Unicode

Question

I try to convert a UTF8 string to a Java Unicode string.

Click to copy

String question = request.getParameter("searchWord");
byte[] bytes = question.getBytes();
question = new String(bytes, "UTF-8");

The input are Chinese Characters and when I compare the hex code of each caracter it is the same Chinses character. So I'm pretty sure that the charset is UTF8.

Where do I go wrong?

Answer 1

There's no such thing as a "UTF-8 string" in Java. Everything is in Unicode.

When you call String.getBytes() without specifying an encoding, that uses the platform default encoding - that's almost always a bad idea.

You shouldn't have to do anything to get the right characters here - the request should be handling it all for you. If it's not doing so, then chances are it's lost data already.

Could you give an example of what's actually going wrong? Specify the Unicode values of the characters in the string you're receiving (e.g. by using toCharArray() and then converting each char to an int) and what you expected to receive.

EDIT: To diagnose this, use something like this:

Click to copy

public static void dumpString(String text) {
    for (int i = 0; i < text.length(); i++) {
        System.out.println(i + ": " + (int) text.charAt(i));
    }
}

Note that that will give the decimal value of each Unicode character. If you have a handy hex library method around, you may want to use that to give you the hex value. The main point is that it will dump the Unicode characters in the string.

Answer 2

First make sure that the data is actually encoded as UTF-8.

There are some inconsistency between browsers regarding the encoding used when sending HTML form data. The safest way to send UTF-8 encoded data from a web form is to put that form on a page that is served with the Content-Type: text/html; charset=utf-8 header or contains a <meta http-equiv="Content-Type" content="text/html; charset=UTF-8" /> meta tag.

---

Now to properly decode the data call request.setCharacterEncoding("UTF-8") in your servlet before the first call to request.getParameter().

The servlet container takes care of the encoding for you. If you use setCharacterEncoding() properly you can expect getParameter() to return normal Java strings.