Sending UTF-8 string using HttpURLConnection

Question

till now I've used the following code snippet in order to send and recieve JSON strings:

static private String sendJson(String json,String url){
    HttpClient httpClient = new DefaultHttpClient();
    String responseString = "";
    try {
        HttpPost request = new HttpPost(url);
        StringEntity params =new StringEntity(json, "UTF-8");
        request.addHeader("content-type", "application/json");
        request.setEntity(params);
        HttpResponse response = httpClient.execute(request);
        HttpEntity entity = response.getEntity();
        responseString = EntityUtils.toString(entity, "UTF-8");

    }catch (Exception ex) {
        ex.printStackTrace();
        // handle exception here
    } finally {
        httpClient.getConnectionManager().shutdown();
    }
    return responseString;
}

The code above worked perfect even if the json string contained UTF-8 chars, and everything worked fine.

For several reasons I had to change the way I send HTTP post requests and use HttpURLConnection instead apache's HttpClient. Here's my code:

static private String sendJson(String json,String url){
    String responseString = "";
    try {
        URL m_url = new URL(url);
        HttpURLConnection conn = (HttpURLConnection)m_url.openConnection();
        conn.setDoInput(true);
        conn.setDoOutput(true);
        conn.setUseCaches(false);
        conn.setRequestMethod("POST");
        conn.setRequestProperty("content-type", "application/json");
        DataOutputStream outputStream = new DataOutputStream(conn.getOutputStream());
        outputStream.writeBytes(json);
        outputStream.flush();
        outputStream.close();

        BufferedReader br = new BufferedReader(new InputStreamReader(conn.getInputStream()));
        StringBuilder sb = new StringBuilder();
        String line;
        while ((line = br.readLine()) != null) {
            sb.append(line+"\n");
        }
        br.close();
        responseString = sb.toString();
    } catch (MalformedURLException e) {
        // TODO Auto-generated catch block
        e.printStackTrace();
    } catch (IOException e) {
        // TODO Auto-generated catch block
        e.printStackTrace();
    }

    return responseString;
}

This code works well for normal English characters, but doesn't seem to support UTF-8 characters in the json string, since it fails each time. (when sending json to server, server crushes saying that utf8 cant decode a certain byte, but when recieving utf8 json from server I think it does work since I manage to view the special characters).

Server didn't change at all and worked fine with previous code, so the problem is 100% on this new code snippet.

Any idea how to fix the json string sending so it would support UTF 8? Thanks

Answer 1

I think the problem is in this part:

DataOutputStream outputStream = new DataOutputStream(conn.getOutputStream());
outputStream.writeBytes(json);
outputStream.flush();
outputStream.close();

Instead of doing this you need to encode json as UTF-8
and send those bytes which represent the UTF-8 encoding.

Try using this:

Charset.forName("UTF-8").encode(json)

See:

Charset.encode

An even simpler approach is to use e.g. a BufferedWriter wrapping an
OutputStreamWriter. The OutputStreamWriter knows about its own encoding
and so it will do the work for you (the encoding work of the json String).

BufferedWriter bw = new BufferedWriter(new OutputStreamWriter(conn.getOutputStream(), "UTF-8"));
bw.write(json);
bw.flush();
bw.close();

Answer 2

When writing a String to an output stream (bytes), you need to specify the encoding to do the conversion. One way to do is to wrap the output stream in a OutputStreamWriter that will use UTF-8 charset for the encoding.

        conn.setRequestProperty("content-type", "application/json;  charset=utf-8");
        Writer writer = new BufferedWriter(new OutputStreamWriter(conn.getOutputStream(), "UTF-8"));
        writer.write(json);
        writer.close();

The flush() is also optional if you call close().

Another option, as mentionned by peter.petrov is to first convert your String to bytes (in memory) and then output the byte array to your output stream.

And to make it obvious on the server side, you can pass the charset used in the content-type header ("content-type", "application/json; charset=utf-8").

Answer 3

StringEntity uses a Charset to make sure the encoding is right. It does ~that:

byte[] content = s.getBytes(charset);

Without much change in your code, your write can be:

outputStream.write(json.getBytes("UTF-8"));

As for your read, there is no point in using a BufferedReader with readLine, except for normalizing the end of line. It is much slower than other methods, as it requires to read each byte individually.

EntityUtils does mostly that:

    Reader reader = new InputStreamReader(conn.getInputStream(), "UTF-8");
    StringBuilder buffer = new StringBuilder();
    char[] tmp = new char[1024];
    int l;
    while((l = reader.read(tmp)) != -1) {
        buffer.append(tmp, 0, l);
    }
    responseString = buffer.toString();