Send utf-8 encoded strings in Android HTTP request

Question

I have made a HTTP-post inside my android application. Values are sent as strings from my app to my webserver. Problem is, the values are not in UTF-8 as I want them to be. My webserver has UTF-8 encoding so I know that there is code inside my app that I need to change. See my snippet below:

private void sendPostRequest(String facebookId, String name, String email) {

        class SendPostReqAsyncTask extends AsyncTask<String, Void, String>{

            @Override
            protected String doInBackground(String... bcs) {

                String bcFacebookId = bcs[0];
                String bcName = bcs[1];
                String bcEmail = bcs[2];



                HttpClient httpClient = new DefaultHttpClient();

                HttpPost httpPost = new HttpPost("URL");

                BasicNameValuePair facebookIdBasicNameValuePair = new BasicNameValuePair("bcFacebookId", bcFacebookId);
                BasicNameValuePair nameBasicNameValuePair = new BasicNameValuePair("bcName", bcName);
                BasicNameValuePair emailBasicNameValiePair = new BasicNameValuePair("bcEmail", bcEmail);

                List<NameValuePair> nameValuePairList = new ArrayList<NameValuePair>();
                nameValuePairList.add(facebookIdBasicNameValuePair);
                nameValuePairList.add(nameBasicNameValuePair);
                nameValuePairList.add(emailBasicNameValiePair);
                try {

                    UrlEncodedFormEntity urlEncodedFormEntity = new UrlEncodedFormEntity(nameValuePairList);

                    httpPost.setEntity(urlEncodedFormEntity);

                    try {
                        HttpResponse httpResponse = httpClient.execute(httpPost);

                        InputStream inputStream = httpResponse.getEntity().getContent();

                        InputStreamReader inputStreamReader = new InputStreamReader(inputStream);

                        BufferedReader bufferedReader = new BufferedReader(inputStreamReader);

                        StringBuilder stringBuilder = new StringBuilder();

                        String bufferedStrChunk = null;

                        while((bufferedStrChunk = bufferedReader.readLine()) != null){
                            stringBuilder.append(bufferedStrChunk);
                        }

                        return stringBuilder.toString();

                    } catch (ClientProtocolException cpe) {

                        cpe.printStackTrace();
                    } catch (IOException ioe) {
                        System.out.println("Second Exception caz of HttpResponse :" + ioe);
                        ioe.printStackTrace();
                    }

                } catch (UnsupportedEncodingException uee) {
                    System.out.println("An Exception given because of UrlEncodedFormEntity argument :" + uee);
                    uee.printStackTrace();
                }

                return null;
            }

For an example, the letter 'ö' becomes a '?'. How do I fix this? Cheers!

Answer 1

The biggest single reason that characters get converted into question marks is the conversion of characters to bytes, and then back into characters, not matching.

The code you have supplied has this line:

InputStreamReader inputStreamReader = new InputStreamReader(inputStream);

This is problematic because you are not specifying how to convert the bytes into characters. Instead you probably want this:

InputStreamReader inputStreamReader = new InputStreamReader(inputStream, "UTF-8");

What you specify for the character encoding will depend upon the character encoding that you have specified elsewhere. Without specifying the character encoding, you will get the "default" character encoding, and that depends upon settings in both the client and the server. Java uses Unicode, and UTF-8 is the only encoding that will preserve all the characters that Java allows.

For debugging, you may want to use the InputStream and retrieve bytes from that, and print out the byte values, in order to verify that they are indeed UTF-8 encoded representations of the original character values. The proper encoding of 'ö' (x00F6) is 'ö' (x00C3 x00B6).

You will also need to assure that the original POST request is properly UTF-8 encoded. The UrlEncodedFormEntity class also uses the default character encoding, which might not be UTF-8. Change this:

UrlEncodedFormEntity uefe = new UrlEncodedFormEntity(nameValuePairList);

to

UrlEncodedFormEntity uefe = new UrlEncodedFormEntity(nameValuePairList, "UTF-8");