Google App Engine datastore encoding?

Question

I'm using the GAE datastore for a Java application, and storing some text that will be in numerous languages. In my servlet, I'm first checking to see if there's any data in the data store, and, if not, I'm creating some, similar to the following:

ArrayList<Lang> list = new ArrayList<Lang>();
list.add(new Lang("EN", "English", 1));
list.add(new Lang("ES", "Español", 0));
//more languages here...

PersistenceManager pm = PMF.get().getPersistenceManager();
for(Lang l : list) {
  pm.makePersistent(l);
}

Since this is using JDO, I guess I should include the relevent parts of the Lang class too:

@PersistenceCapable
public class Lang {
 @PrimaryKey
 private String code;
 @Persistent
 private String name;
 @Persistent
 private int popularity;
// getters & setters & constructors...
}

However, the non-ASCII characters are giving me grief. I've set my Eclipse project to use the UTF-8 encoding instead of the default Cp1252, so I think I'm okay from that perspective, but when I use the App Engine Data Viewer to look at my data, that Español entry becomes Español, and when I click on it to view it, I get a 500 Server Error. (There are some other entries with right-to-left text that don't even show up in the Data Viewer at all, but one problem at a time...)

Is there anything special I can do in my code to set the character encoding, or specify to GAE that the data I'm storing is UTF-8? Or is the problem on the Eclipse side, and is there something I should be doing with my Java code?

Answer 1

Fixed same issue by setting both request and response encoding to utf-8. Request encoding results in valid string stored in datastore, without it values will be stored as "????..."

Requests: if you use Apache HTTP Client, this is done in the following way:

Get request:

NameValuePair... params;
...
String url = urlBase + URLEncodedUtils.format(Arrays.asList(params), "UTF-8");
HttpGet httpGet = new HttpGet(url);

Post request:

NameValuePair... params;
...
HttpPost httpPost = new HttpPost(url);
httpPost.setEntity(new UrlEncodedFormEntity(Arrays.asList(params), "UTF-8"));

Response: if you build your response in HttpServlet, this is done in a following way:

HttpServletResponse resp;
...
resp.setContentType("text/html; charset=utf-8");