Java/Android: java.lang.OutOfMemoryError while building a JSON object

Question

I am importing JSON data from a public database URI http://data.seattle.gov/api/views/3k2p-39jp/rows.json and the rows go as far as 445454. Using the following code I am constructing the JSON object of the entire data.

   HttpGet get = new HttpGet(uri);
   HttpClient client = new DefaultHttpClient();
   HttpResponse response = client.execute(get);
   BufferedReader reader = new BufferedReader(new InputStreamReader(response.getEntity().getContent(), "UTF-8"));
    StringBuilder builder=new StringBuilder();
for(String line=null;(line = reader.readLine()) != null;){
      builder.append(line).append("\n");
    }
  JSONTokener jsonTokener=new JSONTokener(builder.toString());
  JSONObject finalJson=new JSONObject(jsonTokener);
  JSONArray data=finalJson.getJSONArray("data");

Because the data is too large, i am getting 03-21 03:41:49.714: E/AndroidRuntime(666): Caused by: java.lang.OutOfMemoryError pointing the source of error at buildr.append(line).append("\n"). Is there anyway I can handle large datasets without getting memory allocation issues?

Answer 1

That JSON is huge!

You definitely need to use a streaming JSON parser. There are two out there for Android: GSON and Jackson.

GSON Streaming is explained at: https://sites.google.com/site/gson/streaming

I like how GSON explains the problem you're having:

Most applications should use only the object model API. JSON streaming is useful in just a few situations:

When it is impossible or undesirable to load the entire object model into memory. This is most relevant on mobile platforms where memory is limited.

Jackson Streaming is documented at: http://wiki.fasterxml.com/JacksonInFiveMinutes#Streaming_API_Example

Answer 2

If possible only request parts of the data - this also reduces time for network io and thus saves battery.

Otherwise you could try to not keep the incoming data in memory, but to 'stream' it onto the sd-card. When it is stored there you can then iterate over it. Most likely this will mean to use your own JSON tokenizer that does not build a full tree, but which is able to (like a SAX parser) only look at a part of the object tree at a time.

You may have a look at Jackson, which has a streaming mode, which may be applicable.