Android huge JSONObject toString goes OutOfMemoryError

Question

I need to read a lot of data from a SQLite database and create a properly formatted JSON.

I'm currently achieving this by having an object called RequestPayload which contains some ArrayLists in which I put the data I read from SQLite.

The RequestPayload class has a parseJson() method which returns a JSONObject on which I eventually call the toString() method to obtain my JSON String that finally got written on a file.

The Problem

Now, when I've got "small" quantities of data in SQLite everything goes fine. When I've got a lot of data this is what happens:

06-28 09:55:34.121 10857-6799/it.example.sampler E/AndroidRuntime: FATAL EXCEPTION: Thread-195240
    Process: it.example.sampler, PID: 10857
    Theme: themes:{com.cyanogenmod.trebuchet=overlay:system, com.android.settings=overlay:system, default=overlay:system, iconPack:system, fontPkg:system, com.android.systemui=overlay:system, com.android.systemui.navbar=overlay:system}
    java.lang.OutOfMemoryError: Failed to allocate a 52962812 byte allocation with 16764752 free bytes and 41MB until OOM
      at java.lang.StringFactory.newStringFromChars(Native Method)
      at java.lang.AbstractStringBuilder.toString(AbstractStringBuilder.java:629)
      at java.lang.StringBuilder.toString(StringBuilder.java:663)
      at org.json.JSONStringer.toString(JSONStringer.java:430)
      at org.json.JSONObject.toString(JSONObject.java:690)
      at it.example.sampler.controllers.network.RequestBodyEncoder.serialise(RequestBodyEncoder.java:69)
      at it.example.sampler.controllers.network.RequestBodyEncoder.createPacket(RequestBodyEncoder.java:50)
      at it.example.sampler.services.FileStoreRunnable.run(FileStoreRunnable.java:57)
      at java.lang.Thread.run(Thread.java:818)

06-28 09:55:34.509 10857-10857/it.example.sampler E/WindowManager: android.view.WindowLeaked: Activity it.example.sampler.ui.StartSamplingActivity has leaked window com.android.internal.policy.PhoneWindow$DecorView{74710d V.E...... R......D 0,0-1026,494} that was originally added here
    at android.view.ViewRootImpl.<init>(ViewRootImpl.java:372)
    at android.view.WindowManagerGlobal.addView(WindowManagerGlobal.java:299)
    at android.view.WindowManagerImpl.addView(WindowManagerImpl.java:86)
    at android.app.Dialog.show(Dialog.java:319)
    at it.example.sampler.ui.StartSamplingActivity.executeStop(StartSamplingActivity.java:305)
    at it.example.sampler.ui.StartSamplingActivity.onClickedButton(StartSamplingActivity.java:256)
    at it.example.sampler.ui.StartSamplingActivity$3.onClick(StartSamplingActivity.java:190)
    at android.view.View.performClick(View.java:5204)
    at android.view.View$PerformClick.run(View.java:21158)
    at android.os.Handler.handleCallback(Handler.java:739)
    at android.os.Handler.dispatchMessage(Handler.java:95)
    at android.os.Looper.loop(Looper.java:148)
    at android.app.ActivityThread.main(ActivityThread.java:5461)
    at java.lang.reflect.Method.invoke(Native Method)
    at com.android.internal.os.ZygoteInit$MethodAndArgsCaller.run(ZygoteInit.java:726)
    at com.android.internal.os.ZygoteInit.main(ZygoteInit.java:616)

---

 The Code

the error line is from this method:

private String serialise() throws JSONException {
    // Serialise
    String sampleString = mPayload.parseJSON().toString(); // THIS LINE

    Logger.get().d(LOG_TAG, "Serialised payload: -> " + sampleString);
    return sampleString;
}

and if I got the LogCat correctly the error is during the execution of the toString() method and it appears that the JSON object is too big.

How can I handle this situation?

---

UPDATE: To answer to @YashJain comment asking for the size of the JSONObject.

After sampling a sampling lasted 2 hours I had a JSON containing:

• 12 JSONArray containing 155.432 floats
• 2 JSONArray containing 8.393 floats
• 1 JSONArray containing 8.393 custom object containing 5 floats each (and a String)
• 1 2D array containing 155.432 ints

In terms of bytes since a Float is made by 4 bytes (I hope I've done the calculus correctly) I've got circa:

(12 * 155432 * 4) + (2 * 8393 * 4) + (1 * 8393 * 5) + (1 * 155432 * 4) ~= 8.191.573 bytes

UPDATE: I've been asked about the used compression algorithm. I use ZLIB via the Deflater and DeflaterOutputStream Java classes.

Thus the flow is:

• Sample data -> Store them in SQLite
• Read from SQLite -> Build RequestPayload object in memory
• Convert RequestPayload object to JSONObject (with a custom parser).
• Convert the JSONObject to String (using JSONObject toString() method)
• Compress the String bytes (using getBytes()) -> encode it in Base64
• Send final Base64 string to server

EDIT: To address to the "possibile duplicate flag" I haven't seen that question during my researches by the way that didn't help me since it makes use of largeHeap directive (which I don't use). Also there's no accepted answer and the only answer doesn't actually provide a practical solution.

Answer 1

Sorry for not giving any more feedbacks I've been busy with other works and this got last in my priority queue.

Anyhow what actually worked for me is using Google GSON library.

My serialisation method has become simply:

private String serialise() {
    // Serialise
    return mPayload.toStringFromGSON();
}

And method toStringFromGSON() in class Payload is:

public String toStringFromGSON() {
    GsonBuilder gsonBuilder = new GsonBuilder();
    Gson gson = gsonBuilder.create();

    return gson.toJson(this);
}

---

Anyhow I think this can be considered only a temporary patch. In fact I noticed that the memory used by the application increases dramatically during this phase hence I think the problem is only postponed and it will eventually occur with heavier loads..