Android: Convert Stream to String without running out of memory

Question

I have an android client that communicates with the server via REST-ful endpoints and JSON. Because of this, I have the need to retrieve the full server response prior to converting it into a Hash. I have this code in place to do that (found on the internet someplace):

private static String convertStreamToString(InputStream is) {

    BufferedReader reader = new BufferedReader(new InputStreamReader(is));
    StringBuilder sb = new StringBuilder();

    String line = null;
    try {
        while ((line = reader.readLine()) != null) {
            sb.append(line + "\n");
        }
    } catch (IOException e) {
        e.printStackTrace();
    } finally {
        try {
            is.close();
        } catch (IOException e) {
            e.printStackTrace();
        }
    }
    return sb.toString();
}

The code works for the most part, however I am seeing reports of crashes in the field from clients with an OutOfMemory exception on the line:

    while ((line = reader.readLine()) != null) {

The full stack trace is:

java.lang.RuntimeException: An error occured while executing doInBackground()
    at android.os.AsyncTask$3.done(AsyncTask.java:200)
    at java.util.concurrent.FutureTask$Sync.innerSetException(FutureTask.java:273)
    at java.util.concurrent.FutureTask.setException(FutureTask.java:124)
    at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:307)
    at java.util.concurrent.FutureTask.run(FutureTask.java:137)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1068)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:561)
    at java.lang.Thread.run(Thread.java:1102)
Caused by: java.lang.OutOfMemoryError
    at java.lang.String.(String.java:468)
    at java.lang.AbstractStringBuilder.toString(AbstractStringBuilder.java:659)
    at java.lang.StringBuilder.toString(StringBuilder.java:664)
    at java.io.BufferedReader.readLine(BufferedReader.java:448)
    at com.appspot.myapp.util.RestClient.convertStreamToString(RestClient.java:303)
    at com.appspot.myapp.util.RestClient.executeRequest(RestClient.java:281)
    at com.appspot.myapp.util.RestClient.Execute(RestClient.java:178)
    at com.appspot.myapp.$LoadProfilesTask.doInBackground(GridViewActivity.java:1178)
    at com.appspot.myapp.$LoadProfilesTask.doInBackground(GridViewActivity.java:1)
    at android.os.AsyncTask$2.call(AsyncTask.java:185)
    at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:305)
    ... 4 more

My question: is there any way to solve this problem aside from sending smaller chunks of data from the server?

Thanks!

Answer 1

In general, the answer is no, but you can certainly tune the conditions that cause you to run out of memory. In particular, if you send string length ahead of your stream, you will be able to create a StringBuilder with correct array size inside of it. Arrays cannot be resized after creation, so if you run out of array capacity in StringBuilder, the implementation has to allocate a new array (typically twice the size to avoid too many resizes) and then copy the old array contents. Consider stream of size X, to resize StringBuilder that just happened to be a X-1 capacity, you need almost X*3 amount of memory. Sizing StringBuilder such that resizes are avoided will allow you squeeze larger streams into memory.

Another thing you may want to do is to tune the amount of memory available to your server process. Use a switch like -Xmx1024m when launching the server process.

Of course, it would be much better to revise your algorithm to not require the entire stream to be held in memory. It will enable you to handle more clients with the same amount of hardware.

Answer 2

Android has restrictions on the maximum amount of memory you can allocate to your application. You could consider reading the stream on the fly and not save the whole thing into a String if the response is very large. But you should consider following best practice.

You should store the data in a sqlite database or to a regular file. It's not best practice to do what you're doing as the user might hit the home button or receive a phone call when you're in the middle of saving the response. It is better to use a database so that you can come back to the state where you were interrupted. Then you also do not have to worry about running out of memory.

Have you watched this talk on best practices for communicating with REST services from Android? http://www.youtube.com/watch?v=xHXn3Kg2IQE?8m50s (8:50 and 11:20). Highly recommended to clear out best practices and why one should not retrieve REST data without using a database.

In short, consider saving to an sqlite database or a file. If the data is very large you could perhaps consider compressing it before you store it.

Answer 3

There are different ways to tackle these kinds of problems. One way would be to use a hash function that doesn't require the entire stream to be in memory, i.e. you feed it a character or block of characters at a time. Another is to reduce the size of the response.

If you can't do that and you need the entire stream, then I would avoid using readLine() and just call read() on the buffered input stream and append the character you get from read to the string builder. This will reduce the number of strings you are creating and discarding quite dramatically. (A simple optimization to the code above is to take out the newline in the append() call -- you're creating another string unnecessarily there as well.) Also, if you have any idea how long the resulting string will be, you can also set the initial capacity of the string builder upon construction so that you'll know immediately if you'll be out of memory.

Once you get beyond that you have to start splitting the string into blocks that you store on the filesystem.... gets complicated pretty fast.

Answer 4

Maybe this code helps to avoid using StringBuilder and out-of-memory errors:

private String convertStreamToString(InputStream is) {
    ByteArrayOutputStream oas = new ByteArrayOutputStream();
    copyStream(is, oas);
    String t = oas.toString();
    try {
        oas.close();
        oas = null;
    } catch (IOException e) {
        // TODO Auto-generated catch block
        e.printStackTrace();
    }
    return t;
}

private void copyStream(InputStream is, OutputStream os)
{
    final int buffer_size = 1024;
    try
    {
        byte[] bytes=new byte[buffer_size];
        for(;;)
        {
          int count=is.read(bytes, 0, buffer_size);
          if(count==-1)
              break;
          os.write(bytes, 0, count);
        }
    }
    catch(Exception ex){}
}