java outOfMemoryError with stringbuilder

Question

I'm getting a java outOfMemoryError when I call this method - i'm using it in a loop to parse many large files in sequence. my guess is that result.toString() is not getting garbage collected properly during the loop. if so, how should i fix it?

private String matchHelper(String buffer, String regex, String method){
    Pattern abbrev_p = Pattern.compile(regex);//norms U.S.A., B.S., PH.D, PH.D.
    Matcher abbrev_matcher = abbrev_p.matcher(buffer);
    StringBuffer result = new StringBuffer();
    while (abbrev_matcher.find()){
            abbrev_matcher.appendReplacement(result, abbrevHelper(abbrev_matcher));
    }
    abbrev_matcher.appendTail(result);
    String tempResult = result.toString(); //ERROR OCCURS HERE
  return tempResult;

}

Answer 1

Written this way, you'll need roughly 6 bytes of memory for every character in the file.

Each character is two bytes. You have the raw input, the substituted output (in the buffer), and you are asking for a third copy when you run out of memory.

If the file is encoded in something like ASCII or ISO-8859-1 (a single-byte character encoding), that means it will be six times larger in memory than on disk.

You could allocate more memory to the process, but a better solution might be to process the input "streamwise"—read, scan, and write the data without loading it all into memory at once.

Answer 2

If your files to be processed are all very large, say more than a few hundred MB, then you really should go with stream processing instead of this "loading all into memory" way, just as @erickson suggested.

Otherwise, there are a few things you could try, all to reduce memory usage as much as possible:

1. Try properly enlarge your heap size if not yet (when applicable).
2. Give StringBuffer an initial size same as the lenght of the given String buffer. This should reduce the unnecessary memory usage while expanding the StringBuffer in the process. I assume it is only replacing certain words of the original string and should be more or less the same in length.
3. If possible, maybe you could return the generated StringBuffer object instead. Calling its toString() only after you get rid of the original String object.