How to deal with big strings and limited memory

Question

I have a file from which I read data. All the text from this file is stored in a String variable (a very big variable). Then in another part of my app I want to walk through this string and extract useful information, step-by-step (parsing the string).

In the meanwhile my memory gets full and an OutOfMemory exception keeps me from further processing. I think it would be better to process the data directly while reading the inputstream from the file. But for organizational aims, I would like to pass the String to another part in my application.

What should I do to keep the memory from overflowing?

Answer 1

You should be using the BufferedInputReader instead of storing this all into one large string.

If what you want to parse happens to be on the same line, then StringTokenizer will work quite nicely, else you have to devise a way to read what you want from the file to parse out statements, then apply StringTokenizer to each statement.

Answer 2

If you can loosen your requirements a bit you could implement a java.lang.CharSequence backed by your file.

The CharSequence is supported many places in the JDK (A String is a CharSequence) . So this is a good alternative to a Reader-based implementation.