I want to print each line from a huge textfile (more than 600 000 MB).
But when I try the code below I get "...OutOfMemoryError: Java heap space" right before reaching line number 1 000 000.
Is there a better way to handle the input rather than FileReader and LineNumberReader?
FileReader fReader = new FileReader(new File("C:/huge_file.txt"));
LineNumberReader lnReader = new LineNumberReader(fReader);
String line = "";
while ((line = lnReader.readLine()) != null) {
System.out.println(lnReader.getLineNumber() + ": " + line);
}
fReader.close();
lnReader.close();
Thanks in advance!
Thanks all for your answers!
I finally found the memory leak, an unused java class instance which duplicated it self for each row iteration. In other words, it had nothing to do with the file loading part.
Java objects reside in an area called the heap. The heap is created when the JVM starts up and may increase or decrease in size while the application runs. When the heap becomes full, garbage is collected. During the garbage collection objects that are no longer used are cleared, thus making space for new objects.
Usually, this error is thrown when there is insufficient space to allocate an object in the Java heap. In this case, The garbage collector cannot make space available to accommodate a new object, and the heap cannot be expanded further.
The heap is cleared by the garbage collector whenever it feels like it. You can ask it to run (with System. gc() ) but it is not guaranteed to run.
LineNumberReader
extends BufferedReader
. It may be that the buffered reader is buffering too much. Running the program through a profiler should prove this without a doubt.
One of the constructors of the BufferedReader
takes a buffer size, this constructor is also available in the line number reader.
replace:
LineNumberReader lnReader = new LineNumberReader(fReader);
with:
LineNumberReader lnReader = new LineNumberReader(fReader, 4096);
If you love us? You can donate to us via Paypal or buy me a coffee so we can maintain and grow! Thank you!
Donate Us With