Logo Questions Linux Laravel Mysql Ubuntu Git Menu
 

Printing rows, Java heap space [closed]

I want to print each line from a huge textfile (more than 600 000 MB).

But when I try the code below I get "...OutOfMemoryError: Java heap space" right before reaching line number 1 000 000.

Is there a better way to handle the input rather than FileReader and LineNumberReader?

FileReader fReader = new FileReader(new File("C:/huge_file.txt"));
LineNumberReader lnReader = new LineNumberReader(fReader);
String line = "";
while ((line = lnReader.readLine()) != null) {
    System.out.println(lnReader.getLineNumber() + ": " + line);
}
fReader.close();
lnReader.close();

Thanks in advance!


Thanks all for your answers!

I finally found the memory leak, an unused java class instance which duplicated it self for each row iteration. In other words, it had nothing to do with the file loading part.

like image 728
carloscloud Avatar asked May 03 '11 11:05

carloscloud


People also ask

What happens when heap memory is full in Java?

Java objects reside in an area called the heap. The heap is created when the JVM starts up and may increase or decrease in size while the application runs. When the heap becomes full, garbage is collected. During the garbage collection objects that are no longer used are cleared, thus making space for new objects.

What causes Java heap space error?

Usually, this error is thrown when there is insufficient space to allocate an object in the Java heap. In this case, The garbage collector cannot make space available to accommodate a new object, and the heap cannot be expanded further.

How do I clean heap memory?

The heap is cleared by the garbage collector whenever it feels like it. You can ask it to run (with System. gc() ) but it is not guaranteed to run.


1 Answers

LineNumberReader extends BufferedReader. It may be that the buffered reader is buffering too much. Running the program through a profiler should prove this without a doubt.

One of the constructors of the BufferedReader takes a buffer size, this constructor is also available in the line number reader.

replace:

LineNumberReader lnReader = new LineNumberReader(fReader);

with:

LineNumberReader lnReader = new LineNumberReader(fReader, 4096);
like image 182
Andrew Wheat Avatar answered Oct 16 '22 19:10

Andrew Wheat