out of memory error, java heap space

Question

I try to read log file with more than 4 million lines and size more than 400 MB, but I get Out of Memory Error : java heap space. This is my code :

File file = new File("C:\\file.log");
        FileReader fileReader = new FileReader(file);
        BufferedReader bufferedReader = new BufferedReader(fileReader);
        StringBuilder stringBuffer = new StringBuilder();
        String line;
        while ((line = bufferedReader.readLine()) != null) {
            stringBuffer.append(line);
        }

I tried to increase heap memory to 1GB, but still get that message. What would be the possible cause?

Answer 1

Ok, you already should have a clue, reading the comments you got.

Problem explanation:

Your log file has a size of 400MB. Note, that this is measured in bytes. Now you are reading it line by line with line = bufferedReader.readLine() thus converting some bytes to a string.

A String instance in Java internally holds a char[]. But a char in Java takes 2 bytes! So you need at least 800MB of heap space just for storing all the characters. As you are also allocating several other objects, and the JVM itself needs some memory, it is very probable that 1 GB is not enough.

Additionally, the StringBuffer (by the way: better use StringBuilder for that) internally uses again a char[], which is expanded (in length) automatically when needed. This expansion is done by doubling the length. So for a 400MB file it has a char[] with a length of 512M. Still remind: A char takes 2 bytes.

So what is the solution? Simply put: Do not read the entire file into memory!

Do that instead:

class LogAnalyzer {
    private final File logFile;

    LogAnalyzer(File logFile) {
        this.logFile = logFile;
    }

    void analyze() throws IOException {
        try(FileReader fileReader = new FileReader(logFile)) {
            try(BufferedReader bufferedReader = new BufferedReader(fileReader)) {
                String line;
                while ((line = bufferedReader.readLine()) != null) {
                    analyzeLine(line);
                }
            }
        }
    }

    private void analyzeLine(String line) {
        // do whatever you need here
    }
}

If you need to keep some lines, you should store them in some instance fields of the LogAnalyzer, and/or have this class behave like a state machine.