How do you parse large files in Groovy without exceeding heap size?

Question

When parsing a large file I get the following error Caught: java.lang.OutOfMemoryError: Java heap space

How do you parse large files in Groovy without exceeding heap size?

example code that fails with large files...

import java.io.File

def inputFile = new File("c:/dev/test.txt")
    inputFile.getText().eachLine{ it, i ->
            ... do something with each line
        }

Answer 1

Ensure that the you're iterating over the file in a way that doesn't load the whole file into memory...

• In this case specifically turn inputFile.getText().eachLine into inputFile.eachLine
• Don't use .readLines() as it will try and load the whole file into memory, .eachLine{..} should be used instead
• You can also extend the heap size with a jvm flag, eg to 1GB by using groovy -Xmx1024M myscript.groovy See also answer here

See this page on the groovy mailing list for more info and further discussion

Code that works without a heap space error...

def inputFile = new File("c:/dev/test.txt")
inputFile.eachLine{ it, i ->
        ... do something with each line
    }