Debugging Java Out of Memory Error

Question

I'm still a relatively new programmer, and an issue I keep having in Java is Out of Memory Errors. I don't want to increase the memory using -Xmx, because I feel that the error is due to poor programming, and I want to improve my coding rather than rely on more memory.

The work I do involves processing lots of text files, each around 1GB when compressed. The code I have here is meant to loop through a directory where new compressed text files are being dropped. It opens the second most recent text file (not the most recent, because this is still being written to), and uses the Jsoup library to parse certain fields in the text file (fields are separated with custom delimiters: "|nTa|" designates a new column and "|nLa|" designates a new row).

I feel there should be no reason for using a lot of memory. I open a file, scan through it, parse the relevant bits, write the parsed version into another file, close the file, and move onto the next file. I don't need to store the whole file in memory, and I certainly don't need to store files that have already been processed in memory.

I'm getting errors when I start parsing the second file, which suggests that I'm not dealing with garbage collection. Please have a look at the code, and see if you can spot things that I'm doing that mean I'm using more memory than I should be. I want to learn how to do this right so I stop getting memory errors!

import java.io.File;
import java.io.FileInputStream;
import java.io.FileOutputStream;
import java.util.ArrayList;
import java.util.Collections;
import java.util.Scanner;
import java.util.TreeMap;
import java.util.zip.GZIPInputStream;
import java.util.zip.GZIPOutputStream;

import org.jsoup.Jsoup;

public class ParseHTML {

    public static int commentExtractField = 3;
    public static int contentExtractField = 4;
    public static int descriptionField = 5;

    public static void main(String[] args) throws Exception {

        File directoryCompleted = null;     
        File filesCompleted[] = null;

        while(true) {

            // find second most recent file in completed directory  
            directoryCompleted = new File(args[0]);     
            filesCompleted = directoryCompleted.listFiles();

            if (filesCompleted.length > 1) {

                TreeMap<Long, File> timeStamps = new TreeMap<Long, File>(Collections.reverseOrder());

                for (File f : filesCompleted) {
                    timeStamps.put(getTimestamp(f), f);
                }

                File fileToProcess = null;

                int counter = 0;

                for (Long l : timeStamps.keySet()) {
                    fileToProcess = timeStamps.get(l);
                    if (counter == 1) {
                        break;
                    }
                    counter++;
                }   

                // start processing file
                GZIPInputStream gzipInputStream = null;

                if (fileToProcess != null) {
                    gzipInputStream = new GZIPInputStream(new FileInputStream(fileToProcess));
                }

                else {
                    System.err.println("No file to process!");
                    System.exit(1);
                }

                Scanner scanner = new Scanner(gzipInputStream);
                scanner.useDelimiter("\\|nLa\\|");

                GZIPOutputStream output = new GZIPOutputStream(new FileOutputStream("parsed/" + fileToProcess.getName()));

                while (scanner.hasNext()) {
                    Scanner scanner2 = new Scanner(scanner.next()); 
                    scanner2.useDelimiter("\\|nTa\\|");

                    ArrayList<String> row = new ArrayList<String>();

                    while(scanner2.hasNext()) {
                        row.add(scanner2.next());   
                    }

                    for (int index = 0; index < row.size(); index++) {
                        if (index == commentExtractField ||
                                index == contentExtractField ||
                                index == descriptionField) {
                            output.write(jsoupParse(row.get(index)).getBytes("UTF-8"));
                        }

                        else {
                            output.write(row.get(index).getBytes("UTF-8"));
                        }   

                        String delimiter = "";

                        if (index == row.size() - 1) {
                            delimiter = "|nLa|";
                        }

                        else {
                            delimiter = "|nTa|";
                        }

                        output.write(delimiter.getBytes("UTF-8"));
                    }
                }

                output.finish();
                output.close();
                scanner.close();
                gzipInputStream.close();


            }
        }
    }

    public static Long getTimestamp(File f) {
        String name = f.getName();
        String removeExt = name.substring(0, name.length() - 3);
        String timestamp = removeExt.substring(7, removeExt.length());
        return Long.parseLong(timestamp);
    }

    public static String jsoupParse(String s) {
        if (s.length() == 4) {
            return s;
        }

        else {
            return Jsoup.parse(s).text();
        }
    }
}

How can I make sure that when I finish with objects, they are destroyed and not using any resources? For example, each time I close the GZIPInputStream, GZIPOutputStream and Scanner, how can I make sure they're completely destroyed?

For the record, the error I'm getting is:

Exception in thread "main" java.lang.OutOfMemoryError: Java heap space
at java.util.Arrays.copyOf(Arrays.java:2882)
at java.lang.AbstractStringBuilder.expandCapacity(AbstractStringBuilder.java:100)
at java.lang.AbstractStringBuilder.append(AbstractStringBuilder.java:572)
at java.lang.StringBuilder.append(StringBuilder.java:203)
at org.jsoup.parser.TokeniserState$47.read(TokeniserState.java:1171)
at org.jsoup.parser.Tokeniser.read(Tokeniser.java:42)
at org.jsoup.parser.TreeBuilder.runParser(TreeBuilder.java:101)
at org.jsoup.parser.TreeBuilder.parse(TreeBuilder.java:53)
at org.jsoup.parser.Parser.parse(Parser.java:24)
at org.jsoup.Jsoup.parse(Jsoup.java:44)
at ParseHTML.jsoupParse(ParseHTML.java:125)
at ParseHTML.main(ParseHTML.java:81)

Answer 1

I haven't spent very long analysing your code (nothing stands out), but a good general-purpose start would be to familiarise yourself with the free VisualVM tool. This is a reasonable guide to its use, though there are many more articles.

There are better commercial profilers in my opinion - JProfiler for one - but it will at the very least show you what objects/classes most memory is being assigned to, and possibly the method stack traces that caused that to happen. More simply it shows you heap allocation over time, and you can use this to judge whether you are failing to clear something or whether it is an unavoidable spike.

I suggest this rather than looking at the specifics of your code because it is a useful diagnostic skill to have.