Out of heap space during serialization

Question

The following code is causing a OutOfMemmoryError: heap space for some 3 million rows.

Memory allocated to JVM is 4 GB, using 64 bit installation.

while (rs.next())
{    
    ArrayList<String> arrayList = new ArrayList<String>();
    for (int i = 1; i <= columnCount; i++)
    {
        arrayList.add(rs.getString(i));
    }

    objOS.writeObject(arrayList);
}

The memory referenced by the ArrayList is eligible for garbage collection in each iteration of the while loop, and internally JVM calls garbage collection (System.gc()) before throwing an OutOfMemoryError because of heap space.

So why is the exception occurring?

Answer 1

Is objOS an ObjectOutputStream?

If so, then that's your problem: An ObjectOutputStream keeps a strong reference to every object that was ever written to it in order to avoid writing the same object twice (it will simply write a reference saying "that object that I wrote before with id x").

This means that you're effectively leaking all ArrayList istances.

You can reset that "cache" by calling reset() on your ObjectOutputStream. Since you don't seem to be making use of that cache between writeObject calls anyway, you could call reset() directly after the writeObject() call.

Answer 2

I agree with @Joachim.

The below suggestion was a myth

In addition, it is recommended (in good coding convention) that do not declare any object inside the loop. Instead, declare it just before the loop start and use the same reference for initialization purpose. This will ask your code to use the same reference for each iterations and cause less burden on memory release thread (i.e. Garbage collection).

The Truth
I have edited this because I feel that there may be many people who (like me before today) still believe that declaring an object inside loop could harm the memory management; which is wrong.
To demonstrate this, I have used the same code posted on stackOverflow for this.
Following is my code snippet

package navsoft.advskill.test;

import java.util.ArrayList;

public class MemoryTest {

    /**
     * @param args
     */
    public static void main(String[] args) {
        /* Total number of processors or cores available to the JVM */
        System.out.println("Available processors (cores): "
                + Runtime.getRuntime().availableProcessors());
        /*
         * Total amount of free memory available to the JVM
         */
        long freeMemory = Runtime.getRuntime().freeMemory();
        System.out.println("Free memory (bytes): "
                + freeMemory);
        /*
         * This will return Long.MAX_VALUE if there is no preset limit
         */
        long maxMemory = Runtime.getRuntime().maxMemory();
        /*
         * Maximum amount of memory the JVM will attempt to use
         */
        System.out.println("Maximum memory (bytes): "
                + (maxMemory == Long.MAX_VALUE ? "no limit" : maxMemory));
        /*
         * Total memory currently in use by the JVM
         */
        System.out.println("Total memory (bytes): "
                + Runtime.getRuntime().totalMemory());
        final int LIMIT_COUNTER = 1000000;
        
        //System.out.println("Testing Only for print...");
        System.out.println("Testing for Collection inside Loop...");
        //System.out.println("Testing for Collection outside Loop...");
        //ArrayList<String> arr;
        for (int i = 0; i < LIMIT_COUNTER; ++i) {
            //arr = new ArrayList<String>();
            ArrayList<String> arr = new ArrayList<String>();
            System.out.println("" + i + ". Occupied(OldFree - currentFree): "+ (freeMemory - Runtime.getRuntime().freeMemory()));
        }
        System.out.println("Occupied At the End: "+ (freeMemory - Runtime.getRuntime().freeMemory()));
        System.out.println("End of Test");
    }

}

The result from the output is clearly shows that there is no difference in occupying/freeing the memory if you either declare the object inside or outside the loop. So it is recommended to have the declaration to as small scope as it can.
I pay my thanks to all the experts on StackOverflow (specially @Miserable Variable) for guiding me on this.

Hope this would clear your doubts too.