java.lang.OutOfMemoryError when reading from a large table

Question

I'm trying to pre generate reports from a very large table(300GB) in PostgreSQL. I do something like this:

rs = stmt.executeQuery("SELECT * FROM tbl");
System.out.println("select all finished");
while (rs.next()) {
    /* generate report and save it in report table */
    /* generated reports are not in memory, 
     * They are saved in a summary table in each iteration */
}

When I start the application it gives Exception in thread "main" java.lang.OutOfMemoryError: Java heap space. I tried to use stmt.setFetchSize(1000) But it doesn't solve the problem.

What is the solution? I'm using PostgreSQL 8.4.11 on Debian 6.0.5 and openJDK 6.

[UPDATE]

The printed stack trace shows that the OutOfMemoryError exception has been generated in rs = stmt.executeQuery("SELECT * FROM tbl"); line. Also System.out.println("select all finished"); never shows.

1. I'm running in autocommit mode.
2. stmt.getResultSetConcurrency() returns 1007.
3. stmt.getResultSetHoldability() return 2.
4. rs.getType() returns 1003.

Answer 1

The problem is probably that PostgreSQL only uses the fetchSize in a narrow set of circumstances. See: http://jdbc.postgresql.org/documentation/91/query.html#fetchsize-example

• The connection to the server must be using the V3 protocol. This is the default for (and is only supported by) server versions 7.4 and later.
• The Connection must not be in autocommit mode. The backend closes cursors at the end of transactions, so in autocommit mode the backend will have closed the cursor before anything can be fetched from it.
• The Statement must be created with a ResultSet type of ResultSet.TYPE_FORWARD_ONLY. This is the default, so no code will need to be rewritten to take advantage of this, but it also means that you cannot scroll backwards or otherwise jump around in the ResultSet.
• The query given must be a single statement, not multiple statements strung together with semicolons.

So if you are executing this in auto-commit, or with a resultset type other than TYPE_FORWARD_ONLY PostgreSQL will fetch all rows. Also looking at the sources of the PostgreSQL JDBC 9.0-801 driver it looks like using a holdable resultset will also make it fetch all rows.