Spring batch JdbcCursorItemReader cause a memory not enough issue

Question

I am using the spring batch framework to do a data migration. The reader I use is JdbcCursorItemReader. I set chunk size as 500 and set the reader fetch size as 1000. But when run the service with spring batch it just seems to read all the data once in the memory and run out of the memory . then throw a memory not enough issue. Below is how I define the reader:

   private JdbcCursorItemReader<Map<String, Object>> buildItemReader(final DataSource dataSource, String tableName,String tenant) {
        String tenantName = tenantHelper.determineTenant(tableName);
        JdbcCursorItemReader<Map<String, Object>> itemReader = new JdbcCursorItemReader<>();
        itemReader.setDataSource(dataSource);
        itemReader.setSql("select * from " + tableName + " where " + tenantName + " ='" + tenant + "'");
        itemReader.setRowMapper(new ColumnMapRowMapper());
        itemReader.setFetchSize(100);
        return itemReader;
    }

Whats more, from the spring batch document here, we should be able to avoid the memory issue by using the jdbcCursorItemReader

Answer 1

You can try using a JdbcPagingItemReader instead of JdbcCursorItemReader where the page size can be set while configuring it

Answer 2

Figured this out by using the jdbcPagingItemReader. The root cause that the the cursor reader consume massive memory is because it just read all the data into the memory and then process it, which will be considered as a big object by the JVM and it will be allocated into the old generation directlly, Until the whole process finished, it could not be collected.