Grails hibernate session in batches

Question

GORM works fine out of the box as long as there is no batch with more than 10.000 objects. Without optimisation you will face the outOfMemory problems.

The common solution is to flush() and clear() the session each n (e.g.n=500) objects:

Session session = sessionFactory.currentSession
Transaction tx = session.beginTransaction();
def propertyInstanceMap = org.codehaus.groovy.grails.plugins.DomainClassGrailsPlugin.PROPERTY_INSTANCE_MAP

Date yesterday = new Date() - 1

Criteria c = session.createCriteria(Foo.class)
c.add(Restrictions.lt('lastUpdated',yesterday))
ScrollableResults rawObjects = c.scroll(ScrollMode.FORWARD_ONLY)

int count=0;
while ( rawObjects.next() ) {
    def rawOject = rawObjects.get(0);

    fooService.doSomething()

    int batchSize = 500
    if ( ++count % batchSize == 0 ) {
        //flush a batch of updates and release memory:
        try{
            session.flush();
        }catch(Exception e){
            log.error(session)
            log.error(" error: " + e.message)
            throw e
        }
        session.clear();
        propertyInstanceMap.get().clear()
    }
}

session.flush()
session.clear()
tx.commit()

But there are some problems I can't solve:

1. If I use currentSession, then the controller fails because of session is empty
2. If I use sessionFactory.openSession(), then the currentSession is still used inside FooService. Of cause I can use the session.save(object) notation. But this means, that I have to modify fooService.doSomething() and duplicate code for single operation (common grails notation like fooObject.save() ) and batch operation (session.save(fooObject() ).. notation).
3. If I use Foo.withSession{session->} or Foo.withNewSession{session->}, then the objects of Foo Class are cleared by session.clear() as expected. All the other objects are not cleared(), what leads to memory leak.
4. Of cause I can use evict(object) to manualy clear the session. But it is nearly impossible to get all relevant objects, due to autofetching of assosiations.

So I have no idea how to solve my problems without making the FooService.doSomething() more complex. I'm looking for something like withSession{} for all domains. Or to save session at the begin (Session tmp = currentSession) and do something like sessionFactory.setCurrentSession(tmp). Both doesn't exists!

Any idea is wellcome!

Answer 1

I would recommend to use stateless session for this kind of batch processing. See this post: Using StatelessSession for Batch processing