How do I perform batch insert with Playframewok JPA?

Question

I need to parse a very large file and store the resulting entities in a database. I am expecting up to 150k records per file and would liek to process those in batches.

Is there a way to perform batch insert of a Play entity with JPA?

Answer 1

To save memory you must make sure that the session is cleared regularly, so

Customer.em().getTransaction().begin(); 
for ( int i=1; i<=100000; i++ ) {
   ....
   myCustomer.save();
   if (i%1000==0) { 
       //Customer.em().getTransaction().commit();           
       Customer.em().flush(); 
       Customer.em().clear();
       //Customer.em().getTransaction().begin(); 
   }
}
Customer.em().getTransaction().commit();           

How ever you can improve your performance if you optimize the batch size hibernate.jdbc.batch_size 100

You can simply add the last line to the application.conf, all hibernate.* properties go directly to hibernate. See the code of the JPAPlugin for details.

Answer 2

As Play uses Hibernate under the hood, you should be able to use the standard batch processing that Hibernate uses. See http://docs.jboss.org/hibernate/core/3.3/reference/en/html/batch.html

However, Play automatically manages transactions for you, so, if you need to prevent Play from interfering with the Transaction management, you can annotate your method with @play.db.jpa.NoTransaction.

You can read more about Play's JPA and transaction support here http://www.playframework.org/documentation/1.2.1/jpa