How to do bulk inserts with JpaRepository with dynamic batch_size i.e batch_size is not known

Question

I am working in a java spring boot API where I need to insert a bulk data in my database. I know how could I achieve this.

To get a bulk insert with Sring Boot and Spring Data JPA you need only two things:

• set the option spring.jpa.properties.hibernate.jdbc.batch_size = 50
• use saveAll() method of your repo with the list of entities prepared for inserting.

What I want to know is that how could I achieve dynamic batch_size i.e. in some classes, I need to save/insert just 5 to 10 records, while in some classes this number could be 200 to 500 or more records.

Now, How could I achieve this dynamic batch_size option.

Answer 1

You cannot dynamically change the batch_size just by using the EntityManager AP Ioption during runtime but you can have some sort of control:

1) Set the batch_size to the highest expected value (500 i.e.)

2) Set these props in order for hibernate not to try to use the previously done save/update statements.

hibernate.order_inserts=true
hibernate.order_updatest=true

3) Use save instead of saveAll. Loop through your list and flush every number of times that are relevant to the class being saved:

int = flushAfterThisNumber = 10;

for ( int i=0; i<entities.length; i++ ) {
    session.save(entities[i]);
    if ( i % flushAfterThisNumber == 0 ) { 
        //flush a batch of inserts and release memory:
        session.flush();
        session.clear();
    }
}

UPDATE

There is a workaround possible, but you need to create a custom repo so that you will be able to inject an EntityManager, then unwrap the Hibernate Session and then have access to dynamic setting of batch_size.

public class CustomerRepositoryImpl implements CustomCustomerRepository {

    @PersistenceContext
    private EntityManager em;

    @Override
    public void saveAllInBatch(List<Ent> entities, int batchSize) {
        Session session = em.unwrap(Session.class);
        session.setJdbcBatchSize(batchSize);

        em.saveAll(entities);
    }
}