How to override Spring Batch CompositeItemWriter manage transaction for delegate writers in case of exception arises?

Question

I am extending this How does Spring Batch CompositeItemWriter manage transaction for delegate writers? question here:

In my case I've a below CompositeItemWriter which writes data into the multiple tables of same database, before writing data it transforms data by implementing various business rules. Here one record may satisfy differet business rules etc. Hence one writer may get more data than others.

@Bean
public CompositeItemWriter<Employee> EmployeeCompositeWriter() throws Exception {
    List<ItemWriter<? super Employee>> employee = new ArrayList<>();
    employee.add(employeeWriter());
    employee.add(departmentWriter());
    employee.add(stockWriter());
    employee.add(purchaseWriter());

    CompositeItemWriter<Employee> compositeItemWriter = new CompositeItemWriter<>();
    compositeItemWriter.setDelegates(employee);
    compositeItemWriter.afterPropertiesSet();
    return compositeItemWriter;
}

Scenario - Assume 1st writer works very well, 2nd writer generates exception, then 3rd and 4th writers are not getting called this is what the Automic nature defaulted in Spring Batch happening due to Transaction roll back.

Here even if any exception arises at 2nd writer, I want to successfully call the 3rd and 4th writers and save the data, I also wanted to successfully save the data of 1st writer and 2nd writers.. only exception data I want to store into the Error Table with the help of SkipListener to identify which records was junk or garbage.

Solution - To achive above scenario, we've added @Transactional(propagation = Propagation.REQUIRES_NEW) on each writers write method, 1st writer saved the data now and 2nd writer generates exception (using namedJdbcTemplate.batchUpdate() to bulk update data) we're caching it and rethrowing it, but we could see commit level is reduced to 1 (off-course to identify extact garbage record) and the moment exception arises from 2nd writer again 1st writer is getting called and it's saving the duplicate data and writer 2nd, 3rd and 4th is getting called, but also that junk record is not flowing to 3rd and 4th writer.

Here I dont want the whole Batch Job to stop if single or couple of records are garbage, because this Job is critical for us to run everytime. Is there any way if we can save all the data where exception doesn't arise and only save exception data into the error table with the help of SkipListener if possible or any other way?

Is there any way if we can reused the Batch Components like (READER or PROCESSOR)part of any step into another step ?

Answer 1

I can't see a way you could align spring-batch's single transaction for writing the whole chunk as atomic vs your idea of keeping the atomicity to individual writers as long as you want skiplistener.

I am not sure if this is possible but may be you will be able to test it quickly. This is how the message carries the exception in some integration frameworks like camel from one processor to error handling flow.

• You item reader should return a EmployeeWrapper which contains employee record and has a field to store Exception.

• your CompositeItemWriter receives List<EmployeeWrapper> and composite writer has 5 writers instead of 4. And the 5th writer will do what your SkipListener would have done.

    List<ItemWriter<? super EmployeeWrapper>> employee = new ArrayList<>();
    employee.add(employeeWriter());
    employee.add(departmentWriter());
    employee.add(stockWriter());
    employee.add(purchaseWriter());
    employee.add(errorRecordWriter());

• Your first 4 individual writers never throw exception, instead mark it as processed but add the caught exception as attribute of EmployeeWrapper.

• Your 5th errorRecordWriter receives all the records, check any record that has exception attribute added and writes them to error table. Incase it failed to write error record, you can throw the exception and all 5 writers will be retried.

• Regarding how you would know which record is error record when the batch update fails. It seems when an error occur in chunk, spring rollbacks the chunk and start retrying it record by record in that chunk so it knows which record is the problematic. So you can do the same thing in your individual writers. I.e catch the batch update exception and then retry them one by one to separate the error records