I am a newbie in spring batch. The task I need to achieve in spring batch as follows:
My queries are the following:
a. For the 1st requirement, I needed to map the whole resultset to a single object, where Person related data are in 1 table and Pets related data are in another table and joined by person id.
public class PersonPetDetails {
private String personName;
private String personAddr;
private int personAge;
private List<Pet> pets;
For this I have written a custom Item reader which extends JdbcCursorItemReader.
public class CustomJDBCCusrorItemReader<T> extends JdbcCursorItemReader<T> {
private ResultSetExtractor<T> resultSetExtractor;
public void setResultSetExtractor(ResultSetExtractor<T> resultSetExtractor) {
this.resultSetExtractor = resultSetExtractor;
}
@Override
public void afterPropertiesSet() throws Exception {
setVerifyCursorPosition(false);
Assert.notNull(getDataSource(), "DataSource must be provided");
Assert.notNull(getSql(), "The SQL query must be provided");
Assert.notNull(resultSetExtractor, "ResultSetExtractor must be provided");
}
@Override
protected T readCursor(ResultSet rs, int currentRow) throws SQLException {
return resultSetExtractor.extractData(rs);
}
}
Is this the correct way to achieve my requirement? Or is there a much better way?
b. AFAIK, in spring batch there cannot be a step with just a reader, without a writer. Hence, I cannot call another set of reader in a different step of the Job. Then, how can I call multiple readers in a single step?
c. Also, based on some condition I may need to call a third set of Reader. How can I conditionally call a reader in a step?
Thanks for going through my post. I know it is long. Any help is much appreciated. Also, i guess an example code snippet would help me better understand the point. :)
You need to use MultiResourceItemReader to read lines from CSV file. It reads items from multiple resources sequentially.
Multiple jobs can be run simultaneously. There are two main types of Spring Batch Parallel Processing: Single Process, Multi-threaded, or Multi-process. These are also divided into subcategories, as follows: Multi-threaded Step (Step with many threads, single process)
Multithreaded steps. By default, Spring Batch uses the same thread to execute a batch job from start to finish, meaning that everything runs sequentially. Spring Batch also allows multithreading at the step level. This makes it possible to process chunks using several threads.
Spring Batch uses a 'Chunk-oriented' processing style within its most common implementation. Chunk oriented processing refers to reading the data one at a time and creating 'chunks' that are written out within a transaction boundary.
I would recommend as below
High Level Design:
Partitioner It will deal with list of persons. Note: there is not Pet data pulled at this point of time.
Reader It will get a list of Pet which are belong to a Person. Note: Reader will return a list of Pet specific to a Person only.
Processor Base on a Pet-Person you will process base on your requirement.
Writer Based on your requirement to write to DB.
Low Level Code snippet:
Partitioner
public class PetPersonPartitioner implements Partitioner {
@Autowired
private PersonDAO personDAO;
@Override
public Map<String, ExecutionContext> partition(int gridSize) {
Map<String, ExecutionContext> queue = new HashMap<String, ExecutionContext>();
List<Person> personList = this.personDAO.getAllPersons();
for (Person person : personList) {
ExecutionContext ec = new ExecutionContext();
ec.put("person", person);
ec.put("personId", person.getId());
queue.put(person.getId(), ec);
}
return queue;
}
}
Reader
<bean id="petByPersonIdRowMapper" class="yourpackage.PetByPersonIdRowMapper" />
<bean id="petByPesonIdStatementSetter" scope="step"
class="org.springframework.batch.core.resource.ListPreparedStatementSetter">
<property name="parameters">
<list>
<value>#{stepExecutionContext['personId']}</value>
</list>
</property>
</bean>
public class PetByPersonIdRowMapper implements RowMapper<PersonPetDetails> { @Override public BillingFeeConfigEntity mapRow(ResultSet rs, int rowNum) throws SQLException { PersonPetDetails record = new PersonPetDetails(); record.setPersonId(rs.getLong("personId")); record.setPetId(rs.getLong("petid"); ... ... }
Processor
You can continue working on each PersonPetDetails
object.If you love us? You can donate to us via Paypal or buy me a coffee so we can maintain and grow! Thank you!
Donate Us With