Logo Questions Linux Laravel Mysql Ubuntu Git Menu
 

Spring Batch: Multiple Item Readers in a single step

I am a newbie in spring batch. The task I need to achieve in spring batch as follows:

  1. Need to read some metadata from database.
  2. Based on this metadata, I need to read some files.
  3. After some processing, need to write those values from file to database.

My queries are the following:

a. For the 1st requirement, I needed to map the whole resultset to a single object, where Person related data are in 1 table and Pets related data are in another table and joined by person id.

public class PersonPetDetails {

    private String personName;
    private String personAddr;

    private int personAge;

    private List<Pet> pets;

For this I have written a custom Item reader which extends JdbcCursorItemReader.

public class CustomJDBCCusrorItemReader<T> extends JdbcCursorItemReader<T> {

    private ResultSetExtractor<T> resultSetExtractor;


    public void setResultSetExtractor(ResultSetExtractor<T> resultSetExtractor) {
        this.resultSetExtractor = resultSetExtractor;
    }

    @Override
    public void afterPropertiesSet() throws Exception {
        setVerifyCursorPosition(false);
        Assert.notNull(getDataSource(), "DataSource must be provided");
        Assert.notNull(getSql(), "The SQL query must be provided");
        Assert.notNull(resultSetExtractor, "ResultSetExtractor must be provided");
    }


    @Override
    protected T readCursor(ResultSet rs, int currentRow) throws SQLException {      
        return resultSetExtractor.extractData(rs);
    }
}

Is this the correct way to achieve my requirement? Or is there a much better way?

b. AFAIK, in spring batch there cannot be a step with just a reader, without a writer. Hence, I cannot call another set of reader in a different step of the Job. Then, how can I call multiple readers in a single step?

c. Also, based on some condition I may need to call a third set of Reader. How can I conditionally call a reader in a step?

Thanks for going through my post. I know it is long. Any help is much appreciated. Also, i guess an example code snippet would help me better understand the point. :)

like image 734
bidisha mukherjee Avatar asked Sep 07 '16 15:09

bidisha mukherjee


People also ask

Can we have multiple readers in Spring Batch?

You need to use MultiResourceItemReader to read lines from CSV file. It reads items from multiple resources sequentially.

Can a Spring Batch have multiple jobs?

Multiple jobs can be run simultaneously. There are two main types of Spring Batch Parallel Processing: Single Process, Multi-threaded, or Multi-process. These are also divided into subcategories, as follows: Multi-threaded Step (Step with many threads, single process)

How does multithreading work in Spring Batch?

Multithreaded steps. By default, Spring Batch uses the same thread to execute a batch job from start to finish, meaning that everything runs sequentially. Spring Batch also allows multithreading at the step level. This makes it possible to process chunks using several threads.

What is chunking in Spring Batch?

Spring Batch uses a 'Chunk-oriented' processing style within its most common implementation. Chunk oriented processing refers to reading the data one at a time and creating 'chunks' that are written out within a transaction boundary.


1 Answers

I would recommend as below

High Level Design:

  1. Partitioner It will deal with list of persons. Note: there is not Pet data pulled at this point of time.

  2. Reader It will get a list of Pet which are belong to a Person. Note: Reader will return a list of Pet specific to a Person only.

  3. Processor Base on a Pet-Person you will process base on your requirement.

  4. Writer Based on your requirement to write to DB.

Low Level Code snippet:

  1. Partitioner

    public class PetPersonPartitioner implements Partitioner {
    
      @Autowired
      private PersonDAO personDAO;
    
      @Override
      public Map<String, ExecutionContext> partition(int gridSize) {
    
        Map<String, ExecutionContext> queue = new HashMap<String, ExecutionContext>();
    
        List<Person> personList = this.personDAO.getAllPersons();
        for (Person person : personList) {
    
          ExecutionContext ec = new ExecutionContext();
          ec.put("person", person);
          ec.put("personId", person.getId());
    
          queue.put(person.getId(), ec);
        }
    
      return queue;
      }
    }
    
  2. Reader

    <bean id="petByPersonIdRowMapper" class="yourpackage.PetByPersonIdRowMapper" />
    
    <bean id="petByPesonIdStatementSetter" scope="step"
          class="org.springframework.batch.core.resource.ListPreparedStatementSetter">
        <property name="parameters">
            <list>
                <value>#{stepExecutionContext['personId']}</value>
            </list>
        </property>
    </bean>
    
public class PetByPersonIdRowMapper implements RowMapper<PersonPetDetails> {
    @Override
    public BillingFeeConfigEntity mapRow(ResultSet rs, int rowNum) throws SQLException {
        PersonPetDetails record = new PersonPetDetails();

        record.setPersonId(rs.getLong("personId"));
      record.setPetId(rs.getLong("petid");
      ...
      ...
}
  1. Processor You can continue working on each PersonPetDetails object.
like image 102
Nghia Do Avatar answered Nov 07 '22 18:11

Nghia Do