Spring Batch and multiple dependent reader

Question

I'm writing a Spring Batch Job and I can't find a way to handle multiple datasource and merge then together.

• I have a web api that return a list of objects and their id.

• I also have a database with some other objects

• For each object read from the web api call, I need to send a sql request to the database to get the object with the id of my current object i'm reading.

• Then I need to merge those objects together and save them to an another web api.

---

• I have a customHttpReader to call the web api and get the list of data.
• I have a JdbcCursorItemReader to send a sql query to the database.

    @Bean
    public Job todoJob() {
        final String jobName = "todos-migration-job";

        Step step1 = stepBuilderFactory.get("get-todo-from-api").<TodoDto, TodoDto>chunk(30)
                .reader(todoItemReader)
                .writer(todoItemWriter)
                .build();

        Step step2 =
                stepBuilderFactory.get("get-todo-from-database").<TodoDto, TodoBackendDto>chunk(30)
                        .reader(todoItemReader)
                        .processor(todoItemDatabaseProcessor)
                        .writer(todoItemBackendWriter)
                        .build();

        // what should I do here? should I do 1 or 2 steps?
        Job todoJob = jobBuilderFactory.get(jobName)
                .incrementer(new RunIdIncrementer())
                .start(step1) 
                .next(step2)
                .build();

        this.jobService.registerJob(jobName);

        return todoJob;
    }

    @Bean
    public TodoItemRestReader todoItemReader() {
        return new TodoItemRestReader();
    }

    @Bean
    public JdbcCursorItemReader<TodoDto> todoItemDatabaseReader() {
        return new JdbcCursorItemReaderBuilder<TodoDto>()
                .dataSource(this.datasourceConfig.todoDataSource())
                .name("todoItemDatabaseReader")
                .sql("select id, title from todo_data where id = :id") // how do I get the id ?
                .rowMapper(new TodoItemRowMapper())
                .build();
    }

So, how can I do to share all my read data from the first reader to the second ?

Thank's

Answer 1

Spring Batch does not allow to use more than one reader in one step.

There is more than one way to solve your problem:

• Query the web api in the reader. In the processor, execute the sql query for each item received from the reader and pass the combination on to the writer (or next processor).
• Write a single reader that queries the web api and directly executes the sql query for each item read.
• Use two steps: First, query the web api and write the results in a staging table. Then, use a reader that joins the existing data in the DB with the results from the web api.