Spring Boot + Spring Data: Concurrent/parallel save/insert into databases

Question

Small question regarding SpringBoot and SpringData, and how to save a pojo into many databases concurrently, in parallel please.

I have a very simple SpringBoot application which does nothing but expose a rest endpoint to save a pojo:

@RestController
public class SaveController {

    @Autowired
    MyElasticRepository myElasticRepository;
    @Autowired
    MyMongoRepository myMongoRepository;
    @Autowired
    MyAARepository myAARepository;
    
    //@Autowired MyBBRepository, MyCCRepository, ... MyYYRepository
    
    @Autowired
    MyZZRepository myZZRepository;

    @GetMapping("/saveSequential")
    public String saveSequential(@RequestBody MyPojo myPojo) {
        MyPojo myPojoFromElastic = myElasticRepository.save(myPojo);
        MyPojo myPojoFromMongo = myMongoRepository.save(myPojo);
        MyPojo myPojoFromAA = myAARepository.save(myPojo);
        // myBBRepository.save(myPojo) myCCRepository.save(myPojo) ... myYYRepository.save(myPojo)
        MyPojo myPojoFromZZ = myZZRepository.save(myPojo);
        return ...;
    }
}

However, the pojo needs to be saved in many databases, by many, imagine a good dozens of different databases.

As of now, as you can see from the code, the pojo is saved in each of the databases sequentially. I timed the application, as well as monitoring the DBs, the inserts come one after another.

Hypothetically, if one save takes one second, and I have 20 DB, the rest endpoints takes 20ish seconds to complete.

Since the operation is not dependent of any others, i.e. saving the pojo in Mongo, has no dependency on the data saved in Oracle, etc... I would like to optimize the performance by doing the operation in parallel.

I.e, if each save takes one second, and I have 20 DBs, to parallel the save, which should still take something like oneish second. (I am exaggerating)

For the sake of the question, let us imagine the machine doing the save has many cores, is a very good machine, etc.

What I tried:

I tried using the @Async annotation on the repository, such as:

@Async
@Repository
public interface MyElasticRepository extends ElasticsearchRepository<MyPojo, String> {

}

But unfortunately, timing the endpoint, it still takes a sequential time.

May I ask how to achieve this parallel, concurrent save please? If possible, I would like to leverage existing features of Spring Framework, and not having to rewrite boiler plate concurrency code.

Thank you!

Answer 1

I think you are best off creating a service layer with an async method.

import org.springframework.data.repository.Repository;
import java.util.concurrent.CompletableFuture;
import org.springframework.stereotype.Service;
import org.springframework.scheduling.annotation.AsyncResult;

@Service
public class MyPojoPersister {

  @Async
  @CompletableFuture<MyPojo> savePojo(Repository repo, MyPojo pojo) {
    return CompletableFuture.completedFuture(repo.save(pojo));
  }
}

Then your controller would look something like this:

@RestController
public class SaveController {

    @Autowired
    MyElasticRepository myElasticRepository;
    @Autowired
    MyMongoRepository myMongoRepository;
    @Autowired
    MyAARepository myAARepository;
    
    //@Autowired MyBBRepository, MyCCRepository, ... MyYYRepository
    
    @Autowired
    MyZZRepository myZZRepository;

    @Autowired MyPojoPersister myPojoPersister;

    @GetMapping("/saveSequential")
    public String saveSequential(@RequestBody MyPojo myPojo) {
        var futureList = Stream.of(myElasticRepository, myMongoRepository, myAARepository, myZZRepository)
          .map(repo -> myPojoPersister.savePojo(repo, myPojo))
          .collect(Collectors.toList());
        CompletableFuture.allOf(futureList.toArray(new CompletableFuture[list.size()])).join();

        var someString = futureList.stream()
          .map(CompletableFuture::get())
          .map(MyPojo::getId())
          .collect(Collectors.joining(","));      
  
        return someString;
    }
}

I added some assumptions that you want to return a comma separated list of the ids of the pojos since they would presumably be different for each repo. But do whatever you need to with the values of the futures.

Don't forget to enable asynchronicity!

@SpringBootApplication
@EnableAsync
public class MyPojoApplication {
  public static void main(String[] args) {
    SpringApplication.run(AsyncMethodApplication.class, args).close();
  }
}