What is the optimal way to call N blocking api calls in parallel to get most out of CPU while waiting for api calls to complete?

Question

I am working on a micro-service which have following flow of operations :

A request comes to execute some number of tasks. After some pre-processing I know that I need to execute some number of tasks let's say 10. Now the tasks are independent of each other, so can be executed in parallel. And each task have some processing step and some external API calls. And after all tasks are completed the combined results need to be returned.

This is one request, so obviously this micro service can get many such requests in parallel too.

Here API calls are most time-consuming operations and other work executes in very less time comparatively. So I want to design this in a way that as many tasks can be executed in parallel because for tasks mostly would be blocked for API calls.

A simple solution I see is using a thread-pool using ExecutorService, but it doesn't seem like a ideal solution because let's say I create a thread pool of 32 threads, and I get 60 tasks. So only 32 would be executing at a time, even though those 32 tasks are blocked for api call and not using much CPU time.

Is this possible to achieve without breaking the task as a single unit ?

Answer 1

The optimal number of threads depends on the number of cores the server has and the time the I/O workload takes. See http://baddotrobot.com/blog/2013/06/01/optimum-number-of-threads/ to calculate that.

In short it states: threads = number of cores * (1 + wait time / service time)

The timings has to come from your observation and measurements. For the rest, you can use the CompletableFuture as mentioned in the comments or you can use the Executors class: Executors.newFixedThreadPool(<num of threads>);

Answer 2

You will have to do some benchmarking to figure out what is optimal for your setup. You might want to look into using ThreadPoolExecutor which can scale up and down the number of threads according to how many threads are available in the pool. There are a few parameters you can adjust in your benchmarks, namely corePoolSize and maximumPoolSize.