Sending large amounts of HTTP requests concurrently with a small number of threads

Question

We have the need to send large amounts of HTTP requests to a server endpoint concurrently (hundreds of requests per second) continuously for some time, in our Java code. The server takes ~4-5 seconds to respond to each request.

The underlying code is making those requests using Apache HttpClient (classic). Since this client is blocking, I'm having to use large amounts of threads (in a ExecutorService.newFixedThreadPool()) to send requests concurrently in order to achieve the goal: start these threads, where each thread basically does something like this:

while (time_elapsed < 3600 seconds) {
    sendHttpRequest(); // this takes ~4-5 seconds, during which the current thread is blocked due to the nature of Apache 
}

There are 2 problems:

• as the server responds to each HTTP call in 4-5 seconds, theoretically we will need 400-500 threads in order to send ~100 requests/second, but this huge amount of threads will consume lots of resources, and will leads to a lot of context switch as well as other problems, so in practice they won't really send as many as 100 requests/second
• as each thread has to wait until the current request to finish (4-5 seconds) before sending the next, we can't control the precise number of requests we send every second.

This would have been easy if we have the flexibility to switch to async http clients like Apache HttpAsyncClient or async-http-client, but unfortunately we can't due to the codebase's limitation.

My question is, is there another way to achieve hundreds of concurrent requests/second, with a small number of threads and without using an async HTTP client? Is CompletableFuture an option?

Answer 1

Virtual threads

Virtual threads (fibers) is a new feature in Java 21. This is part of Project Loom.

In conventional Java concurrency, threads in Java are mapped directly one-to-one with host OS threads. The problem is that when code running in a thread blocks, the thread blocks too. That thread pauses until the executing code is unblocked. So much of the time a thread may go unused, sitting idle, while waiting for a call to storage I/O, network I/O, database access, etc. to return.

In contrast, many of the proposed virtual threads are mapped to each “real” thread provided by the host OS. When the code being executed by a virtual thread blocks, the virtual threads is “parked”, set aside, while another virtual thread takes its place on the host OS thread. When the prior virtual thread’s task is no longer blocked, it gets assigned again to a host OS thread for continued execution.

This parking and unparking is extremely efficient. This means you can have thousands, even millions, of simultaneous threads.

So you could set up your hundreds of HTTP requests as individual Runnable/Callabletasks, and then submit them all to an appropriate executor service for immediate execution.

---

By the way, Java now includes a new HTTP client library.