Java: Does making async calls increase number of threads?

Question

So let's say I have a web app and for every request we spawn a new thread. Hundreds of requests come in, somewhere in the web server code we make synchronous calls to several services, we block and wait. This approach bloats the number of threads that we have as the sync calls create a bottleneck.

Supposedly, if we switch these calls to async requests we get rid of the bottleneck as the threads can continue and the callbacks will handle whatever needs to happen.

As far as I understand, in Java, in order to make an async call we spawn a new thread that makes the network call and contains the callback (I won't be implementing this, I'm assuming thats how some of the Java http libraries work).

So my question, how is this solving the problem of many threads? Async requests end up creating more threads (one for each request) and then go to sleep until something is returned, doesn't this create many sleeping threads?

The problem I am trying to solve is that at some point, when there's too many threads, the JVM explodes.

Answer 1

Specifically in web service / servlet environments:

In the simplest configuration, common web servers (Jetty, Tomcat) are configured with a fixed number of threads, or range of number of threads. If more requests arrive than there are threads, then those requests will pile up in the kernel connection queue. A thread accepts a connection and does all the work. When the response is sent, the thread is available for another connection. Adding your own thread pool or executor service won't help that.

In more complex configurations, the web container accepts connections on one pool of threads, and then dispatches the work on another, with a queue in between. Then, instead of blocking clients on connect, or having them fail to connect, they just wait.

In async Servlet processing, such as the JAX-RS @suspended AsyncResponse object, you get to control the details of this yourself. The servlet calls you with a data structure that includes the connection. Your code can put that object into some queue (possibly just the queue built into an Executor Service), and return. That frees the web server thread to accept another container. Your threads, probably from an Executor Service, work through the queue, processing requests and sending responses.

What you never do is create an unbounded number of threads.

Answer 2

Asynchronous means the request is processed by another thread. It doesn't have to be a dedicated thread, let alone a new thread.

For instance, consider JAX-RS asynchronous client callbacks:

target().path("http://example.com/resource/")
    .request().async().get(new InvocationCallback<String>() {
        @Override
        public void completed(String dataFromBackendServer) {
            respondWith(dataFromBackendServer);
        }

        @Override
        public void failed(Throwable throwable) {
            respondWithError(throwable);
        }
    });

Here, the InvocationCallback is executed in a thread provided by the JAX-RS implementation, that waits for a response to any pending backend request, then processes that response using the appropriate InvocationCallback. Because a single thread can wait for any number of pending backend requests, fewer threads are needed.

That said, synchronous processing is often easier to implement, and while it does not scale quite as well as asynchronous processing, it scales sufficiently for many applications. That is, unless you have thousands of concurrent requests, the plain old synchronous processing model will do.