Reuse tomcat threads while waiting "long" time

Question

THE CONFIGURATION
Web server : Nginx
App server : Tomcat with default configuration of 200 request serving threads
Expected response time for my server : ~30 seconds(There are lots of third party dependencies)

THE SCENARIO
Every 10 seconds the application will need to generate the token for its use. The expected time for token generation is around 5 seconds, but since its a third party system being contacted over network, this is obviously not consistent and can go up to 10 seconds.
During the token generation process, nearly 80% of the incoming requests per second will need to wait.

WHAT I BELIEVE SHOULD HAPPEN
Since the requests waiting for the token generation will have to wait a "long" time, there is no reason for these request serving to be reused to serve other incoming requests while waiting for token generation process to complete.
Basically, it would make sense if my 20% to keep being served. If the waiting threads are not being utilized for other requests, tomcat request serving limit will be reached and server would essentially choke, not really something any developer will like.

WHAT I TRIED
Initially I expected switching to tomcat NIO connector would do this job. But after looking at this comparison, I was really not hopeful. Nevertheless, I tried by forcing the requests to wait for 10 second and it did not work.
Now I am thinking on the lines that I need to, sort of, shelve the request while its waiting and need to signal the tomcat that this thread is free to reuse. Similarly, I will need tomcat to give me a thread from its threadpool when the request is ready to be moved forward. But I am blindsided on how to do it or even if this is possible.

Any guidance or help?

Answer 1

This problem is essentially the reason so many "reactive" libraries and toolkits exist.

It's not a problem that can be solved by tweaking or swapping out the tomcat connector.
You basically need to remove all blocking IO calls, replacing them with non-blocking IO will likely require rewriting large parts of the application.
Your HTTP server needs to be non-blocking, you need to be using a non blocking API to the server(like servlet 3.1) and your calls to the third party API need to be non blocking.
Libraries like Vert.x and RxJava provide tooling to help with all of this.

Otherwise the only other alternative is to just increase the size of the threadpool, the operating system already takes care of scheduling the CPU so that the inactive threads don’t cause too much performance loss, but there is always going to be more overhead compared to a reactive approach.

Without knowing more about your application it is hard to offer advice on a specific approach.

Answer 2

You need an asynchronous servlet but you also need asynchronous HTTP calls to the external token generator. You will gain nothing by passing the requests from the servlet to an ExecutorService with a thread pool if you still create one thread somewhere per token request. You have to decouple threads from HTTP requests so that one thread can handle multiple HTTP requests. This can be achieved with an asynchronous HTTP client like Apache Asynch HttpClient or Async Http Client.

First you have to create an asynchronous servlet like this one

public class ProxyService extends HttpServlet {

    private CloseableHttpAsyncClient httpClient;

    @Override
    public void init() throws ServletException {
        httpClient = HttpAsyncClients.custom().
                setMaxConnTotal(Integer.parseInt(getInitParameter("maxtotalconnections"))).             
                setMaxConnPerRoute(Integer.parseInt(getInitParameter("maxconnectionsperroute"))).
                build();
        httpClient.start();
    }

    @Override
    public void destroy() {
        try {
            httpClient.close();
        } catch (IOException e) { }
    }

    @Override
    public void doGet(HttpServletRequest request, HttpServletResponse response) {
        AsyncContext asyncCtx = request.startAsync(request, response);
        asyncCtx.setTimeout(ExternalServiceMock.TIMEOUT_SECONDS * ExternalServiceMock.K);       
        ResponseListener listener = new ResponseListener();
        asyncCtx.addListener(listener);
        Future<String> result = httpClient.execute(HttpAsyncMethods.createGet(getInitParameter("serviceurl")), new ResponseConsumer(asyncCtx), null);
    }

}

This servlet performs an asynchronous HTTP call using Apache Asynch HttpClient. Note that you may want to configure the maximum connections per route because as per RFC 2616 spec HttpAsyncClient will only allow a maximum of two concurrent connections to the same host by default. And there are plenty of other options that you can configure as shown in HttpAsyncClient configuration. HttpAsyncClient is expensive to create, therefore you do not want to create an instace of it on each GET operation.

One listener is hooked to the AsyncContext, this listener is only used in the example above to handle timeouts.

public class ResponseListener implements AsyncListener {

    @Override
    public void onStartAsync(AsyncEvent event) throws IOException {
    }

    @Override
    public void onComplete(AsyncEvent event) throws IOException {
    }

    @Override
    public void onError(AsyncEvent event) throws IOException {
        event.getAsyncContext().getResponse().getWriter().print("error:");
    }

    @Override
    public void onTimeout(AsyncEvent event) throws IOException {
        event.getAsyncContext().getResponse().getWriter().print("timeout:");
    }

}

Then you need a consumer for the HTTP client. This consumer informs the AsyncContext by calling complete() when buildResult() is executed internally by HttpClient as a step to return a Future<String> to the caller ProxyService servlet.

public class ResponseConsumer extends AsyncCharConsumer<String> {

    private int responseCode;
    private StringBuilder responseBuffer;
    private AsyncContext asyncCtx;

    public ResponseConsumer(AsyncContext asyncCtx) {
        this.responseBuffer = new StringBuilder();
        this.asyncCtx = asyncCtx;
    }

    @Override
    protected void releaseResources() { }

    @Override
    protected String buildResult(final HttpContext context) {
        try {
            PrintWriter responseWriter = asyncCtx.getResponse().getWriter();
            switch (responseCode) {
                case javax.servlet.http.HttpServletResponse.SC_OK:
                    responseWriter.print("success:" + responseBuffer.toString());
                    break;
                default:
                    responseWriter.print("error:" + responseBuffer.toString());
                }
        } catch (IOException e) { }
        asyncCtx.complete();        
        return responseBuffer.toString();
    }

    @Override
    protected void onCharReceived(CharBuffer buffer, IOControl ioc) throws IOException {
        while (buffer.hasRemaining())
            responseBuffer.append(buffer.get());
    }

    @Override
    protected void onResponseReceived(HttpResponse response) throws HttpException, IOException {        
        responseCode = response.getStatusLine().getStatusCode();
    }

}

The web.xml configuration for ProxyService servlet may be like

<?xml version="1.0" encoding="UTF-8"?>
<web-app xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
         xmlns="http://java.sun.com/xml/ns/javaee"
         xmlns:web="http://java.sun.com/xml/ns/javaee/web-app_3_0.xsd"
         xsi:schemaLocation="http://java.sun.com/xml/ns/javaee http://java.sun.com/xml/ns/javaee/web-app_3_0.xsd"
         id="WebApp_ID" version="3.0" metadata-complete="true">
  <display-name>asyncservlet-demo</display-name>

  <servlet>
    <servlet-name>External Service Mock</servlet-name>
    <servlet-class>ExternalServiceMock</servlet-class>
    <load-on-startup>1</load-on-startup>
  </servlet>

  <servlet>
    <servlet-name>Proxy Service</servlet-name>
    <servlet-class>ProxyService</servlet-class>
    <load-on-startup>1</load-on-startup>
    <async-supported>true</async-supported>
    <init-param>
      <param-name>maxtotalconnections</param-name>
      <param-value>200</param-value>
    </init-param>
    <init-param>
      <param-name>maxconnectionsperroute</param-name>
      <param-value>4</param-value>
    </init-param>
    <init-param>
      <param-name>serviceurl</param-name>
      <param-value>http://127.0.0.1:8080/asyncservlet/externalservicemock</param-value>
    </init-param>
  </servlet>

  <servlet-mapping>
    <servlet-name>External Service Mock</servlet-name>
    <url-pattern>/externalservicemock</url-pattern>
  </servlet-mapping>

  <servlet-mapping>
    <servlet-name>Proxy Service</servlet-name>
    <url-pattern>/proxyservice</url-pattern>
  </servlet-mapping>

</web-app>

And a mock servlet for the token generator with a delay in seconds may be:

public class ExternalServiceMock extends HttpServlet{

    public static final int TIMEOUT_SECONDS = 13;
    public static final long K = 1000l;

    public void doGet(HttpServletRequest request, HttpServletResponse response) throws IOException {
        Random rnd = new Random();
        try {
            Thread.sleep(rnd.nextInt(TIMEOUT_SECONDS) * K);
        } catch (InterruptedException e) { }
        final byte[] token = String.format("%10d", Math.abs(rnd.nextLong())).getBytes(ISO_8859_1);
        response.setContentType("text/plain");
        response.setCharacterEncoding(ISO_8859_1.name());
        response.setContentLength(token.length);
        response.getOutputStream().write(token);
    }

}

You can get a fully working example at GitHub.