Avoid waiting on Servlet streams

Question

My Servler spends quite some time in reading request.getInputStream() and writing to response.getOutputStream(). In the long run, this can be a problem as its blocking a thread for nothing but reading/writing literally a few bytes per second. (*)

I'm never interested in a partial request data, the processing should not start before the request is completely available. Similarly for the response.

I guess, asynchronous IO would solve it, but I wonder what's the proper way. Maybe a servlet Filter replacing the ServletInputStream by a wrapped ByteArrayInputStream, using request.startAsync and calling the chained servlet after having collected the whole input?

• Is there already such a filter?
• Should I write one or should I use a different approach?

---

Note that what I mean is to avoid wasting threads on slow servlet streams. This isn't the same as startAsync which avoids wasting threads just waiting for some event.

And yes, at the moment it'd be a premature optimization.

My read loop as requested

There's nothing interesting in my current input stream reading method, but here you are:

private byte[] getInputBytes() throws IOException {
    ServletInputStream inputStream = request.getInputStream();
    final int len = request.getContentLength();
    if (len >= 0) {
        final byte[] result = new byte[len];
        ByteStreams.readFully(inputStream, result);
        return result;
    } else {
        return ByteStreams.toByteArray(inputStream);
    }
}

That's all and it blocks when data aren't available; ByteStreams come from Guava.

Summary of my understanding so far

As the answers clearly state, it's impossible to work with servlet streams without wasting a thread on them. Neither the servlet architecture nor the common implementation expose anything allowing to say "buffer the whole data and call me only when you collected everything", albeit they use NIO and could do it.

The reason may be that usually a reverse proxy like nginx gets used, which can do it. nginx does this buffering by default and it couldn't be even switched off until two years ago.

Actually a supported case???

Given that many negative answer, I'm not sure, but it looks like my goal

to avoid wasting threads on slow servlet streams

is actually fully supported: Since 3.1, there's ServletInputStream.html#setReadListener which seems to be meant exactly for this. The thread allocated for processing Servlet#Service initially calls request.startAsync(), attaches the listener and gets returned to the pool by simply returning from service. The listener implements onDataAvailable(), which gets called when it's possible to read without blocking, adds a piece of data and returns. In onAllDataRead(), I can do the whole processing of the collected data.

There's an example, how it can be done with Jetty. It seems to cover non-blocking output as well.

---

(*) In the logfiles, I can see requests taking up to eight seconds which get spend on reading the input (100 bytes header + 100 bytes data). Such cases are rare, but they do happen, although the server is mostly idle. So I guess, it's a mobile client on a very bad connection (some users of ours connect from places having such bad connectivity).

Answer 1

HttpServletRequest#startAsync() isn't useful for this. That's only useful for push things like web sockets and the good 'ol SSE. Moreover, JSR356 Web Socket API is built on top of it.

Your concrete problem is understood, but this definitely can't be solved from the servlet on. You'd only end up wasting yet more threads for the very simple reason because the container has already dedicated the current thread to the servlet request until the request body is read fully up to the last bit, even if it's ultimately read by a newly spawned async thread.

To save threads, you actually need a servletcontainer which supports NIO and if necessary turn on that feature. With NIO, a single thread can handle as many TCP connections as the available heap memory allows it, instead of that a single thread is allocated per TCP connection. Then, in your servlet you don't at all need to worry about this delicate I/O task.

Almost all modern servletcontainers support it: Undertow (WildFly), Grizzly (GlassFish/Payara), Tomcat, Jetty, etc. Some have it by default enabled, others require extra configuration. Just refer their documentation using the keyword "NIO".

If you'd actually also want to save the servlet request thread itself, then you'd basically need to go a step back, drop servlets and implement a custom NIO based service on top of an existing NIO connector (Undertow, Grizzly, Jetty, etc).

Answer 2

1. You can't. The Servlet container allocates the thread to the request, and that's the end of it, it's allocated. That's the model. If you don't like that, you will have to stop using Servlets.
2. Even if you could solve (1), you can't start async I/O on an input stream.
3. The way to handle slow requests is to time them out, by setting the appropriate setting for whatever container you're using ... if you actually have a problem, and it's far from clear that you really do, with a mostly idle server and this only happening rarely.
4. Your read loop makes a distinction without a difference. Just read the request input stream to its end. The servlet container already ensures that end of stream happens at the content-length if provided.

Answer 3

There's a class called org.apache.catalina.connector.CoyoteAdapter, which is the class that receives the marshaled request from TCP worker thread. It has a method called "service" which does the bulk of the heavy lifting. This method is called by another class: org.apache.coyote.http11.Http11Processor which also has a method of the same name.

I find it interesting that I see so many hooks in the code to handle async io, which makes me wonder if this is not a built in feature of the container already? Anyway, with my limited knowledge, the best way that I can think of to implement the feature you are talking about, would be to create a class:

public class MyAsyncReqHandlingAdapter extends CoyoteAdapter and @Override service() method and roll your own... I don't have the time to devote to doing this now, but I may revisit in the future.

In this method you would need a way to identify slow requests and handle them, by handing them off to a single threaded nio processor and "complete" the request at that level, which, given the source code:

https://github.com/apache/tomcat/blob/075920d486ca37e0286586a9f017b4159ac63d65/java/org/apache/coyote/http11/Http11Processor.java

https://github.com/apache/tomcat/blob/3361b1321201431e65d59d168254cff4f8f8dc55/java/org/apache/catalina/connector/CoyoteAdapter.java

You should be able to figure out how to do. Interesting question and yes it can be done. Nothing I see in the spec says that it cannot...