Properly streaming of both input and output in HTTP Servlet

Question

I'm trying to write servlet that will handle POST request and stream both input and output. I mean it should read one line of input, do some work on this line, and write one line of output. And it should be able to handle arbitrary long request (so also will produce arbitrary long response) without out of memory exceptions. Here's my first attempt:

protected void doPost(HttpServletRequest request, HttpServletResponse response) {
    ServletInputStream input = request.getInputStream();
    ServletOutputStream output = response.getOutputStream();

    LineIterator lineIt = lineIterator(input, "UTF-8");
    while (lineIt.hasNext()) {
        String line = lineIt.next();
        output.println(line.length());
    }
    output.flush();
}

Now I tested this servlet using curl and it works, but when I've written client using Apache HttpClient both the client thread and server thread hangs. The client looks like that:

HttpClient client = HttpClientBuilder.create().build();
HttpPost post = new HttpPost(...);

// request
post.setEntity(new FileEntity(new File("some-huge-file.txt")));
HttpResponse response = client.execute(post);

// response
copyInputStreamToFile(response.getEntity().getContent(), new File("results.txt"));

The issue is obvious. Client does it's job sequentially in one thread - first it sends request completely, and only then starts reading the response. But server for every line of input writes one line of output, if client is not reading output (and sequential client isn't) then server is blocked trying to write to the output stream. This in turn blocks client trying to send the input to the server.

I suppose curl works, because it somehow sends input and receives output concurrently (in separate threads?). So the first question is can Apache HttpClient be configured to behave similarly as curl?

The next question is, how to improve the servlet so ill-behaving client won't cause server threads to be hanging? My first attempt is to introduce intermediate buffer, that will gather the output until the client finishes sending input, and only then servlet will start sending output:

ServletInputStream input = request.getInputStream();
ServletOutputStream output = response.getOutputStream();

// prepare intermediate store
int threshold = 100 * 1024; // 100 kB before switching to file store
File file = File.createTempFile("intermediate", "");
DeferredFileOutputStream intermediate = new DeferredFileOutputStream(threshold, file);

// process request to intermediate store
PrintStream intermediateFront = new PrintStream(new BufferedOutputStream(intermediate));
LineIterator lineIt = lineIterator(input, "UTF-8");
while (lineIt.hasNext()) {
    String line = lineIt.next();
    intermediateFront.println(line.length());
}
intermediateFront.close();

// request fully processed, so now it's time to send response
intermediate.writeTo(output);

file.delete();

This works, and ill-behaving client can use my servlet safely, but on the other hand for these concurrent clients like curl this solution adds an unnecessary latency. The parallel client is reading the response in separate thread, so it will benefit when the response will be produced line by line as the request is consumed.

So I think I need a byte buffer/queue that:

• can be written by one thread, and read by another thread
• will initially be only in memory
• will overflow to disk if necessary (similarly as DeferredFileOutputStream).

In the servlet I'll spawn new thread to read input, process it, and write output to the buffer, and the main servlet thread will read from this buffer and send it to the client.

Do you know any library like to do that? Or maybe my assumptions are wrong and I should do something completely different...

Answer 1

To achieve simultaneously writing and reading you can use Jetty HttpClient http://www.eclipse.org/jetty/documentation/current/http-client-api.html

I've created pull request to your repo with this code.

HttpClient httpClient = new HttpClient();
httpClient.start();

Request request = httpClient.newRequest("http://localhost:8080/line-lengths");
final OutputStreamContentProvider contentProvider = new OutputStreamContentProvider();
InputStreamResponseListener responseListener = new InputStreamResponseListener();

request.content(contentProvider).method(HttpMethod.POST).send(responseListener); //async request
httpClient.getExecutor().execute(new Runnable() {
    public void run() {
        try (OutputStream outputStream = contentProvider.getOutputStream()) {
            writeRequestBodyTo(outputStream); //writing to stream in another thread
        } catch (IOException e) {
            e.printStackTrace();
        }
    }
});

readResponseBodyFrom(responseListener.getInputStream()); //reading response
httpClient.stop();