How to download and process large data reactively?

Question

I need to initiate download of some content over HTTP and then read the data as a reactive stream.

So, even though the downloaded data are big, I can almost immediately read the first few bytes of the response body (no need to wait for the whole response body). Then, do some computations and in a few seconds read another portion of the data. There has to be some limit of the cached data, because operation memory can't handle the whole content (its tens of GB).

I've been trying to use HttpClient's sendAsync method with BodyHandlers.ofInputStream(), but it always blocks and waits for all the data to arrive.

HttpClient client = HttpClient.newHttpClient();

HttpRequest request = HttpRequest.newBuilder()
    .uri(URI.create("https://..."))
    .build();

HttpResponse<InputStream> response = client
    .sendAsync(request, HttpResponse.BodyHandlers.ofInputStream())
    .get(); // this finishes as soon as the header is received

try {
    InputStream stream = response.body();
    byte[] test = stream.readNBytes(20); // trying to read just a few bytes
                                         // but it waits for the whole body
} catch (IOException ex) {}

What do I need to change so the response body is downloaded gradually?

Answer 1

This is a bug. It has been fixed in Java 11.0.2: https://bugs.openjdk.java.net/browse/JDK-8212926