Can Elasticsearch stream the SearchResponse?

Question

I have a rest application that can export some report data from Elasticsearch. It is easy to do with the Java API:

SearchResponse response = getClient()
            .prepareSearch("my_index_name")
            .setQuery(QueryBuilders.someQuery())
            .addAggregation(AggregationBuilders.someAggregation())
            .get();

The problem starts with the big responses. Using this code snippet, the response is read to build the SearchResponse object in memory. In my case, the response does not fits in memory.

Paging cannot help because we often need to return the full data and Aggregations do not support paging yet.

I know that I can use the Elasticsearch REST API to read the response as stream, but manually build the request it is cumbersome. I really want something like this:

// my dream API
InputStream response = getClient()
            .prepareSearch("my_index_name")
            .setQuery(QueryBuilders.someQuery())
            .addAggregation(AggregationBuilders.someAggregation())
            .getStream();

So, can the Elasticsearch Java API stream the SearchResponse?

Answer 1

A proposal for streaming results does exist but it doesn't seem to have picked up steam so far and was closed (for now).

There's a way to do it with XContentBuilder but that still requires the whole response to be in memory before being sent.

It might not be what you want, but that's the closest thing that I know which could fulfill your need. Worth giving it a try.