How to use Scroll on Elasticsearch aggregation?

Question

I am using Elasticsearch 5.3. I am aggregating on some data but the results are far too much to return in a single query. I tried using size = Integer.MAX_VALUE; but even that has proved to be less. In ES search API, there is a method to scroll through the search results. Is there a similar feature to use for the org.elasticsearch.search.aggregations.AggregationBuilders.terms aggregator and how do I use it? Can the search scroll API be used for the aggregators?

Answer 1

In ES 5.3, you can partition the terms buckets and retrieve one partition per request.

For instance, in the query below, you can request to partition your buckets into 10 partitions and only return the first partition. It will return ~10x less data than if you wanted to retrieve all buckets at once.

{
   "size": 0,
   "aggs": {
      "my_terms": {
         "terms": {
            "field": "my_field",
            "include": {
               "partition": 0,
               "num_partitions": 10
            },
            "size": 10000
         }
      }
   }
}

You can then make the second request by increasing the partition to 1 and so on

{
   "size": 0,
   "aggs": {
      "my_terms": {
         "terms": {
            "field": "my_field",
            "include": {
               "partition": 1,           <--- increase this up until partition 9
               "num_partitions": 10
            },
            "size": 10000
         }
      }
   }
}

To add this in your Java code, you can do it like this:

TermsAggregationBuilder agg = AggregationBuilders.terms("my_terms");
agg.includeExclude(new IncludeExclude(0, 10));