How do you convert an Elasticsearch JSON String Response, with an Aggregation, to an Elasticsearch SearchResponse Object

Question

I want to serialize a json string to an Elasticsearch SearchResponse object. It works fine if the json string doesn't contains an aggregation.

If the json string contains an aggregation the XContentParser throws an ParsingException[Could not parse aggregation keyed as [target_field] exception.

The code I use to serialize the json string to an Elasticsearch SearchResponse object:

    Settings settings = Settings.builder().build();
    SearchModule searchModule = new SearchModule(settings, false, new ArrayList<>());

    NamedXContentRegistry xContentRegistry = new NamedXContentRegistry(searchModule.getNamedXContents());

    JsonXContentParser xContentParser = new JsonXContentParser(xContentRegistry,
            new JsonFactory().createParser(json));
    SearchResponse response = SearchResponse.fromXContent(xContentParser);

It seems that I have to register aggregations to the NamedXContentRegistry but i don't know how to.

Answer 1

Background:
I'm writing this answer from my experience of creating a SearchResponse object for the purpose of writing a Java Unit Test. The goal was to take any JSON response object from an Elasticsearch query, marshall that into a SearchResponse object, and Unit Test the business logic of creating a consumable output.

We're using Elasticsearch 6.7, the high-level rest client, and parsing the SearchResponse using Elastic's POJOs (vs just doing a .toString() and manipulating it with GSON or Jackson).

Explanation of the solution:
Elasticsearch's high-level rest client generically parses results from the low-level rest client. The SearchRequest's response JSON is converted into a SearchResponse Object in the RestHighLevelClient on line 129 in the search method. This method calls performRequestAndParseEntity on line 1401, which accepts an entityParser as a CheckedFunction<XContentParser, Resp, IOException>. Finally, we can see that when invoking the entityParser on line 1401, it calls the parseEntity method on line 1714 which determines the XContentType for the entity and ultimately performs the parse. Notably, when the parser is created on line 1726 a registry is passed into the parser. This registry contains all the possible XContent values a response field may be. The registry is created when the RestHighLevelClient is constructed on line 288. The full list of types, including Aggregation types, is listed on line 1748.

Onto the solution:
After reading the Elasticsearch discussion on this, it would appear that if you want to inject a JSON Response from Elastic into the SearchResponse object, it is necessary to create a NamedXContentRegistry and list of XContents testing you have to re-create the parsing. A helper method to do that, sourced from Elastic's discussion:

public static List<NamedXContentRegistry.Entry> getDefaultNamedXContents() {
    Map<String, ContextParser<Object, ? extends Aggregation>> map = new HashMap<>();
    map.put(TopHitsAggregationBuilder.NAME, (p, c) -> ParsedTopHits.fromXContent(p, (String) c));
    map.put(StringTerms.NAME, (p, c) -> ParsedStringTerms.fromXContent(p, (String) c));
    List<NamedXContentRegistry.Entry> entries = map.entrySet().stream()
            .map(entry -> new NamedXContentRegistry.Entry(Aggregation.class, new ParseField(entry.getKey()), entry.getValue()))
            .collect(Collectors.toList());
  return entries;
}

The map in the above code needs to have ALL of the Aggregations that's necessary for your test. There are more than two, two are here for brevity.

Using this helper getNamedXContents() method, you can now use the following method to take a JSON String and inject it into the SearchResponse. Also sourced from Elastic's Discussion:

public static SearchResponse getSearchResponseFromJson(String jsonResponse){
    try {
        NamedXContentRegistry registry = new NamedXContentRegistry(getDefaultNamedXContents());
        XContentParser parser = JsonXContent.jsonXContent.createParser(registry, jsonResponse);
        return SearchResponse.fromXContent(parser);
    } catch (IOException e) {
        System.out.println("exception " + e);
    }catch (Exception e){
        System.out.println("exception " + e);
    }
    return new SearchResponse();
}

Applying the solution with an Aggregation result:
Elasticsearch needs a hint to know what type of aggregation to parse this as. The hint is provided by elastic when adding ?typed_keys to the query. An example is shown in the Elasticsearch documentation on Aggregation Type Hints.

To inject the JSON String into a SearchResponse object, one must (1) Use the methods above and (2) Inject a string with type hints in it.

Primary Sources:

1. https://discuss.elastic.co/t/elasticsearch-json-response-to-searchresponse-object/124394/6
2. https://github.com/elastic/elasticsearch/blob/master/client/rest-high-level/src/main/java/org/elasticsearch/client/RestHighLevelClient.java
3. https://github.com/elastic/elasticsearch/blob/master/test/framework/src/main/java/org/elasticsearch/test/InternalAggregationTestCase.java
4. https://www.elastic.co/guide/en/elasticsearch/reference/current/returning-aggregation-type.html

Note: There are a lot of articles from circa-2015 that say this is impossible. That is obviously incorrect.

Answer 2

Based on the answer above, I managed to do it like this:

I wrote an JSON like this:

XContentBuilder builder = XContentFactory.jsonBuilder();
response.toXContent(builder, ToXContent.EMPTY_PARAMS);
String result = Strings.toString(builder);

and then I manged to read it like this:

 try {
     NamedXContentRegistry registry = new NamedXContentRegistry(getDefaultNamedXContents());
     XContentParser parser = JsonXContent.jsonXContent.createParser(registry, DeprecationHandler.THROW_UNSUPPORTED_OPERATION, result);
     SearchResponse searchResponse = SearchResponse.fromXContent(parser);
 } catch (IOException e) {
     System.out.println("exception " + e);
 } catch (Exception e) {
     System.out.println("exception " + e);
 }

public static List<NamedXContentRegistry.Entry> getDefaultNamedXContents() {
    Map<String, ContextParser<Object, ? extends Aggregation>> map = new HashMap<>();
    map.put(TopHitsAggregationBuilder.NAME, (p, c) -> ParsedTopHits.fromXContent(p, (String) c));
    map.put(StringTerms.NAME, (p, c) -> ParsedStringTerms.fromXContent(p, (String) c));
    List<NamedXContentRegistry.Entry> entries = map.entrySet().stream()
            .map(entry -> new NamedXContentRegistry.Entry(Aggregation.class, new ParseField(entry.getKey()), entry.getValue()))
            .collect(Collectors.toList());
    return entries;
}

Hope it works :)