co.elastic.clients.transport.TransportException: [es/search] Missing [X-Elastic-Product] header

Question

I'm following the tutorial from elastic search java api client here: https://www.elastic.co/guide/en/elasticsearch/client/java-api-client/current/connecting.html

My code is as following.

// Create the low-level client
RestClient restClient = RestClient.builder(
 new HttpHost("localhost", 9200)).build();

// Create the transport with a Jackson mapper
ElasticsearchTransport transport = new RestClientTransport(
 restClient, new JacksonJsonpMapper());

// And create the API client
ElasticsearchClient client = new ElasticsearchClient(transport);

try {
 SearchResponse<Object> search = client.search(s -> s
   .index("*:*"),
   Object.class);
} catch (IOException e) {
 System.out.println(e.getMessage());
}

This code is throwing out the following exception:

co.elastic.clients.transport.TransportException: [es/search] Missing [X-Elastic-Product] header. Please check that you are connecting to an Elasticsearch instance, and that any networking filters are preserving that header.

I've tried manually putting this header via the setDefaultHeaders method like this:

RestClientBuilder builder = RestClient.builder(
 new HttpHost("localhost", 9200, "http"));
Header[] defaultHeaders = new Header[]{new BasicHeader("X-Elastic-Product", "Elasticsearch")};
builder.setDefaultHeaders(defaultHeaders);
RestClient restClient = builder.build();

But the error is the same.

I've tried both version 7.16 and 8.0.0, same result.

Answer 1

The default headers the RestClientBuilder allows you to specify are the request headers, not the response headers. The error you are getting is because older Elasticsearch [server] versions do not include the X-Elastic-Product=Elasticsearch header in any of the API responses, but the recent distributions do (7.14+?), so the newer versions of elasticsearch-java (i.e. client) expects them.

I am in the same boat — I use 8.4.2 of elasticsearch-java with an Elasticsearch server version of 7.2.0.

I ran into two format-based compatibility issues:

1. The client passing a Content-Type not known to the server, and so its request getting rejected with a 406
2. The client validating if the response has X-Elastic-Product=Elasticsearch header

Fortunately, the RestClientBuilder allows you to customize the underlying http client through: setHttpClientConfigCallback. The callback looks like this, so basically you can intercept the request and responses, manipulate headers, and thereby get around these issues:

    public interface HttpClientConfigCallback {
        /**
         * Allows to customize the {@link CloseableHttpAsyncClient} being created and used by the {@link RestClient}.
         * Commonly used to customize the default {@link org.apache.http.client.CredentialsProvider} for authentication
         * or the {@link SchemeIOSessionStrategy} for communication through ssl without losing any other useful default
         * value that the {@link RestClientBuilder} internally sets, like connection pooling.
         */
        HttpAsyncClientBuilder customizeHttpClient(HttpAsyncClientBuilder httpClientBuilder);
    }

Specifically, here's what worked for me:

var httpClientConfigCallback = httpClientBuilder ->
        httpClientBuilder
            .setDefaultCredentialsProvider(credentialsProvider)
            // this request & response header manipulation helps get around newer (>=7.16) versions
            // of elasticsearch-java client not working with older (<7.14) versions of Elasticsearch
            // server
            .setDefaultHeaders(
                List.of(
                    new BasicHeader(
                        HttpHeaders.CONTENT_TYPE, ContentType.APPLICATION_JSON.toString())))
            .addInterceptorLast(
                (HttpResponseInterceptor)
                    (response, context) ->
                        response.addHeader("X-Elastic-Product", "Elasticsearch"));
var restClient =
        RestClient.builder(elasticsearchHosts)
            .setHttpClientConfigCallback(httpClientConfigCallback)
            .build();

Note that there could still be behavioral differences between the aforementioned product and API versions as they are way too apart. The above only fixes format-based incompatibilities. For this reason, it's always best to use at least the same major versions of these components, if not the exact versions.