How to manage properly Elastic Java Rest Client timeout

Question

We are using elastic java rest high client 6.1, but often and randomly the rest client return an timeout.

We alredy customize timeout in this way:

        client.setRequestConfigCallback(new RestClientBuilder.RequestConfigCallback {
        override def customizeRequestConfig(requestConfigBuilder: RequestConfig.Builder): RequestConfig.Builder = {
            requestConfigBuilder
                .setConnectionRequestTimeout(0)
                .setConnectTimeout(parameters.getInt(s"$propertyGroup.timeout.connection"))
                .setSocketTimeout(parameters.getInt(s"$propertyGroup.timeout.socket"))
        }
    }).setMaxRetryTimeoutMillis(parameters.getInt(s"$propertyGroup.timeout.socket"))
        .setFailureListener(new RestClient.FailureListener {
            override def onFailure(host: HttpHost): Unit = {
                val a = 0
            }
        })

And during our test we set timeout threshold in 60000ms, 100000ms, 180000ms; and still randomly get a timeout exception.

This issue happens only when we run a FLINK JOB that run as much fast it can many queries in parallel, or when we query elastic into a ForEach with billions of iterations.

There is a strategy, idea, snippet of code, configuration, best practise, etc. to: Wait few seconds (or milliseconds) and try again to execute the failed query?

Answer 1

For who have same issue...

By my test seems that a timeout can be avoided just adding more ram.

Of course it depend by the reason of timeout; while I was figuring out to my issue I discovered three potential reasons:

1. request timeout: Exist a reported bug about java http client, the suggestion is to set to 0 the request connection timeout (look to my code above)
2. many concurrent request: if your application is executing many concurrent request, then is possible that elastic node starts to queue queries producing unwanted timeout. This was my issue, to resolve i change JVM configuration about heap memory bringing from 8gb to 20gb and resizing data node from 32gb to 64gb, also I gain boost of performance adding a client node and configuring a load balancer between two client node, and again I gain more performance moving from 10k disk to ssd read intensive.
3. query timeout: sometimes simply the query is inefficient and can occure a timeout, my suggestion is to simplify the query adding a field that contains a value to represent the combination of a more complex filter condition, so instead of making a complex query you can query just one indexed field