Elasticsearch timeout true but still get result

Question

I'm setting the timeout to 10ms to my search query, so I'm expecting that elasticsearch search query should timeout in 10ms.

In the response, I do get "timed_out":true but the query doesnt seem to timeout. It still runs for a few hundred milliseconds.

Sample response:

{
    "took": 460,
    "timed_out": true,
....

Is this the expected behavior or am I missing something here ? My goal is to terminate the query if its taking too long so that it doesnt put load on the cluster.

Answer 1

What to expect from query timeout?

Elasticsearch query running with timeout set may return partial or empty results (if timeout has expired), from the Elasticsearch Guide:

The timeout parameter tells shards how long they are allowed to process data before returning a response to the coordinating node. If there was not enough time to process all data, results for this shard will be partial, even possibly empty.

The documentation of the Request Body Search parameters also tells this:

timeout

A search timeout, bounding the search request to be executed within the specified time value and bail with the hits accumulated up to that point when expired. Defaults to no timeout.

For further details please consult this page in the guide.

How to terminate queries that run too long?

Looks like Elasticsearch does not have an ultimate answer, rather several workarounds for particular cases. Here they are.

There isn't a way to protect system from DoS attacks (as of year 2015). Long-running queries can be limited with timeout or terminate_after query parameters. terminate_after is like timeout but it counts the number of documents per shard. Both of these parameters are more like recommendations to Elasticsearch, means that some long-running queries can still pass through the desired max execution time (like a script query for instance).

Since then Task Management API was introduced and monitoring and cancelling long-running tasks became possible. This means that you will have to write some additional code that will check the health of the cluster and cancel the tasks.