ElasticSearch: is there a way to get new documents inserted in all indexes after a previous SearchRequest?

Question

I have multiple ElasticSearch indices sharded over multiple machines into which I ingest logs using logstash. I expose the data via a REST API. The API layer searches for several terms (using the ElasticSearch Java API) and presents the user with the results.

Now, if the user is looking at the results in a browser, there may be new logs ingested which match the same queries and filters that the user is now looking at. I would like the API layer to be notified that there are new items which match the query. So, if I get the same API request, I can send back only new data (the delta) or send a HTTP 304 "Not modified" response.

I looked at the percolator API but this does not seem to do what I want - it appears to give the list of queries a given document that is about to be inserted will match. My requirements are these:

1. The same user may want to check for new log messages after a few seconds or a few days.
2. Multiple users may search for the same terms but they will be in different pages (I will implement pagination using the SearchRequestBuilder.setSize API.) So, the changed items will be different for different users as they are on different pages of the same output.

Is there any way to do this in a scalable manner?

Answer 1

There is popular "Change API" proposal in Elasticsearch issue tracker, but it is not implemented yet - there are some major implementation problems and it has dependencies on some other tasks.

There is 3rd party plugin which tries to implement this feature, but it looks unmaintained.

I hope we can finally have this feature available in one of the next major ES releases.