Logo Questions Linux Laravel Mysql Ubuntu Git Menu
 

Check if Elasticsearch has finished indexing

Is there a way to check if Elasticsearch has finished processing my request?
I want to perform integration tests for my application checking if a record can be found after insertion. For example if I make a following request:

POST /_all/_bulk
{  
   "update":{  
      "_id":419,
      "_index":"popc",
      "_type":"offers"
   }
}
{  
   "doc":{  
      "id":"419",
      "author":"foo bar",
      "number":"642-00419"
   },
   "doc_as_upsert":true
}

And I check immediately, the test fails, because it takes some time for Elasticsearch to complete my request.
If I sleep for 1 second before the assertion it works most of the time, but not always.
I could extend the sleep time to eg. 3 seconds, but it makes the tests very slow, hence my question.

I have tried using cat pending tasks and pending cluster tasks endpoints, but the responses are always empty.

If any of this is relevant, I'm using Elasticsearch 5.4, Laravel Scout 3.0.5 and tamayo/laravel-scout-elastic 3.0.3

like image 393
Kuba Szymanowski Avatar asked Aug 29 '17 10:08

Kuba Szymanowski


People also ask

How do I check my Elasticsearch data?

You can use the search API to search and aggregate data stored in Elasticsearch data streams or indices. The API's query request body parameter accepts queries written in Query DSL. The following request searches my-index-000001 using a match query. This query matches documents with a user.id value of kimchy .

How can I check my Kibana index?

Open Kibana's main menu and click Stack Management > Index Management. The Index Management page contains an overview of your indices. Badges indicate if an index is a follower index, a rollup index, or frozen. Clicking a badge narrows the list to only indices of that type.

What does closing an index do in Elasticsearch?

A closed index is blocked for read/write operations and does not allow all operations that opened indices allow. It is not possible to index documents or to search for documents in a closed index.

How do I search all indexes in Elasticsearch?

To search multiple data streams and indices, add them as comma-separated values in the search API's request path. The following request searches the my-index-000001 and my-index-000002 indices. You can also search multiple data streams and indices using an index pattern.


2 Answers

I found this PR: https://github.com/elastic/elasticsearch/pull/17986

You can use refresh: wait_for and Elasticsearch will only respond once your data is available for search.

like image 77
olvlvl Avatar answered Sep 22 '22 19:09

olvlvl


You can wait for the response; when you receive the response to the update request, it's done (and you won't see it in pending or current tasks). I think the problem you're having is probably with the refresh interval (see dynamic settings). Indexed documents are not available for search right away, and this is the (maximum) amount of time before they will be available. (You can change this setting for what makes sense for your use case, or use this setting to let you know how long you need to sleep before searching for the integration tests.)

If you want to see at in-progress tasks, you can use the tasks api.

like image 24
dshockley Avatar answered Sep 25 '22 19:09

dshockley