Elasticsearch unlimited size

Question

I am listening a traffic and inserting these traffic's data to elasticsearch continually. And I want to search these datas with my python script.

Here is small part of my python code,

     test = es.search(
     index="argus_data",
     body=dict(query=search_body["query"],
                size= "1000") # I want to do this "unlimited"
  )

  pprint(test)

I dont know what is my size because I have new data continually. how to manage this situation please help me to solve this issue,Thanks.

Answer 1

You can do it this way:

test=es.search(index=['test'],doc_type=['test'],size=1000, from_=0)

Then change from_ gradually until you get all of the data.

from_ – Starting offset (default: 0) Elasticsearch-Py API Documentation

Answer 2

First get number of hits by using test['hits']['total'] into a variable, then pass it to size.

You have to use the query two times. The first time you use it to get number of hits(don't pass size argument).

test=es.search(index=['test'],doc_type=['test'])
size=test['hits']['total']

Second time use the query along with size

test=es.search(index=['test'],doc_type=['test'],"size":size)

Answer 3

You will get error if there will be more than 10000 observation..