How to increase search size > 10,000 in elasticsearch in R?

Question

I have been trying to query elasticsearch from R using elastic package.

I was able to query and get data with

`Search(index = "tmp_test_data", 
           q = "_type: random AND log.type: regular", size = 10000)`

However, when I try to increase the size by adding body

`body1 <- '{"settings" : {"index" : {"max_result_window" : "170000"}}}'`

to the search query

`Search(index = "tmp_test_data", 
           q = "_type: random AND log.type: regular", body = body1)`

it returns, Error: 400 - Unknown key for a START_OBJECT in [settings].

Edit:
I have tried looping through using the from argument in search function with each search size = 1000 which is returning upto 10,000 records and throws Error: 500 - all shards failed after that.

I have also tried with elastic::scroll, tm_scroll = "5m" in the search function by following some of the examples given in the R documentation/help but it is returning the same error Error: 500 - all shards failed.

What is the appropriate way to increase the size of query in R elasticsearch?

Answer 1

You are trying to update a dynamic index setting. You cannot include it as part of a search.

To update it, you need to update it using the Update Settings API. I'm not sure how to do it with R but here is a http request example.

PUT http://myserver:9200/tmp_test_data/_settings

{  
    "index" : {
        "max_result_window": 170000
    }
}