version_conflict_engine_exception with multiple _update_by_query

Question

I have situation where i am getting version_conflict_engine_exception. I have daemon which continuously pushing data for update to elasticsearch. I am using _updat_by_query. Here is the sample call:

curl -XPOST 'localhost:9200/my_index/my_type/_update_by_query?pretty' -H 'Content-Type: application/json' -d '
{
   "query":{
      "term":{
         "userid":1234
      }
   },
   "script":{
      "lang":"painless",
      "inline":"if (ctx._source.containsKey(\"newfield\")) {ctx._source.newfiled.add(params.value)} else {ctx._source.newfield = [params.value]}",
      "params":{
         "value":{"new":"value"}
      }
   }
}'

Whenever i have same userid (on query.terms), there is multiple update query hits one after another and it gives me error version_conflict_engine_exception . I know its because of version conflicting but how we can overcome from this where i have frequently update queries. I also not finding any bulk option for _update_by_query. I am getting following error:

{
    "took": 1,
    "timed_out": false,
    "total": 1,
    "updated": 0,
    "deleted": 0,
    "batches": 1,
    "version_conflicts": 1,
    "noops": 0,
    "retries": {
        "bulk": 0,
        "search": 0
    },
    "throttled_millis": 0,
    "requests_per_second": -1.0,
    "throttled_until_millis": 0,
    "failures": [{
        "index": "my_index",
        "type": "my_type",
        "id": "Gc-_SWIBUzg1_4kxJ5uD",
        "cause": {
            "type": "version_conflict_engine_exception",
            "reason": "[logs][Gc-_SWIBUzg1_4kxJ5uD]: version conflict, current version [2] is different than the one provided [1]",
            "index_uuid": "dbtt5uS9R3ClcPt6Oar1MQ",
            "shard": "3",
            "index": "my_index"
        },
        "status": 409
    }]
}

Answer 1

Currently this is the expected behavior in Elasticsearch. You can try to workaround this by using the conflicts and refresh parameters.

You can use

$ curl -XPOST 'localhost:9200/my_index/my_type/_update_by_query?conflicts=proceed&refresh=wait_for'

as mentioned in here. Hope that helps!