There were concurrent policy changes. Please retry the whole read-modify-write with exponential backoff

Question

POST METHOD URL

https://cloudresourcemanager.googleapis.com/v1/projects/project-name:setIamPolicy

Request:

{
    "resource": "projects/project-name",
    "policy": {
        "bindings": [
            {
                "role": "roles/resourcemanager.organizationAdmin",
                "members": [
                    "user:[email protected]"
                ]
            }
         ],
        "etag": "BwWWja0YfJA=",
        "version": 3
     }
}

Response:

{

"error": { "code": 409, "message": "There were concurrent policy changes. Please retry the whole read-modify-write with exponential backoff.", "status": "ABORTED" }

}

Answer 1

Documentation recommends using the read-modify-write pattern to update policy for a resource.

Reading the current policy by calling getIamPolicy(). Editing the returned policy, either by using a text editor or programmatically, to add or remove any desired members and their role grants. Writing the updated policy by calling setIamPolicy().

Looks like in your case the policy you're trying to set and the policy that is currently active on the resource have diverged. One of the ways this can happen is if you did:

1. getIamPolicy() > policy.json
2. addIamPolicyBinding() or removeIamPolicyBinding()
3. setIamPolicy() policy.json

The policy version on the resource after #2, is out of sync with what #3 is trying to set, and so it throws an exception. To confirm you can compare the etag field in the policy your strying to set with the etag currently on the resource. There should be a mismatch.

Answer 2

This means that more than one change was performed at the same time. You should try to perform only one request to change policies at the same time.

Implementing Exponential backoff should help you with this error. It is as simple as handle your request retry with a time magnitude of n+1 + random_number_milliseconds seconds and retry the request