"500 Backend Error" using Gmail API - safe to retry?

Question

I'm sending messages via the Gmail API. In particular, I am trying to send 5-7 emails from the same account to different users (1 each) within about 2 seconds.

About 8% of those emails are failing with this error:

&googleapi.Error{
    Code:500, 
    Message:"Backend Error", Body:`{
        "error": {
            "errors": [
            {
                "domain": "global",
                "reason": "backendError",
                "message": "Backend Error"
            }
            ],
            "code": 500,
            "message": "Backend Error"
        }
    }`, 
    Header:http.Header(nil),
    Errors:[]googleapi.ErrorItem{
        googleapi.ErrorItem{Reason:"backendError", Message:"Backend Error"}
    }
}

It doesn't seem like it's specific to a particular account, as 6/7 emails may succeed.

I'm hesitant to retry this for fear of sending 2 emails to the same person.

Is there any way to tell whether this message is safe to retry?

Answer 1

"code": 500, "message": "Backend Error"

Is basically an issue with Google server. Either the request you are making took to long or the server preforming the request is busy and the request again took to long. It doesn't sound like what you are doing should be causing the problem.

Tips when not to run: Don't run on the hour you will be completing with everyone who has cron jobs set up also don't run at midnight (PDT) as this is when quotas reset and again you will be completing with everyone who blew out yesterdays quota.

Solution:

The normal solution is to wait a few seconds then send the same request again. (Implementing Exponential Backoff)

The flow for implementing simple exponential backoff is as follows.

1. Make a request to the API
2. Receive an error response that has a retry-able error code
3. Wait 1s + random_number_milliseconds seconds
4. Retry request
5. Receive an error response that has a retry-able error code
6. Wait 2s + random_number_milliseconds seconds
7. Retry request
8. Receive an error response that has a retry-able error code
9. Wait 4s + random_number_milliseconds seconds
10. Retry request
11. Receive an error response that has a retry-able error code
12. Wait 8s + random_number_milliseconds seconds
13. Retry request
14. Receive an error response that has a retry-able error code
15. Wait 16s + random_number_milliseconds seconds
16. Retry request

If you still get an error, stop and log the error.