AWS Cloudfront error? invalidation max times while deploying?

Question

I got the error while I'm deploying something from gitlab.

$ aws cloudfront create-invalidation --distribution-id=EVB1B2B3B4B56 --paths "/*"
**An error occurred (ServiceUnavailable) when calling the CreateInvalidation operation (reached max retries: 2): CloudFront encountered an internal error. Please try again.**

I sort it out by creating invalidation manually from AWS console and it works . This is the case happens sometimes causes this error, at that time I'm doing manually . What was the permanent solution ? Please help me to sort it out this.

Answer 1

I experienced a similar issue when creating multiple invalidations while deploying the same application to many distributions. The error still happens randomly, sometimes just retrying the deployment later works fine, and others fail on a different distribution.

I found a related issue showing the same error in the github aws-cdk repo, there seems to be an error with the AWS Cloudfront API during peak hours.

From the CloudFront team:

CreateInvalidation API suffers from high fault rate during the daily traffic peaks. It will return faults upto 50% of the requests. It is primarily due to the limited capacity of the API.

We have evidence that some requests failed even after six retries , during the peak. We are working on improving this , but there is no quick fix for this and we are expecting it will get better by the end of Q1 2022.

I still didn´t find a "clean" solution to this, what I have done is handle the error within my CLI script and automatically retry until it eventually succeeds.

Answer 2

You can use increase the number of retries directly in your command:

AWS_MAX_ATTEMPTS=10 aws cloudfront create-invalidation --distribution-id=EVB1B2B3B4B56 --paths "/*"

You usually get this error when CloudFront API is experiencing a traffic peak. No guarantee to hit successfully the endpoint but you still have less chance to miss all the retries.