CannotPullContainerError: ref pull has been retried 1 time(s): failed to copy: httpReadSeeker: failed open with 403 error

Question

I've recently set up an ECS Fargate service in a private subnet having CannotPullContainerError: ref pull has been retried 1 time(s): failed to copy: httpReadSeeker: failed open: unexpected status code https://xxx.dkr.ecr.ap-southeast-2.amazonaws.com/v2/xxx/blobs/sha256:xxx: 403 Forbidden error during startup. I've done some troubleshooting myself, checked private subnet routes, ACLs, service task execution role and security group, which all seem correct (comparing to a running env). But I'm still getting this error, I'm hoping to get some help, many thanks!

Checked VPC private subnet and ACL - routes to local within VPC, to 0.0.0.0 via NAT GW. ACL now allows all traffic for troubleshooting purpose.

ECS task exec role - allow ecr:* for troubleshooting purpose.

ECS security group - all traffic on service port (8080)

I also found something confusing, the error containing an URL:

https://xxxx.dkr.ecr.ap-southeast-2.amazonaws.com/v2/xxx/blobs/sha256:xxxx, and the sha is different from the ECR image sha, not sure if this is expected.

Answer 1

The AWS ECR service stores image layers in S3. So you should make sure that the Policy of the S3 Gateway Endpoint is not denying access to ECR.

For more information about how to adjust the Policy make sure you check the following AWS documentation.