Given a failed AWS API request, how can I debug what permissions I need?

Question

I'm using Terraform to provision some resources on AWS. Running the "plan" step of Terraform fails with the following vague error (for example):

Error: Error loading state: AccessDenied: Access Denied
        status code: 403, request id: ABCDEF12345678, host id: SOMELONGBASE64LOOKINGSTRING===

Given a request id and a host id is it possible to see more in depth what went wrong?

Setting TF_LOG=DEBUG (or some other level) seems to help, but I was curious if there is a CLI command to get more information from CloudTrail or something.

Thanks!

Answer 1

Terraform won't have any privileged information about the access denial, but AWS does. Because you mentioned S3 was the problem I based my answer on finding the S3 request id. You have a couple options to find the request given a request id in AWS.

1. Create a trail in AWS CloudTrail. CloudTrail will log the API calls (including request id) at the bucket level by default. If the request was for a specific object, you need to enable S3 data events when you create the trail.
2. Turn on S3 server access logs.

You can manually search for the request id in the log files in S3 or use Athena. For CloudTrail, you can also configure CloudWatch Logs and search within the Log Group that gets created via the search bar.

CloudTrail records API calls from all services, not just S3. It could be a useful tool for diagnosing issues besides those related to S3. Note that there can be an up to 15-minute delay for logs to appear in CloudTrail.