Access denied to AWS Secrets Manager API for BatchGetSecretValue, while GetSecretValue works

Question

I have configured an AWS IAM policy to allow all access to secretsmanager on all resources:

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Action": [
                "secretsmanager:*"
            ],
            "Resource": [
                "arn:aws:secretsmanager:us-east-2:MYACCOUNTID:secret:*"
            ],
            "Effect": "Allow"
        }
    ]
}

And running the AWS CLI command get-secret-value works fine, while batch-get-secret-value gives the error:

An error occurred (AccessDeniedException) when calling the BatchGetSecretValue operation:
 User: arn:aws:sts::MYACCOUNTID:assumed-role/my-server-staging-us-east-2/i-XXXXXXXXXXXXXXXXX is not
 authorized to perform: secretsmanager:BatchGetSecretValue because no identity-based policy allows the
 secretsmanager:BatchGetSecretValue action

Here's how it goes:

ubuntu@vapp-3-145-XX-XXX:~$ /snap/bin/aws --region us-east-2 secretsmanager get-secret-value --secret-id secret-secret-account
{
    "ARN": "arn:aws:secretsmanager:us-east-2:MYACCOUNTID:secret:secret-secret-account-2eHacl",
    "Name": "secret-secret-account",
    "VersionId": "b72697d0-4d9d-4bbe-8504-d174d7444510",
    "SecretString": "FOOBAR",
    "VersionStages": [
        "AWSCURRENT"
    ],
    "CreatedDate": "2024-11-12T14:05:56.132000+00:00"
}
ubuntu@vapp-3-145-XX-XXX:~$ /snap/bin/aws --region us-east-2 secretsmanager batch-get-secret-value --secret-id-list secret-secret-account
An error occurred (AccessDeniedException) when calling the BatchGetSecretValue operation: User: arn:aws:sts::MYACCOUNTID:assumed-role/my-server-staging-us-east-2/i-XXXXXXXXXXXXXXXXX is not authorized to perform: secretsmanager:BatchGetSecretValue because no identity-based policy allows the secretsmanager:BatchGetSecretValue action

What gives?

Searching the web I found suggestions to add a "Principal" to the policy, but if I try to do that, the IAM policy editor says that "Unsupported Principal: The policy type IDENTITY_POLICY does not support the Principal element. Remove the Principal element".

Update 1

I verified that there is only one policy that applies to said instance that has anything about secretsmanager - and it is what I expect to find:

$ (
function aws() { /snap/bin/aws --profile MYPROFILE "$@"; };
iamProfile=$(aws ec2 describe-instances --instance-id i-XXXXXXXXXXXXXXXXX | \
  jq -r '.Reservations[].Instances[].IamInstanceProfile.Arn|split("/")|last');
aws iam get-instance-profile --instance-profile "$iamProfile" | \
  jq -r '.InstanceProfile.Roles[].RoleName' | while read roleName; do
aws iam list-attached-role-policies --role-name vapp-role-staging-us-east-2 | \
  jq -r '.AttachedPolicies[].PolicyArn' | while read policyArn; do
aws iam get-policy --policy-arn "$policyArn" | \
  jq -r '.Policy.DefaultVersionId' | while read defVer; do
aws iam get-policy-version --policy-arn "$policyArn" --version-id "$defVer" | \
  jq -r '.PolicyVersion.Document.Statement[].Action|if type == "array" then . else [.] end|.[]';
done;
done;
done;
)|grep secretsmanager
secretsmanager:*

Update 2

We do have Service Control Policies enabled in the organization:

But there's only one - the AWS managed "FullAWSAccess":

And that is applied to the root account, all sub accounts and all organizational unit.

Update 3

I found out that if I replace more ARN parts with wild cards in the Resource field, it doesn't help, but doing "Resource": [ "*" ] does allow the EC2 instance to successfully call BatchGetSecretValue.

Even more interestingly, using the original policy (that I started with but didn't post here) that allows secretsmanager:GetSecretValue on the specific secrets ARNs that I have, and then also adding a statement for secretsmanager:BatchGetSecretValue with no resource limit - i.e.:

{"Version": "2012-10-17","Statement": [
  { "Action": [ "secretsmanager:GetSecretValue" ],
    "Resource": [ "arn:aws:secretsmanager:us-east-2:MYACCUOUNTID:secret:secret-secret-account", ... ],
    "Effect": "Allow" },
  { "Action": [ "secretsmanager:BatchGetSecretValue" ],
    "Resource": [ "*" ],
    "Effect": "Allow" }
]}

This causes the batch-get-secret-value call to return this weird error message:

{
    "SecretValues": [],
    "Errors": [
        {
            "SecretId": "secret-secret-account",
            "ErrorCode": "AccessDeniedException",
            "Message": "User: arn:aws:sts::MYACCOUNTID:assumed-role/my-server-staging-us-east-2/i-XXXXXXXXXXXXXXXXX is not authorized to perform: secretsmanager:GetSecretValue on resource: secret-secret-account because no identity-based policy allows the secretsmanager:GetSecretValue action"
        }
    ]
}

The format is fine, I believe, it is the batch API returning a successful result that has an error for one of the requested secrets (the only one), but the internal error is "access denied" even though there is access. Even weirder is that with this configuration, the get-secret-value call fails sometimes (!!!).

My current workaround is to use the horrible policy statement "Action":["secretsmanager:GetSecretValue","secretsmanager:BatchGetSecretValue"],"Resource":"*","Effect":"Allow" which is not good solution because I'm running several applications on the same region and account, and they should not have access to each other's secrets.

Update 4

The inconsistent behavior seems to be about the calls hitting different API servers that take differing amounts of time to get updated with new policy versions - sometimes as long as 10 minutes, but it eventually settles.

Answer 1

I've raised the same issue on AWS re:Post where I was pointed at the documentation and BKM - which is that BatchGetSecretValue by definition does not act on a specific resource and therefor breaks if it has any resource limit. Instead, what BatchGetSecretValue does is that it just takes the argument list and perform multiple GetSecretValue API calls for the caller, with the caller's security context - so access to specific secrets can be controlled by setting the GetSecretValue permissions correctly.

The BKM is to use the following security policy:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": [
        "secretsmanager:BatchGetSecretValue",
        "secretsmanager:ListSecrets"
      ],
      "Resource": "*"
    },
    {
      "Effect": "Allow",
      "Action": [
        "secretsmanager:GetSecretValue"
      ],
      "Resource": [
        "SecretARN1",
        "SecretARN2",
        "SecretARN3"
     ]
   }
 ]
}

The BatchGetSecretValue API needs to be able to list all the secrets, so that it can tell which of the secrets that were asked for actually exist, and then it could try to get each of them and will return an errors item in the response (as can be seen in the OP) if it can't access a secret that it saw.