Secrets Manage: Fail to rotate the secret, cannot invoke the specified Lambda function

Question

Error: Secrets Manager cannot invoke the specified Lambda function. Ensure that the function policy grants access to the principal secretsmanager.amazonaws.com

I'm using Secret Manager to store my key for verifying JWTs.
My planned configuration is to rotate deprecate the keys with the following logic:

my secret looks like this:

{
  current:'my-current-secret',
  previous:'my-previous-secret',
  alg:'encoding alg',
}

*It seemed like overkill to use two secrets and rotating them -- I'm only keeping a memory of the previous token to handle fringe cases for a hand-off. If auth fails I'll check if it verifies with the previous, if it does it'll return an updated cookie using the current key

createSecret:

putSecretValue({
      current: getRandomPassword(...),
      previous: getSecretValue(...)['current'],
      alg: env.param ? env.param : getSecretValue(...)['alg']
})

I'm not using setSecret, testSecret, finishSecret

I'm not using serverless (I will at some point, but I wanted to familiarize myself w/ AWS/GUI first before short-cutting w/ the CLI) I've looked at:

• How do I grant a rotation Lambda access to AWS Secrets Manager
• https://docs.aws.amazon.com/secretsmanager/latest/userguide/rotating-secrets-create-generic-template.html

I can't figure out what IAM setting I'm missing.

When I try to set the rotation lambda:

This flashes (So quickly, I had to record my screen to take a look):

And I immediately get the following error:

I started by giving the lambda full control of secrets manager and lambdas to work backwards into minimal controls, but even throwing the kitchen sink at it I couldn't get it to work:

{
  "permissionsBoundary": {},
  "roleName": "secrets_manager-role-REDACTED",
  "policies": [
    {
      "document": {
        "Version": "2012-10-17",
        "Statement": [
          {
            "Sid": "VisualEditor0",
            "Effect": "Allow",
            "Action": [
              "secretsmanager:GetRandomPassword",
              "secretsmanager:CreateSecret",
              "secretsmanager:ListSecrets"
            ],
            "Resource": "*"
          },
          {
            "Sid": "VisualEditor1",
            "Effect": "Allow",
            "Action": "secretsmanager:*",
            "Resource": "arn:aws:secretsmanager:us-east-1:REDACTED:secret:REDACTED"
          }
        ]
      },
      "name": "ReadWriteREDACTEDSecret",
      "id": "REDACTED",
      "type": "managed",
      "arn": "arn:aws:iam::REDACTED:policy/ReadWriteREDACTEDSecret"
    },
    {
      "document": {
        "Version": "2012-10-17",
        "Statement": [
          {
            "Sid": "VisualEditor0",
            "Effect": "Allow",
            "Action": [
              "lambda:InvokeFunction",
              "lambda:InvokeAsync"
            ],
            "Resource": "arn:aws:lambda:us-east-1:REDACTED:function:secrets_manager"
          }
        ]
      },
      "name": "invoke_secrets_manager_lambda",
      "id": "REDACTED",
      "type": "managed",
      "arn": "arn:aws:iam::REDACTED:policy/invoke_secrets_manager_lambda"
    },
    {
      "document": {
        "Version": "2012-10-17",
        "Statement": [
          {
            "Effect": "Allow",
            "Action": "logs:CreateLogGroup",
            "Resource": "arn:aws:logs:us-east-1:REDACTED:*"
          },
          {
            "Effect": "Allow",
            "Action": [
              "logs:CreateLogStream",
              "logs:PutLogEvents"
            ],
            "Resource": [
              "arn:aws:logs:us-east-1:REDACTED:log-group:/aws/lambda/secrets_manager:*"
            ]
          }
        ]
      },
      "name": "AWSLambdaBasicExecutionRole-REDACTED",
      "id": "REDACTED",
      "type": "managed",
      "arn": "arn:aws:iam::REDACTED:policy/service-role/AWSLambdaBasicExecutionRole-REDACTED"
    },
    {
      "document": {
        "Version": "2012-10-17",
        "Statement": [
          {
            "Effect": "Allow",
            "Action": [
              "cloudformation:DescribeChangeSet",
              "cloudformation:DescribeStackResources",
              "cloudformation:DescribeStacks",
              "cloudformation:GetTemplate",
              "cloudformation:ListStackResources",
              "cloudwatch:*",
              "cognito-identity:ListIdentityPools",
              "cognito-sync:GetCognitoEvents",
              "cognito-sync:SetCognitoEvents",
              "dynamodb:*",
              "ec2:DescribeSecurityGroups",
              "ec2:DescribeSubnets",
              "ec2:DescribeVpcs",
              "events:*",
              "iam:GetPolicy",
              "iam:GetPolicyVersion",
              "iam:GetRole",
              "iam:GetRolePolicy",
              "iam:ListAttachedRolePolicies",
              "iam:ListRolePolicies",
              "iam:ListRoles",
              "iam:PassRole",
              "iot:AttachPrincipalPolicy",
              "iot:AttachThingPrincipal",
              "iot:CreateKeysAndCertificate",
              "iot:CreatePolicy",
              "iot:CreateThing",
              "iot:CreateTopicRule",
              "iot:DescribeEndpoint",
              "iot:GetTopicRule",
              "iot:ListPolicies",
              "iot:ListThings",
              "iot:ListTopicRules",
              "iot:ReplaceTopicRule",
              "kinesis:DescribeStream",
              "kinesis:ListStreams",
              "kinesis:PutRecord",
              "kms:ListAliases",
              "lambda:*",
              "logs:*",
              "s3:*",
              "sns:ListSubscriptions",
              "sns:ListSubscriptionsByTopic",
              "sns:ListTopics",
              "sns:Publish",
              "sns:Subscribe",
              "sns:Unsubscribe",
              "sqs:ListQueues",
              "sqs:SendMessage",
              "tag:GetResources",
              "xray:PutTelemetryRecords",
              "xray:PutTraceSegments"
            ],
            "Resource": "*"
          }
        ]
      },
      "name": "AWSLambdaFullAccess",
      "id": "REDACTED",
      "type": "managed",
      "arn": "arn:aws:iam::aws:policy/AWSLambdaFullAccess"
    }
  ],
  "trustedEntities": [
    "secretsmanager.amazonaws.com",
    "lambda.amazonaws.com"
  ]
}

My lambda trust policy is as follows:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Principal": {
        "Service": [
          "secretsmanager.amazonaws.com",
          "lambda.amazonaws.com"
        ]
      },
      "Action": "sts:AssumeRole"
    }
  ]
}

Answer 1

After combing through the aws gui for ~10 hours and googling extensively I came across a s/o post for a different resource that linked to the docs saying that a lambda's function policy cannot be set in the gui.

I ran the following command in the cli and everything worked:

aws lambda add-permission \
          --function-name secrets_manager \
          --principal secretsmanager.amazonaws.com \
          --action lambda:InvokeFunction \
          --statement-id SecretsManagerAccess

--function-name secrets_manager is because my lambda function is named secrets_manager

source: https://docs.aws.amazon.com/secretsmanager/latest/userguide/rotating-secrets-create-generic-template.html