AWS CloudWatch Rule returns FailedInvocation with AWS batch as Target

Question

Hi I've scheduled a Cloudwatch rule in order to run every Wednesday at 14.15 GTM by having as target an AWS Batch, which always returns FailedInvocation. I'm seeing the FailedInvocation event from associated metrics

However there are no logs regarding the error, I cannot understand the problem.

I've followed this tutorial: https://docs.aws.amazon.com/batch/latest/userguide/batch-cwe-target.html I'm stucked here from hours any suggestion?

CONFIGURATIONS

The AWS batch target is configured as is:

• job Queue= arn:..
• job Definition = arn:...
• Job Name = name

The role associated to the target has the following policies:

• arn:aws:iam::aws:policy/service-role/AWSBatchServiceEventTargetRole

  {
  "Version": "2012-10-17",
  "Statement": [
      {
          "Effect": "Allow",
          "Action": [
              "batch:SubmitJob"
          ],
          "Resource": "*"
      }
    ]
    }
  

• arn:aws:iam::216314997889:role/awsInvokeActionOnEc2

  {
  "Version": "2012-10-17",
  "Statement": [
      {
          "Effect": "Allow",
          "Action": [
              "cloudwatch:Describe*",
              "ec2:Describe*",
              "ec2:RebootInstances",
              "ec2:StopInstances",
              "ec2:TerminateInstances"
          ],
          "Resource": "*"
      }
  ]
  }
  

• and Trust relationships

  {
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Principal": {
        "Service": "events.amazonaws.com"
      },
      "Action": "sts:AssumeRole"
    }
   ]
  }
  

Answer 1

Enable CloudTrail to find out the FailedInvocation reason in its logs. I agree going through CloudTrail to find out the failure reason is terrible. But for now, that's all there is. Faced the same issue and found out the Input

Answer 2

If anyone ever encounters FailedInvocations from event rules targeting Cloudwatch log groups, this is most likely due to the absence of a "Cloudwatch log resource policy" permitting the AWS Events service to create Cloudwatch logs. If you create the rule through the console, there should be an appropriate one automatically provisioned. You can check whether you have one provisioned:

aws logs describe-resource-policies

If you already have an appropriate Cloudwatch log resource policy configured, you should see something like:

{
    "resourcePolicies": [
        {
            "policyName": "TrustEventsToStoreLogEvents",
            "policyDocument": "{\"Version\":\"2012-10-17\",\"Statement\":[{\"Sid\":\"TrustEventsToStoreLogEvent\",\"Effect\":\"Allow\",\"Principal\":{\"Service\":[\"events.amazonaws.com\",\"delivery.logs.amazonaws.com\"]},\"Action\":[\"logs:PutLogEvents\",\"logs:CreateLogStream\"],\"Resource\":\"arn:aws:logs:eu-central-1:1234567890:log-group:/aws/events/*:*\"}]}",
            "lastUpdatedTime": 1641611871623
        }
    ]
}

However, if you've configured your rules with Terraform (maybe even Cloudformation), then this will probably not automatically be provisioned.

Here's an example Terraform excerpt to provision a policy matching the one auto-configured through the console:

data "aws_iam_policy_document" "events_delivery_logs_write_logs" {
  statement {
    sid = "TrustEventsToStoreLogEvent"

    actions = [
      "logs:CreateLogStream",
      "logs:PutLogEvents",
    ]

    resources = ["arn:${data.aws_partition.current.partition}:logs:${data.aws_region.current.name}:${data.aws_caller_identity.current.account_id}:log-group:/aws/events/*:*"]

    principals {
      identifiers = [
        "events.amazonaws.com",
        "delivery.logs.amazonaws.com"
      ]
      type = "Service"
    }
  }
}

resource "aws_cloudwatch_log_resource_policy" "events_delivery_logs_write_logs" {
  policy_document = data.aws_iam_policy_document.events_delivery_logs_write_logs.json
  # This is the standard name this is utilized when creating a CW event rule -> CW log group through the console
  policy_name = "TrustEventsToStoreLogEvents"
}

Infra resources:

• Cloudformation resource
• Terraform resource