Including an exit code 1 event in CloudWatch using Terraform for ECS

Question

I've been running containers on ECS, and using AWS Cloudwatch events to notify me when my tasks complete. All of the infrastructure has been created using Terraform. However, I'm unable to get the correct syntax in my event pattern so that I am only notified for non-zero exit codes.

The following resource works great, and sends notifications to SNS every time one of my containers exits:

resource "aws_cloudwatch_event_rule" "container-stopped-rule" {
  name        = "container-stopped"
  description = "Notification for containers that exit for any reason. (error)."

  event_pattern = <<PATTERN
{
  "source": [
    "aws.ecs"
    ],
  "detail-type": [
    "ECS Task State Change"
    ],
  "detail": {

    "lastStatus": [
      "STOPPED"
    ],
"stoppedReason" : [
    "Essential container in task exited"
  ]
}
}
PATTERN
}

However, I'm trying to modify the pattern slightly so that I'm only notified when a container exits with an error code- since we get so many notifications, we've started to tune out the emails and sometimes don't notice the email notifications where containers are exiting with errors:

resource "aws_cloudwatch_event_rule" "container-stopped-rule" {
  name        = "container-stopped"
  description = "Notification for containers with exit code of 1 (error)."

  event_pattern = <<PATTERN
{
  "source": [
    "aws.ecs"
    ],
  "detail-type": [
    "ECS Task State Change"
    ],
  "detail": {
    "containers": [
      {
      "exitCode": 1
      }
    ],
    "lastStatus": [
      "STOPPED"
    ],
"stoppedReason" : [
    "Essential container in task exited"
  ]
}
}
PATTERN
}

This triggers the following error when I terraform apply:

aws_cloudwatch_event_rule.container-stopped-rule: Updating CloudWatch Event Rule failed: InvalidEventPatternException: Event pattern is not valid. Reason: Match value must be String, number, true, false, or null at [Source: (String)"{"detail":{"containers":[{"exitCode":1}],"lastStatus":["STOPPED"],"stoppedReason":["Essential container in task exited"]},"detail-type":["ECS Task State Change"],"source":["aws.ecs"]}"; line: 1, column: 27] status code: 400

This is perplexing to me, since I'm following the exact structure laid out in the AWS CloudWatch documentation for containers. I've even attempted to put double quotes around 1 in case Terraform wants a string instead of a number.

I also tried to use AWS Console to manually edit the event pattern JSON, but received this error:

Validation error. Details: Event pattern contains invalid value (can only be a nonempty array or nonempty object)

I'm honestly a bit stumped at this point and would appreciate any tips on where my syntax is incorrect.

Answer 1

The event pattern syntax is pretty weird, I ran into the same issue. The following will work:

{
  "source": [
    "aws.ecs"
  ],
  "detail-type": [
    "ECS Task State Change"
  ],
  "detail": {
    "lastStatus": [
      "STOPPED"
    ],
    "stoppedReason": [
      "Essential container in task exited"
    ],
    "containers": {
      "exitCode": [
        1
      ]
    }
  }
}

I used $.detail.group in the Input Transformer to get the task family name in the notification message.

Answer 2

As per https://docs.aws.amazon.com/AmazonCloudWatch/latest/events/CloudWatchEventsandEventPatterns.html, For a pattern to match an event, the event must contain all the field names listed in the pattern. The field names must appear in the event with the same nesting structure.

Can you try adding more fields lister here https://docs.aws.amazon.com/AmazonECS/latest/developerguide/ecs_cwe_events.html like clusterArn, containerInstanceArn etc?