Can I trigger an ECS/Fargate task from a specific file upload in S3?

Question

I know that I can trigger a task when a file is uploaded (per https://docs.aws.amazon.com/AmazonCloudWatch/latest/events/CloudWatch-Events-tutorial-ECS.html) however, how can I trigger a task when a specific file is uploaded?

Amazon seems not to have anticipated people having multiple jobs watching the same bucket for different files :(

Answer 1

You can accomplish this with CloudWatch Events from CloudTrail Data Events.

Head over to CloudTrail, and create a Trail for your account.

• For Apply trail to all regions, choose No.
• Under Management events, Read/Write Events, select none.
• Under Data events, select S3. Input your S3 bucket name and folder name (prefix) to log data events for, and select Write (don't set read).
• Under Storage location, create a new bucket or provide a bucket to be used to store the log files.
• Select Create

Next, create a CloudWatch Event rule that targets your ECS Task when the CloudTrail Data Event happens.

Head over to CloudWatch and Create a new Event rule.

• For the Event Source select Event Pattern
• Change the dropdown that says "Build event pattern to match events by service" to select "Custom Event Pattern"
• Enter the event pattern below:

{
  "source": [
    "aws.s3"
  ],
  "detail-type": [
    "AWS API Call via CloudTrail"
  ],
  "detail": {
    "eventSource": [
      "s3.amazonaws.com"
    ],
    "eventName": [
      "PutObject"
    ],
    "requestParameters": {
      "bucketName": [
        "your-bucket-name" // this is the bucket where your events are happening
      ],
      "key": [
        "your-object-key" // this is the object key you want to trigger starting your ECS task, note that it's an array.
      ]
    }
  }
}

• Customize the bucketName and key above as appropriate for your use.
• For your target, select ECS Task, configure the task as appropriate.
• Select Configure details, give the rule a name and set the State to Enabled, and click Create rule.

Now that your rule is enabled, when you upload an object with the specified key to the specified bucket, CloudWatch Events will trigger the ECS Task you specified.

Answer 2

Looks like you have a wildcard in your comment. To add to the event pattern from hephalump, you can indicate a prefix for the key as well, which will match to any key with that prefix, not just a specific key:

{
  "source": [
    "aws.s3"
  ],
  "detail-type": [
    "AWS API Call via CloudTrail"
  ],
  "detail": {
    "eventSource": [
      "s3.amazonaws.com"
    ],
    "eventName": [
      "PutObject"
    ],
    "requestParameters": {
      "bucketName": [
        "your-bucket-name" // this is the bucket where your events are happening
      ],
      "key": [
        {"prefix": "path/to/key"}
      ]
    }
  }
}