Executing Step Function "Tasks" using ECS Fargate

Question

In Re:Invent 2018, AWS seems to have launched new integrations with Step Functions, which includes ECS Fargate support.

https://docs.aws.amazon.com/step-functions/latest/dg/connectors-ecs.html

I have been digging up on this, and I am able to get the ECS task to startup, and do its thing, but I am not able to send desired output back to the step function. The response syntax too does not seem to have anything that corresponds to the output of a particular state.

https://docs.aws.amazon.com/AmazonECS/latest/APIReference/API_RunTask.html#API_RunTask_ResponseSyntax

Is it possible for an ECS task to send back output to the step function? Any leads would help.

P.S: I am aware of activities, and have ECS poll for activity, and send task success on completion. But, I was looking to leverage step function calling ECS if that works.

Answer 1

You can use the "Wait for Callback (.waitForTaskToken)" integration pattern to callback to Step Functions in the ECS task. Using .waitForTaskToken instead of .sync, the execution will pause and wait for SendTaskSuccess or SendTaskFailure, which can be done from the ECS task with any output you want. It's similar to activities, but the task token is pushed to downstream service instead of polling for it.

The output of a arn:aws:states:::ecs:runTask.sync task is the result of ECS DescribeTasks.

https://docs.aws.amazon.com/step-functions/latest/dg/connect-to-resource.html#connect-wait-token

Answer 2

I am currently working on a project that is leveraging Step Functions to launch multiple Fargate Tasks in sequence and I ran into this same issue. Currently there does not appear to be a way to return anything other than the standard ResponseSyntax although hopefully this is an area AWS will expand on.

Our work around for this limitation has simply been to use S3 to store the output from each Task so it can be accessed by subsequent Tasks (or other Steps). We provide the S3 bucket and S3 key prefix to each Step Function invocation as environment variables and since container overrides are returned as part of the ResponseSyntax they can be passed to additional Steps with the following snippet.

"Overrides": {
    "ContainerOverrides": [
     {
       "Name": "<your container name>",
       "Environment.$": "$.Overrides.ContainerOverrides[0].Environment"
     }
    ]
}