How to Auto scale workers in AWS step functions

Question

We are implementing a process using AWS step functions. Some of the tasks in this process take long time. We are using Activities(workers) running in ECS instances for this step. The state machine is like this:

Trigger -> step 1 (Lambda) -> step2 (ECS) -> final step

Is there a way to find the number of tasks in queue waiting to be picked up by the step2 workers and use this information to auto scale the workers (ECS cluster)?

Answer 1

You can call the api list_executions to get the list of running executions and than for each of this executions call get_execution_history.

In the response you can iterate the execution events. If you find in the list ActivityScheduled and you don't see ActivityStarted it means it is waiting for a worker to take this task.