Logo Questions Linux Laravel Mysql Ubuntu Git Menu
 

How to limit concurrency of a step in step functions

I have a state machine in AWS. I want to limit concurrency of a task (created via lambda) to reduce traffic to one of my downstream API.

I can restrict the lambda concurrency, but the task fails with "Lambda.TooManyExecutions" failure. Can someone please share a simple approach to limit concurrency of a lambda task?

Thanks, Vinod.

like image 467
user7365614 Avatar asked Mar 11 '20 00:03

user7365614


People also ask

Do Step Functions have limits?

Step Functions is engineered for limits of 300 new executions per second in N. Virginia, Oregon, and Ireland and 150 per second in all other regions. So if we scale to 1 Million events per day, that's 11.57 events per second.

Can step function run concurrently?

Each Step Functions state machine can have multiple simultaneous executions, which you can initiate from the Step Functions console , or by using the AWS SDKs, the Step Functions API actions, or the AWS Command Line Interface (AWS CLI). An execution receives JSON input and produces JSON output.

How many Step Functions can run concurrently?

Large data sets that need to be processed concurrently sounds like a great use for Step Functions. However, if you are doing parallel processing via a Map state, the max concurrency limit is 40. Meaning you will be processing the data in “batches” of 40. So your parallel processing might not be as fast as you think.


1 Answers

Within the same state machine execution

You can use a Map state to run these tasks in parallel, and use the maximum concurrency setting to reduce excessive lambda executions.

The Map state ("Type": "Map") can be used to run a set of steps for each element of an input array. While the Parallel state executes multiple branches of steps using the same input, a Map state will execute the same steps for multiple entries of an array in the state input.

  • MaxConcurrency (Optional) The MaxConcurrency field’s value is an integer that provides an upper bound on how many invocations of the Iterator may run in parallel. For instance, a MaxConcurrency value of 10 will limit your Map state to 10 concurrent iterations running at one time.

This should reduce the likelihood of issues. That said, you would still benefit from adding a retry statement for these cases. Here's an example:

{
  "Retry": [ {
    "ErrorEquals": ["Lambda.TooManyRequestsException", "Lambda.ServiceException"],
    "IntervalSeconds": 2,
    "MaxAttempts": 6,
    "BackoffRate": 2
  } ]
}

Across different executions

If you want to control this concurrency across different executions, you'll have to implement some kind of separate control yourself. One way to prepare your state machine for that is to request the data you need and then using an Activity to wait for a response.

like image 89
Ricardo Nolde Avatar answered Oct 16 '22 21:10

Ricardo Nolde