Working around AWS Step Function Map concurrency limit

Question

I have a Map task in an AWS Step Function which executes 100-200 lambdas in parallel, each running for a few minutes, then collects the results. However, I'm running into throttling where not all lambdas are started for some time. The relevant AWS documentation says you may experience throttling with more than ~40 items, which is what I believe I'm running into.

Does anyone have any experience with working around this concurrency limitation? Can i have nested Maps, or could I bucket my tasks into several Maps I run in parallel?

Answer 1

Use nested state machine inside your map state, so you can have ~40 child state machines executing in parallel. Then inside each child state machine use a map state to process ~40 items in parallel. This way you can reach processing ~1600 items in parallel.

But before reaching that you will reach AWS Step Functions Quotas:

https://docs.aws.amazon.com/step-functions/latest/dg/limits.html