How does the MaxConcurrency attribute work for the Map Task in AWS Step Functions?

Question

Update: Creating a step function from the Map State step template and running that also throws an error. This is strong evidence that the MaxConcurrency attribute together with the Parameters value is not working.

I am not able to use the MaxConcurrency attribute successfully in the step function definition.

This can be demonstrated by using the example provided in the documentation for the Map Task (new as of 18 sept 2019):

{
  "StartAt": "ExampleMapState",
  "States": {
    "ExampleMapState": {
      "Type": "Map",
      "MaxConcurrency": 2,
      "Parameters": {
        "ContextIndex.$": "$$.Map.Item.Index",
        "ContextValue.$": "$$.Map.Item.Value"
      },
      "Iterator": {
         "StartAt": "TestPass",
         "States": {
           "TestPass": {
             "Type": "Pass",    
             "End": true
           }
         }
      },
      "End": true
    }
  }
}

By executing the step function with the following input:

[
  {
    "who": "bob"
  },
  {
    "who": "meg"
  },
  {
    "who": "joe"
  }
]

We can observe in the Execution event history that we get:

• ExecutionStarted
• MapStateEntered
• MapStateStarted
• MapIterationStarted (index 0)
• MapIterationStarted (index 1)
• PassStateEntered (index 0)
• PassStateExited (index 0)
• MapIterationSucceeded (index 0)
• ExecutionFailed

The step function fails. The ExecutionFailed step has the following output (execution id omitted):

{
  "error": "States.Runtime",
  "cause": "Internal Error (omitted)"
}

Trying to catch the error with a Catch step has no effect.

What am I doing wrong here? Is this a bug?

Answer 1

Response to a private ticket submitted to AWS this morning;

Thank you for contacting AWS Premium Support. My name is Akanksha and I will be assisting you with this case.

I understand that you have been working with the new Map state feature of step functions and have noticed that when we use Parameters along with MaxConcurrency set to lower value than the number of iterations (with only first iteration successful) it fails with ‘States.Runtime’ and looks like a bug with the functionality.

Thank you for providing the details. It helped me during troubleshooting. In order to confirm the behavior, I used the below state machine example with Pass:

{ "StartAt": "Map State", "TimeoutSeconds": 3600, "States": { "Map State": { "Type": "Map”, "Parameters": { “ContextValue.$”: "$$.Map.Item.Value" }, "MaxConcurrency": 1, "Iterator": { "StartAt": "Run Task", "States": { "Run Task": { "Type": "Pass", "End": true } } }, "Next": "Final State" }, "Final State": { "Type": "Pass", "End": true } } }

I tested with multiple input lists and MaxConcurrency values and below are my observations:

1. Input size list: 4 MaxConcurrency:1/2/3 - Fails and MaxConcurrency:0/4/5 or above - Works
2. Input size list: 3 MaxConcurrency: 1/2 - Fails and MaxConcurrency:0/3/4 or above - Works
3. Similarly, I performed tests by removing the parameters from state machine as well and could see that it works as expected with different MaxConcurrency values.
4. I also tested the same by changing the Task type of “Pass” with “Lambda” and observed the same behavior.

Hence, I can confirm that the state machine fails when we have parameters in the code and specify MaxConcurrency value as anything other than zero or the number greater than or equal to the list size.

After doing some research regarding this behavior to check if this is intended, I could not find much information regarding the same as this is a new feature. So, I will be reaching out to the internal team with all the details and the example state machine that you have provided. Thank you for bringing this to our notice. I will get back to you as soon as I have an update from the internal team. Please be assured that I will regularly follow up with the team and work with them to investigate further.

Meanwhile, if you have any other queries or concerns, please do let me know.

Have a great day ahead!

I will update here when I get more information.