AWS Step Functions: Combine task input with *partial* task output

Question

I'm working on AWS Step functions. I've read the documentation on InputPath, OutputPath, and ResultPath. My problem is that I want to combine values from the input to a Lambda Task with a partial output from the task.

The input to the task is similar to below. I want to continue passing these values on to subsequent tasks.

{
    "previous_task_result": 100,
    "next_task_input": "asdf"
}

What I would like, is to have the output of my Lambda Task look like this:

{
    "previous_task_result": 100,
    "next_task_input": "asdf",
    "current_task_result": {
        "val1": "ghjk",
        "val2": [0,2,13,100]
    }
}

The specific problem is that the raw output of the Lambda Task looks like this. I only care about the Payload node. The rest of the output is boilerplate I would rather not pass through the Step Function.

{
  "resourceType": "lambda",
  "resource": "invoke",
  "output": {
    "ExecutedVersion": "$LATEST",
    "Payload": {
        "val1": "ghjk",
        "val2": [0,2,13,100]
    },
    "SdkHttpMetadata": {
      "HttpHeaders": {
        "Connection": "keep-alive",
        "Content-Length": "42",
        "Content-Type": "application/json",
        "Date": "Tue, 25 Feb 2020 14:36:29 GMT",
        "X-Amz-Executed-Version": "$LATEST",
        "x-amzn-Remapped-Content-Length": "0"
      },
      "HttpStatusCode": 200
    },
    "SdkResponseMetadata": {
      "RequestId": "redacted"
    },
    "StatusCode": 200
  }
}

I understand how to use ResultPath and OutputPath to combine the input with the output, or assign the output to a specific node, but what I can't find is a way to merge just the Payload node from the result with the existing input.

Answer 1

When working with Lambda Task States there's 3 ways in which you can structure the your task state:

1. Request Response:

This method returns the full API response including the SdkHttpMetadata field. I suspect the reason for returning the full HTTP response is because this is the only way in which you can invoke a Lambda function asynchronously from Step Function (Asynchronous Lambda invocations only return a status code). Example:

"CallLambda": {
    "Type": "Task",
    "Resource": "arn:aws:states:::lambda:invoke",
    "Parameters": {
        "FunctionName": "MyFunction",
        "InvocationType": "Event|RequestResponse|DryRun",
        "Payload.$": "$"
    },
    "End": true
}

2. Lambda ARN as the Resource:

I believe this is what you are looking for. When specifying the ARN of the Lambda function as the Resource, Step Functions will invoke your Lambda function synchronously and return only the result from your Lambda function without the API response metadata. In this case your State Input will be passed as the payload to your Lambda function or you can use InputPath/Parameters to filter/modify the data sent as the Payload.

"CallLambda": {
    "Type": "Task",
    "Resource": "arn:aws:lambda:us-west-2:123456789012:function:my-function",
    "ResultPath": "$.current_task_result",
    "End": true
}

Using the input as in your example the above Task will give you the output as follows:

{
    "previous_task_result": 100,
    "next_task_input": "asdf",
    "current_task_result": {
        <Your Lambda Functions Result>
    }
}

3. .WaitForTaskToken:

Passes a token to your function, pauses the execution untill either a call to SendTaskSuccess or SendTaskFailed has been received (this method will only return the Result/Error from SendTaskSuccess or SendTaskFailed with no additional HTTP metadata).

"CallLambda": {
    "Type": "Task",
    "Resource":"arn:aws:states:::lambda:invoke.waitForTaskToken",
    "Parameters": {
        "FunctionName": "MyFunction",
        "Payload":{  
           "token.$":"$$.Task.Token"
        }
    },
    "End": true
}