AWS Step Functions with dynamic steps

Question

I am exploring AWS step functions, however, I am not able to find the answer to a question that my use case requires. Let us say we have 10 tasks we can do in the system overall. Sometimes we have to execute tasks 1, 3 and some times 4, 8 while other times 1, 9, 5 in that order.

The tasks to execute and the order in which they have to execute are determined by incoming json data into the system.

I was wondering if I could somehow dynamically create the step functions based on the needs that arise.

Answer 1

If it's all dynamic, you have to get somewhat creative to model the process in Step Functions.

One approach could be to build a state machine with all your tasks in it and a orchestrating decider function. The SM would start with the decider and then execute the correct job based on its output. Each job would in turn call the decider again. If it determines the process is done, the SM would end successfully. This is how it might look:

{
    "StartAt": "Decide next",
    "States": {
        "Decide next": {
            "Type": "Task",
            "Resource":"arn:aws:lambda:::function:decider",
            "ResultPath": "$.nextAction",
            "Next": "Choose action"
        },
        "Choose action": {
            "Type": "Choice",
            "Choices": [
                {
                    "Variable": "$.nextAction",
                    "StringEquals": "DONE",
                    "Next": "Process finished"
                },
                {
                    "Variable": "$.nextAction",
                    "StringEquals": "1",
                    "Next": "Job 1"
                },
                {
                    "Variable": "$.nextAction",
                    "StringEquals": "2",
                    "Next": "Job 2"
                },
                {
                    "Variable": "$.nextAction",
                    "StringEquals": "3",
                    "Next": "Job 3"
                }
            ]
        },
        "Job 1": {
            "Type": "Task",
            "Resource":"arn:aws:lambda:::function:job1",
            "ResultPath": "$.jobResult",
            "Next": "Decide next"
        },
        "Job 2": {
            "Type": "Task",
            "Resource":"arn:aws:lambda:::function:job2",
            "ResultPath": "$.jobResult",
            "Next": "Decide next"
        },
        "Job 3": {
            "Type": "Task",
            "Resource":"arn:aws:lambda:::function:job3",
            "ResultPath": "$.jobResult",
            "Next": "Decide next"
        },
        "Process finished": {
            "Type": "Succeed"
        }
    }
}

You will have to go through the execution log to see the progress through the SM, you won't be able to see it very well visually (all the jobs that ran will be green, which doesn't tell you about their succession).

Answer 2

Note that you can alway create a State Machine dynamically. (from here)

package com.example;

import static com.amazonaws.services.stepfunctions.builder.StepFunctionBuilder.*;
import com.amazonaws.services.stepfunctions.builder.ErrorCodes;

public class StepFunctionsDemo {

    public static void main(String[] args) {
        final StateMachine stateMachine = stateMachine()
                .comment("A Hello World example of the Amazon States Language using an AWS Lambda Function")
                .startAt("Hello World")
                .state("Hello World", taskState()
                        .resource("arn:aws:lambda:REGION:ACCOUNT_ID:function:FUNCTION_NAME")
                        .transition(end()))
                .build();
        System.out.println(stateMachine.toPrettyJson());
    }
}