Deploy StepFunctions with CloudFormation, from external definition file

Question

I'm trying to deploy stepfunctions with CloudFormation, and I'd like to reference the actual stepfunction definition from an external file in S3.

Here's how the template looks like:

StepFunction1: 
    Type: "AWS::StepFunctions::StateMachine"
    Properties:
      StateMachineName: !Ref StepFunction1SampleName
      RoleArn: !GetAtt StepFunctionExecutionRole.Arn
      DefinitionString:  
        Fn::Transform:
          Name: AWS::Include
          Parameters:
            Location: 
              Fn::Sub: 's3://${ArtifactsBucketName}/StepFunctions/StepFunction1/definition.json'

However, this doesn't seem to be supported, as we are getting error

Property validation failure: [Value of property {/DefinitionString} does not match type {String}]

I am doing something similar for APIs, referencing the actual API definition from an external swagger file, and that seems to be working fine.

Example:

SearchAPI:
    Type: "AWS::Serverless::Api"
    Properties:
      Name: myAPI
      StageName: latest
      DefinitionBody: 
        Fn::Transform:
          Name: AWS::Include
          Parameters:            
            Location: 
              Fn::Sub: 's3://${ArtifactsBucketName}/ApiGateway/myAPI/swagger.yaml'

How can I make this work?

Answer 1

The trick is to escape the StepFunction DefinitionString property, and include the actual property, DefinitionString, in the external CloudFormation referenced file. Escaping only the stepfunction definition string would fail, CloudFormation complaining that the referenced Transform/Include template, is not a valid yaml/json. Here's how it looks like: Template:

StepFunction1: 
    Type: "AWS::StepFunctions::StateMachine"
    Properties:
      StateMachineName: !Ref StepFunction1SampleName
      RoleArn: !GetAtt StepFunctionExecutionRole.Arn      
      Fn::Transform:
        Name: AWS::Include
        Parameters:
          Location: 
            Fn::Sub: 's3://${ArtifactsBucketName}/StepFunctions/StepFunction1/definition.json'

External stepfunction definition file:

{
    "DefinitionString" : {"Fn::Sub" : "{\r\n  \"Comment\": \"A Retry example of the Amazon States Language using an AWS Lambda Function\",\r\n  \"StartAt\": \"HelloWorld\",\r\n  \"States\": {\r\n    \"HelloWorld\": {\r\n      \"Type\": \"Task\",\r\n      \"Resource\": \"arn:aws:lambda:${AWS::Region}:${AWS::AccountId}:function:${HelloWorldLambdaFunctionName}\",      \r\n      \"End\": true\r\n    }\r\n  }\r\n}"}
}

Now, although this solves the problem, it's a bit more difficult to maintain the StepFunction definition, in this form, in source control.

So I've thought about using a CloudFormation custom resource backed by a lambda function. The lambda function would deal with the actual StepFunction DefinitionString escaping part.

Here's how it looks like: Template:

StepFunctionParser:
    Type: Custom::AMIInfo
    Properties:
      ServiceToken: myLambdaArn
      DefinitionString: 
        Fn::Transform:
          Name: AWS::Include
          Parameters:
            Location: 
              Fn::Sub: 's3://${ArtifactsBucketName}/StepFunctions/StepFunctionX/definition.json'   
  StepFunctionX: 
    Type: "AWS::StepFunctions::StateMachine"
    Properties:
      StateMachineName: StepFunction1SampleNameX
      RoleArn: !GetAtt StepFunctionExecutionRole.Arn      
      DefinitionString: !GetAtt StepFunctionParser.DefinitionString

External StepFunction definition file:

{
  "Comment": "A Retry example of the Amazon States Language using an AWS Lambda Function",
  "StartAt": "HelloWorld",
  "States": {
    "HelloWorld": {
      "Type": "Task",
      "Resource": {"Fn::Sub" : "arn:aws:lambda:${AWS::Region}:${AWS::AccountId}:function:${HelloWorldLambdaFunctionName}" },
      "End": true
    }
  }
}

Here's the documentation for creating AWS Lambda-backed Custom Resources.

There's still a problem with this. Transform/Include converts external template boolean properties into string properties. Therefore, DefinitionString

"DefinitionString": {
            "States": {
                "HelloWorld": {
                    "Type": "Task",
                    "Resource": "arn:aws:lambda:${AWS::Region}:${AWS::AccountId}:function:${HelloWorldLambdaFunctionName}",
                    **"End": true**
                }
            },
            "Comment": "A Retry example of the Amazon States Language using an AWS Lambda Function",
            "StartAt": "HelloWorld"
        }

becomes

"DefinitionString": {
            "States": {
                "HelloWorld": {
                    "Type": "Task",
                    "Resource": _realLambdaFunctionArn_,
                    **"End": "true"**
                }
            },
            "Comment": "A Retry example of the Amazon States Language using an AWS Lambda Function",
            "StartAt": "HelloWorld"
        }

CloudFormation then complains about the StepFunction definition not being valid:

Invalid State Machine Definition: 'SCHEMA_VALIDATION_FAILED: Expected value of type Boolean at /States/HelloWorld/End' 

Is this a CloudFormation Transform/Include issue? Can someone from AWS give a statement on this?