Add an unknown-sized list of security groups to an EC2 instance

Question

We have a CloudFormation template that creates an EC2 instance and a security group (among many other resources), but we need to be able to add some additional pre-existing security groups to that same EC2 instance.

The issue we have is that the number of pre-existing security groups won't always be the same, and we want to have a single template that will handle all cases.

Currently, we have an input parameter that looks like this:

"WebTierSgAdditional": {
  "Type": "String",
  "Default": "",
  "Description": ""
}

We pass into this parameter a comma-delimited string of pre-existing security groups, such as 'sg-abc123,sg-abc456'.

The SecurityGroup tag of the EC2 instance look like this:

"SecurityGroups": [
  {
    "Ref": "WebSg"
  },
  {
    "Ref": "WebTierSgAdditional"
  }
]

With this code, when the instance gets created, we get this error in the AWS console:

Must use either use group-id or group-name for all the security groups, not both at the same time

The 'WebSg' ref above is one of the security groups being created elsewhere in the template. This same error appears if we pass in a list of group names rather than a list of group Id's through the input parameter.

When we change the "Type" field of the input parameter to be 'CommaDelimitedList', we get an error saying:

Value of property SecurityGroups must be of type List of String

It obviously can't join a list with a string to make it a new list.

When the parameter only contains a single sg id, everything gets created successfully, however, we need to have the capability to add more than just one sg id.

We have tried many different combinations of using Fn::Join within the SecurityGroups tag, but nothing seems to work. What we really need is some sort of 'Explode' function to extract the individual id's from the parameter string.

Does anyone know of a nice way to get this to work?

Answer 1

AWS introduced Fn::Split in January 2017 and this is now possible. It's not pretty, but you're essentially converting two lists into strings with Fn::Join, and then converting the strings back to a list with Fn::Split.

Parameters:

    WebTierSgAdditional:
        Type: CommaDelimitedList
        Default: ''

Conditions:

    HasWebTierSgAdditional: !Not [ !Equals [ '', !Select [ 0, !Ref WebTierSgAdditional ] ] ]

Resources:

    WebSg:
        Type: AWS::EC2::SecurityGroup
        Properties:
            # ...

    Ec2Instance:
        Type: AWS::EC2::Instance
        Properties:
            # ...
            SecurityGroupIds: !Split
                  - ','
                  - !Join
                      - ','
                      -   - !Ref WebSg
                          - !If [ HasWebTierSgAdditional, !Join [ ',', !Ref WebTierSgAdditional ], !Ref 'AWS::NoValue' ]

WebTierSgAdditional starts as a list, but gets converted into a string with each item separated by a comma via !Join [ ',', !Ref WebTierSgAdditional ]. That is included in another list that includes !Ref WebSg that also gets converted to a string with each item separated by a comma. !Split will take a string and split items into a list as separated by a comma.

Answer 2

As you've found out, the problem is that you need to send a list of strings as the security groups, and although CloudFormation gives you a method to join a list of strings into a single delimited string it doesn't provide an easy method for splitting a delimited string into a list of strings.

Unfortunately the only way I know how to do this is with a nested stack. You can make use of the parameter type "CommaDelimitedList" to split a comma delimited string into a list of strings.

The basic method is this:

1. Create your security group in your cloudformation template.
2. Merge that security group ID with your list of security groups using Fn::Join.
3. Pass that list to a nested stack (a resource of type AWS::CloudFormation::Stack).
4. Take that parameter as type "CommaDelimitedList" in the separate template.
5. Pass the parameter ref into your EC2 instance declaration.

I've tested this out with the following 2 templates and it's working:

Main template:

{
    "AWSTemplateFormatVersion": "2010-09-09",                                                                                                  
    "Parameters" : {
        "SecurityGroups" : {                                                                        
            "Description" : "A comma separated list of security groups to merge with the web security group",
            "Type" : "String"
        }
    },
    "Resources" : {
        "WebSg" : ... your web security group here,
        "Ec2Instance" : {
            "Type" : "AWS::CloudFormation::Stack",
            "Properties" : {
                "TemplateURL" : "s3 link to the other stack template",
                "Parameters" : {
                    "SecurityGroups" : { "Fn::Join" : [ ",", [ { "Ref" : "WebSg" }, { "Ref" : "SecurityGroups" } ] ] },
                }
            }
        }
    }
}

The nested template (linked to by the "TemplateURL" above):

{
    "AWSTemplateFormatVersion": "2010-09-09",
    "Parameters" : {
        "SecurityGroups" : {
            "Description" : "The Security Groups to launch the instance with",
            "Type" : "CommaDelimitedList"
        },
    }
    "Resources" : {
        "Ec2Instance" : {
            "Type" : "AWS::EC2::Instance",
            "Properties" : {                                     
                ... etc           
                "SecurityGroupIds" : { "Ref" : "SecurityGroups" }
            }
        }
    }
}

I'd love to know if there's a better way to do this. As you mention it really needs an explode function.

Answer 3

There is another solution which I got from the support and I found nicer.

Basically it is just getting each security group by its index from the list and then adds the internal one at the end.

"SecurityGroupList": {
    "Description": "List of existing security groups",
    "Type": "CommaDelimitedList"
},
...
"InternalSecurityGroup": {
    "Type": "AWS::EC2::SecurityGroup"
},
...
"SecurityGroupIds": [
    {
        "Fn::Select": [
            "0",
            {
                "Ref": "SecurityGroupList"
            }
        ]
    },
    {
        "Fn::Select": [
            "1",
            {
                "Ref": "SecurityGroupList"
            }
        ]
    },
    {
        "Fn::Select": [
            "2",
            {
                "Ref": "SecurityGroupList"
            }
        ]
    },
    {
        "Ref": "InternalSecurityGroup"
    }
],

Hope this helps anyone else.

Answer 4

You can actually write the necessary conversion function in lambda without creating a substack, to get the necessary List-String.

For example: http://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/walkthrough-custom-resources-lambda-lookup-amiids.html

In fact, this function doesn't really need to do anything other than receive list and return it back. I had a similar scenario where I had a List-AWS::EC2::Subnet::Id- and needed to pass that to a resource that required a List-String.

Process:

Define a LambdaExecutionRole.

Define a ListToStringListFunction of type AWS::Lambda::Function, add code that returns in the cfnresponse data {'Result':event['ResourceProperties']['List']}.

Define a custom resource MyList type Custom::MyList, give the ServiceToken of GetAtt the ListToStringListFunction Arn. Also pass List as a property that Refs the original list.

Reference MyList with Fn::GetAtt(MyList,Result)

Answer 5

Following the approach of @alanthing I came up with this solution in JSON format. Just make sure the parameter is not empty, as in my case I'm not checking for that. And also note that the template is incomplete, just showing the relevant parts.

Posting it here in case it's useful for anybody.

  "Parameters": {
    "SecurityGroups": {
      "Type": "List<AWS::EC2::SecurityGroup::Id>",
      "Description": "List of security groups"
    }
  },
  "Resources": {
    "EC2Instance": {
      "Type": "AWS::EC2::Instance",
      "Properties": {
        "SecurityGroupIds": {
          "Fn::Split": [
            ",",
            {
              "Fn::Sub": [
                "${SGIdsByParam},${SGByLogicalId}",
                {
                  "SGIdsByParam": {
                    "Fn::Join": [",", {
                      "Ref": "SecurityGroups"
                    }]
                  },
                  "SGByLogicalId": {
                    "Fn::GetAtt": ["InstanceSecurityGroup", "GroupId"]
                  }
                }
              ]
            }
          ]
        }
      }