Cycle error when trying to create AWS VPC security groups using Terraform

Question

I want to create 2 VPC security groups.

One for the Bastion host of the VPC and one for the Private subnet.

# BASTION # 
resource "aws_security_group" "VPC-BastionSG" {
  name        = "VPC-BastionSG"
  description = "The sec group for the Bastion instance"
  vpc_id      = "aws_vpc.VPC.id"

  ingress {
      from_port = 22
      to_port   = 22
      protocol  = "tcp"
      cidr_blocks = ["my.super.ip/32"]
  } 

  egress {
      # Access to the Private subnet from the bastion host[ssh]
      from_port = 22
      to_port   = 22
      protocol  = "tcp"
      security_groups = ["${aws_security_group.VPC-PrivateSG.id}"]
  }
  egress {
      # Access to the Private subnet from the bastion host[jenkins]
      from_port = 8686
      to_port   = 8686
      protocol  = "tcp"
      security_groups = ["${aws_security_group.VPC-PrivateSG.id}"]
  }

  tags = {
    Name = "VPC-BastionSG"
  }
}

# PRIVATE #
resource "aws_security_group" "VPC-PrivateSG" {
  name        = "VPC-PrivateSG"
  description = "The sec group for the private subnet"
  vpc_id      = "aws_vpc.VPC.id"

  ingress {
      from_port = 22
      to_port   = 22
      protocol  = "tcp"
      security_groups = ["${aws_security_group.VPC-BastionSG.id}"]
  }
  ingress {
      from_port = 80
      to_port   = 80
      protocol  = "tcp"
      security_groups = ["${aws_security_group.VPC-PublicSG.id}"]
  }
  ingress {
      from_port = 443
      to_port   = 443
      protocol  = "tcp"
      security_groups = ["${aws_security_group.VPC-PublicSG.id}"]
  }
  ingress {
      from_port = 3306
      to_port   = 3306
      protocol  = "tcp"
      security_groups = ["${aws_security_group.VPC-PublicSG.id}"]
  }
  ingress {
      from_port = 8686
      to_port   = 8686
      protocol  = "tcp"
      security_groups = ["${aws_security_group.VPC-BastionSG.id}"]
  }
  ingress {
      # ALL TRAFFIC from the same subnet
      from_port = 0
      to_port   = 0
      protocol  = "-1"
      self      = true
  }
  egress {
      # ALL TRAFFIC to outside world
      from_port = 0
      to_port   = 0
      protocol  = "-1"
      cidr_blocks = ["0.0.0.0/0"]
  }
  tags = {
    Name = "VPC-PrivateSG"
  }
}

When I terraform plan it, this error is returned:

**`Error configuring: 1 error(s) occurred:
* Cycle: aws_security_group.VPC-BastionSG, aws_security_group.VPC-PrivateSG`**

If I comment out the ingress rules for the BastionSG from the PrivateSG the plan executes fine.

Also, if I comment out the egress rules for the PrivateSG from the BastionSG it also executes fine.

The AWS Scenario 2 for building a VPC with Public/Private subnets and Bastion host describes the architecture I am trying to setup.

I have the exact same settings configured via the AWS console and it plays fine.

Why isn't Terraform accepting it? Is there another way to connect the Bastion security group with the Private security group?

EDIT

As I understand there is a circular reference between the two sec groups that somehow needs to break even though in AWS it is valid.

So, I thought of allowing all outbound traffic (0.0.0.0/0) from the Bastion sec group and not specifying it to individual security groups.

Would it have a bad security impact?

Answer 1

Terraform attempts to build a dependency chain for all of the resources defined in the folder that it is working on. Doing this enables it to work out if it needs to build things in a specific order and is pretty key to how it all works.

Your example is going to fail because you have a cyclic dependency (as Terraform helpfully points out) where each security group is dependent on the other one being created already.

Sometimes these can be tricky to solve and may mean you need to rethink what you're trying to do (as you mention, one option would be to simply allow all egress traffic out from the bastion host and only restrict the ingress traffic on the private instances) but in this case you have the option of using the aws_security_group_rule resource in combination with the aws_security_group resource.

This means we can define empty security groups with no rules in them at first which we can then use as targets for the security group rules we create for the groups.

A quick example might look something like this:

resource "aws_security_group" "bastion" {
  name = "bastion"
  description = "Bastion security group"
}

resource "aws_security_group_rule" "bastion-to-private-ssh-egress" {
    type = "egress"
    from_port = 22
    to_port = 22
    protocol = "tcp"
    security_group_id = "${aws_security_group.bastion.id}"
    source_security_group_id = "${aws_security_group.private.id}"
}

resource "aws_security_group" "private" {
  name = "private"
  description = "Private security group"
}

resource "aws_security_group_rule" "private-from-bastion-ssh-ingress" {
    type = "ingress"
    from_port = 22
    to_port = 22
    protocol = "tcp"
    security_group_id = "${aws_security_group.private.id}"
    source_security_group_id = "${aws_security_group.bastion.id}"
}

Now, Terraform can see that the dependency chain says that both security groups must be created before either of those security group rules as both of them are dependent on the groups already having been created.