Terraform: All security group rules are destroyed and replaced when adding a single rule

Question

Terraform Info:

• Terraform v0.12.18
• provider.aws v2.43.0
• provider.template v2.1.2

I have a security group defined:

If I run terraform plan or terraform apply against my existing stack, there are NO changes pending, state is fully up-to-date.

resource "aws_security_group" "sg_apps" {
  name        = "Custom apps ${var.env}"
  description = "Custom apps ${var.env}"
  vpc_id      = data.terraform_remote_state.vpc.outputs.vpc_east_id
  tags = {
    Name = "Custom apps ${var.env} - TF"
  }

  ingress {
    from_port = 3306
    to_port   = 3306
    protocol  = "tcp"
    cidr_blocks = [
      data.terraform_remote_state.vpc.outputs.m-1a_cidr,
      data.terraform_remote_state.vpc.outputs.m-1b_cidr,
      data.terraform_remote_state.vpc.outputs.p-1b_cidr,
      data.terraform_remote_state.vpc.outputs.p-1a_cidr,
    ]
  }

  ingress {
    from_port = 8800
    to_port   = 8808
    protocol  = "tcp"
    cidr_blocks = [
      data.terraform_remote_state.vpc.outputs.m-1a_cidr,
      data.terraform_remote_state.vpc.outputs.m-1b_cidr,
      data.terraform_remote_state.vpc.outputs.p-1b_cidr,
      data.terraform_remote_state.vpc.outputs.p-1a_cidr,
    ]
  }

  ingress {
    from_port = 8503
    to_port   = 8503
    protocol  = "tcp"
    cidr_blocks = [
      data.terraform_remote_state.vpc.outputs.m-1a_cidr,
      data.terraform_remote_state.vpc.outputs.m-1b_cidr,
      data.terraform_remote_state.vpc.outputs.p-1b_cidr,
      data.terraform_remote_state.vpc.outputs.p-1a_cidr,
    ]
  }

  ingress {
    from_port = 8889
    to_port   = 8889
    protocol  = "tcp"
    cidr_blocks = [
      data.terraform_remote_state.vpc.outputs.m-1a_cidr,
      data.terraform_remote_state.vpc.outputs.m-1b_cidr,
      data.terraform_remote_state.vpc.outputs.p-1b_cidr,
      data.terraform_remote_state.vpc.outputs.p-1a_cidr,
    ]
  }
}

If I add just one more rule:

resource "aws_security_group" "sg_tlapps" {
  name        = "Custom apps ${var.env}"
  description = "Custom apps ${var.env}"
  vpc_id      = data.terraform_remote_state.vpc.outputs.vpc_east_id
  tags = {
    Name = "Custom apps ${var.env} - TF"
  }

  ingress {
    from_port = 3306
    to_port   = 3306
    protocol  = "tcp"
    cidr_blocks = [
      data.terraform_remote_state.vpc.outputs.m-1a_cidr,
      data.terraform_remote_state.vpc.outputs.m-1b_cidr,
      data.terraform_remote_state.vpc.outputs.p-1b_cidr,
      data.terraform_remote_state.vpc.outputs.p-1a_cidr,
    ]
  }

  ingress {
    from_port = 8800
    to_port   = 8808
    protocol  = "tcp"
    cidr_blocks = [
      data.terraform_remote_state.vpc.outputs.m-1a_cidr,
      data.terraform_remote_state.vpc.outputs.m-1b_cidr,
      data.terraform_remote_state.vpc.outputs.p-1b_cidr,
      data.terraform_remote_state.vpc.outputs.p-1a_cidr,
    ]
  }

  ingress {
    from_port = 8503
    to_port   = 8503
    protocol  = "tcp"
    cidr_blocks = [
      data.terraform_remote_state.vpc.outputs.m-1a_cidr,
      data.terraform_remote_state.vpc.outputs.m-1b_cidr,
      data.terraform_remote_state.vpc.outputs.p-1b_cidr,
      data.terraform_remote_state.vpc.outputs.p-1a_cidr,
    ]
  }

  ingress {
    from_port = 8889
    to_port   = 8889
    protocol  = "tcp"
    cidr_blocks = [
      data.terraform_remote_state.vpc.outputs.m-1a_cidr,
      data.terraform_remote_state.vpc.outputs.m-1b_cidr,
      data.terraform_remote_state.vpc.outputs.p-1b_cidr,
      data.terraform_remote_state.vpc.outputs.p-1a_cidr,
    ]
  }

  ingress {
    from_port = 9200
    to_port = 9200
    protocol = "tcp"
    cidr_blocks = [
      data.terraform_remote_state.vpc.outputs.m-1a_cidr,
      data.terraform_remote_state.vpc.outputs.m-1b_cidr,
      data.terraform_remote_state.vpc.outputs.p-1b_cidr,
      data.terraform_remote_state.vpc.outputs.p-1a_cidr,
    ]
  }
}

The apply wants to remove all the rules and recreate them -- I can't figure out why it wants to replace them all.

This is fine for non-production environments but I really don't want to go destroying my firewall rules in prod just in case the task fails halfway through or something. I'm just trying to append a rule to the sg.

  # aws_security_group.sg_apps will be updated in-place
  ~ resource "aws_security_group" "sg_apps" {
        arn                    = "arn:aws:ec2:us-east-1:xxxxx:security-group/sg-xxxxxxxx"
        description            = "Custom apps prod"
        egress                 = []
        id                     = "sg-xxxxxxxxx"
      ~ ingress                = [
          - {
              - cidr_blocks      = [
                  - "10.10.100.0/24",
                  - "10.10.200.0/24",
                  - "10.10.1.0/24",
                  - "10.10.0.0/24",
                ]
              - description      = ""
              - from_port        = 3306
              - ipv6_cidr_blocks = []
              - prefix_list_ids  = []
              - protocol         = "tcp"
              - security_groups  = []
              - self             = false
              - to_port          = 3306
            },
          - {
              - cidr_blocks      = [
                  - "10.10.100.0/24",
                  - "10.10.200.0/24",
                  - "10.10.1.0/24",
                  - "10.10.0.0/24",
                ]
              - description      = ""
              - from_port        = 8503
              - ipv6_cidr_blocks = []
              - prefix_list_ids  = []
              - protocol         = "tcp"
              - security_groups  = []
              - self             = false
              - to_port          = 8503
            },
          - {
              - cidr_blocks      = [
                  - "10.10.100.0/24",
                  - "10.10.200.0/24",
                  - "10.10.1.0/24",
                  - "10.10.0.0/24",
                ]
              - description      = ""
              - from_port        = 8800
              - ipv6_cidr_blocks = []
              - prefix_list_ids  = []
              - protocol         = "tcp"
              - security_groups  = []
              - self             = false
              - to_port          = 8808
            },
          - {
              - cidr_blocks      = [
                  - "10.10.100.0/24",
                  - "10.10.200.0/24",
                  - "10.10.1.0/24",
                  - "10.10.0.0/24",
                ]
              - description      = ""
              - from_port        = 8889
              - ipv6_cidr_blocks = []
              - prefix_list_ids  = []
              - protocol         = "tcp"
              - security_groups  = []
              - self             = false
              - to_port          = 8889
            },
          + {
              + cidr_blocks      = [
                  + "10.10.100.0/24",
                  + "10.10.200.0/24",
                  + "10.10.1.0/24",
                  + "10.10.0.0/24",
                ]
              + description      = ""
              + from_port        = 9200
              + ipv6_cidr_blocks = []
              + prefix_list_ids  = []
              + protocol         = "tcp"
              + security_groups  = []
              + self             = false
              + to_port          = 9200
            },
          + {
              + cidr_blocks      = [
                  + "10.10.100.0/24",
                  + "10.10.200.0/24",
                  + "10.10.1.0/24",
                  + "10.10.0.0/24",
                ]
              + description      = null
              + from_port        = 3306
              + ipv6_cidr_blocks = []
              + prefix_list_ids  = []
              + protocol         = "tcp"
              + security_groups  = []
              + self             = false
              + to_port          = 3306
            },
          + {
              + cidr_blocks      = [
                  + "10.10.100.0/24",
                  + "10.10.200.0/24",
                  + "10.10.1.0/24",
                  + "10.10.0.0/24",
                ]
              + description      = null
              + from_port        = 8503
              + ipv6_cidr_blocks = []
              + prefix_list_ids  = []
              + protocol         = "tcp"
              + security_groups  = []
              + self             = false
              + to_port          = 8503
            },
          + {
              + cidr_blocks      = [
                  + "10.10.100.0/24",
                  + "10.10.200.0/24",
                  + "10.10.1.0/24",
                  + "10.10.0.0/24",
                ]
              + description      = null
              + from_port        = 8800
              + ipv6_cidr_blocks = []
              + prefix_list_ids  = []
              + protocol         = "tcp"
              + security_groups  = []
              + self             = false
              + to_port          = 8808
            },
          + {
              + cidr_blocks      = [
                  + "10.10.100.0/24",
                  + "10.10.200.0/24",
                  + "10.10.1.0/24",
                  + "10.10.0.0/24",
                ]
              + description      = null
              + from_port        = 8889
              + ipv6_cidr_blocks = []
              + prefix_list_ids  = []
              + protocol         = "tcp"
              + security_groups  = []
              + self             = false
              + to_port          = 8889
            },
        ]
        name                   = "Custom apps prod"
        owner_id               = "xxxxxxxxxx"
        revoke_rules_on_delete = false
        tags                   = {
            "Name" = "Custom apps ${var.env} - TF"
        }
        vpc_id                 = "vpc-xxxxxxxxxxxxxxx"

        timeouts {}
    }

Answer 1

If you convert your ingress and egress blocks within your aws_security_group resource to individual aws_security_group_rule resources, then the rules will not be re-created when any one or more other rule is modified. For example:

resource "aws_security_group_rule" "mysql" {
  type              = "ingress"
  from_port         = 3306
  to_port           = 3306
  protocol          = "tcp"
  cidr_blocks       = [
    data.terraform_remote_state.vpc.outputs.m-1a_cidr,
    data.terraform_remote_state.vpc.outputs.m-1b_cidr,
    data.terraform_remote_state.vpc.outputs.p-1b_cidr,
    data.terraform_remote_state.vpc.outputs.p-1a_cidr,
  ]
  security_group_id = aws_security_group.sg_apps.id
}

would replace your first ingress block.