Terraform: how to not duplicate security groups when creating it using modules?

Question

Update

I don't get it, but I reran terraform apply and it did not try to duplicate resources (no errors). Now it checks resources correctly. Kind of unexpected end of events.

---

I'm learning Terraform, and I created module to allow creating some basic security groups. It runs fine first time and creates resources as expected. But if I run terraform apply second time, it tries to create same groups again and then I get duplicate error, because such security groups already exist.

If I would create security groups directly without module, Terraform recognizes it and does not try to recreate existing resources.

I probably do something wrong here.

Here is my module and how I try to use it:

My project structure looks like this

├── main.tf
├── modules
│   ├── security_group_ec2
│   │   ├── main.tf
│   │   ├── outputs.tf
│   │   └── variables.tf
│   └── security_group_rds
│       ├── main.tf
│       ├── outputs.tf
│       └── variables.tf
├── scripts
│   └── update-odoo-cfg.py
├── security_groups.tf
├── terraform.tfstate
├── terraform.tfstate.backup
├── variables.tf
└── vpc.tf

Now my security_group_ec2 content:

main.tf:

resource "aws_security_group" "sg" {
  name = "${var.name}"
  description = "${var.description}"
  vpc_id = "${var.vpc_id}"

  ingress {
    from_port   = 80
    to_port     = 80
    protocol    = "tcp"
    cidr_blocks = ["0.0.0.0/0"]
  }

  ingress {
    from_port   = 443
    to_port     = 443
    protocol    = "tcp"
    cidr_blocks = ["0.0.0.0/0"]
  }

  ingress {
    from_port   = 22
    to_port     = 22
    protocol    = "tcp"
    cidr_blocks = ["0.0.0.0/0"]
  }

  egress {
    from_port       = 0
    to_port         = 0
    protocol        = "-1"
    cidr_blocks     = ["0.0.0.0/0"]
  }
}

variables.tf:

variable "name" {
  description = "Name of security group"
}

variable "description" {
  description = "Description of security group"
}
variable "vpc_id" {
  description = "Virtual Private Cloud ID to assign"
}

outputs:

output "sg_id" {
  value = "${aws_security_group.sg.id}"
}

And this is file were I call module to create two security groups.

security_groups.tf:

# EC2
module "security_group_staging_ec2" {
  source = "modules/security_group_ec2"
  name = "ec2_staging_sg"
  description = "EC2 Staging Security Group"
  vpc_id = "${aws_default_vpc.default.id}"
}

module "security_group_prod_ec2" {
  source = "modules/security_group_ec2"
  name = "ec2_prod_sg"
  description = "EC2 Production Security Group"
  vpc_id = "${aws_default_vpc.default.id}"
}

This is error output when running terraform apply:

module.security_group_staging.aws_security_group.sg: Destruction complete after 1s
module.security_group_prod.aws_security_group.sg: Destruction complete after 1s

Error: Error applying plan:

2 error(s) occurred:

* module.security_group_staging_ec2.aws_security_group.sg: 1 error(s) occurred:

* aws_security_group.sg: Error creating Security Group: InvalidGroup.Duplicate: The security group 'ec2_staging_sg' already exists for VPC 'vpc-2a84a741'
    status code: 400, request id: 835004f0-d8a1-4ed5-8e21-17f01eb18a23
* module.security_group_prod_ec2.aws_security_group.sg: 1 error(s) occurred:

* aws_security_group.sg: Error creating Security Group: InvalidGroup.Duplicate: The security group 'ec2_prod_sg' already exists for VPC 'vpc-2a84a741'
    status code: 400, request id: 953b23e8-20cb-4ccb-940a-6a9ddab54d53

Terraform does not automatically rollback in the face of errors.
Instead, your Terraform state file has been partially updated with
any resources that successfully completed. Please address the error
above and apply again to incrementally change your infrastructure.

P.S. I probably need to somehow indicate that resource is created when calling module?

Answer 1

This looks like a race condition. Terraform tries to parallelise the creation of resources which do not depend on each other, and in this case it looks like it tried to destroy the security groups from module.security_group_staging while simultaneously trying to create them in module.security_group_staging_ec2 with the same names. Did you rename security_group_staging into security_group_staging_ec2?

The destruction succeeded, but the creation failed, because it ran in parallel with destruction.

The second time you ran it there was no race condition, because module.security_group_staging was already destroyed.

As a side note, it is usually a good idea to not keep separate environments in the same state file.