Terraform Error creating Lambda function: ResourceConflictException with the resource just created by Terraform apply

Question

I am deploying a Lambda function in each AWS Region of our account and encountering weird issue where the Apply is failing with the following error message for some of the AWS Regions

Error while Terraform Apply

Error: Error creating Lambda function: ResourceConflictException: Function already exist: log-forwarder
{
  RespMetadata: {
    StatusCode: 409,
    RequestID: "8cfd7260-7c4a-42d2-98c6-6619c7b2804f"
  },
  Message_: "Function already exist: log-forwarder",
  Type: "User"
}

The above Lambda function has just been created by same Terraform Apply that is failing.

The terraform plan and init doesn't throw any errors about having TF config issues.

Both plan and init runs successfully.

Below is my directory structure

.
├── log_forwarder.tf
├── log_forwarder_lambdas
│   └── main.tf
└── providers.tf

Below is my providers.tf file

provider "aws" {
  region  = "us-east-1"
  version = "3.9.0"
}
provider "aws" {
  alias   = "us-east-2"
  region  = "us-east-2"
  version = "3.9.0"
}
provider "aws" {
  alias   = "us-west-2"
  region  = "us-west-2"
  version = "3.9.0"
}  
provider "aws" {
  alias   = "us-west-1"
  region  = "us-west-1"
  version = "3.9.0"
}
provider "aws" {
  alias   = "ca-central-1"
  region  = "ca-central-1"
  version = "3.9.0"
}


... with all the AWS Regions.

Below is the tf config of log_forwarder.tf

terraform {
  required_version = "0.12.25"

  backend "s3" {
    All the backend Config
  }
}


resource "aws_iam_role" "log_forwarder" {
  name = "LogForwarder"

  assume_role_policy = <<EOF
{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "",
      "Effect": "Allow",
      "Principal": {
        "Service": ["lambda.amazonaws.com"]
      },
      "Action": "sts:AssumeRole"
    }
  ]
}
EOF
}

resource "aws_iam_role_policy" "log_forwarder" {
  name = "LogForwarder"
  role = aws_iam_role.log_forwarder.id

  policy = <<EOF
{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": [
        "lambda:ListTags",
        "logs:CreateLogGroup",
        "logs:CreateLogStream",
        "logs:PutLogEvents"
      ],
      "Resource": [
        "arn:aws:logs:*",
        "arn:aws:lambda:*"
      ]
    },
    {
      "Effect": "Allow",
      "Action": [
        "lambda:InvokeFunction"
      ],
      "Resource": "*"
    },
        {
    "Sid": "AWSDatadogPermissionsForCloudtrail",
    "Effect": "Allow",
    "Action": ["s3:ListBucket", "s3:GetBucketLocation", "s3:GetObject", "s3:ListObjects"],
    "Resource": [
        "arn:aws:s3:::BucketName",
        "arn:aws:s3:::BucketName/*"
    ]
}

  ]
}
EOF
}


module "DDLogForwarderUSEast1" {
  source                = "./log_forwarder_lambdas"
  dd_log_forwarder_role = aws_iam_role.log_forwarder.arn
  region                = "us-east-1"

}

module "DDLogForwarderUSEast2" {
  source                = "./log_forwarder_lambdas"
  dd_log_forwarder_role = aws_iam_role.log_forwarder.arn
  providers             = { aws = aws.us-east-2 }
  region                = "us-east-2"
}

module "DDLogForwarderUSWest1" {
  source                = "./log_forwarder_lambdas"
  dd_log_forwarder_role = aws_iam_role.log_forwarder.arn
  providers             = { aws = aws.us-west-1 }
  region                = "us-west-1"
}

module "DDLogForwarderUSWest2" {
  source                = "./log_forwarder_lambdas"
  dd_log_forwarder_role = aws_iam_role.log_forwarder.arn
  region                = "us-west-2"
  providers             = { aws = aws.us-west-2 }
}

module "DDLogForwarderAPEast1" {
  source                = "./log_forwarder_lambdas"
  dd_log_forwarder_role = aws_iam_role.log_forwarder.arn
  providers             = { aws = aws.ap-east-1 }
  region                = "ap-east-1"
}

module "DDLogForwarderAPSouth1" {
  source                = "./log_forwarder_lambdas"
  dd_log_forwarder_role = aws_iam_role.log_forwarder.arn
  region                = "ap-south-1"
  providers             = { aws = aws.ap-south-1 }
}

... All AWS Regions with different providers 

TF Config of log_forwarder_lambdas/main.tf

variable "region" {}

variable "account_id" {
  default = "AWS Account Id"
}

variable "dd_log_forwarder_role" {}

variable "exclude_at_match" {
  default = "([A-Z]* RequestId: .*)"
}

data "aws_s3_bucket" "cloudtrail_bucket" {
  count  = var.region == "us-west-2" ? 1 : 0
  bucket = "BucketName"
}


resource "aws_lambda_function" "log_forwarder" {
  filename      = "${path.cwd}/log_forwarder_lambdas/aws-dd-forwarder-3.16.3.zip"
  function_name = "log-forwarder"
  role          = var.dd_log_forwarder_role
  description   = "Gathers logs from targetted Cloudwatch Log Groups and sends them to DataDog"
  handler       = "lambda_function.lambda_handler"
  runtime       = "python3.7"
  timeout       = 600
  memory_size   = 1024
  layers        = ["arn:aws:lambda:${var.region}:464622532012:layer:Datadog-Python37:11"]


  environment {
    variables = {
      DD_ENHANCED_METRICS = false
      EXCLUDE_AT_MATCH    = var.exclude_at_match
    }
  }
}

resource "aws_cloudwatch_log_group" "log_forwarder" {
  name              = "/aws/lambda/${aws_lambda_function.log_forwarder.function_name}"
  retention_in_days = 90
}

resource "aws_lambda_permission" "cloudtrail_bucket" {
  count         = var.region == "us-west-2" ? 1 : 0
  statement_id  = "AllowExecutionFromS3Bucket"
  action        = "lambda:InvokeFunction"
  function_name = aws_lambda_function.log_forwarder.arn
  principal     = "s3.amazonaws.com"
  source_arn    = element(data.aws_s3_bucket.cloudtrail_bucket.*.arn, count.index)
}

resource "aws_s3_bucket_notification" "cloudtrail_bucket_notification" {
  count  = var.region == "us-west-2" ? 1 : 0
  bucket = element(data.aws_s3_bucket.cloudtrail_bucket.*.id, count.index)


  lambda_function {
    lambda_function_arn = aws_lambda_function.log_forwarder.arn
    events              = ["s3:ObjectCreated:*"]
  }

  depends_on = [aws_lambda_permission.cloudtrail_bucket, aws_cloudwatch_log_group.log_forwarder]
}

I am using TF 0.12.25 in this case.

The things I have tried so far.

1. Remove the .terraform folder from the root module every time I run the Terraform init/plan/apply cycle
2. I have tried to refactor code as much as possible.
3. I am running the TF Plan/Apply cycle locally without any CI.

Answer 1

At first glance it looks as though the Lambda function may not be in your Terraform state (for whatever reason). Have you changed backends / deleted data off your backend?

Run a terraform show and/or terraform state show and see if the conflicting Lambda function is in your state.
If it is not, but it already exists in AWS, you can import it.
See here: https://registry.terraform.io/providers/hashicorp/aws/latest/docs/resources/lambda_function#import

---

Update:

As per your coment, since the resource exists in AWS but not in the state, this is an expected error. (Terraform doesn't know the resource exists, therefore tries to create it; AWS knows it already exists, therefore returns an error.)

You have two choices:

• Delete the resource in AWS and run Terraform again; or
• Import the existing recource into Terraform (recomended).

Try something like:

terraform import module.DDLogForwarderUSEast1.aws_lambda_function.log-forwarder log-forwarder

(Make sure you have the correct provider/region set up if trying this for other regions!)