ConflictException: Stage already exist from aws_api_gateway_deployment with stage_name

Question

Problem

When first create an API Gateway deployment with the stage name, and also create a stage to configure X-RAY or CloudWatch logging, it will cause the "Stage already exist".

resource "aws_api_gateway_deployment" "this" {
  rest_api_id = aws_api_gateway_rest_api.mysfit.id
  stage_name  = "${var.ENV}"
  variables = {
    deployed_at = timestamp()
  }
  lifecycle {
    create_before_destroy = true
  }
}

resource "aws_api_gateway_stage" "this" {
  stage_name    = var.ENV
  rest_api_id   = aws_api_gateway_rest_api.mysfit.id
  deployment_id = aws_api_gateway_deployment.this.id

  dynamic "access_log_settings" {
    for_each = var.enable_apigw_stage_cloudwatch_access_log ? [1] : []
    content {
      destination_arn = module.cloudwatch.cloudwatch_loggroup_arn
      format          = file("${path.module}/apigw_access_log_format.json")
    }
  }

  xray_tracing_enabled = var.xray_tracing_enabled

  tags = {
    Project     = var.PROJECT
    Environment = var.ENV
  }
}

The workaround is to omit the stage_name in aws_api_gateway_deployment as stage is optional for the API Gateway deployment. However, the the invoke_url of the deployment does not have the stage part in the URL path.

Is this a Terraform specific issue or an API Gateway issue?

Reference

• Error creating API Gateway Stage: ConflictException: Stage already exists #2918

Answer 1

In my understanding, there is a logical defect in the API Gateway deployment design.

API deployment is to deploy to a stage as in Deploying a REST API in Amazon API Gateway.

To deploy an API, you create an API deployment and associate it with a stage. A stage is a logical reference to a lifecycle state of your API (for example, dev, prod, beta, v2).

However, create-stage API requires an existing deployment. This circular dependency between deployment and stage is, I believe, the source of the problems.

Problem 1

The create-stage API requires a deployment with --deployment-id argument. Hence we need to create an API deployment first.

Here is the first problem. If we specifies a stage for create-deployment, it creates the stage. Then we cannot create the stage by ourselves.

When we use a configuration management tool e.g. CloudFormation or Terraform, this causes "Stage already exists" exception as we will try to create the stage resource ourselves.

• Can't create a STAGE / DEPLOYMENT for API Gateway, circular reference error
• API Gateway Cloudformation Support Not Working As Expected
• Error creating API Gateway Stage: ConflictException: Stage already exists #1153
• Error creating API Gateway Stage: ConflictException: Stage already exists #2918

Hence we cannot specify a stage when we first create a API deployment.

Problem 2

For us to manage the creation of the stage resource due to the problem 1, we need to first create a dummy deployment so that we can create a stage using the dummy. This step of wasteful deployment creation is the 2nd problem. Although the stage points to the deployment, the deployment does not fully recognize the stage because if try to get the invoke URL from the deployment, it does not include the stage.

Once the stage is created, then finally we can create another API deployment which specifies the stage. As the stage already exists, the deployment will refer to the stage and the invoke URL will include the stage.

Example

Terraform

#--------------------------------------------------------------------------------
# Dummy API Deployment
#--------------------------------------------------------------------------------
resource "aws_api_gateway_deployment" "dummy" {
  rest_api_id = "${aws_api_gateway_rest_api.this.id}"

  #--------------------------------------------------------------------------------
  # To avoid State already exists
  # https://github.com/terraform-providers/terraform-provider-aws/issues/2918
  #--------------------------------------------------------------------------------
  #stage_name  = "${var.ENV}"

  #--------------------------------------------------------------------------------
  # Force re-deployment at each run. Alternative is to verify MD5 of API GW files.
  #--------------------------------------------------------------------------------
  # https://medium.com/coryodaniel/til-forcing-terraform-to-deploy-a-aws-api-gateway-deployment-ed36a9f60c1a
  # https://github.com/hashicorp/terraform/issues/6613
  # Terraform’s aws_api_gateway_deployment won’t deploy subsequent releases in the event
  # that something has changed in an integration, method, etc
  #--------------------------------------------------------------------------------
  stage_description = "Deployment at ${timestamp()}"

  lifecycle {
    create_before_destroy = true
  }

  depends_on = [
    #--------------------------------------------------------------------------------
    # [aws_api_gateway_account.this]
    # To avoid the error: Updating API Gateway Stage failed:
    # BadRequestException: CloudWatch Logs role ARN must be set in account settings to enable logging.
    #--------------------------------------------------------------------------------
    "aws_api_gateway_account.this",

    #--------------------------------------------------------------------------------
    # To avoid NotFoundException: Invalid Integration identifier specified
    #--------------------------------------------------------------------------------
    "aws_api_gateway_integration.ping_put",
  ]
  #--------------------------------------------------------------------------------
}

#--------------------------------------------------------------------------------
# Create a stage refering to the dummy.
# The 2nd/true deployment will later refer to this stage
#--------------------------------------------------------------------------------
resource "aws_api_gateway_stage" "this" {
  stage_name    = var.ENV
  rest_api_id   = aws_api_gateway_rest_api.this.id
  deployment_id = aws_api_gateway_deployment.dummy.id

  xray_tracing_enabled = var.apigw_xray_tracing_enabled

  tags = {
    Project     = var.PROJECT
    Environment = var.ENV
  }

  depends_on = [
    aws_api_gateway_deployment.dummy
  ]
}

#--------------------------------------------------------------------------------
# Legitimate API Deployment
#--------------------------------------------------------------------------------
resource "aws_api_gateway_deployment" "this" {
  rest_api_id = aws_api_gateway_rest_api.this.id
  stage_name  = aws_api_gateway_stage.this.stage_name

  lifecycle {
    create_before_destroy = true
  }
}

CloudFormation

AWSTemplateFormatVersion: "2010-09-09"
Description: "My API Gateway and Lambda function"

Parameters:
  apiGatewayStageName:
    Type: "String"
    Default: "devStage"

  lambdaFunctionName:
    Type: "String"
    Default: "my-lambda-function"

Resources:
  apiGateway:
    Type: "AWS::ApiGateway::RestApi"
    Properties:
      Name: "test-api"
      Description: "My Test API"

  apiGatewayRootMethod:
    Type: "AWS::ApiGateway::Method"
    Properties:
      AuthorizationType: "NONE"
      HttpMethod: "GET"
      Integration:
        IntegrationHttpMethod: "POST"
        Type: "AWS_PROXY"
        Uri: !Sub
          - "arn:aws:apigateway:${AWS::Region}:lambda:path/2015-03-31/functions/${lambdaArn}/invocations"
          - lambdaArn: !GetAtt "lambdaFunction.Arn"
      ResourceId: !GetAtt "apiGateway.RootResourceId"
      RestApiId: !Ref "apiGateway"

  apiGatewayDeployment:
    Type: "AWS::ApiGateway::Deployment"
    DependsOn:
      - "apiGatewayRootMethod"
    Properties:
      RestApiId: !Ref "apiGateway"
      StageName: ""

  apiGatewayStage:
    Type: "AWS::ApiGateway::Stage"
    Properties:
      StageName: !Ref "apiGatewayStageName"
      RestApiId: !Ref "apiGateway"
      DeploymentId: !Ref "apiGatewayDeployment"
      MethodSettings:
        - ResourcePath: /
          HttpMethod: "GET"
          MetricsEnabled: 'true'
          DataTraceEnabled: 'true'

  lambdaFunction:
    Type: "AWS::Lambda::Function"
    Properties:
      Code:
        ZipFile: |
          def handler(event,context):
            return {
              'body': 'Hello World from Lambda',
              'headers': {
                'Content-Type': 'text/plain'
              },
              'statusCode': 200
            }
      Description: "My function"
      FunctionName: !Ref "lambdaFunctionName"
      Handler: "index.handler"
      MemorySize: 256
      Role: !GetAtt "lambdaIAMRole.Arn"
      Runtime: "python3.7"
      Timeout: 30

  lambdaApiGatewayInvoke:
    Type: "AWS::Lambda::Permission"
    Properties:
      Action: "lambda:InvokeFunction"
      FunctionName: !GetAtt "lambdaFunction.Arn"
      Principal: "apigateway.amazonaws.com"
      SourceArn: !Sub "arn:aws:execute-api:${AWS::Region}:${AWS::AccountId}:${apiGateway}/*/GET/"

  lambdaIAMRole:
    Type: "AWS::IAM::Role"
    Properties:
      AssumeRolePolicyDocument:
        Version: "2012-10-17"
        Statement:
          - Action:
              - "sts:AssumeRole"
            Effect: "Allow"
            Principal:
              Service:
                - "lambda.amazonaws.com"
      Policies:
        - PolicyDocument:
            Version: "2012-10-17"
            Statement:
              - Action:
                  - "logs:CreateLogGroup"
                  - "logs:CreateLogStream"
                  - "logs:PutLogEvents"
                Effect: "Allow"
                Resource:
                  - !Sub "arn:aws:logs:${AWS::Region}:${AWS::AccountId}:log-group:/aws/lambda/${lambdaFunctionName}:*"
          PolicyName: "lambda"

  lambdaLogGroup:
    Type: "AWS::Logs::LogGroup"
    Properties:
      LogGroupName: !Sub "/aws/lambda/${lambdaFunctionName}"
      RetentionInDays: 90

Outputs:
  apiGatewayInvokeURL:
    Value: !Sub "https://${apiGateway}.execute-api.${AWS::Region}.amazonaws.com/${apiGatewayStageName}"

  lambdaArn:
    Value: !GetAtt "lambdaFunction.Arn"

Answer 2

To quote the documentation

The AWS::ApiGateway::Deployment resource deploys an API Gateway RestApi resource to a stage

This means that if you're creating an AWS::ApiGateway::Stage resource with the same stage name that you pass into AWS::ApiGateway::Deployment, then the Cloudformation script will try to create 2 stages with the same name - thus prompting the error.

Therefore, the solution is to not pass in a Stage name into the Deployment resource. This will work:

Parameters:
  ENVIRONMENT:
    Type: String

Resources:
  ContactsStage:
    Type: AWS::ApiGateway::Stage
    Properties:
      StageName: !Ref ENVIRONMENT
      RestApiId: !Ref MyApiGateway
      DeploymentId: !Ref MyDeployment
      MethodSettings:
        - ResourcePath: /
          HttpMethod: POST
          MetricsEnabled: 'true'
          DataTraceEnabled: 'false'

  MyDeployment:
    Type: AWS::ApiGateway::Deployment
    DependsOn: MyMethod
    Properties:
      RestApiId: !Ref MyApiGateway
#      StageName: !Ref ENVIRONMENT      # uncommenting this will cause the error