Resource handler returned message: "Error occurred during operation 'ECS Deployment Circuit Breaker was triggered'."

Question

I'm encountering this error when trying to create a service using Task Definition in my ECS Cluster:

Encountered error deploying NoteAppService. Resource handler message: "Operation 'ECS Deployment Circuit Breaker was triggered' encountered an error." (RequestToken: a9b4170a-b382-355d-0dc5-ae55cc25d314, HandlerErrorCode: GeneralServiceException)

Consequently, I'm unable to create the service in the cluster. I have tried multiple times but error still persists. Lacking experience, I'm unsure about the specific error and how to resolve it. The following is my Task Definition file:

{
  "taskDefinitionArn": "arn:aws:ecs:us-east-1:755204033406:task-definition/NoteappTaskDefin:1",
  "containerDefinitions": [
    {
      "name": "Container1",
      "image": "public.ecr.aws/d4a1q9y6/note_app_public_repo",
      "cpu": 0,
      "portMappings": [
        {
          "name": "container1-3000-tcp",
          "containerPort": 3000,
          "hostPort": 3000,
          "protocol": "tcp",
          "appProtocol": "http"
        }
      ],
      "essential": true,
      "environment": [],
      "environmentFiles": [],
      "mountPoints": [],
      "volumesFrom": [],
      "ulimits": [],
      "logConfiguration": {
        "logDriver": "awslogs",
        "options": {
          "awslogs-create-group": "true",
          "awslogs-group": "/ecs/NoteappTaskDefin",
          "awslogs-region": "us-east-1",
          "awslogs-stream-prefix": "ecs"
        },
        "secretOptions": []
      }
    }
  ],
  "family": "NoteappTaskDefin",
  "executionRoleArn": "arn:aws:iam::755204033406:role/ecsTaskExecutionRole",
  "networkMode": "awsvpc",
  "revision": 1,
  "volumes": [],
  "status": "ACTIVE",
  "requiresAttributes": [
    {
      "name": "com.amazonaws.ecs.capability.logging-driver.awslogs"
    },
    {
      "name": "ecs.capability.execution-role-awslogs"
    },
    {
      "name": "com.amazonaws.ecs.capability.docker-remote-api.1.19"
    },
    {
      "name": "com.amazonaws.ecs.capability.docker-remote-api.1.18"
    },
    {
      "name": "ecs.capability.task-eni"
    },
    {
      "name": "com.amazonaws.ecs.capability.docker-remote-api.1.29"
    }
  ],
  "placementConstraints": [],
  "compatibilities": [
    "EC2",
    "FARGATE"
  ],
  "requiresCompatibilities": [
    "EC2"
  ],
  "cpu": "1024",
  "memory": "3072",
  "runtimePlatform": {
    "cpuArchitecture": "X86_64",
    "operatingSystemFamily": "LINUX"
  },
  "registeredAt": "2023-08-22T09:37:03.162Z",
  "registeredBy": "arn:aws:iam::755204033406:root",
  "tags": []
}

Answer 1

Edit 2025-01-29

Advice 1:

Given that you're using Fargate, you probably configured variables and secrets in the service manifest file and don't have ENV variables in the Dockerfile itself. It's often worth copying Dockerfile to Dockerfile.local, add all ENV variables to it, and try to run the image locally to check if it succeeds, has any errors or warnings in the output, plus it's easier to debug using container logs straight in the Docker Desktop app. Only once you have zero issues locally, should you try to redeploy.

Advice 2:

Sometime rollback isn't as successful as it appears to be in the AWS CloudFormation > Stacks > {My Stack}, and it is helpful to

1. let whatever process is ongoing to finish
2. delete the problematic stack
3. again, wait for it to completely finish
4. redeploy (manually with copilot svc deploy to confirm it's working, then commit+push, if you have CodeBuild+CodePipeline setup to auto deploy)

---

Debugging

I got nowhere with that error message alone, but going to ECS > Clusters > {My Cluster} > Tasks and filtering them by status Stopped revealed the tasks with more meaningful error messages. Since there were several and in an odd order, I had to edit the view to add the column Stopped at and then sorted the list by this value, so I could always try to resolve the last error.

My case

In my case the error was with how I wanted to get the secrets from Secrets Manager to the (AWS Copilot CLI generated) service manifest.

Docs: https://aws.github.io/copilot-cli/docs/developing/secrets/

Example

AWS Secrets Manager

Secret name:

PaxstoreSDK

Secret value:

{
    "API_KEY": "...",
    "API_SECRET": "..."
}

Had to add tags for copilot-application and copilot-environment too.

/copilot/{service name}/manifest.yml

secrets:
  PAXSTORE_API_KEY:
    secretsmanager: 'PaxstoreSDK:API_KEY::'
  PAXSTORE_API_SECRET:
    secretsmanager: 'PaxstoreSDK:API_SECRET::'

Answer 2

This issue is commonly attributed to a failure in initiating the container. Please examine the logs for more insights.

Additionally, consider specifying the entryPoint and command parameters within the containerDefinitions section.

{
  "taskDefinitionArn": "arn:aws:ecs:us-east-1:755204033406:task-definition/NoteappTaskDefin:1",
  "containerDefinitions": [
    {
      "name": "Container1",
      "image": "public.ecr.aws/d4a1q9y6/note_app_public_repo",
      "cpu": 0,
      "portMappings": [
        {
          "name": "container1-3000-tcp",
          "containerPort": 3000,
          "hostPort": 3000,
          "protocol": "tcp",
          "appProtocol": "http"
        }
      ],
      "essential": true,
      "entryPoint": [
        "sh",
        "-c"
      ],
      "command": [
        "your command here"
      ]
    }
  ]
  
  […]
}