Logo Questions Linux Laravel Mysql Ubuntu Git Menu
HTML CSS JAVASCRIPT SQL PYTHON PHP BOOTSTRAP JAVA JQUERY R React Kotlin
 

AWS ECS - exit code 137

Tags:

docker

amazon-web-services

amazon-ecs

I am trying deploy my application with AWS ECS and docker. running the environment locally on my machine with docker-compose up works. however once converted and moved to ECS I am getting error 137

I have searched the exit code and it could be ram related so I launched a larger container with 8gb of ram, still get the same issue. (there is only two images running)

I've run the images manually by connecting to the container and running "docker run -it "imagename" /bin/bash and they run without issue.

Can anyone see the cause or how I can troubleshoot this?

This is my task:

{
  "requiresAttributes": [
    {
      "value": null,
      "name": "com.amazonaws.ecs.capability.docker-remote-api.1.17",
      "targetId": null,
      "targetType": null
    },
    {
      "value": null,
      "name": "com.amazonaws.ecs.capability.ecr-auth",
      "targetId": null,
      "targetType": null
    }
  ],
  "taskDefinitionArn": "arn:aws:ecs:eu-west-1:****:task-definition/it-app:1",
  "networkMode": "bridge",
  "status": "ACTIVE",
  "revision": 1,
  "taskRoleArn": null,
  "containerDefinitions": [
    {
      "volumesFrom": [],
      "memory": 300,
      "extraHosts": null,
      "dnsServers": null,
      "disableNetworking": null,
      "dnsSearchDomains": null,
      "portMappings": [],
      "hostname": null,
      "essential": true,
      "entryPoint": null,
      "mountPoints": [],
      "name": "it-app",
      "ulimits": null,
      "dockerSecurityOptions": null,
      "environment": [],
      "links": null,
      "workingDirectory": "/itapp/itapp",
      "readonlyRootFilesystem": null,
      "image": "****.dkr.ecr.eu-west-1.amazonaws.com/itapp",
      "command": [
        "bash",
        "-c",
        "python manage.py collectstatic --noinput && python manage.py makemigrations && python manage.py migrate && exec gunicorn itapp.wsgi -b 0.0.0.0:8000"
      ],
      "user": null,
      "dockerLabels": null,
      "logConfiguration": null,
      "cpu": 0,
      "privileged": null,
      "memoryReservation": null
    },
    {
      "volumesFrom": [],
      "memory": 300,
      "extraHosts": null,
      "dnsServers": null,
      "disableNetworking": null,
      "dnsSearchDomains": null,
      "portMappings": [
        {
          "hostPort": 80,
          "containerPort": 8000,
          "protocol": "tcp"
        }
      ],
      "hostname": null,
      "essential": true,
      "entryPoint": null,
      "mountPoints": [],
      "name": "itapp-ngnix",
      "ulimits": null,
      "dockerSecurityOptions": null,
      "environment": [],
      "links": [
        "it-app"
      ],
      "workingDirectory": null,
      "readonlyRootFilesystem": null,
      "image": "****.dkr.ecr.eu-west-1.amazonaws.com/itapp-ngnix:latest",
      "command": null,
      "user": null,
      "dockerLabels": null,
      "logConfiguration": null,
      "cpu": 0,
      "privileged": null,
      "memoryReservation": null
    }
  ],
  "placementConstraints": [],
  "volumes": [],
  "family": "it-app"
}

EDIT:

I think I found the issue, but am not sure why I'm getting it?

service itapp-web was unable to place a task because no container instance met all of its requirements. 
The closest matching container-instance 315a7d85-1e1a-4a66-9173-e1fc2c17fda2 is already using a port required by your task. 
For more information, see the Troubleshooting section.

here are the container logs:

2017-09-18T15:10:07Z [INFO] Waiting for any previous stops to complete module="TaskEngine" task="it-app:1 arn:aws:ecs:eu-west-1:****:task/2d1f357a-1350-40bb-801c-d3d552225d3c, Status: (NONE->RUNNING) Containers: [itapp-ngnix (NONE->RUNNING),it-app (NONE->RUNNING),]" seqnum="109"
2017-09-18T15:10:07Z [INFO] Wait over; ready to move towards status: RUNNING module="TaskEngine" task="it-app:1 arn:aws:ecs:eu-west-1:****:task/2d1f357a-1350-40bb-801c-d3d552225d3c, Status: (NONE->RUNNING) Containers: [itapp-ngnix (NONE->RUNNING),it-app (NONE->RUNNING),]"
2017-09-18T15:10:07Z [INFO] Pulling container it-app(****.dkr.ecr.eu-west-1.amazonaws.com/itapp) (NONE->RUNNING) concurrently. Task: it-app:1 arn:aws:ecs:eu-west-1:****:task/2d1f357a-1350-40bb-801c-d3d552225d3c, Status: (NONE->RUNNING) Containers: [itapp-ngnix (NONE->RUNNING),it-app (NONE->RUNNING),]
2017-09-18T15:10:08Z [INFO] Updating container reference it-app in Image State - sha256:a5a8ed72293792e2c1a51f2083d61049e9cda26b5d39a78d14087b0a2ffda675
2017-09-18T15:10:08Z [INFO] Saving state! module="statemanager"
2017-09-18T15:10:08Z [INFO] Finished pulling container ****.dkr.ecr.eu-west-1.amazonaws.com/itapp in 124.116556ms. Task: it-app:1 arn:aws:ecs:eu-west-1:****:task/2d1f357a-1350-40bb-801c-d3d552225d3c, Status: (NONE->RUNNING) Containers: [itapp-ngnix (NONE->RUNNING),it-app (PULLED->RUNNING),]
2017-09-18T15:10:08Z [INFO] Creating container module="TaskEngine" task="it-app:1 arn:aws:ecs:eu-west-1:****:task/2d1f357a-1350-40bb-801c-d3d552225d3c, Status: (NONE->RUNNING) Containers: [itapp-ngnix (NONE->RUNNING),it-app (PULLED->RUNNING),]" container="it-app(****.dkr.ecr.eu-west-1.amazonaws.com/itapp) (PULLED->RUNNING)"
2017-09-18T15:10:08Z [INFO] Created container name mapping for task it-app:1 arn:aws:ecs:eu-west-1:****:task/2d1f357a-1350-40bb-801c-d3d552225d3c, Status: (NONE->RUNNING) Containers: [itapp-ngnix (NONE->RUNNING),it-app (PULLED->RUNNING),] - it-app(****.dkr.ecr.eu-west-1.amazonaws.com/itapp) (PULLED->RUNNING) -> ecs-it-app-1-it-app-d280ddb2fed09c640000
2017-09-18T15:10:08Z [INFO] Saving state! module="statemanager"
2017-09-18T15:10:08Z [INFO] Created docker container for task it-app:1 arn:aws:ecs:eu-west-1:****:task/2d1f357a-1350-40bb-801c-d3d552225d3c, Status: (NONE->RUNNING) Containers: [itapp-ngnix (NONE->RUNNING),it-app (CREATED->RUNNING),]: it-app(****.dkr.ecr.eu-west-1.amazonaws.com/itapp) (CREATED->RUNNING) -> c2ce8f61094cd93865c4bce71e12d8264d45c91e8c8ad1436a614380541e7e64
2017-09-18T15:10:08Z [INFO] Starting container module="TaskEngine" task="it-app:1 arn:aws:ecs:eu-west-1:****:task/2d1f357a-1350-40bb-801c-d3d552225d3c, Status: (NONE->RUNNING) Containers: [itapp-ngnix (NONE->RUNNING),it-app (CREATED->RUNNING),]" container="it-app(****.dkr.ecr.eu-west-1.amazonaws.com/itapp) (CREATED->RUNNING)"
2017-09-18T15:10:08Z [INFO] Redundant container state change for task it-app:1 arn:aws:ecs:eu-west-1:****:task/2d1f357a-1350-40bb-801c-d3d552225d3c, Status: (NONE->RUNNING) Containers: [itapp-ngnix (NONE->RUNNING),it-app (CREATED->RUNNING),]: it-app(****.dkr.ecr.eu-west-1.amazonaws.com/itapp) (CREATED->RUNNING) to CREATED, but already CREATED
2017-09-18T15:10:08Z [INFO] TaskHandler, batching container event: arn:aws:ecs:eu-west-1:****:task/2d1f357a-1350-40bb-801c-d3d552225d3c it-app -> RUNNING, Known Sent: NONE
2017-09-18T15:10:08Z [INFO] Redundant container state change for task it-app:1 arn:aws:ecs:eu-west-1:****:task/2d1f357a-1350-40bb-801c-d3d552225d3c, Status: (NONE->RUNNING) Containers: [itapp-ngnix (NONE->RUNNING),it-app (RUNNING->RUNNING),]: it-app(****.dkr.ecr.eu-west-1.amazonaws.com/itapp) (RUNNING->RUNNING) to RUNNING, but already RUNNING
2017-09-18T15:10:08Z [INFO] Pulling container itapp-ngnix(****.dkr.ecr.eu-west-1.amazonaws.com/itapp-ngnix:latest) (NONE->RUNNING) concurrently. Task: it-app:1 arn:aws:ecs:eu-west-1:****:task/2d1f357a-1350-40bb-801c-d3d552225d3c, Status: (NONE->RUNNING) Containers: [itapp-ngnix (NONE->RUNNING),it-app (RUNNING->RUNNING),]
2017-09-18T15:10:08Z [INFO] Updating container reference itapp-ngnix in Image State - sha256:fb38044bb0d9b78d8da038b54d5afc1b3091b35d04524976b716500b0fcabb4f
2017-09-18T15:10:08Z [INFO] Saving state! module="statemanager"
2017-09-18T15:10:08Z [INFO] Finished pulling container ****.dkr.ecr.eu-west-1.amazonaws.com/itapp-ngnix:latest in 105.243653ms. Task: it-app:1 arn:aws:ecs:eu-west-1:****:task/2d1f357a-1350-40bb-801c-d3d552225d3c, Status: (NONE->RUNNING) Containers: [itapp-ngnix (PULLED->RUNNING),it-app (RUNNING->RUNNING),]
2017-09-18T15:10:07Z [INFO] Pulling container it-app(****.dkr.ecr.eu-west-1.amazonaws.com/itapp) (NONE->RUNNING) concurrently. Task: it-app:1 arn:aws:ecs:eu-west-1:****:task/2d1f357a-1350-40bb-801c-d3d552225d3c, Status: (NONE->RUNNING) Containers: [itapp-ngnix (NONE->RUNNING),it-app (NONE->RUNNING),]
2017-09-18T15:10:08Z [INFO] Updating container reference it-app in Image State - sha256:a5a8ed72293792e2c1a51f2083d61049e9cda26b5d39a78d14087b0a2ffda675
2017-09-18T15:10:08Z [INFO] Saving state! module="statemanager"
2017-09-18T15:10:08Z [INFO] Finished pulling container ****.dkr.ecr.eu-west-1.amazonaws.com/itapp in 124.116556ms. Task: it-app:1 arn:aws:ecs:eu-west-1:****:task/2d1f357a-1350-40bb-801c-d3d552225d3c, Status: (NONE->RUNNING) Containers: [itapp-ngnix (NONE->RUNNING),it-app (PULLED->RUNNING),]
2017-09-18T15:10:08Z [INFO] Creating container module="TaskEngine" task="it-app:1 arn:aws:ecs:eu-west-1:****:task/2d1f357a-1350-40bb-801c-d3d552225d3c, Status: (NONE->RUNNING) Containers: [itapp-ngnix (NONE->RUNNING),it-app (PULLED->RUNNING),]" container="it-app(****.dkr.ecr.eu-west-1.amazonaws.com/itapp) (PULLED->RUNNING)"
2017-09-18T15:10:08Z [INFO] Created container name mapping for task it-app:1 arn:aws:ecs:eu-west-1:****:task/2d1f357a-1350-40bb-801c-d3d552225d3c, Status: (NONE->RUNNING) Containers: [itapp-ngnix (NONE->RUNNING),it-app (PULLED->RUNNING),] - it-app(****.dkr.ecr.eu-west-1.amazonaws.com/itapp) (PULLED->RUNNING) -> ecs-it-app-1-it-app-d280ddb2fed09c640000
2017-09-18T15:10:08Z [INFO] Saving state! module="statemanager"
2017-09-18T15:10:08Z [INFO] Created docker container for task it-app:1 arn:aws:ecs:eu-west-1:****:task/2d1f357a-1350-40bb-801c-d3d552225d3c, Status: (NONE->RUNNING) Containers: [itapp-ngnix (NONE->RUNNING),it-app (CREATED->RUNNING),]: it-app(****.dkr.ecr.eu-west-1.amazonaws.com/itapp) (CREATED->RUNNING) -> c2ce8f61094cd93865c4bce71e12d8264d45c91e8c8ad1436a614380541e7e64
2017-09-18T15:10:08Z [INFO] Starting container module="TaskEngine" task="it-app:1 arn:aws:ecs:eu-west-1:****:task/2d1f357a-1350-40bb-801c-d3d552225d3c, Status: (NONE->RUNNING) Containers: [itapp-ngnix (NONE->RUNNING),it-app (CREATED->RUNNING),]" container="it-app(****.dkr.ecr.eu-west-1.amazonaws.com/itapp) (CREATED->RUNNING)"
2017-09-18T15:10:08Z [INFO] Redundant container state change for task it-app:1 arn:aws:ecs:eu-west-1:****:task/2d1f357a-1350-40bb-801c-d3d552225d3c, Status: (NONE->RUNNING) Containers: [itapp-ngnix (NONE->RUNNING),it-app (CREATED->RUNNING),]: it-app(****.dkr.ecr.eu-west-1.amazonaws.com/itapp) (CREATED->RUNNING) to CREATED, but already CREATED
2017-09-18T15:10:08Z [INFO] TaskHandler, batching container event: arn:aws:ecs:eu-west-1:****:task/2d1f357a-1350-40bb-801c-d3d552225d3c it-app -> RUNNING, Known Sent: NONE
2017-09-18T15:10:08Z [INFO] Redundant container state change for task it-app:1 arn:aws:ecs:eu-west-1:****:task/2d1f357a-1350-40bb-801c-d3d552225d3c, Status: (NONE->RUNNING) Containers: [itapp-ngnix (NONE->RUNNING),it-app (RUNNING->RUNNING),]: it-app(****.dkr.ecr.eu-west-1.amazonaws.com/itapp) (RUNNING->RUNNING) to RUNNING, but already RUNNING
2017-09-18T15:10:08Z [INFO] Pulling container itapp-ngnix(****.dkr.ecr.eu-west-1.amazonaws.com/itapp-ngnix:latest) (NONE->RUNNING) concurrently. Task: it-app:1 arn:aws:ecs:eu-west-1:****:task/2d1f357a-1350-40bb-801c-d3d552225d3c, Status: (NONE->RUNNING) Containers: [itapp-ngnix (NONE->RUNNING),it-app (RUNNING->RUNNING),]
2017-09-18T15:10:08Z [INFO] Updating container reference itapp-ngnix in Image State - sha256:fb38044bb0d9b78d8da038b54d5afc1b3091b35d04524976b716500b0fcabb4f
2017-09-18T15:10:08Z [INFO] Saving state! module="statemanager"
2017-09-18T15:10:08Z [INFO] Finished pulling container ****.dkr.ecr.eu-west-1.amazonaws.com/itapp-ngnix:latest in 105.243653ms. Task: it-app:1 arn:aws:ecs:eu-west-1:****:task/2d1f357a-1350-40bb-801c-d3d552225d3c, Status: (NONE->RUNNING) Containers: [itapp-ngnix (PULLED->RUNNING),it-app (RUNNING->RUNNING),]
2017-09-18T15:10:08Z [INFO] Creating container module="TaskEngine" task="it-app:1 arn:aws:ecs:eu-west-1:****:task/2d1f357a-1350-40bb-801c-d3d552225d3c, Status: (NONE->RUNNING) Containers: [itapp-ngnix (PULLED->RUNNING),it-app (RUNNING->RUNNING),]" container="itapp-ngnix(****.dkr.ecr.eu-west-1.amazonaws.com/itapp-ngnix:latest) (PULLED->RUNNING)"
2017-09-18T15:10:08Z [WARN] Warning, link with no linkalias module="api client" linkName="it-app" task="it-app:1 arn:aws:ecs:eu-west-1:****:task/2d1f357a-1350-40bb-801c-d3d552225d3c, Status: (NONE->RUNNING) Containers: [itapp-ngnix (PULLED->RUNNING),it-app (RUNNING->RUNNING),]" container="itapp-ngnix(****.dkr.ecr.eu-west-1.amazonaws.com/itapp-ngnix:latest) (PULLED->RUNNING)"
2017-09-18T15:10:08Z [INFO] Created container name mapping for task it-app:1 arn:aws:ecs:eu-west-1:****:task/2d1f357a-1350-40bb-801c-d3d552225d3c, Status: (NONE->RUNNING) Containers: [itapp-ngnix (PULLED->RUNNING),it-app (RUNNING->RUNNING),] - itapp-ngnix(****.dkr.ecr.eu-west-1.amazonaws.com/itapp-ngnix:latest) (PULLED->RUNNING) -> ecs-it-app-1-itapp-ngnix-c4e29c838bb19df6cb01
2017-09-18T15:10:08Z [INFO] Saving state! module="statemanager"
2017-09-18T15:10:08Z [INFO] Saving state! module="statemanager"
2017-09-18T15:10:09Z [INFO] Created docker container for task it-app:1 arn:aws:ecs:eu-west-1:****:task/2d1f357a-1350-40bb-801c-d3d552225d3c, Status: (CREATED->RUNNING) Containers: [itapp-ngnix (CREATED->RUNNING),it-app (RUNNING->RUNNING),]: itapp-ngnix(****.dkr.ecr.eu-west-1.amazonaws.com/itapp-ngnix:latest) (CREATED->RUNNING) -> d06b6e4300665f0f8c18f0be04345e4797827736574d977b2a81cfb5d7eb471a
2017-09-18T15:10:09Z [INFO] Starting container module="TaskEngine" task="it-app:1 arn:aws:ecs:eu-west-1:****:task/2d1f357a-1350-40bb-801c-d3d552225d3c, Status: (CREATED->RUNNING) Containers: [itapp-ngnix (CREATED->RUNNING),it-app (RUNNING->RUNNING),]" container="itapp-ngnix(****.dkr.ecr.eu-west-1.amazonaws.com/itapp-ngnix:latest) (CREATED->RUNNING)"
2017-09-18T15:10:09Z [INFO] Task change event module="TaskEngine" event="{TaskArn:arn:aws:ecs:eu-west-1:****:task/2d1f357a-1350-40bb-801c-d3d552225d3c Status:RUNNING Reason: Containers:[] Task:it-app:1 arn:aws:ecs:eu-west-1:****:task/2d1f357a-1350-40bb-801c-d3d552225d3c, Status: (RUNNING->RUNNING) Containers: [itapp-ngnix (RUNNING->RUNNING),it-app (RUNNING->RUNNING),]}"
2017-09-18T15:10:09Z [INFO] TaskHandler, batching container event: arn:aws:ecs:eu-west-1:****:task/2d1f357a-1350-40bb-801c-d3d552225d3c itapp-ngnix -> RUNNING, Ports [{8000 80 0.0.0.0 0}], Known Sent: NONE
2017-09-18T15:10:09Z [INFO] TaskHandler, Adding event: TaskChange: arn:aws:ecs:eu-west-1:****:task/2d1f357a-1350-40bb-801c-d3d552225d3c -> RUNNING, Known Sent: NONE
2017-09-18T15:10:09Z [INFO] TaskHandler, Sending task change: TaskChange: arn:aws:ecs:eu-west-1:****:task/2d1f357a-1350-40bb-801c-d3d552225d3c -> RUNNING, Known Sent: NONE
2017-09-18T15:10:09Z [INFO] Redundant container state change for task it-app:1 arn:aws:ecs:eu-west-1:****:task/2d1f357a-1350-40bb-801c-d3d552225d3c, Status: (RUNNING->RUNNING) Containers: [itapp-ngnix (RUNNING->RUNNING),it-app (RUNNING->RUNNING),]: itapp-ngnix(****.dkr.ecr.eu-west-1.amazonaws.com/itapp-ngnix:latest) (RUNNING->RUNNING) to RUNNING, but already RUNNING
2017-09-18T15:10:09Z [INFO] Stopping container module="TaskEngine" task="it-app:1 arn:aws:ecs:eu-west-1:****:task/2d1f357a-1350-40bb-801c-d3d552225d3c, Status: (RUNNING->STOPPED) Containers: [itapp-ngnix (STOPPED->STOPPED),it-app (RUNNING->STOPPED),]" container="it-app(****.dkr.ecr.eu-west-1.amazonaws.com/itapp) (RUNNING->STOPPED)"
2017-09-18T15:10:09Z [INFO] Error retrieving stats for container d06b6e4300665f0f8c18f0be04345e4797827736574d977b2a81cfb5d7eb471a: context canceled
2017-09-18T15:10:09Z [INFO] Container d06b6e4300665f0f8c18f0be04345e4797827736574d977b2a81cfb5d7eb471a is terminal, stopping stats collection
2017-09-18T15:10:09Z [INFO] TaskHandler, batching container event: arn:aws:ecs:eu-west-1:****:task/2d1f357a-1350-40bb-801c-d3d552225d3c itapp-ngnix -> STOPPED, Exit 1, , Known Sent: NONE
2017-09-18T15:10:18Z [INFO] Saving state! module="statemanager"
2017-09-18T15:10:40Z [WARN] Error converting stats for container c2ce8f61094cd93865c4bce71e12d8264d45c91e8c8ad1436a614380541e7e64: Invalid container statistics reported
2017-09-18T15:10:40Z [INFO] Task change event module="TaskEngine" event="{TaskArn:arn:aws:ecs:eu-west-1:****:task/2d1f357a-1350-40bb-801c-d3d552225d3c Status:STOPPED Reason: Containers:[] Task:it-app:1 arn:aws:ecs:eu-west-1:****:task/2d1f357a-1350-40bb-801c-d3d552225d3c, Status: (STOPPED->STOPPED) Containers: [itapp-ngnix (STOPPED->STOPPED),it-app (STOPPED->STOPPED),]}"
2017-09-18T15:10:40Z [INFO] Error retrieving stats for container c2ce8f61094cd93865c4bce71e12d8264d45c91e8c8ad1436a614380541e7e64: context canceled
2017-09-18T15:10:40Z [INFO] TaskHandler, batching container event: arn:aws:ecs:eu-west-1:****:task/2d1f357a-1350-40bb-801c-d3d552225d3c it-app -> STOPPED, Exit 137, , Known Sent: NONE
2017-09-18T15:10:40Z [INFO] TaskHandler, Adding event: TaskChange: arn:aws:ecs:eu-west-1:****:task/2d1f357a-1350-40bb-801c-d3d552225d3c -> STOPPED, Known Sent: RUNNING
2017-09-18T15:10:40Z [INFO] TaskHandler, Sending task change: TaskChange: arn:aws:ecs:eu-west-1:****:task/2d1f357a-1350-40bb-801c-d3d552225d3c -> STOPPED, Known Sent: RUNNING
2017-09-18T15:10:40Z [INFO] Container c2ce8f61094cd93865c4bce71e12d8264d45c91e8c8ad1436a614380541e7e64 is terminal, stopping stats collection
2017-09-18T15:10:40Z [INFO] Redundant container state change for task it-app:1 arn:aws:ecs:eu-west-1:****:task/2d1f357a-1350-40bb-801c-d3d552225d3c, Status: (STOPPED->STOPPED) Containers: [itapp-ngnix (STOPPED->STOPPED),it-app (STOPPED->STOPPED),]: it-app(****.dkr.ecr.eu-west-1.amazonaws.com/itapp) (STOPPED->STOPPED) - Exit: 137 to STOPPED, but already STOPPED
2017-09-18T15:10:40Z [INFO] Saving state! module="statemanager"
2017-09-18T15:10:40Z [INFO] Redundant container state change for task it-app:1 arn:aws:ecs:eu-west-1:****:task/2d1f357a-1350-40bb-801c-d3d552225d3c, Status: (STOPPED->STOPPED) Containers: [itapp-ngnix (STOPPED->STOPPED),it-app (STOPPED->STOPPED),]: it-app(****.dkr.ecr.eu-west-1.amazonaws.com/itapp) (STOPPED->STOPPED) - Exit: 137 to STOPPED, but already STOPPED
2017-09-18T15:10:50Z [INFO] Saving state! module="statemanager"
like image 452
AlexW Avatar asked Sep 18 '17 15:09

AlexW

People also ask

What is exit code 137?

Exit code 137 is triggered when a pod or a container within your Kubernetes environment exceeds the amount of memory that they're assigned. Typically, this exit code is accompanied by or simply known as OOMKilled. Killed in this description refers to the result of this error, which causes a pod to terminate.

Why is ECS task stopping?

If you have trouble starting a task, your task might be stopping because of an error. For example, you run the task and the task displays a PENDING status and then disappears. You can view stopped task errors like this in the Amazon ECS console by viewing the stopped task and inspecting it for error messages.

1 Answers

In Linux, there are a number of exit codes with Special Meanings, of note here is the 128+n section, which are the Kill levels for a process. in this case, 137 = 128 + 9, so this process was killed with the highest level.

This usually happens in ECS when ECS sends a STOP to the process, but it hasn't exited within 30 seconds.

Two common causes for stuff like this that I'd check:

  1. In your Container Definition, you have "memory": 300. This is a hard limit on the amount of memory that a specific Docker task can use. If it reaches that limit, it gets terminated (I don't know what exit code that would be, but 137 might be reasonable). Based on your example, you could try to run something similar with docker run -m 300M -it "imagename" /bin/bash. If this hits the memory limit, it should also get terminated.
  2. If your container is connected to a Load Balancer and has a health check enabled, ensure that your application is responding to the health check within the health check interval. If not, it will count against the container, and after the configured number of health check fails, the container will be considered unhealthy, and will create a new task and destroy the old one.

One thing to note here as that ECS supports memory and memoryReservation in the task definition, and only one of them need be set. memory as mentioned above is a hard limit, and anything that hits that gets destroyed. memoryReservation is a soft limit, and your container can exceed that, up to the total memory of the ECS Instance. You can also combine the two, in which case memoryReservation is used in determining the amount of memory available for extra tasks on an instance, and memory is used to kill the task if it over runs it overruns the number provided.

like image 116
Jamie Starke Avatar answered Sep 28 '22 09:09

Jamie Starke
Sign in to Comment

Related questions

AWS SageMaker on GPU
How to use the AWS Python SDK while connecting via SSO credentials
Can not find table using NoSQL Workbench for DynamoDB when connecting to DynamoDB Docker
Difference between AWS Elastic Load Balancing and Auto Scaling
Ubuntu AMI not loading user-data
Processing SQS queues with boto
Why using a common hash key with AWS DynamoDB is a bad thing?
Amazon S3 Folder Level Permissions
Hex string to integer conversion in Amazon Redshift
AWS S3 - Listing all objects inside a folder without the prefix
CloudFormation: Template error: every Ref object must have a single String value
Avoid throttle dynamoDB
How to check when is the last time S3 bucket has been updated?
Aws s3 reduced redundancy costs the same as standard?
AWS CloudFormation w/ AWS::ApiGateway::RestApi gives Invalid REST API identifier specified
missing region; use :region option or export region name to ENV['AWS_REGION']
Stopping task on AWS ECS via CLI (program output as argument input bash)
What is the maximum number of event source mappings per AWS Lambda?
How to create a nested Resource path in AWS RestAPI using Cloudformation?
Aws lambda not connecting with Dynamo Db

Donate For Us

If you love us? You can donate to us via Paypal or buy me a coffee so we can maintain and grow! Thank you!