Unable to start Elastic Beanstalk MultiContainer instance

Question

My Dockerrun.aws.json

{
  "AWSEBDockerrunVersion": 2,
  "Authintication": {
    "Bucket": "abc",
    "key": "config.json"
  },
  "containerDefinitions": [
    {
      "name": "data",
      "image": "id.dkr.ecr.eu-west-2.amazonaws.com/dbdata:latest",
      "essential": false,
      "memory": 512,
      "command" : [
       "/bin/bash"
      ]
    },
    {
      "name": "codebase",
      "image": "id.dkr.ecr.eu-west-2.amazonaws.com/codebase:latest",
      "essential": false,
      "memory": 512,
      "command" : [
       "/bin/bash"
      ]
    },
    {
      "name": "postgres",
      "image": "id.dkr.ecr.eu-west-2.amazonaws.com/postgres:latest",
      "essential": true,
      "memory": 1024,
      "volumesFrom": [
        {
          "sourceContainer": "data"
        }
      ],
      "portMappings": [
        {
          "hostPort": 5432,
          "containerPort": 5432
        }
      ],
      "links": [
        "data"
      ]
    },
    {
      "name": "boxer-api",
      "image": "id.dkr.ecr.eu-west-2.amazonaws.com/boxer-api:latest",
      "essential": true,
      "memory": 1024,
      "volumesFrom": [
        {
          "sourceContainer": "codebase"
        }
      ],
      "portMappings": [
        {
          "hostPort": 8080,
          "containerPort": 8080
        }
      ],
      "links": [
        "codebase",
        "postgres",
        "data"
      ]
    }
  ]
}

Error eb-activity.log

[2017-04-05T08:29:00.728Z] INFO [3738] - [Application deployment First

Release@1/StartupStage1/AppDeployEnactHook/02update-credentials.sh] :

Starting activity... [2017-04-05T08:29:00.913Z] INFO [3738] -

[Application deployment First

Release@1/StartupStage1/AppDeployEnactHook/02update-credentials.sh] :

Completed activity. [2017-04-05T08:29:00.914Z] INFO [3738] - [Application deployment First Release@1/StartupStage1/AppDeployEnactHook/03start-task.sh] : Starting activity... [2017-04-05T08:30:37.520Z] INFO [3738] - [Application deployment First Release@1/StartupStage1/AppDeployEnactHook/03start-task.sh] : Activity execution failed, because: + . /opt/elasticbeanstalk/hooks/common.sh
+ RETRY=0 + initctl start eb-ecs EB_EVENT_FILE=/tmp/eventsfile20170405-3738-17c2sc0 initctl: Job

failed to start + '[' -f /etc/elasticbeanstalk/.eb-ecs-start-no-retry ']' + warn 'Failed to start ECS task, retrying...' + echo 'Failed to start ECS task, retrying...' Failed to start ECS task, retrying... + eventHelper.py --msg 'Failed to start ECS task, retrying...' --severity WARN + RETRY=1 + sleep 3 + '[' 1 -gt 1 ']' + initctl start eb-ecs

EB_EVENT_FILE=/tmp/eventsfile20170405-3738-17c2sc0 initctl: Job failed to start + '[' -f /etc/elasticbeanstalk/.eb-ecs-start-no-retry ']' + warn 'Failed to start ECS task, retrying...' + echo 'Failed to start ECS task, retrying...' Failed to start ECS task, retrying... + eventHelper.py --msg 'Failed to start ECS task, retrying...' --severity WARN + RETRY=2 + sleep 3 + '[' 2 -gt 1 ']' + error_exit 'Failed to start ECS task after retrying 2 times.' 1 + error 'Failed to start ECS task after retrying 2 times.' + echo 'Failed to start ECS task after retrying 2 times.' Failed to start

ECS task after retrying 2 times. + eventHelper.py --msg 'Failed to start ECS task after retrying 2 times.' --severity ERROR + exit 1 (ElasticBeanstalk::ExternalInvocationError) caused by: + . /opt/elasticbeanstalk/hooks/common.sh + RETRY=0 + initctl start eb-ecs EB_EVENT_FILE=/tmp/eventsfile20170405-3738-17c2sc0 initctl: Job failed to start + '[' -f /etc/elasticbeanstalk/.eb-ecs-start-no-retry ']' + warn 'Failed to start ECS task, retrying...' + echo 'Failed to start ECS task, retrying...' Failed to start ECS task, retrying... + eventHelper.py --msg 'Failed to start ECS task, retrying...' --severity WARN + RETRY=1 + sleep 3 + '[' 1 -gt 1 ']' + initctl start eb-ecs

EB_EVENT_FILE=/tmp/eventsfile20170405-3738-17c2sc0 initctl: Job failed to start + '[' -f /etc/elasticbeanstalk/.eb-ecs-start-no-retry ']' + warn 'Failed to start ECS task, retrying...' + echo 'Failed to start ECS task, retrying...' Failed to start ECS task, retrying... + eventHelper.py --msg 'Failed to start ECS task, retrying...' --severity WARN + RETRY=2 + sleep 3 + '[' 2 -gt 1 ']' + error_exit 'Failed to start ECS task after retrying 2 times.' 1 + error 'Failed to start ECS task after retrying 2 times.' + echo 'Failed to start ECS task after retrying 2 times.' Failed to start

ECS task after retrying 2 times. + eventHelper.py --msg 'Failed to start ECS task after retrying 2 times.' --severity ERROR + exit 1 (Executor::NonZeroExitStatus)

ECS agent log:

2017-04-05T09:13:11Z [INFO] Saving state! module="statemanager"

2017-04-05T09:13:11Z [INFO] Finished pulling container postgres(id.dkr.ecr.eu-west-2.amazonaws.com/postgres:latest) (PULLED->RUNNING). Lock acquisition and pull took 233ns.

2017-04-05T09:13:11Z [CRITICAL] Task in a bad state; it's not steadystate but no containers want to transition module="TaskEngine" task="awseb-ebsBoxer-env-y9jmnnjnnm:3 arn:aws:ecs:eu-west-2:443611395156:task/f510bd3f-5b97-4fbe-af6c-61e037b08a29, Status: (NONE->RUNNING) Containers: [codebase (STOPPED->RUNNING),postgres (PULLED->RUNNING),data (STOPPED->RUNNING),]"

2017-04-05T09:13:11Z [CRITICAL] Moving task to stopped due to bad state module="TaskEngine" task="awseb-ebsBoxer-env-y9jmnnjnnm:3 arn:aws:ecs:eu-west-2:443611395156:task/f510bd3f-5b97-4fbe-af6c-61e037b08a29, Status: (NONE->RUNNING) Containers: [codebase (STOPPED->RUNNING),postgres (PULLEDRUNNING),data(STOPPED->RUNNING),]" 2017-04-05T09:13:11Z [INFO] Task change event module="TaskEngine"event="{TaskArn:arn:aws:ecs:eu-west-2:443611395156:task/f510bd3f-5b97-4fbe-af6c-61e037b08a29 Status:STOPPED Reason: Task:awseb-ebsBoxer-env-y9jmnnjnnm:3 arn:aws:ecs:eu-west-2:443611395156:task/f510bd3f-5b97-4fbe-af6c-61e037b08a29, Status: (STOPPEDSTOPPED) Containers: [codebase(STOPPED->STOPPED),postgres (STOPPED STOPPED),data (STOPPED->STOPPED),]}"

Please let me know if you need any more information.

Answer 1

After breaking my head for two days finally found it.

It was happening because my data and codebase container were exiting with exit code "0". I changed the command to "tail -f /bin/bash/".

Before:

    {
      "name": "data",
      "image": "id.dkr.ecr.eu-west-2.amazonaws.com/dbdata:latest",
      "essential": false,
      "memory": 512,
      "command" : [
       "/bin/bash"
      ]
    },
    {
      "name": "codebase",
      "image": "id.dkr.ecr.eu-west-2.amazonaws.com/codebase:latest",
      "essential": false,
      "memory": 512,
      "command" : [
       "/bin/bash"
      ]
    }

After:

       {
          "name": "data",
          "image": "id.dkr.ecr.eu-west-2.amazonaws.com/dbdata:latest",
          "essential": false,
          "memory": 512,
          "command": [
            "tail",
            "-f",
            "/bin/bash"
          ]
        },
        {
          "name": "codebase",
          "image": "id.dkr.ecr.eu-west-2.amazonaws.com/codebase:latest",
          "essential": false,
          "memory": 512,
         "command": [
           "tail",
           "-f",
           "/bin/bash"
         ]
        }

I still don't understand why "essential" flag was ignored!!