Is it possible to launch privileged docker containers on Amazon elasticbeanstalk?

Question

I have tried numerous different ways to include the privileged flag in my task definition per the task definition documentation here: http://docs.aws.amazon.com/AmazonECS/latest/developerguide/task_definition_parameters.html#container_definition_security

I have also found forum postings over at amazon here: https://forums.aws.amazon.com/thread.jspa?threadID=180014&tstart=0&messageID=687696#687696 where the amazon employee "ChrisB@AWS" said "ECS now supports privileged mode."

I have successfully launched privileged containers on ECS using the aforementioned privileged key/val in the task definition and can confirm using docker command on the ec2 host. However the same task definition stanza does not prove successful on an elastic beanstalk multi-container solution stack host.

I see a ~year old post on the amazon forum specifically about support in elasticbeanstalk here: https://forums.aws.amazon.com/thread.jspa?messageID=687694&#687694 where amazon employee "DhanviK@AWS" says: "EB does not yet cleanly support the privileged mode of docker execution. We'll take this into account as feedback as we continue to release the next versions of our docker containers."

I also see some old discussion from last april on github here: https://github.com/awslabs/eb-docker-virtual-hosting/issues/1 where they say it's not supported on ECS. But clearly it has been implemented their at this point per my experiment above.

So what gives? If EB multi-conatiner solution stack simply wraps the ECS service why can't my privileged flag be accepted by the ecs agent when passed from elasticbeanstalk? Is elasticbeanstalk simply deleting the flag before it get's to the ecs agent? If so that is wack. Can anyone shed any light on this?

UPDATE: I found this SO question that pertains to the single container elasticbeanstalk solution stack. This is not what I am using. I am using the multi-container solution stack. How can I run a Docker container in AWS Elastic Beanstalk with non-default run parameters?

Answer 1

Turns out that Elastic Beanstalk does simply delete the privileged flag from the task definition. You can confirm this by including it in the Dockerrun.aws.json file that you upload in your app bundle to EB and then going to the ECS control panel in aws and viewing the Task Definition that is created by EB for your container cluster. The privileged flag will now be set to false. This is in fact wack.

To deal with this issue I had to spend many hours hacking together an ebextension that waits for a deploy to start all of the containers, then loops over the Dockerrun.aws.json and extracts any container definition that is supposed to be privileged and then does a docker inspect on the running unprivileged versions of those containers and then stops and reruns them using their existing running configs from docker inspect but with the privileged flag set back to true. The files for the ebextension are provided in a gist here: https://gist.github.com/therealjessesanford/5a012218889831926169

NOTE: You cannot use essential: true and privileged: true in the same container definition stanza in the Dockerrun.aws.json file. Those two params are mutually exclusive with this hack.

I will also include them inline here for googlers:

.ebextensions/0001_restart_privileged_containers.config

container_commands:
  01-move-restart-hook:
    command: cp -f .ebextensions/files/00_restart_containers_with_privileges.sh /opt/elasticbeanstalk/hooks/appdeploy/post/00_restart_containers_with_privileges.sh && chmod 755 /opt/elasticbeanstalk/hooks/appdeploy/post/00_restart_containers_with_privileges.sh
  02-move-stop-hook:
    command: cp -f .ebextensions/files/02stop_privileged_containers.sh /opt/elasticbeanstalk/hooks/appdeploy/pre/02stop_privileged_containers.sh && chmod 755 /opt/elasticbeanstalk/hooks/appdeploy/pre/02stop_privileged_containers.sh

.ebextensions/files/00_restart_containers_with_privileges.sh

#!/bin/bash

set -ex

. /opt/elasticbeanstalk/hooks/common.sh

EB_CONFIG_APP_STAGING=$(/opt/elasticbeanstalk/bin/get-config container -k app_deploy_dir)
export DOCKERRUN_AWS_JSON=$EB_CONFIG_APP_STAGING/Dockerrun.aws.json

while read -r container_short_name; do
  CURRENT_CONTAINER_ID=$(docker ps --no-trunc -q --filter=name=.$container_short_name)
  CONTAINER_LONG_NAME=$(docker inspect --format='{{.Name}}' $CURRENT_CONTAINER_ID)
  CURRENT_CONFIG=$(docker inspect --format='{{json .Config}}' $CURRENT_CONTAINER_ID)
  NEW_HOST_CONFIG=$(docker inspect --format='"HostConfig":{{json .HostConfig}}' $CURRENT_CONTAINER_ID | sed 's/\"Privileged\":false/\"Privileged\":true/I')
  echo "Stopping unprivileged $CONTAINER_LONG_NAME"
  docker stop $CURRENT_CONTAINER_ID
  docker rm $CURRENT_CONTAINER_ID
  NEW_CONTAINER_ID=$(curl --unix-socket /var/run/docker.sock -X POST -H "Content-Type: application/json" http:/containers/create?name=$CONTAINER_LONG_NAME -d "${CURRENT_CONFIG%?},$NEW_HOST_CONFIG}" | jq -r '.Id')
  echo "Starting privileged $CONTAINER_LONG_NAME"
  docker start $NEW_CONTAINER_ID
  sed -i "s/$CURRENT_CONTAINER_ID/$NEW_CONTAINER_ID/g" /var/lib/ecs/data/ecs_agent_data.json
done <<< "$(jq -r '.containerDefinitions[] | select(.privileged == true) | .name' $DOCKERRUN_AWS_JSON)"

.ebextensions/files/02stop_priviliged_containers.sh

#!/bin/bash

set -ex

. /opt/elasticbeanstalk/hooks/common.sh

EB_CONFIG_APP_CURRENT=$(/opt/elasticbeanstalk/bin/get-config container -k app_deploy_dir)
export DOCKERRUN_AWS_JSON=$EB_CONFIG_APP_CURRENT/Dockerrun.aws.json

while read -r container_short_name; do
  CURRENT_CONTAINER_ID=$(docker ps -q --filter=name=.$container_short_name)
  if [[ ! -z $CURRENT_CONTAINER_ID && "FOOBAR$CURRENT_CONTAINER_ID" != "FOOBAR" ]]; then
    CONTAINER_LONG_NAME=$(docker inspect --format='{{.Name}}' $CURRENT_CONTAINER_ID)
    echo "Stopping unprivileged $CONTAINER_LONG_NAME"
    docker stop $CURRENT_CONTAINER_ID || true
    docker rm $CURRENT_CONTAINER_ID || true
  fi
done <<< "$(jq -r '.containerDefinitions[] | select(.privileged == true) | .name' $DOCKERRUN_AWS_JSON)"

./Dockerrun.aws.json

{
  "AWSEBDockerrunVersion": 2,
  "containerDefinitions": [
    {
      "name": "happy_container_name",
      "image": "tutum.co/happy/happy_container",
      "memory": 128,
      "essential": false,
      "privileged": true
    }
  ]
}

Answer 2

Just figured out how to solve this, now Elastic Beanstalk supports running a privileged containers and you just need to add the "privileged": "true" to your Dockerrun.aws.json as the following sample (please take a look at the container-1):

{
  "AWSEBDockerrunVersion": 2,
  "containerDefinitions": [{
    "name": "container-0",
    "image": "ubuntu",
    "memory": "512"
  }, {
    "name": "container-1",
    "image": "ubuntu",
    "memory": "512",
    "privileged": "true"
  }]
}