howto: elastic beanstalk + deploy docker + graceful shutdown

Question

Hi great people of stackoverflow,

Were hosting a docker container on EB with an nodejs based code running on it. When redeploying our docker container we'd like the old one to do a graceful shutdown.

I've found help & guides on how our code could receive a sigterm signal produced by 'docker stop' command.

However further investigation into the EB machine running docker at: /opt/elasticbeanstalk/hooks/appdeploy/enact/01flip.sh shows that when "flipping" from current to the new staged container, the old one is killed with 'docker kill'

Is there any way to change this behaviour to docker stop?
Or in general a recommended approach to handling graceful shutdown of the old container?

Thanks!

Answer 1

Self answering as I've found a solution that works for us:

tl;dr: use .ebextensions scripts to run your script before 01flip, your script will make sure a graceful shutdown of whatevers inside the docker takes place

first, your app (or whatever your'e running in docker) has to be able to catch a signal, SIGINT for example, and shutdown gracefully upon it.
this is totally unrelated to Docker, you can test it running wherever (locally for example) There is a lot of info about getting this kind of behaviour done for different kind of apps on the net (be it ruby, node.js etc...)

Second, your EB/Docker based project can have a .ebextensions folder that holds all kinda of scripts to execute while deploying. we put 2 custom scripts into it, gracefulshutdown_01.config and gracefulshutdown_02.config file that looks something like this:

# gracefulshutdown_01.config
commands:
  backup-original-flip-hook:
    command: cp -f /opt/elasticbeanstalk/hooks/appdeploy/enact/01flip.sh /opt/elasticbeanstalk/hooks/appdeploy/01flip.sh.bak
    test: '[ ! -f /opt/elasticbeanstalk/hooks/appdeploy/01flip.sh.bak ]'
  cleanup-custom-hooks:
    command: rm -f 05gracefulshutdown.sh
    cwd: /opt/elasticbeanstalk/hooks/appdeploy/enact
    ignoreErrors: true

and:

# gracefulshutdown_02.config
commands:
  reorder-original-flip-hook:
    command: mv /opt/elasticbeanstalk/hooks/appdeploy/enact/01flip.sh /opt/elasticbeanstalk/hooks/appdeploy/enact/10flip.sh
    test: '[ -f /opt/elasticbeanstalk/hooks/appdeploy/enact/01flip.sh ]'

files:
  "/opt/elasticbeanstalk/hooks/appdeploy/enact/05gracefulshutdown.sh":
    mode: "000755"
    owner: root
    group: root
    content: |
      #!/bin/sh

      # find currently running docker
          EB_CONFIG_DOCKER_CURRENT_APP_FILE=$(/opt/elasticbeanstalk/bin/get-config container -k app_deploy_file)
      EB_CONFIG_DOCKER_CURRENT_APP=""

      if [ -f $EB_CONFIG_DOCKER_CURRENT_APP_FILE ]; then
        EB_CONFIG_DOCKER_CURRENT_APP=`cat $EB_CONFIG_DOCKER_CURRENT_APP_FILE | cut -c 1-12`
        echo "Graceful shutdown on app container: $EB_CONFIG_DOCKER_CURRENT_APP"
      else
        echo "NO CURRENT APP TO GRACEFUL SHUTDOWN FOUND"
        exit 0
      fi

      # give graceful kill command to all running .js files (not stats!!)
      docker exec $EB_CONFIG_DOCKER_CURRENT_APP sh -c "ps x -o pid,command | grep -E 'workers' | grep -v -E 'forever|grep' " |  awk '{print $1}' | xargs docker exec $EB_CONFIG_DOCKER_CURRENT_APP kill -s SIGINT
      echo "sent kill signals"

      # wait (max 5 mins) until processes are done and terminate themselves
      TRIES=100
      until [ $TRIES -eq 0 ]; do
        PIDS=`docker exec $EB_CONFIG_DOCKER_CURRENT_APP sh -c "ps x -o pid,command | grep -E 'workers' | grep -v -E 'forever|grep' " | awk '{print $1}' | cat`
        echo TRIES $TRIES PIDS $PIDS
        if [ -z "$PIDS" ]; then
          echo "finished graceful shutdown of docker $EB_CONFIG_DOCKER_CURRENT_APP"
          exit 0
        else
          let TRIES-=1
          sleep 3
        fi
      done

      echo "failed to graceful shutdown, please investigate manually"
      exit 1

gracefulshutdown_01.config is a small util that backups the original flip01 and deletes (if exists) our custom script.

gracefulshutdown_02.config is where the magic happens. it creates a 05gracefulshutdown enact script and makes sure flip will happen afterwards by renaming it to 10flip.

05gracefulshutdown, the custom script, does this basically:

• find current running docker
• find all processes that need to be sent a SIGINT (for us its processes with 'workers' in its name
• send a sigint to the above processes
• loop:
• check if processes from before were killed
• continue looping for an amount of tries
• if tries are over, exit with status "1" and dont continue to 10flip, manual interference is needed.

this assumes you only have 1 docker running on the machine, and that you are able to manually hop on to check whats wrong in the case it fails (for us never happened yet).
I imagine it can also be improved in many ways, so have fun.