AWS ECS leader commands (django migrate)

Question

We are currently deploying our Django APP on AWS Elastic Beanstalk. There we execute the django db migrations using container commands, where we assure we only run migrations on one instance by using the "leader_only" restriction.

We are considering to move our deployment to AWS EC2 Container Service. However, we cannot figure out a way to enforce the migrate to only be run on one container when new image is being deployed.

Is it possible to configure leader_only commands in AWS EC2 Container Service?

Answer 1

There is a possibility to use ECS built-in functionality to handle deployments that involve migrations. Basically, the idea is the following:

1. Make containers fail their health checks if they are running against an unmigrated database, e.g. via making a custom view and checking for the existence of the migrations execution plan executor.migration_plan(executor.loader.graph.leaf_nodes()) status = 503 if plan else 200
2. Make a task definition that does nothing more then just migrates the database and make sure it is scheduled for execution with the rest of the deployment process.

The result is deployment process will try to bring one new container. This new container will fail health checks as long as database is not migrated and thus will block all the further deployment process (so you will still have old instances running to serve requests). Once migration is done - health check will now succeed, so the deployment will unblock and proceed.

This is by far the most elegant solution I was able to find in terms of running Django migrations on Amazon ECS.

Source: https://engineering.instawork.com/elegant-database-migrations-on-ecs-74f3487da99f

Answer 2

For the ones using task definitions JSON all we need to do is flag a container as not essential on containerDefinitions

    {
      "name": "migrations",
      "image": "your-image-name",
      "essential": false,
      "cpu": 24,
      "memory": 200,
      "logConfiguration": {
        "logDriver": "awslogs",
        "options": {
          "awslogs-group": "your-logs-group",
          "awslogs-region": "your-region",
          "awslogs-stream-prefix": "your-log-prefix"
        }
      },
      "command": [
        "python3", "manage.py", "migrate"
      ],
      "environment": [
        {
          "name": "ENVIRON_NAME",
          "value": "${ENVIRON_NAME}"
        }
      ]
    }

I flagged this container as "essential": false.

"If the essential parameter of a container is marked as true, and that container fails or stops for any reason, all other containers that are part of the task are stopped. If the essential parameter of a container is marked as false, then its failure does not affect the rest of the containers in a task. If this parameter is omitted, a container is assumed to be essential."

source: https://docs.aws.amazon.com/AmazonECS/latest/developerguide/task_definition_parameters.html