Django's migrate command on Amazon Elastic Beanstalk is killed

Question

I'm using Amazon's Elastic Beanstalk and Django 1.8.2. Here is my container commands,

container_commands:
  01_wsgipass:
    command: 'echo "WSGIPassAuthorization On" >> ../wsgi.conf'
  02_makemigrations:
    command: "source /opt/python/run/venv/bin/activate && python manage.py makemigrations --merge --noinput"
    leader_only: true
  03_migrate:
    command: "source /opt/python/run/venv/bin/activate && python manage.py migrate --noinput"
    leader_only: true

For some reasons the migrate command is being killed. All migrations are working fine even with a fresh database in my local. But following is the error appearing on eb-activity.log.

Synchronizing apps without migrations:
  Creating tables...
  Running deferred SQL...
  Installing custom SQL...
  Running migrations:
  Rendering model states.../bin/sh: line 1: 21228 Killed                  python manage.py migrate --noinput
   (ElasticBeanstalk::ExternalInvocationError)

Note: The same container commands were working fine without any issues earlier at Elastic Beanstalk. I tried with --verbose 3 with migrate command but didn't get any other debug messages.

Any solutions? Thanks in advance.

Answer 1

AWS is not developer friendly when it comes to troubleshooting with the poor logging mechanism.

As an avid AWS user who recently eval'd EBS for a Django project, I totally agree with this for the same reasons. I ended up going with Heroku for this and reasons I won't go into but I think the following pattern helps either way.

The steps to prepare your prod environment can go in different places; they don't have to happen on your target web-server environment.

I ended up pulling my make/migrate tasks out of my deployment automation and into tasks that happen just before it. The only things that happen in my target web-server environment are directly related to the code on that server.

In other words: I recommend if you have a CI tool for builds/tests, you pull your make/migrate and any other prep to the environment outside your webserver into your deployment pipeline. Something like:

• Checkout code
• Run Tests (including make/migrate on ephemeral database to test it if possible)
• Put app in maintenance mode (or similar, if required)
• Snapshot database
• Make/Migrate on production
• Deploy
• If deploy fails, rollback DB, rollback app.

Then you are separating the concerns of automation for your app server and automation for the rest of your prod environment and letting your CI handle that. You could handle them in the same place, but clearly its a bit clunky to do that using EBS's facilities.

Answer 2

My migrations were being killed because the memory reserved in the Dockerrun.aws.json file was too low. The example provided in the documentation gave "128" as a sample value, and I had just used that. Increasing the value for "memory" resolved the problem.

e.g. Dockerrun.aws.json excerpt:

  "containerDefinitions": [
    {
      "name": "php-app",
      "image": "php:fpm",
      "essential": true,
      "memory": 512,
      // ... etc. ...
    }
  ]