How can I run Rails background jobs on AWS Elastic Beanstalk?

Question

I just started to use AWS Elastic Beanstalk with my rails app and I need to use the Resque gem for background jobs. However, despite all effort to search how to run Resque worker on Elastic Beanstalk, I haven't been able to figure out how?

How can I run Rails background jobs with Resque on AWS Elastic Beanstalk? talks about running those as services in Elastic Beanstalk containers, however, it is still very confusing.

Here my ebextentions resque.config file:

services: 
  sysvinit:
  resque_worker:
    enabled: true
    ensureRunning: true
    commands: 
      resque_starter: 
        rake resque:work QUEUE='*'

EDIT Now my resque.config file looks like this:

container_commands:
  resque_starter: "rake resque:work QUEUE='*'"
services: 
  sysvinit:
    resque_worker:
      enabled: true
        ensureRunning: true
      commands: 
        resque_starter

And it is still not working. EDIT 2

container_commands:
  resque_starter: 
    command: "rake resque:work QUEUE=sqs_message_sender_queue"
    cwd: /var/app/current/
    ignoreErrors: true

Still it shows 0 workers.

Answer 1

I think it is suboptimal to run queues, like Resque, inside Elastic Beanstalk web environments. A web environment is intended to host web applications and spawn new instances when traffic and load increases. However, it would not make sense to have multiple Resque queues, each running on one of the instances.

Elastic Beanstalk offers worker environments which are intended to host code that executes background tasks. These worker environments pull jobs from an Amazon SQS queue (which makes an additional queue solution, like Resque, obsolete). An Amazon SQS queue scales easily and is easier to maintain (AWS just does it for you).

To use worker environments, which come with Amazon SQS queues, makes more sense, as it is out of the box supported and fits nicely into the Elastic Beanstalk landscape. There also exists a gem, Active Elastic Job, which makes it simple for Rails >= 4.2 applications to run background tasks in worker environments.

Disclaimer: I'm the author of Active Elastic Job.

Answer 2

First at all, I would recommend to run resque with help of supervisord this will help you make sure that worker will be restarted if process die.

On how to run command when you do deploy every time: Log in by ssh to your beanstalk instance go to folder: /opt/elasticbeanstalk/hooks/appdeploy/ Here you will find list of hooks that execute every time when you do deploy. Also here you can put own script that will be executed every time when you do deploy. Also with same approach you can put script to hooks that responsible to application server restart and have ability to restart you background job without connection by ssh.

Another option, put your command that start background worker is use container_commands instead of commands.

Also, please have a look to best articles that I have found about customization of beanstalk: http://www.hudku.com/blog/tag/elastic-beanstalk/ it would be good start point for customization of beanstalk environment for your need. \