How do I separate workers into pools of jobs with delayed job + heroku?

Question

My environment is rails 3.1, heroku bamboo stack, delayed_job_active_record, (https://github.com/collectiveidea/delayed_job ) and experimenting with hirefire. (https://github.com/meskyanichi/hirefire) - I can see the delayed_job queue documentation, but how do I apply this on heroku?

I have a max priority set of tasks that get spawned off every hour that I need to dedicate 3 workers to, it takes approx 26 minutes to complete. During that time, less important background tasks need to continue, with perhaps 1 worker dedicated to them.

So I'll set that block of priority tasks to being in a named queue, e.g. 'hourtask', and then name a queue for everything else 'everythingelse' :)

The question is, how do I dedicate heroku workers to specific queues? Is it something to do with the environment variables as per the documentation? It says:

# Set the --queue or --queues option to work from a particular queue.
$ RAILS_ENV=production script/delayed_job --queue=tracking start
$ RAILS_ENV=production script/delayed_job --queues=mailers,tasks start

But I'm not familiar enough with heroku setup to work out how to apply this to my heroku production environment?

Answer 1

It's in the README for Delayed Job 3:

DJ 3 introduces Resque-style named queues while still retaining DJ-style priority. The goal is to provide a system for grouping tasks to be worked by separate pools of workers, which may be scaled and controlled individually.

Jobs can be assigned to a queue by setting the queue option:

object.delay(:queue => 'tracking').method

Delayed::Job.enqueue job, :queue => 'tracking'

handle_asynchronously :tweet_later, :queue => 'tweets'

script/delayed_job can be used to manage a background process which will start working off jobs.

To do so, add gem "daemons" to your Gemfile and make sure you’ve run rails generate delayed_job.

You can then do the following:

$ RAILS_ENV=production script/delayed_job start
$ RAILS_ENV=production script/delayed_job stop

# Runs two workers in separate processes.
$ RAILS_ENV=production script/delayed_job -n 2 start
$ RAILS_ENV=production script/delayed_job stop

# Set the --queue or --queues option to work from a particular queue.
$ RAILS_ENV=production script/delayed_job --queue=tracking start
$ RAILS_ENV=production script/delayed_job --queues=mailers,tasks start

Work off queues by setting the QUEUE or QUEUES environment variable.

QUEUE=tracking rake jobs:work
QUEUES=mailers,tasks rake jobs:work

On Heroku, In your procfile, create two entries:

worker1: QUEUE=tracking rake jobs:work
worker2: QUEUES=mailers,tasks rake jobs:work

and scale them individually:

heroku ps:scale worker1=2 worker2=1 

etc

Answer 2

Original question asked about HireFire as well. At this time HireFire does not support named queues (see HireFire website) which makes auto-scaling difficult.