Ruby long running process to react to queue events

Question

I have a rails 3 app that writes certain events to a queue.

Now on the server I want to create a service that polls the queue every x seconds, and performs other tasks on a scheduled basis.

Other than creating a ruby script and running it via a cron job, are there other alternatives that are stable?

Answer 1

Although spinning up a persistent Rails-based task is an option, you may want to look at more orderly systems like delayed_job or Starling to manage your workload.

I'd advise against running something in cron since the expense of spinning up a whole Rails stack can be significant. Running it every few seconds isn't practical as the ramp-up time on Rails is usually 5-15 seconds depending on your hardware. Doing this a few times a day is usually no big deal, though.

A simple alternative is to create a work loop in a script you can engage with runner:

interval = 15.minutes
next_time = Time.now + interval

while (true)
  if (stuff_to_do?)
    do_stuff
  end

  # Figure out how much time is left before the next iteration
  delay = next_time.to_i - Time.now.to_i

  if (delay > 0)
     # If ahead of schedule, take a break
    sleep(delay)
  end
end

The downside to this is that the Rails stack will remain in memory as long as this background process is running, but this is a trade-off between huge CPU hits and a memory hit.

Answer 2

You have several options for that, including DelayedJob and Resque.

Resque relies on Redis and is the solution I use all the time (and am very happy with).