How to throttle Delayed job to not anger the Facebook API

Question

I'm building an app that will be hitting the Facebook graph api a lot. I learned they have a rate limit of 600 requests every 600 seconds.

I'm using delayed job for all my background processing. What is a good way to schedule delayed job to stay under the fb api rate limit? Are there any tricks with delayed job or do I need to build a separate background task processor to not go over my rate limit?

Thanks

Answer 1

600 requests every 600 sec is 1 per sec on avg.

Not very fast!

1) Depending on your company's size and heft, I'd investigate with FB to see if you can get the limit raised for you.

2) You can stick with DelayedJob, no need to re-invent the wheel. You just need to change the scheduler.

In my DelayedJob installation, I use the "run_at" column for more than just setting the time to retry the jobs--I also use it as the time to run the job in the first place. You can also use it to throttle your jobs.

Changed in the DelayedJob file job.rb:

# added run_at param
# eg   Delayed::Job.enqueue NewsletterJob.new('lorem ipsum...'), 0,
#                           Delayed::Job.db_time_now + 15.minutes
def self.enqueue(object, priority = 0, run_at = nil)
  unless object.respond_to?(:perform)
    raise ArgumentError, 'Cannot enqueue items which do not respond to perform' 
  end

  Job.create(:payload_object => object, :priority => priority,
    :run_at => run_at)    
end                    

For your goal, I would keep track of the last time a FB api call was enqueued, and schedule the next one to run_at a time at least a sec greater.

Benefit: you would be able to interleave other, non-FB tasks, in with the FB api calls.