Consequent attribute calculations with a queuing system

Question

For all of the following assume these:

• rails v3.0
• ruby v1.9
• resque

We have 3 models:

• Product belongs_to :sku, belongs_to :category
• Sku has_many :products, belongs_to :category
• Category has_many :products, has_many :skus

When we update the product (let's say we disable it) we need to have some things happen to the relevant sku and category. The same is true for when a sku is updated.

The proper way of achieving this is have an after_save on each model that triggers the other models' update events.

example:

products.each(&:disable!)
# after_save triggers self.sku.products_updated
#   and self.category.products_updated (self is product)

Now if we have 5000 products we are in for a treat. The same category might get updated hundreds of times and hog the database while doing so.

We also have a nice queueing system, so the more realisting way of updating products would be products.each(&:queue_disable!) which would simply toss 5000 new tasks to the working queue. The problem of 5000 category updates still exists though.

Is there a way to avoid all those updates on the db?

How can we concatenate all the category.products_updated for each category in the queue?

Answer 1

You can ensure a single category update for all the products by using a couple Resque plugins: Resque Unique Job and Resque Scheduler.

Delay the execution of the job to update the category slightly (however long it takes to typically call all the product updates) and ensure each job is unique by including the Unique Job module. Unique Job uses the paramaters of the job, so if you try to queue 2 jobs with category_id 123, it ignores the 2nd one since the job is already queued.

class Product
  after_save :queue_category_update

  def queue_category_update
    Resque.enqueue_at(1.minute.from_now, Jobs::UpdateCategory, category.id) if need_to_update_category?
  end
end

module Jobs
  module UpdateCategory
    include Resque::Plugins::UniqueJob

    def self.perform(category_id)
      category = Category.find_by_id(category_id)
      category.update_some_stuff if category
    end
  end
end