Background workers view communication in rails app

Question

Lets say I have a blog application with posts. After creating a post a worker is created to handle some background operations. My case is that after submitting a post form I want to display some kind of loading message (gif loader or such) and when the worker finishes I want to hide the loading message and display some data supplied by the worker. My question is what's the best way to communicate that a worker has finished its job and display it on the frontend for user. Worker callback looks something like this

def worker_finish
  #message the user
end

Answer 1

I think you may be missing the point of having background workers, basically, what you're attempting to do is self defeating. -- If the user submits the form and you queue the job in your controller, only to make the user wait for the worker to both start and finish, you not only accomplished exactly what the controller could have done by itself, but you've made the process far more complicated (and this type of functionality is not built in to either resque, or sidekiq).

When offloading a job to be processed by a queue you want to return a response to the client immediately, i.e.

class PostsController < ApplicationController
  def create
    @post = Post.create(params[:post])
    BackgroundBlogOperation.enque(@post.id)
    respond_with(@post)
  end
end

The BackgroundBlogOperation class would then be placed into a queue, for a worker to go through and work off. If you need the BackgroundBlogOperation worker to do something else after it's finished, you can do so, but that should take place within the job itself, so that the worker can be responsible for that.

If you are just trying to display and hide the spinner after a post is created, without page reloading, just display the javascript spinner before click of the submit button, and make sure the request type is js (add :remote => true to form). Then create a javascript view response that looks like:

class PostsController < ApplicationController
  respond_to :js, :only => [:create]
  def create
    @post = Post.create(params[:post])
    BackgroundBlogOperation.enque(@post.id)
    respond_with(@post)
  end
end

And the create.js.erb, could also append a message telling them that whatever you are doing in the background has been queued, if its more complex than creating a post or whatever.

$("#spinner").hide();

Now -- Although what you were asking originally doesent serve any purpose (because to display and hide the spinner on completion of the job would require waiting for the controller to complete the action) -- There are situations where displaying to the client that the job has finished processing is useful.

First lets define a scenario where background processing is actually useful. Lets say you have a button that when click, will pull in data from some external api, and after getting data from the external site, you are performing a database operation based on the response. This would be a good example of when to use background process. Basically in the controller you would do something like:

class ApiController < ApplicationController
  respond_to :js, :only => [:create]

  def sync_tweets
    TwitterApiJob.enque(current_user.twitter_username)
    respond_with(message: 'Syncing Tweets')
  end

end

Now, to tell the user when the tweets have finished syncing is a bit more complicated, and you have 3 basic options:

1) Notify user via email (generally worst option imo) 2) Use some sort of html5 or websocket capable rails server to run rails, and send a push through the websocket to the client, which although very cool, is in most cases overkill and outside the scope of this response. Google rails websocket pushing if you want to see your options. 3) IMO best option for most cases, Create a notifcations model to handle all sorts of user notifications, and in your job, after the work is complete, do something like

Notification.create(:user_id => user.id, :message => 'Sync has finished', type => 'sync_complete')

Then, next page the user requests, up in a header toolbar or wherever, I would have a notice that the user has unread notifications, to alert the user to click on it.

--- Also I think in your post you mentioned you are using resque. I tried resque and it was decent, but I found it overcomplicated to debug in production, and it used crazy memory -- I would advise you to check out sidekiq if you haven't already, it uses redis also but much faster as it uses threads:

https://github.com/mperham/sidekiq

It is much faster, cleaner, and easy to get setup, imho.