Logo Questions Linux Laravel Mysql Ubuntu Git Menu
 

How can I run a continuous background job with Sidekiq?

I have been using Sidekiq successfully to run background jobs that are initiated by user actions in a Rails 3.2 application. My particular application involves sending and receiving data from and external source via a third party API.

I need to keep data synchronized with this external source by continuously checking for each user if there is data available to be downloaded.

All of my background jobs thus far are user-initiated as I mentioned above. My question is, if I want to continuously check for the existence of external data for each user, in a while loop I imagine:

# pseudocode
while true

  for user in User.active
    data = user.get_data
    next if data.blank?

    UserDataWorker.perform_async(user.id)

  end
end

then where do I put the code above? I get confused on how to spawn a continuous task. I don't want to put the code in a Sidekiq worker, because it will just be a job that never completes. Or should I do that?

I don't want to put it in a rake task, because then how do I monitor that rake task? Everything I've read about Sidekiq seems to point to a Worker running a task that is finite and is able to be completed or errored out. I appreciate any help you can offer.

like image 330
AKWF Avatar asked Jul 23 '13 14:07

AKWF


People also ask

How many jobs can Sidekiq handle?

Today Sidekiq uses a default concurrency of 25. These means Sidekiq will spawn 25 worker threads and execute up to 25 jobs concurrently in a process.

How do I run a Sidekiq job?

To run sidekiq, you will need to open a terminal, navigate to your application's directory, and start the sidekiq process, exactly as you would start a web server for the application itself. When the command executes you will see a message that sidekiq has started.

How can I test my Sidekiq worker locally?

To test your Sidekiq Worker jobs array, run WorkerNameHere.jobs in terminal and see if it contains your job with your jid. If it does, then it was enqueued in Sidekiq to be run as a job.


1 Answers

Ruby multithreading is just for that. Spawn a thread in your initializers folder:

Thread.new do

  loop do

    for user in User.active
      data = user.get_data
      next if data.blank?

      UserDataWorker.perform_async(user.id)
    end

    sleep 1

  end
end

If your checks don't need much resources, should be allright.

like image 110
Utgarda Avatar answered Sep 17 '22 15:09

Utgarda