I have been using Sidekiq successfully to run background jobs that are initiated by user actions in a Rails 3.2 application. My particular application involves sending and receiving data from and external source via a third party API.
I need to keep data synchronized with this external source by continuously checking for each user if there is data available to be downloaded.
All of my background jobs thus far are user-initiated as I mentioned above. My question is, if I want to continuously check for the existence of external data for each user, in a while
loop I imagine:
# pseudocode
while true
for user in User.active
data = user.get_data
next if data.blank?
UserDataWorker.perform_async(user.id)
end
end
then where do I put the code above? I get confused on how to spawn a continuous task. I don't want to put the code in a Sidekiq worker, because it will just be a job that never completes. Or should I do that?
I don't want to put it in a rake task, because then how do I monitor that rake task? Everything I've read about Sidekiq seems to point to a Worker running a task that is finite and is able to be completed or errored out. I appreciate any help you can offer.
Today Sidekiq uses a default concurrency of 25. These means Sidekiq will spawn 25 worker threads and execute up to 25 jobs concurrently in a process.
To run sidekiq, you will need to open a terminal, navigate to your application's directory, and start the sidekiq process, exactly as you would start a web server for the application itself. When the command executes you will see a message that sidekiq has started.
To test your Sidekiq Worker jobs array, run WorkerNameHere.jobs in terminal and see if it contains your job with your jid. If it does, then it was enqueued in Sidekiq to be run as a job.
Ruby multithreading is just for that. Spawn a thread in your initializers folder:
Thread.new do
loop do
for user in User.active
data = user.get_data
next if data.blank?
UserDataWorker.perform_async(user.id)
end
sleep 1
end
end
If your checks don't need much resources, should be allright.
If you love us? You can donate to us via Paypal or buy me a coffee so we can maintain and grow! Thank you!
Donate Us With