Menu
I have several workers that are being run using Sidekiq and scheduled using Sidetiq. I'm looking for advice on the best way to wait for all workers to complete before executing a callback, similar to Sidekiq-Pro's batching functionality. Any advice on possible options would be greatly appreciated!
879
Sidekiq is an open-source framework that provides efficient background processing for Ruby applications. It uses Redis as an in-memory data structure to store all of its job and operational data. It's important to be aware that Sidekiq by default doesn't do scheduling, it only executes jobs.
Today Sidekiq uses a default concurrency of 25. These means Sidekiq will spawn 25 worker threads and execute up to 25 jobs concurrently in a process.
You can add the ability to stop a worker to your own code by having the worker code check regularly to see if it should stop, similar to job cancellation.
At the same time, Sidekiq uses multithreading so it is much more memory-efficient than Rescue.
You can write a method:
def wait_for_sidekiq
sleep(1) until Sidekiq::Workers.new.size == 0 && Sidekiq::Queue.new.size == 0
end
I would also suggest that you make sure that the jobs got queued in the first place:
def wait_for_queuing
sleep(1) until Sidekiq::Queue.new.size > 0 || Sidekiq::Workers.new.size > 0
end
This is required, because sometimes the first method might get executed a few miliseconds before queuing the jobs, so that it won't wait at all.
103
If you love us? You can donate to us via Paypal or buy me a coffee so we can maintain and grow! Thank you!
Donate Us With
