Menu
Currently am running my background jobs using sidekiq. Concurrently am running 10 jobs at a time. These jobs are not a small jobs. It will take 6-8 hours to complete the work.
In the mean time i need to run some smaller jobs(will take 2-3 mins). Currently am not able to run these jobs until any one of the above 10 jobs to complete. So i need to wait for 6-8 hours to perform this small job.
But these jobs should run immediately after adding to the queue. Is there any option to block any one of the process to run this kind of small jobs. I tried the queue option but its also not working in my scenario.
Here is my sidekiq config
:concurrency: 10
:queues:
- [web, 7]
- [default, 3]
Can any one give me the solution for this problem?
496
The author of Sidekiq recommends a limit of 1,000 jobs per bulk enqueue and that's the default in the perform_bulk method.
To run sidekiq, you will need to open a terminal, navigate to your application's directory, and start the sidekiq process, exactly as you would start a web server for the application itself. When the command executes you will see a message that sidekiq has started.
In 2012, Mike Perham was the first to introduce a multi-threading background job system. Resque is multi-process and single-thread. Delayed_job can work for single thread processing only. Sidekiq was the first multi-threading tool in the Ruby community.
The long jobs suffocate the smaller jobs since they fight over the same resources.
I can suggest that you run two sidekiq processes, each using a different config file, and each listening to a different queue:
sidekiq_long.yml:
:concurrency: 10
:queues:
- default
sidekiq_normal.yml:
:concurrency: 3
:queues:
- web
This way one sidekiq process will always be available to server short jobs:
sidekiq -C config/sidekiq_long.yml.yml
sidekiq -C config/sidekiq_normal.yml
If you love us? You can donate to us via Paypal or buy me a coffee so we can maintain and grow! Thank you!
Donate Us With
