Menu
I have a rake task which is going to call 4 more rake tasks, in order:
rake:one
rake:two
rake:three
rake:four
Rake tasks one, two, and three are getting data and adding it to my database. Then rake:four is going to do something with that data. But I need to make sure that one, two, and three are complete first. Each rake task is actually spinning up Sidekiq workers to run in the background. In this scenario, would all of the workers created by rake:one finish first, then rake:two, etc?
If not, how can I ensure that the workers are executed in order?
538
Sidekiq is usually the background job processor of choice for Ruby-on-Rails, and uses Redis as a data store for the job queues. Background (or asynchronous) job processing is critical to GitLab because there are many tasks that: Shouldn't tie up relatively expensive HTTP workers to perform long-running operations.
It works the same way as the scheduled queue; it contains jobs that have to be executed at a given time in the future. The main difference is that in retries queue, jobs are added by the Sidekiq, and the time when the job should be executed is calculated based on the retries counter.
Sidekiq server process pulls jobs from the queue in Redis and processes them. Like your web processes, Sidekiq boots Rails so your jobs and workers have the full Rails API, including Active Record, available for use. The server will instantiate the worker and call perform with the given arguments.
To run sidekiq, you will need to open a terminal, navigate to your application's directory, and start the sidekiq process, exactly as you would start a web server for the application itself. When the command executes you will see a message that sidekiq has started.
Sidekiq processes jobs in the order which they are created, but by default it processes multiple jobs simultaneously, and there is no guarantee that a given job will finish before another job is started.
Quoting from https://github.com/mperham/sidekiq/wiki/FAQ:
How can I process a certain queue in serial?
You can't, by design. Sidekiq is designed for asynchronous processing of jobs that can be completed in isolation and independent of each other. Jobs will be popped off of Redis in the order in which they were pushed but there's no guarantee that Job #1 will execute fully before Job #2 is started.
If you need serial execution, you should look into other systems which give those types of guarantees.
Note you can create a Sidekiq process dedicated to processing a queue with a single worker. This will give you serial execution but it's a hack.
Also note you can use third-party extensions for sidekiq to achieve that goal.
94
If you love us? You can donate to us via Paypal or buy me a coffee so we can maintain and grow! Thank you!
Donate Us With
