Menu
I've been trying to limit the amount of workers per queue using the sidekiq-limit_fetch gem, and Sidekiq seems to "see" the imposed limits in the log but when I watch the workers the limits are ignored.
Here's the part from the log where Sidekiq sees the limits:
2013-04-02T05:47:19Z 748 TID-11ilcw DEBUG: {:queues=>
["recommendvariations",
"recommendvariations",
"recommendvariations",
"recommendphenotypes",
"recommendphenotypes",
"recommendphenotypes",
"preparse",
"preparse",
"preparse",
"parse",
"parse",
"parse",
"zipgenotyping",
"zipgenotyping",
"zipfulldata",
"deletegenotype",
"fitbit",
"frequency",
"genomegov",
"mailnewgenotype",
"mendeley_details",
"mendeley",
"pgp",
"plos_details",
"plos",
"snpedia",
"fixphenotypes"],
:concurrency=>5,
:require=>".",
:environment=>"production",
:timeout=>8,
:profile=>false,
:verbose=>true,
:pidfile=>"/tmp/sidekiq.pid",
:logfile=>"./log/sidekiq.log",
:limits=>
{"recommendvariations"=>1,
"recommendphenotypes"=>1,
"preparse"=>2,
"parse"=>2,
"zipgenotyping"=>1,
"zipfulldata"=>1,
"fitbit"=>3,
"frequency"=>10,
"genomegov"=>1,
"mailnewgenotype"=>1,
"mendeley_details"=>1,
"mendeley"=>1,
"pgp"=>1,
"plos_details"=>1,
"plos"=>1,
"snpedia"=>1,
"fixphenotypes"=>1},
:strict=>false,
:config_file=>"config/sidekiq.yml",
:tag=>"snpr"}
and here's the sidekiq.yml. Judging from the web-interface of sidekiq the limits are ignored - right now, I got 2 workers on the "recommendvariations"-queue but that should be 1.
I start the workers over bundle exec sidekiq -e production -C config/sidekiq.yml.
Has anyone else ever encountered this?
227
asked Apr 02 '13 07:04
The author of Sidekiq recommends a limit of 1,000 jobs per bulk enqueue and that's the default in the perform_bulk method. Even then you are saving the time of 999 round trips to Redis.
Today Sidekiq uses a default concurrency of 25. These means Sidekiq will spawn 25 worker threads and execute up to 25 jobs concurrently in a process.
Sidekiq server process pulls jobs from the queue in Redis and processes them. Like your web processes, Sidekiq boots Rails so your jobs and workers have the full Rails API, including Active Record, available for use. The server will instantiate the worker and call perform with the given arguments.
Out-of-the-box Sidekiq uses a single queue named "default," but it's possible to create and use any number of other named queues if there is at least one Sidekiq worker configured to look at every queue.
Did you try to set the limit in a sidekiq.rb initializer file?
Like this:
Sidekiq::Queue['recommend'].limit = 1
It worked for me.
93
If you love us? You can donate to us via Paypal or buy me a coffee so we can maintain and grow! Thank you!
Donate Us With
