Sidekiq remote server setup

Question

I want to run sidekiq workers remotly on different servers to split load. Could not find any documentation for it. I could not find anything in the official sidekiq documentation and I have tried in google to search for 'sidekiq remote server', 'sidekiq remote workers' ..., but could not find anything useful. Here is what I am trying to to

My questions are: Do I need to replicate the same application code as on production server to all sidekicks servers? How to configure sidekiq to work this way or what documentation or sources to read about this type of configuration? Thanks allot to everyone for any type of help!

Answer 1

Q1. Do I need to replicate the same application code as on production server to all sidekicks servers?
Answer: Yes

Q2. How to configure Sidekiq to work this way or what documentation or sources to read about this type of configuration?
Answer: You need to connect Sidekiq to a common Redis server. Consider your Sidekiq workers are connected to Redis server on a server, say R. Then when you are enqueueing any job, You need to configure your Sidekiq gem to connect to Redis on server R.

To configure your Sidekiq to use external server you could use

Sidekiq.configure_server do |config|
  config.redis = { url: 'redis://redis.example.com:7372/12' }
end

You could check the documentation for even further details.

Core ideology is, you need a common database server where every Sidekiq workers and queues will connect to.

Answer 2

First question - Sidekiq requires the application code to do its job so you will need to make sure it's replicated on to each task server.

Configuring the Sidekiq is pretty simple, but setting up those servers will be a bit more involved. Personally I like Amazon's Opsworks for this kind of stack, and I use this Chef recipe to configure each machine that's running Sidekiq. (I use Monit to keep it running, which is why I've labelled this as a Monit-specific recipe.)

If you can give more information on your preferred deployment and hosting choices we can probably give you more detailed responses. Good luck. :)