How to decide concurrency number in Sidekiq for CPU/RAM expensive jobs?

Question

I have a Rails 4 application in Heroku which uses Sidekiq as a background job processor. The job basically does two things (which are CPU/RAM expensive):

• Parse sites using PhantomJS
• Do some image manipulation with Rmagick

By default Sidekiq comes with a concurrency of 25 threads per process, but if I have that concurrency, I exceed the memory quota of my worker dyno. When doing 10 threads per process I get more or less this:

Process running mem=936M(182.8%)
Error R14 (Memory quota exceeded)

So, I could low the numbers of threads to 2 or 3 in order to make sure that I will 'never' overcome the memory allowed by the dyno, but there would be sometimes that the worker could have enough memory to have more threads.

My question is: Is there any way Sidekiq can modify dynamically its concurrency depending on the % of memory used by its process? If not, any recommendations on how to find a good threshold to run this in Heroku?

Answer 1

1. Unfortunately not.
2. Sidekiq generally takes 200MB on startup with a decent sized Rails app. Any memory usage beyond that is due to your application code. Running a full browser like PhantomJS can take a huge amount of memory. Your best bet is to upgrade your dyno size.